Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ASM] when loading library catch paths exceptions #6212

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[ASM] when loading library catch paths exceptions #6212

wants to merge 1 commit into from

Conversation

anna-git
Copy link
Contributor

Summary of changes

Some exceptions appear when trying to load the waf library

Reason for change

https://app.datadoghq.com/cases/ET-550

Implementation details

Test coverage

Other details

@anna-git anna-git marked this pull request as ready for review October 28, 2024 15:45
@anna-git anna-git requested a review from a team as a code owner October 28, 2024 15:45
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Oct 28, 2024 •
edited
Loading

Datadog Report

Branch report: anna/asm/fix-illegal-chars
Commit report: 05c79ff
Test service: dd-trace-dotnet

✅ 0 Failed, 365509 Passed, 2071 Skipped, 14h 47m 33.24s Total Time

@andrewlock
Copy link
Member

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6212) - mean (71ms)  : 68, 73
     .   : milestone, 71,
    master - mean (71ms)  : 68, 75
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (6212) - mean (1,120ms)  : 1096, 1144
     .   : milestone, 1120,
    master - mean (1,118ms)  : 1096, 1140
     .   : milestone, 1118,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6212) - mean (110ms)  : 107, 113
     .   : milestone, 110,
    master - mean (110ms)  : 107, 114
     .   : milestone, 110,

    section CallTarget+Inlining+NGEN
    This PR (6212) - mean (777ms)  : 760, 793
     .   : milestone, 777,
    master - mean (778ms)  : 762, 793
     .   : milestone, 778,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6212) - mean (94ms)  : 91, 97
     .   : milestone, 94,
    master - mean (94ms)  : 90, 97
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (6212) - mean (729ms)  : 715, 743
     .   : milestone, 729,
    master - mean (733ms)  : 718, 749
     .   : milestone, 733,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6212) - mean (190ms)  : 187, 194
     .   : milestone, 190,
    master - mean (190ms)  : 188, 193
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6212) - mean (1,204ms)  : 1184, 1224
     .   : milestone, 1204,
    master - mean (1,201ms)  : 1175, 1228
     .   : milestone, 1201,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6212) - mean (276ms)  : 272, 280
     .   : milestone, 276,
    master - mean (276ms)  : 271, 280
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (6212) - mean (945ms)  : 930, 960
     .   : milestone, 945,
    master - mean (949ms)  : 927, 971
     .   : milestone, 949,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6212) - mean (265ms)  : 261, 269
     .   : milestone, 265,
    master - mean (265ms)  : 261, 268
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (6212) - mean (926ms)  : 903, 949
     .   : milestone, 926,
    master - mean (924ms)  : 905, 943
     .   : milestone, 924,
Loading

@andrewlock
Copy link
Member

Benchmarks Report for appsec 🐌

Benchmarks for #6212 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.183
  • 1 benchmarks have fewer allocations
  • 2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6212

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody‑net472 1.183 4,494.64 3,799.88

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 75.2μs 129ns 498ns 0.0746 0 0 6.01 KB
master AllCycleSimpleBody netcoreapp3.1 63μs 80.3ns 300ns 0.0938 0 0 6.95 KB
master AllCycleSimpleBody net472 49.9μs 58ns 225ns 1.31 0 0 8.33 KB
master AllCycleMoreComplexBody net6.0 79μs 50.2ns 188ns 0.12 0 0 9.51 KB
master AllCycleMoreComplexBody netcoreapp3.1 69.5μs 64.7ns 251ns 0.139 0 0 10.36 KB
master AllCycleMoreComplexBody net472 56.9μs 86.4ns 335ns 1.88 0.0285 0 11.85 KB
master ObjectExtractorSimpleBody net6.0 145ns 0.179ns 0.692ns 0.00397 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 202ns 0.294ns 1.14ns 0.00367 0 0 272 B
master ObjectExtractorSimpleBody net472 167ns 0.0968ns 0.375ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.95μs 1.79ns 6.68ns 0.0534 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.04μs 1.53ns 5.72ns 0.0505 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 4.5μs 5.33ns 20.6ns 0.602 0.00448 0 3.8 KB
#6212 AllCycleSimpleBody net6.0 73.1μs 70.2ns 272ns 0.0729 0 0 6 KB
#6212 AllCycleSimpleBody netcoreapp3.1 63.5μs 68.8ns 266ns 0.0952 0 0 6.95 KB
#6212 AllCycleSimpleBody net472 49.9μs 63.9ns 247ns 1.31 0 0 8.33 KB
#6212 AllCycleMoreComplexBody net6.0 80.1μs 83.2ns 322ns 0.119 0 0 9.51 KB
#6212 AllCycleMoreComplexBody netcoreapp3.1 71.2μs 114ns 440ns 0.141 0 0 10.37 KB
#6212 AllCycleMoreComplexBody net472 56.7μs 66.1ns 238ns 1.88 0.0281 0 11.85 KB
#6212 ObjectExtractorSimpleBody net6.0 141ns 0.127ns 0.477ns 0.00398 0 0 280 B
#6212 ObjectExtractorSimpleBody netcoreapp3.1 214ns 0.126ns 0.455ns 0.00367 0 0 272 B
#6212 ObjectExtractorSimpleBody net472 172ns 0.187ns 0.724ns 0.0446 0 0 281 B
#6212 ObjectExtractorMoreComplexBody net6.0 3.26μs 2.09ns 7.8ns 0.0523 0 0 3.78 KB
#6212 ObjectExtractorMoreComplexBody netcoreapp3.1 3.93μs 2.24ns 8.69ns 0.051 0 0 3.69 KB
#6212 ObjectExtractorMoreComplexBody net472 3.81μs 5.93ns 23ns 0.603 0.00567 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 38.4μs 18.6ns 71.9ns 0.46 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 54.2μs 25ns 93.5ns 0.43 0 0 32.4 KB
master EncodeArgs net472 65.7μs 45.1ns 175ns 5.13 0.0658 0 32.5 KB
master EncodeLegacyArgs net6.0 72μs 96ns 333ns 0.0365 0 0 2.14 KB
master EncodeLegacyArgs netcoreapp3.1 106μs 99.4ns 385ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 153μs 93.6ns 363ns 0.306 0 0 2.15 KB
#6212 EncodeArgs net6.0 36.9μs 20.5ns 79.5ns 0.458 0 0 32.4 KB
#6212 EncodeArgs netcoreapp3.1 54.5μs 29ns 112ns 0.433 0 0 32.4 KB
#6212 EncodeArgs net472 67μs 65.8ns 255ns 5.15 0.0669 0 32.5 KB
#6212 EncodeLegacyArgs net6.0 72.4μs 64.8ns 251ns 0 0 0 2.14 KB
#6212 EncodeLegacyArgs netcoreapp3.1 108μs 68ns 245ns 0 0 0 2.14 KB
#6212 EncodeLegacyArgs net472 153μs 125ns 485ns 0.304 0 0 2.15 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 184μs 317ns 1.23μs 0 0 0 2.44 KB
master RunWafRealisticBenchmark netcoreapp3.1 195μs 241ns 901ns 0 0 0 2.39 KB
master RunWafRealisticBenchmark net472 209μs 119ns 461ns 0.311 0 0 2.46 KB
master RunWafRealisticBenchmarkWithAttack net6.0 122μs 72.8ns 282ns 0 0 0 1.47 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 129μs 89.6ns 335ns 0 0 0 1.46 KB
master RunWafRealisticBenchmarkWithAttack net472 138μs 48.2ns 167ns 0.207 0 0 1.49 KB
#6212 RunWafRealisticBenchmark net6.0 188μs 284ns 1.1μs 0 0 0 2.44 KB
#6212 RunWafRealisticBenchmark netcoreapp3.1 197μs 76.1ns 274ns 0 0 0 2.39 KB
#6212 RunWafRealisticBenchmark net472 210μs 149ns 577ns 0.315 0 0 2.46 KB
#6212 RunWafRealisticBenchmarkWithAttack net6.0 123μs 179ns 692ns 0 0 0 1.47 KB
#6212 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 134μs 139ns 536ns 0 0 0 1.46 KB
#6212 RunWafRealisticBenchmarkWithAttack net472 140μs 72.9ns 282ns 0.21 0 0 1.49 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6212

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 263.3 KB 265.6 KB 2.3 KB 0.88%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0 254.19 KB 256.08 KB 1.89 KB 0.74%

Fewer allocations 🎉 in #6212

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 59.36 KB 57.37 KB -1.99 KB -3.36%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 62.4μs 667ns 6.64μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 53.9μs 287ns 1.49μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 38.4μs 64.1ns 222ns 0 0 0 59.36 KB
master StringConcatAspectBenchmark net6.0 309μs 1.7μs 10.7μs 0 0 0 254.19 KB
master StringConcatAspectBenchmark netcoreapp3.1 347μs 1.87μs 11.1μs 0 0 0 263.3 KB
master StringConcatAspectBenchmark net472 288μs 6.44μs 62.5μs 0 0 0 278.53 KB
#6212 StringConcatBenchmark net6.0 61.2μs 644ns 6.37μs 0 0 0 43.44 KB
#6212 StringConcatBenchmark netcoreapp3.1 54.6μs 289ns 1.5μs 0 0 0 42.64 KB
#6212 StringConcatBenchmark net472 38.8μs 203ns 1.03μs 0 0 0 57.37 KB
#6212 StringConcatAspectBenchmark net6.0 328μs 1.85μs 14.6μs 0 0 0 256.08 KB
#6212 StringConcatAspectBenchmark netcoreapp3.1 349μs 1.92μs 11.7μs 0 0 0 265.6 KB
#6212 StringConcatAspectBenchmark net472 291μs 5.75μs 56.1μs 0 0 0 278.53 KB

andrewlock
andrewlock reviewed Oct 28, 2024
View reviewed changes
tracer/src/Datadog.Trace/AppSec/Waf/Initialization/LibraryLocationHelper.cs
{
AddProfilerFolders(paths, frameworkDescription, runtimeId);
Log.Warning(e, "When adding paths from native loader engine, home folders, and profiler folders, some path contained invalid characters or were too long");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems very surprising 🤔 I think we should try to understand why this happens. After all, the profiler runs from exactly the same folder - is this a case of manual-only instrumentation or something? Is there anything we can determine from the telemetry configuration as to what's triggering this?

Copy link
Contributor Author

@anna-git anna-git Oct 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed that's quite strange, but as we do at some point Path.GetDirectoryName(value), and this is the method throwing, maybe, profiler still manages to load from a path with invalid chars, but calling this method would throw 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe, profiler still manages to load from a path with invalid chars

I was just looking at this again, and I have a suspicion about what's going on... " is an invalid char path. It's Very easy to accidentally include that if you run the value in cmd (instead of powershell).

One thing we could/should consider is changing the "find" logic to use a different variable.

We (the native loader) set an environment variable pointing to the location where the native profiler is actually loaded from IIRC (in addition to the COR_/CORECLR_ variables). The WAF should be next to it I believe, so we could consider checking if that variable exists, and if it does, look for where the waf Should be.

Now, in this case, I suspect the native loader wouldn't be loaded correctly. Which then raises the question of "should we bail on ASM in this scenario" 🤔

Oooh, in fact if my theory is correct, this won't happen in newer versions of the tracer, because if auto-instrumentation is not correctly configured they won't get ASM initialized either. And looking at the telemetry, I don't think we have any examples of this in v3, so maybe this is a non-issue now 😄

We may still want to see if we can optimise the loading of the waf, given the extra constraints we have now 🤔

But also, given all that (and the fact that we don't expect to see this error any more) I'm still inclined to say let's not add the try catch, so that we get the details when something does go wrong 😄

@andrewlock
Copy link
Member

Benchmarks Report for tracer 🐌

Benchmarks for #6212 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.180
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.66μs 38.2ns 175ns 0.0119 0.00396 0 5.42 KB
master StartStopWithChild netcoreapp3.1 9.84μs 53.7ns 294ns 0.0147 0.00489 0 5.62 KB
master StartStopWithChild net472 16.3μs 49.8ns 186ns 1.03 0.318 0.0896 6.06 KB
#6212 StartStopWithChild net6.0 7.85μs 44.8ns 317ns 0.0165 0.00823 0 5.42 KB
#6212 StartStopWithChild netcoreapp3.1 9.93μs 53.6ns 298ns 0.0152 0.00505 0 5.62 KB
#6212 StartStopWithChild net472 16.1μs 66.1ns 256ns 1.03 0.312 0.0985 6.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 471μs 231ns 865ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 621μs 295ns 1.14μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 893μs 330ns 1.19μs 0.437 0 0 3.3 KB
#6212 WriteAndFlushEnrichedTraces net6.0 480μs 383ns 1.48μs 0 0 0 2.7 KB
#6212 WriteAndFlushEnrichedTraces netcoreapp3.1 643μs 202ns 757ns 0 0 0 2.7 KB
#6212 WriteAndFlushEnrichedTraces net472 809μs 531ns 2.06μs 0.406 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 196μs 1.1μs 8.59μs 0.212 0 0 18.45 KB
master SendRequest netcoreapp3.1 225μs 1.32μs 12.4μs 0.218 0 0 20.61 KB
master SendRequest net472 9.94E‑06ns 9.94E‑06ns 3.85E‑05ns 0 0 0 0 b
#6212 SendRequest net6.0 191μs 1.05μs 6.71μs 0.195 0 0 18.45 KB
#6212 SendRequest netcoreapp3.1 215μs 1.21μs 8.65μs 0.21 0 0 20.61 KB
#6212 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 572μs 2.04μs 7.37μs 0.584 0 0 41.52 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 678μs 3.36μs 13.9μs 0.327 0 0 41.83 KB
master WriteAndFlushEnrichedTraces net472 885μs 3.04μs 11.4μs 8.74 2.62 0.437 53.37 KB
#6212 WriteAndFlushEnrichedTraces net6.0 585μs 2.93μs 13.4μs 0.581 0 0 41.58 KB
#6212 WriteAndFlushEnrichedTraces netcoreapp3.1 733μs 4.24μs 36.7μs 0.361 0 0 41.76 KB
#6212 WriteAndFlushEnrichedTraces net472 872μs 4.01μs 16μs 8.13 2.57 0.428 53.3 KB
Benchmarks.Trace.DbCommandBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6212

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net6.0 1.136 1,413.89 1,245.13

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.41μs 1.37ns 5.29ns 0.0142 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.76μs 1.52ns 5.9ns 0.0141 0 0 1.02 KB
master ExecuteNonQuery net472 2.06μs 2.19ns 8.48ns 0.156 0 0 987 B
#6212 ExecuteNonQuery net6.0 1.24μs 1.31ns 5.06ns 0.0144 0 0 1.02 KB
#6212 ExecuteNonQuery netcoreapp3.1 1.75μs 1.41ns 5.47ns 0.0132 0 0 1.02 KB
#6212 ExecuteNonQuery net472 2.07μs 1.76ns 6.84ns 0.156 0 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.25μs 0.447ns 1.67ns 0.0138 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.5μs 0.832ns 2.88ns 0.0132 0 0 976 B
master CallElasticsearch net472 2.51μs 1.42ns 5.3ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.36μs 0.515ns 1.93ns 0.0131 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.58μs 0.55ns 2.06ns 0.0135 0 0 1.02 KB
master CallElasticsearchAsync net472 2.54μs 1.6ns 5.99ns 0.166 0 0 1.05 KB
#6212 CallElasticsearch net6.0 1.16μs 0.688ns 2.66ns 0.0134 0 0 976 B
#6212 CallElasticsearch netcoreapp3.1 1.51μs 0.563ns 2.18ns 0.0128 0 0 976 B
#6212 CallElasticsearch net472 2.65μs 2.88ns 10.8ns 0.158 0 0 995 B
#6212 CallElasticsearchAsync net6.0 1.29μs 0.57ns 2.21ns 0.0136 0 0 952 B
#6212 CallElasticsearchAsync netcoreapp3.1 1.62μs 1.54ns 5.77ns 0.0136 0 0 1.02 KB
#6212 CallElasticsearchAsync net472 2.66μs 1.88ns 7.27ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.2μs 0.484ns 1.68ns 0.0132 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.61μs 0.434ns 1.62ns 0.0129 0 0 952 B
master ExecuteAsync net472 1.8μs 0.733ns 2.64ns 0.145 0 0 915 B
#6212 ExecuteAsync net6.0 1.24μs 0.451ns 1.75ns 0.0137 0 0 952 B
#6212 ExecuteAsync netcoreapp3.1 1.6μs 0.694ns 2.6ns 0.0129 0 0 952 B
#6212 ExecuteAsync net472 1.75μs 0.575ns 2.23ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.15μs 2.37ns 8.88ns 0.031 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.09μs 2.33ns 8.71ns 0.0381 0 0 2.76 KB
master SendAsync net472 7.76μs 1.9ns 7.35ns 0.498 0 0 3.15 KB
#6212 SendAsync net6.0 4.09μs 1.01ns 3.78ns 0.0309 0 0 2.22 KB
#6212 SendAsync netcoreapp3.1 5.15μs 1.79ns 6.92ns 0.0359 0 0 2.76 KB
#6212 SendAsync net472 7.79μs 3.53ns 13.2ns 0.496 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.55μs 0.725ns 2.62ns 0.0231 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.08μs 0.824ns 3.08ns 0.022 0 0 1.64 KB
master EnrichedLog net472 2.49μs 1.36ns 5.07ns 0.249 0 0 1.57 KB
#6212 EnrichedLog net6.0 1.56μs 0.705ns 2.73ns 0.0226 0 0 1.64 KB
#6212 EnrichedLog netcoreapp3.1 2.11μs 1.28ns 4.95ns 0.0222 0 0 1.64 KB
#6212 EnrichedLog net472 2.61μs 1.57ns 6.09ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 116μs 153ns 573ns 0.0583 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 122μs 86.3ns 311ns 0.0606 0 0 4.28 KB
master EnrichedLog net472 153μs 148ns 572ns 0.69 0.23 0 4.46 KB
#6212 EnrichedLog net6.0 117μs 298ns 1.15μs 0.0585 0 0 4.28 KB
#6212 EnrichedLog netcoreapp3.1 121μs 107ns 401ns 0 0 0 4.28 KB
#6212 EnrichedLog net472 153μs 85ns 318ns 0.687 0.229 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.06μs 0.767ns 2.87ns 0.0307 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.22μs 1.49ns 5.77ns 0.0294 0 0 2.2 KB
master EnrichedLog net472 4.89μs 2.67ns 10.4ns 0.32 0 0 2.02 KB
#6212 EnrichedLog net6.0 3.02μs 0.945ns 3.54ns 0.0305 0 0 2.2 KB
#6212 EnrichedLog netcoreapp3.1 3.99μs 1.29ns 5ns 0.0299 0 0 2.2 KB
#6212 EnrichedLog net472 4.7μs 5.67ns 22ns 0.319 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.39μs 0.816ns 3.16ns 0.016 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.79μs 0.844ns 2.92ns 0.0156 0 0 1.14 KB
master SendReceive net472 2.18μs 1.04ns 4.05ns 0.183 0.00109 0 1.16 KB
#6212 SendReceive net6.0 1.33μs 0.585ns 2.19ns 0.016 0 0 1.14 KB
#6212 SendReceive netcoreapp3.1 1.78μs 0.777ns 3.01ns 0.0151 0 0 1.14 KB
#6212 SendReceive net472 2.2μs 1.11ns 4ns 0.183 0.0011 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.76μs 0.913ns 3.42ns 0.022 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.89μs 2.25ns 8.44ns 0.0215 0 0 1.65 KB
master EnrichedLog net472 4.46μs 4.67ns 18.1ns 0.322 0 0 2.04 KB
#6212 EnrichedLog net6.0 2.92μs 1.09ns 4.22ns 0.022 0 0 1.6 KB
#6212 EnrichedLog netcoreapp3.1 3.9μs 1.14ns 4.42ns 0.0216 0 0 1.65 KB
#6212 EnrichedLog net472 4.42μs 1.31ns 4.73ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6212

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.226 490.44 399.92

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 490ns 0.31ns 1.2ns 0.00809 0 0 576 B
master StartFinishSpan netcoreapp3.1 603ns 0.916ns 3.3ns 0.00775 0 0 576 B
master StartFinishSpan net472 683ns 0.282ns 1.09ns 0.0916 0 0 578 B
master StartFinishScope net6.0 493ns 0.255ns 0.987ns 0.00968 0 0 696 B
master StartFinishScope netcoreapp3.1 754ns 4.12ns 22.5ns 0.0093 0 0 696 B
master StartFinishScope net472 902ns 0.969ns 3.75ns 0.104 0 0 658 B
#6212 StartFinishSpan net6.0 400ns 0.153ns 0.573ns 0.00807 0 0 576 B
#6212 StartFinishSpan netcoreapp3.1 635ns 0.328ns 1.23ns 0.00778 0 0 576 B
#6212 StartFinishSpan net472 674ns 0.247ns 0.956ns 0.0917 0 0 578 B
#6212 StartFinishScope net6.0 498ns 0.207ns 0.773ns 0.00966 0 0 696 B
#6212 StartFinishScope netcoreapp3.1 668ns 0.36ns 1.39ns 0.00938 0 0 696 B
#6212 StartFinishScope net472 896ns 0.377ns 1.46ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 647ns 0.516ns 2ns 0.00971 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 960ns 0.623ns 2.41ns 0.00958 0 0 696 B
master RunOnMethodBegin net472 1.09μs 0.976ns 3.78ns 0.104 0 0 658 B
#6212 RunOnMethodBegin net6.0 685ns 0.606ns 2.35ns 0.00957 0 0 696 B
#6212 RunOnMethodBegin netcoreapp3.1 881ns 1.31ns 5.07ns 0.00921 0 0 696 B
#6212 RunOnMethodBegin net472 1.08μs 0.367ns 1.37ns 0.104 0 0 658 B

@andrewlock
Copy link
Member

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6212) (11.126M)   : 0, 11125824
    master (11.093M)   : 0, 11093158
    benchmarks/2.9.0 (11.081M)   : 0, 11080577

    section Automatic
    This PR (6212) (7.347M)   : 0, 7346839
    master (7.331M)   : 0, 7331071
    benchmarks/2.9.0 (7.732M)   : 0, 7732233

    section Trace stats
    master (7.785M)   : 0, 7785265

    section Manual
    master (11.052M)   : 0, 11052488

    section Manual + Automatic
    This PR (6212) (6.803M)   : 0, 6802615
    master (6.877M)   : 0, 6876901

    section DD_TRACE_ENABLED=0
    master (10.152M)   : 0, 10152233
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6212) (9.769M)   : 0, 9768867
    master (9.614M)   : 0, 9613522
    benchmarks/2.9.0 (9.798M)   : 0, 9798067

    section Automatic
    This PR (6212) (6.664M)   : 0, 6664263
    master (6.631M)   : 0, 6631372

    section Trace stats
    master (6.812M)   : 0, 6811684

    section Manual
    master (9.609M)   : 0, 9609124

    section Manual + Automatic
    This PR (6212) (6.144M)   : 0, 6143736
    master (6.132M)   : 0, 6132262

    section DD_TRACE_ENABLED=0
    master (9.014M)   : 0, 9013904
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6212) (9.947M)   : 0, 9947003
    master (9.865M)   : 0, 9865196
    benchmarks/2.9.0 (10.067M)   : 0, 10067315

    section Automatic
    This PR (6212) (6.443M)   : 0, 6442942
    master (6.549M)   : 0, 6548649
    benchmarks/2.9.0 (7.552M)   : 0, 7552193

    section Trace stats
    master (7.252M)   : 0, 7251983

    section Manual
    master (9.775M)   : 0, 9775031

    section Manual + Automatic
    This PR (6212) (6.197M)   : 0, 6197135
    master (6.137M)   : 0, 6137438

    section DD_TRACE_ENABLED=0
    master (9.294M)   : 0, 9293578
Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewlock andrewlock andrewlock left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
area:asm identified-by:telemetry
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@anna-git @andrewlock