forked from Azure/azure-functions-host
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Benchmarks: update to latest libraries and add tests (Azure#7994)
This builds on the work @mathewc started with benchmarks. There are a lot of things that would be nice to have in benchmark form as we evaluate improving various parts of pipeline performance like async/await elimination, `Task` vs. `ValueTask`, allocations, general algorithmic improvements, etc. Overall: - Upgrades to latest BenchmarkDotNet - Moves the benchmarks project out of tests/ to benchmarks/ (this allows us to do fun things with `Directory.Build.props` and such down the road for _only_ tests and simplifies some things - the move itself is minor just doing it up front. - Adds some benchmarks as examples - a few areas I'm poking at but haven't gotten into PRs yet (these can be dropped from this PR if wanted). To run benchmarks: ```ps1 dotnet run -c Release -f net6.0 --project .\benchmarks\WebJobs.Script.Benchmarks\ ``` This will present a prompt with all benchmarks: ```text Available Benchmarks: #0 AuthUtilityBenchmarks Azure#1 CSharpCompilationBenchmarks Azure#2 ScriptLoggingBuilderExtensionsBenchmarks You should select the target benchmark(s). Please, print a number of a benchmark (e.g. `0`) or a contained benchmark caption (e.g. `AuthUtilityBenchmarks`). If you want to select few, please separate them with space ` ` (e.g. `1 2 3`). You can also provide the class name in console arguments by using --filter. (e.g. `--filter *AuthUtilityBenchmarks*`). ``` Note the instructions to do similar from the command line, e.g.: ```ps1 dotnet run -c Release -f net6.0 --project .\benchmarks\WebJobs.Script.Benchmarks\ --filter *ScriptLogging* ``` Co-authored-by: Lilian Kasem <[email protected]>
- Loading branch information
1 parent
368d5af
commit 6286c54
Showing
15 changed files
with
385 additions
and
46 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| # Benchmarks | ||
|
|
||
| Welcome to the benchmarks! This folder is for code benchmarking (e.g. components rather than end-to-do testing). | ||
| The intent is to benchmark areas we think are interesting and measure improvements as well as ensuring we don't unintentionally regress over time. | ||
|
|
||
| There's a lot of things that would be nice to have in benchmark form as we evaluate improving various parts of pipeline performance like async/await elimination, | ||
| `Task` vs. `ValueTask`, allocations, general algorithmic improvements, etc. This is where those assessments live. | ||
|
|
||
| To run benchmarks (from solution root - otherwise shorten the project path!): | ||
|
|
||
| ```ps1 | ||
| dotnet run -c Release -f net6.0 --project .\benchmarks\WebJobs.Script.Benchmarks\ | ||
| ``` | ||
|
|
||
| This will present a prompt with all benchmarks discovered - something like this: | ||
|
|
||
| ```text | ||
| Available Benchmarks: | ||
| #0 AuthUtilityBenchmarks | ||
| #1 CSharpCompilationBenchmarks | ||
| #2 ScriptLoggingBuilderExtensionsBenchmarks | ||
| You should select the target benchmark(s). Please, print a number of a benchmark (e.g. `0`) or a contained benchmark caption (e.g. `AuthUtilityBenchmarks`). | ||
| If you want to select few, please separate them with space ` ` (e.g. `1 2 3`). | ||
| You can also provide the class name in console arguments by using --filter. (e.g. `--filter *AuthUtilityBenchmarks*`). | ||
| ``` | ||
|
|
||
| Or, you can directly run a set of benchmarks from the command line as noted above: | ||
|
|
||
| ```ps1 | ||
| dotnet run -c Release -f net6.0 --project .\benchmarks\WebJobs.Script.Benchmarks\ --filter *Grpc* | ||
| ``` |
114 changes: 114 additions & 0 deletions
114
benchmarks/WebJobs.Script.Benchmarks/AuthUtilityBenchmarks.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| // Copyright (c) .NET Foundation. All rights reserved. | ||
| // Licensed under the MIT License. See License.txt in the project root for license information. | ||
|
|
||
| using BenchmarkDotNet.Attributes; | ||
| using Microsoft.Azure.WebJobs.Extensions.Http; | ||
| using Microsoft.Azure.WebJobs.Script.WebHost.Security.Authentication; | ||
| using Microsoft.Azure.WebJobs.Script.WebHost.Security.Authorization; | ||
| using System; | ||
| using System.Collections.Generic; | ||
| using System.Linq; | ||
| using System.Security.Claims; | ||
|
|
||
| namespace Microsoft.Azure.WebJobs.Script.Benchmarks | ||
| { | ||
| public class AuthUtilityBenchmarks | ||
| { | ||
| private ClaimsPrincipal Principal; | ||
| private static List<Claim> TotallyRandomClaims { get; } = new List<Claim>() | ||
| { | ||
| new Claim(SecurityConstants.AuthLevelKeyNameClaimType, "test1"), | ||
| new Claim(SecurityConstants.AuthLevelKeyNameClaimType, "test2"), | ||
| new Claim(SecurityConstants.AuthLevelKeyNameClaimType, "test3"), | ||
| new Claim(SecurityConstants.AuthLevelClaimType, nameof(AuthorizationLevel.Function)), | ||
| new Claim(SecurityConstants.AuthLevelClaimType, nameof(AuthorizationLevel.Anonymous)), | ||
| new Claim(SecurityConstants.AuthLevelClaimType, nameof(AuthorizationLevel.User)), | ||
| new Claim(SecurityConstants.AuthLevelClaimType, nameof(AuthorizationLevel.Admin)), | ||
| new Claim(SecurityConstants.AuthLevelClaimType, nameof(AuthorizationLevel.System)), | ||
| }; | ||
|
|
||
| [Params(null, "code")] | ||
| public string KeyName; | ||
|
|
||
| [Params(0, 4, 8)] | ||
| public int ClaimsCount; | ||
|
|
||
| [GlobalSetup] | ||
| public void Setup() | ||
| { | ||
| var identity = new ClaimsIdentity(TotallyRandomClaims.Take(ClaimsCount)); | ||
| Principal = new ClaimsPrincipal(identity); | ||
| } | ||
|
|
||
| [Benchmark(Baseline = true)] | ||
| public bool PrincipalHasAuthLevelClaim() => | ||
| AuthUtility.PrincipalHasAuthLevelClaim(Principal, AuthorizationLevel.Function); | ||
|
|
||
| [Benchmark] | ||
| public bool PrincipalHasAuthLevelClaimNoArray() => | ||
| PrincipalHasAuthLevelClaimNoArray(Principal, AuthorizationLevel.Function); | ||
|
|
||
| [Benchmark] | ||
| public bool PrincipalHasAuthLevelClaimHasClaim() => | ||
| PrincipalHasAuthLevelClaimHasClaim(Principal, AuthorizationLevel.Function); | ||
|
|
||
| public static bool PrincipalHasAuthLevelClaimNoArray(ClaimsPrincipal principal, AuthorizationLevel requiredLevel, string keyName = null) | ||
| { | ||
| // If the required auth level is anonymous, the requirement is met | ||
| if (requiredLevel == AuthorizationLevel.Anonymous) | ||
| { | ||
| return true; | ||
| } | ||
|
|
||
| // Still allocating a enumerate from Identities -> Claims | ||
| foreach (var claim in principal.Claims) | ||
| { | ||
| if (claim.Type == SecurityConstants.AuthLevelClaimType) | ||
| { | ||
| var level = claim.Value switch | ||
| { | ||
| nameof(AuthorizationLevel.Admin) => AuthorizationLevel.Admin, | ||
| nameof(AuthorizationLevel.Function) => AuthorizationLevel.Function, | ||
| nameof(AuthorizationLevel.System) => AuthorizationLevel.System, | ||
| nameof(AuthorizationLevel.User) => AuthorizationLevel.User, | ||
| _ => AuthorizationLevel.Anonymous | ||
| }; | ||
| if (level == AuthorizationLevel.Admin) | ||
| { | ||
| // If we have a claim with Admin level, regardless of whether a name is required, return true. | ||
| return true; | ||
| } | ||
| if (level == requiredLevel && (keyName == null || string.Equals(principal.FindFirstValue(SecurityConstants.AuthLevelKeyNameClaimType), keyName, StringComparison.OrdinalIgnoreCase))) | ||
| { | ||
| return true; | ||
| } | ||
| } | ||
| } | ||
|
|
||
| return false; | ||
| } | ||
|
|
||
| public static bool PrincipalHasAuthLevelClaimHasClaim(ClaimsPrincipal principal, AuthorizationLevel requiredLevel, string keyName = null) | ||
| { | ||
| // If the required auth level is anonymous, the requirement is met | ||
| if (requiredLevel == AuthorizationLevel.Anonymous) | ||
| { | ||
| return true; | ||
| } | ||
|
|
||
| var levelString = requiredLevel switch | ||
| { | ||
| AuthorizationLevel.Admin => nameof(AuthorizationLevel.Admin), | ||
| AuthorizationLevel.Function => nameof(AuthorizationLevel.Function), | ||
| AuthorizationLevel.System => nameof(AuthorizationLevel.System), | ||
| AuthorizationLevel.User => nameof(AuthorizationLevel.User), | ||
| _ => throw new ArgumentOutOfRangeException(nameof(requiredLevel)) | ||
| }; | ||
|
|
||
| return principal.HasClaim(c => c.Type == SecurityConstants.AuthLevelClaimType | ||
| && (c.Value == nameof(AuthorizationLevel.Admin) | ||
| || (c.Value == levelString && (keyName == null || string.Equals(principal.FindFirstValue(SecurityConstants.AuthLevelKeyNameClaimType), keyName, StringComparison.OrdinalIgnoreCase))) | ||
| )); | ||
| } | ||
| } | ||
| } |
74 changes: 74 additions & 0 deletions
74
benchmarks/WebJobs.Script.Benchmarks/CSharpCompilationBenchmarks.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| // Copyright (c) .NET Foundation. All rights reserved. | ||
| // Licensed under the MIT License. See License.txt in the project root for license information. | ||
|
|
||
| using BenchmarkDotNet.Attributes; | ||
| using Microsoft.Azure.WebJobs.Script.Description; | ||
| using Microsoft.Azure.WebJobs.Script.Extensibility; | ||
| using Microsoft.CodeAnalysis; | ||
| using Microsoft.CodeAnalysis.CSharp.Scripting; | ||
| using Microsoft.CodeAnalysis.Scripting; | ||
| using Microsoft.CodeAnalysis.Scripting.Hosting; | ||
| using Microsoft.Extensions.Logging.Abstractions; | ||
| using System; | ||
| using System.IO; | ||
| using System.Runtime.CompilerServices; | ||
| using System.Threading.Tasks; | ||
|
|
||
| namespace Microsoft.Azure.WebJobs.Script.Benchmarks | ||
| { | ||
| public class CSharpCompilationBenchmarks | ||
| { | ||
| // Set of samples to benchmark | ||
| // TODOO: BlobTrigger, needs assembly refs working | ||
| [Params("DocumentDB", "HttpTrigger", "HttpTrigger-Cancellation", "HttpTrigger-CustomRoute", "NotificationHub")] | ||
| public string BenchmarkTrigger; | ||
|
|
||
| // Script source | ||
| private string ScriptPath; | ||
| private static string GetCSharpSamplePath([CallerFilePath] string thisFilePath = null) => | ||
| Path.Combine(thisFilePath, "..", "..", "..", "sample", "CSharp"); | ||
| private string ScriptSource; | ||
| private FunctionMetadata FunctionMetadata; | ||
|
|
||
| // Dyanmic Compilation | ||
| private readonly InteractiveAssemblyLoader AssemblyLoader = new InteractiveAssemblyLoader(); | ||
| private IFunctionMetadataResolver Resolver; | ||
| private CSharpCompilationService CompilationService; | ||
|
|
||
| private IDotNetCompilation ScriptCompilation; | ||
| private DotNetCompilationResult ScriptAssembly; | ||
|
|
||
| [GlobalSetup] | ||
| public async Task SetupAsync() | ||
| { | ||
| ScriptPath = Path.Combine(GetCSharpSamplePath(), BenchmarkTrigger, "run.csx"); | ||
| ScriptSource = File.ReadAllText(ScriptPath); | ||
| FunctionMetadata = new FunctionMetadata() | ||
| { | ||
| FunctionDirectory = Path.GetDirectoryName(ScriptPath), | ||
| ScriptFile = ScriptPath, | ||
| Name = BenchmarkTrigger, | ||
| Language = DotNetScriptTypes.CSharp | ||
| }; | ||
|
|
||
| Resolver = new ScriptFunctionMetadataResolver(ScriptPath, Array.Empty<IScriptBindingProvider>(), NullLogger.Instance); | ||
| CompilationService = new CSharpCompilationService(Resolver, OptimizationLevel.Release); | ||
|
|
||
| ScriptCompilation = await CompilationService.GetFunctionCompilationAsync(FunctionMetadata); | ||
| ScriptAssembly = await ScriptCompilation.EmitAsync(default); | ||
| } | ||
|
|
||
| [Benchmark(Description = nameof(CSharpScript) + "." + nameof(CSharpScript.Create))] | ||
| public Script<object> ScriptCreation() => | ||
| CSharpScript.Create(ScriptSource, options: Resolver.CreateScriptOptions(), assemblyLoader: AssemblyLoader); | ||
|
|
||
| [Benchmark(Description = nameof(CSharpCompilationService) + "." + nameof(CSharpCompilationService.GetFunctionCompilationAsync))] | ||
| public Task<IDotNetCompilation> GetFunctionCompilationAsync() => CompilationService.GetFunctionCompilationAsync(FunctionMetadata); | ||
|
|
||
| [Benchmark(Description = nameof(CSharpCompilationBenchmarks) + "." + nameof(CSharpCompilationBenchmarks.EmitAsync))] | ||
| public Task<DotNetCompilationResult> EmitAsync() => ScriptCompilation.EmitAsync(default); | ||
|
|
||
| [Benchmark(Description = nameof(DotNetCompilationResult) + "." + nameof(DotNetCompilationResult.Load))] | ||
| public void Load() => ScriptAssembly.Load(FunctionMetadata,Resolver, NullLogger.Instance); | ||
| } | ||
| } |
71 changes: 71 additions & 0 deletions
71
benchmarks/WebJobs.Script.Benchmarks/GrpcMessageConversionBenchmarks.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| // Copyright (c) .NET Foundation. All rights reserved. | ||
| // Licensed under the MIT License. See License.txt in the project root for license information. | ||
|
|
||
| using System.Threading.Tasks; | ||
| using BenchmarkDotNet.Attributes; | ||
| using Google.Protobuf.Collections; | ||
| using Newtonsoft.Json; | ||
| using Newtonsoft.Json.Linq; | ||
| using Microsoft.Extensions.Logging; | ||
| using Microsoft.Extensions.Logging.Abstractions; | ||
| using Microsoft.Azure.WebJobs.Script.Grpc; | ||
| using Microsoft.Azure.WebJobs.Script.Grpc.Messages; | ||
| using Microsoft.Azure.WebJobs.Script.Workers.Rpc; | ||
|
|
||
| namespace Microsoft.Azure.WebJobs.Script.Benchmarks | ||
| { | ||
| public class GrpcMessageConversionBenchmarks | ||
| { | ||
| private static byte[] _byteArray = new byte[2000]; | ||
| private static string _str = new string('-', 2000); | ||
| private static double _dbl = 2000; | ||
| private static byte[][] _byteJaggedArray = new byte[1000][]; | ||
| private static string[] _strArray = new string[]{ new string('-', 1000), new string('-', 1000) }; | ||
| private static double[] _dblArray = new double[1000]; | ||
| private static long[] _longArray = new long[1000]; | ||
| private static JObject _jObj = JObject.Parse(@"{'name': 'lilian'}"); | ||
| internal GrpcCapabilities grpcCapabilities = new GrpcCapabilities(NullLogger.Instance); | ||
|
|
||
| // Not easy to benchmark | ||
| // public static HttpRequest _httpRequest; | ||
|
|
||
| [Benchmark] | ||
| public Task ToRpc_Null() => InvokeToRpc(((object)null)); | ||
|
|
||
| [Benchmark] | ||
| public Task ToRpc_ByteArray() => InvokeToRpc(_byteArray); | ||
|
|
||
| [Benchmark] | ||
| public Task ToRpc_String() => InvokeToRpc(_str); | ||
|
|
||
| [Benchmark] | ||
| public Task ToRpc_Double() => InvokeToRpc(_dbl); | ||
|
|
||
| [Benchmark] | ||
| public Task ToRpc_ByteJaggedArray() => InvokeToRpc(_byteJaggedArray); | ||
|
|
||
| [Benchmark] | ||
| public Task ToRpc_StringArray() => InvokeToRpc(_strArray); | ||
|
|
||
| [Benchmark] | ||
| public Task ToRpc_DoubleArray() => InvokeToRpc(_dblArray); | ||
|
|
||
| [Benchmark] | ||
| public Task ToRpc_LongArray() => InvokeToRpc(_longArray); | ||
|
|
||
| [Benchmark] | ||
| public Task ToRpc_JObject() => InvokeToRpc(_jObj); | ||
|
|
||
| public async Task InvokeToRpc(object obj) => await obj.ToRpc(NullLogger.Instance, grpcCapabilities); | ||
|
|
||
| [GlobalSetup] | ||
| public void Setup() | ||
| { | ||
| MapField<string, string> addedCapabilities = new MapField<string, string> | ||
| { | ||
| { RpcWorkerConstants.TypedDataCollection, "1" } | ||
| }; | ||
| grpcCapabilities.UpdateCapabilities(addedCapabilities); | ||
| } | ||
| } | ||
| } |
15 changes: 15 additions & 0 deletions
15
benchmarks/WebJobs.Script.Benchmarks/Microsoft.Azure.WebJobs.Script.Benchmarks.csproj
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
| <PropertyGroup> | ||
| <TargetFrameworks>net6.0</TargetFrameworks> | ||
| <OutputType>Exe</OutputType> | ||
| <!-- BenchmarkDotNet looks for a matching Project, see https://github.com/dotnet/BenchmarkDotNet/issues/1019 --> | ||
| <!--<AssemblyName>Microsoft.Azure.WebJobs.Script.Benchmarks</AssemblyName>--> | ||
| <RootNamespace>Microsoft.Azure.WebJobs.Script.Benchmarks</RootNamespace> | ||
| <LangVersion>Latest</LangVersion> | ||
| </PropertyGroup> | ||
| <ItemGroup> | ||
| <PackageReference Include="BenchmarkDotNet" Version="0.13.1" /> | ||
| <ProjectReference Include="../../src/WebJobs.Script/WebJobs.Script.csproj" /> | ||
| <ProjectReference Include="../../src/WebJobs.Script.WebHost/WebJobs.Script.WebHost.csproj" /> | ||
| </ItemGroup> | ||
| </Project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| // Copyright (c) .NET Foundation. All rights reserved. | ||
| // Licensed under the MIT License. See License.txt in the project root for license information. | ||
|
|
||
| using BenchmarkDotNet.Running; | ||
| using System.IO; | ||
|
|
||
| namespace Microsoft.Azure.WebJobs.Script.Benchmarks | ||
| { | ||
| public static class Program | ||
| { | ||
| public static void Main(string[] args) => | ||
| BenchmarkSwitcher | ||
| .FromAssembly(typeof(Program).Assembly) | ||
| .Run(args, RecommendedConfig.Create( | ||
| artifactsPath: new DirectoryInfo(Path.Combine(Path.GetDirectoryName(typeof(Program).Assembly.Location), "BenchmarkDotNet.Artifacts")) | ||
| )); | ||
| } | ||
| } |
Oops, something went wrong.