Skip to content

Commit

Permalink
[SF] LabelsParser supports all basic *Options (microsoft#579)
Browse files Browse the repository at this point in the history
It adds support for `LoadBalancingOptions`, `SessionAffinityOptions`, `ProxyHttpRequestOptions`, `PassiveHealthCheckOptions` to `LabelsParser`. This PR also fixes casing of `YARP.Backend.Healthcheck` label's key to be `YARP.Backend.HealthCheck`.

Fixes microsoft#557
  • Loading branch information
alnikola authored Dec 4, 2020
1 parent 0f3e4dd commit dd9ac52
Show file tree
Hide file tree
Showing 8 changed files with 134 additions and 48 deletions.
12 changes: 6 additions & 6 deletions docs/docfx/articles/service-fabric-int.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,11 +74,11 @@ These are the supported parameters:
- `YARP.Enable` - indicates whether the service opt-ins to serving traffic through YARP. Default `false`
- `YARP.EnableDynamicOverrides` - indicates whether application parameters replacement is enabled on the service. Default `false`
- `YARP.Backend.LoadBalancing.Mode` - configures YARP load balancing mode. Optional parameter
- `YARP.Backend.SessionAffinity.*` - configures YARP session affinity. Available parameters and their meanings are provided on [the respective documentation page](xref:session-affinity.md). Optional parameter
- `YARP.Backend.HttpRequest.*` - sets proxied HTTP request properties. Available parameters and their meanings are provided on [the respective documentation page](xref:proxyhttpclientconfig.md) in 'HttpRequest' section. Optional parameter
- `YARP.Backend.HealthCheck.Active.*` - configures YARP active health checks to be run against the given service. Available parameters and their meanings are provided on [the respective documentation page](xref:dests-health-checks.md). There is one label in this group `YARP.Backend.HealthCheck.Active.ServiceFabric.ListenerName` which is not covered by that document because it's SF specific. Its purpose is explained below. Optional parameter
- `YARP.Backend.SessionAffinity.*` - configures YARP session affinity. Available parameters and their meanings are provided on [the respective documentation page](session-affinity.md). Optional parameter
- `YARP.Backend.HttpRequest.*` - sets proxied HTTP request properties. Available parameters and their meanings are provided on [the respective documentation page](proxyhttpclientconfig.md) in 'HttpRequest' section. Optional parameter
- `YARP.Backend.HealthCheck.Active.*` - configures YARP active health checks to be run against the given service. Available parameters and their meanings are provided on [the respective documentation page](dests-health-checks.md). There is one label in this group `YARP.Backend.HealthCheck.Active.ServiceFabric.ListenerName` which is not covered by that document because it's SF specific. Its purpose is explained below. Optional parameter
- `YARP.Backend.HealthCheck.Active.ServiceFabric.ListenerName` - sets an explicit listener name controlling selection of the health probing endpoint for each replica/instance that is used to probe replica/instance health state and is stored on the `Destination.Health` property in YARP's model. Optional parameter
- `YARP.Backend.HealthCheck.Passive.*` - configures YARP passive health checks to be run against the given service. Available parameters and their meanings are provided on [the respective documentation page](xref:dests-health-checks.md). Optional parameter
- `YARP.Backend.HealthCheck.Passive.*` - configures YARP passive health checks to be run against the given service. Available parameters and their meanings are provided on [the respective documentation page](dests-health-checks.md). Optional parameter
- `YARP.Backend.Metadata.*` - sets the cluster's metadata. Optional parameter
- `YARP.Backend.BackendId` - overrides the cluster's Id. Default cluster's Id is the SF service name. Optional parameter
- `YARP.Backend.ServiceFabric.ListenerName` - sets an explicit listener name controlling selection of the main service's endpoint for each replica/instance that is used to route client requests to and is stored on the `Destination.Address` property in YARP's model. Optional parameter
Expand Down Expand Up @@ -154,9 +154,9 @@ Limitations of the current Service Fabric to YARP configuration model conversion
## Architecture
The detailed process of SF cluster polling and model conversion looks as follows.

[ServiceFabricConfigProvider](xref:Microsoft.ReverseProxy.ServiceFabric.ServiceFabricConfigProvider) invokes [IDiscoverer](xref:Microsoft.ReverseProxy.ServiceFabric.IDiscoverer) to discover the current Service Fabric topology, retrieve its metadata and convert everything to YARP configuration model. `IDiscoverer` calls [IServiceFabricCaller](xref:Microsoft.ReverseProxy.ServiceFabric.IServiceFabricCaller) to fetch the following necessary SF entities from the connected SF cluster: Applications, Services, Partitions, Replicas. Basically, `IServiceFabricCaller` forwards all calls to the Service Fabric API, but it also adds a level of resiliency by caching data returned from succesfull calls which later can be served from the cache should subsequent fetches of the same entities fail. Retrieved SF entities get converted to YARP configuration model as explained futher, but it's worth noting that not all of SF topology configurations are supported as it's explained in the section `Known Limitations`.
`ServiceFabricConfigProvider` invokes `IDiscoverer` to discover the current Service Fabric topology, retrieve its metadata and convert everything to YARP configuration model. `IDiscoverer` calls `IServiceFabricCaller` to fetch the following necessary SF entities from the connected SF cluster: Applications, Services, Partitions, Replicas. Basically, `IServiceFabricCaller` forwards all calls to the Service Fabric API, but it also adds a level of resiliency by caching data returned from succesfull calls which later can be served from the cache should subsequent fetches of the same entities fail. Retrieved SF entities get converted to YARP configuration model as explained futher, but it's worth noting that not all of SF topology configurations are supported as it's explained in the section `Known Limitations`.

Service Fabric to YARP model conversion starts by enumerating all SF applications. For each application, `IDiscoverer` retrieves all its services and calls [IServiceExtensionLabelsProvider](xref:Microsoft.ReverseProxy.ServiceFabric.IServiceExtensionLabelsProvider) to fetch all extension labels defined in the service manifests. `IServiceExtensionLabelsProvider` also replaces application parameters references with their actual values if it is enabled for the service by the `YARP.EnableDynamicOverrides` label. At the next step, `IDiscoverer` filters services with enabled YARP integration (`YARP.Enable` label) and calls `LabelsParser` to build YARP's `Clusters`. Then, it begins building `Destinations` by fetching partitions and replicas for each SF service. Partitions are handled as simple replica/instance containers and they don't get converted to any YARP model entities. Replicas and instances are mapped to `Destinations`, but only those which are in `Ready` state are considered. Additionally, there is a control over which stateful service replicas are converted based on their roles as it's specified by `YARP.Backend.ServiceFabric.StatefulReplicaSelectionMode`. By default, all `Primary` and `Active Secondary` replicas are mapped to `Destinations`. A SF replica can expose several endpoints for client requests and health probes, however `IDiscoverer` picks only one of each type to set `Destination`'s `Address` and `Health` properties respectively. This selection logic is controlled by specifying listener names on the two labels `YARP.Backend.ServiceFabric.ListenerName` (for `Destination.Address` endpoint) and `YARP.Backend.HealthCheck.Active.ServiceFabric.ListenerName` (for `Destination.Health` endpoint). If any error is encountered in the conversion process, the given replica gets skipped, but all remaining replicas of the same partition will be considered. Overall, the error handling logic is now quite relaxed and might be tighten up in the future. As the last step of a replica conversion, `IDiscoverer` sends a replica health report to SF cluster. The replica is reported as 'healthy' if the conversion completed successfully, and as 'unhealthy' otherwise.
Service Fabric to YARP model conversion starts by enumerating all SF applications. For each application, `IDiscoverer` retrieves all its services and calls `IServiceExtensionLabelsProvider` to fetch all extension labels defined in the service manifests. `IServiceExtensionLabelsProvider` also replaces application parameters references with their actual values if it is enabled for the service by the `YARP.EnableDynamicOverrides` label. At the next step, `IDiscoverer` filters services with enabled YARP integration (`YARP.Enable` label) and calls `LabelsParser` to build YARP's `Clusters`. Then, it begins building `Destinations` by fetching partitions and replicas for each SF service. Partitions are handled as simple replica/instance containers and they don't get converted to any YARP model entities. Replicas and instances are mapped to `Destinations`, but only those which are in `Ready` state are considered. Additionally, there is a control over which stateful service replicas are converted based on their roles as it's specified by `YARP.Backend.ServiceFabric.StatefulReplicaSelectionMode`. By default, all `Primary` and `Active Secondary` replicas are mapped to `Destinations`. A SF replica can expose several endpoints for client requests and health probes, however `IDiscoverer` picks only one of each type to set `Destination`'s `Address` and `Health` properties respectively. This selection logic is controlled by specifying listener names on the two labels `YARP.Backend.ServiceFabric.ListenerName` (for `Destination.Address` endpoint) and `YARP.Backend.HealthCheck.Active.ServiceFabric.ListenerName` (for `Destination.Health` endpoint). If any error is encountered in the conversion process, the given replica gets skipped, but all remaining replicas of the same partition will be considered. Overall, the error handling logic is now quite relaxed and might be tighten up in the future. As the last step of a replica conversion, `IDiscoverer` sends a replica health report to SF cluster. The replica is reported as 'healthy' if the conversion completed successfully, and as 'unhealthy' otherwise.

Once `Cluster` and all its `Destinations` have been built, `IDiscoverer` calls `IConfigValidator` to check validity of the produced YARP configuration. The validation is performed incrementally on each completed `Cluster` before starting building the next one. In case of validation errors or other service conversion failures, the `Cluster`'s construction gets aborted and a health report gets sent to SF cluster indicating that the respective service is 'unhealthy'. A failure in conversion of one SF service doesn't fail the whole process, so all remaining services in the same SF application will be considered.

Expand Down
10 changes: 5 additions & 5 deletions samples/ReverseProxy.ServiceFabric.Sample/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,10 +76,10 @@ You will need to add the lines indicated below. A brief explanation of labels is
+ <Label Key="YARP.Enable">true</Label>
+ <Label Key="YARP.Routes.route1.Path">{**catchall}</Label>
+ <!-- Optional: enable active health probes -->
+ <Label Key='YARP.Backend.Healthcheck.Active.Enabled'>true</Label>
+ <Label Key='YARP.Backend.Healthcheck.Active.Timeout'>30</Label>
+ <Label Key='YARP.Backend.Healthcheck.Active.Interval'>10</Label>
+ <Label Key='YARP.Backend.Healthcheck.Active.Policy'>ConsecutiveFailures</Label>
+ <Label Key="YARP.Backend.HealthCheck.Active.Enabled">true</Label>
+ <Label Key="YARP.Backend.HealthCheck.Active.Timeout">30</Label>
+ <Label Key="YARP.Backend.HealthCheck.Active.Interval">10</Label>
+ <Label Key="YARP.Backend.HealthCheck.Active.Policy">ConsecutiveFailures</Label>
+ </Labels>
+ </Extension>
</Extensions>
Expand All @@ -97,7 +97,7 @@ Label|Value|Explanation
`YARP.Enable`|`true`|Opt-ins to serving traffic through YARP with Service Fabric dynamic service discovery
`YARP.Routes.route1.Path`|`{**catchall}`|Configures a route, named `route1`, that will match requests with any path. Any [ASP .NET Core route template](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/routing?view=aspnetcore-5.0#route-template-reference) is allowed
`YARP.Routes.route1.Hosts`|`example.com`|Configures a route, named `route1`, that will match requests with host name `example.com`. If both `Hosts` and `Path` are added, only requests matching **both** will resolve to this route.
`YARP.Backend.Healthcheck.Active.*`|`...`|Configures active health checks. YARP will probe each replica of the service at the configured cadence at the provided path.
`YARP.Backend.HealthCheck.Active.*`|`...`|Configures active health checks. YARP will probe each replica of the service at the configured cadence at the provided path.


> NOTE: Label values can use the special syntax `[AppParamName]` to reference an application parameter with the name given within square brackets. This is consistent with Service Fabric conventions, see e.g. [using parameters in Service Fabric](https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-how-to-specify-port-number-using-parameters).
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,7 @@ private async Task DiscoverDestinationsAsync(
}

var listenerName = serviceExtensionLabels.GetValueOrDefault("YARP.Backend.ServiceFabric.ListenerName", string.Empty);
var healthListenerName = serviceExtensionLabels.GetValueOrDefault("YARP.Backend.Healthcheck.Active.ServiceFabric.ListenerName", string.Empty);
var healthListenerName = serviceExtensionLabels.GetValueOrDefault("YARP.Backend.HealthCheck.Active.ServiceFabric.ListenerName", string.Empty);
var statefulReplicaSelectionMode = ParseStatefulReplicaSelectionMode(serviceExtensionLabels, service.ServiceName);
foreach (var partition in partitions)
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
using System.Collections.Generic;
using System.ComponentModel;
using System.Linq;
using System.Net.Http;
using System.Text.RegularExpressions;
using Microsoft.ReverseProxy.Abstractions;

Expand Down Expand Up @@ -106,37 +107,81 @@ internal static List<ProxyRoute> BuildRoutes(Uri serviceName, Dictionary<string,
internal static Cluster BuildCluster(Uri serviceName, Dictionary<string, string> labels)
{
var clusterMetadata = new Dictionary<string, string>();
Dictionary<string, string> sessionAffinitySettings = null;
const string BackendMetadataKeyPrefix = "YARP.Backend.Metadata.";
const string SessionAffinitySettingsKeyPrefix = "YARP.Backend.SessionAffinity.Settings.";
foreach (var item in labels)
{
if (item.Key.StartsWith(BackendMetadataKeyPrefix, StringComparison.Ordinal))
{
clusterMetadata[item.Key.Substring(BackendMetadataKeyPrefix.Length)] = item.Value;
}
else if (item.Key.StartsWith(SessionAffinitySettingsKeyPrefix, StringComparison.Ordinal))
{
if (sessionAffinitySettings == null)
{
sessionAffinitySettings = new Dictionary<string, string>();
}

sessionAffinitySettings[item.Key.Substring(SessionAffinitySettingsKeyPrefix.Length)] = item.Value;
}
}

var clusterId = GetClusterId(serviceName, labels);

var loadBalancingModeLabel = GetLabel<string>(labels, "YARP.Backend.LoadBalancing.Mode", null);
var versionLabel = GetLabel<string>(labels, "YARP.Backend.HttpRequest.Version", null);
#if NET
var versionPolicyLabel = GetLabel<string>(labels, "YARP.Backend.HttpRequest.VersionPolicy", null);
#endif
var cluster = new Cluster
{
Id = clusterId,
LoadBalancing = new LoadBalancingOptions(), // TODO
LoadBalancing = !string.IsNullOrEmpty(loadBalancingModeLabel)
? new LoadBalancingOptions { Mode = (LoadBalancingMode)Enum.Parse(typeof(LoadBalancingMode), loadBalancingModeLabel) }
: null,
SessionAffinity = new SessionAffinityOptions
{
Enabled = GetLabel(labels, "YARP.Backend.SessionAffinity.Enabled", false),
Mode = GetLabel<string>(labels, "YARP.Backend.SessionAffinity.Mode", null),
FailurePolicy = GetLabel<string>(labels, "YARP.Backend.SessionAffinity.FailurePolicy", null),
Settings = sessionAffinitySettings
},
HttpRequest = new ProxyHttpRequestOptions
{
Timeout = ToNullableTimeSpan(GetLabel<double?>(labels, "YARP.Backend.HttpRequest.Timeout", null)),
Version = !string.IsNullOrEmpty(versionLabel) ? Version.Parse(versionLabel + (versionLabel.Contains('.') ? "" : ".0")) : null,
#if NET
VersionPolicy = !string.IsNullOrEmpty(versionLabel) ? (HttpVersionPolicy)Enum.Parse(typeof(HttpVersionPolicy), versionPolicyLabel) : null
#endif
},
HealthCheck = new HealthCheckOptions
{
Active = new ActiveHealthCheckOptions
{
Enabled = GetLabel(labels, "YARP.Backend.Healthcheck.Active.Enabled", false),
Interval = TimeSpan.FromSeconds(GetLabel<double>(labels, "YARP.Backend.Healthcheck.Active.Interval", 0)),
Timeout = TimeSpan.FromSeconds(GetLabel<double>(labels, "YARP.Backend.Healthcheck.Active.Timeout", 0)),
Path = GetLabel<string>(labels, "YARP.Backend.Healthcheck.Active.Path", null),
Policy = GetLabel<string>(labels, "YARP.Backend.Healthcheck.Active.Policy", null),
Enabled = GetLabel(labels, "YARP.Backend.HealthCheck.Active.Enabled", false),
Interval = ToNullableTimeSpan(GetLabel<double?>(labels, "YARP.Backend.HealthCheck.Active.Interval", null)),
Timeout = ToNullableTimeSpan(GetLabel<double?>(labels, "YARP.Backend.HealthCheck.Active.Timeout", null)),
Path = GetLabel<string>(labels, "YARP.Backend.HealthCheck.Active.Path", null),
Policy = GetLabel<string>(labels, "YARP.Backend.HealthCheck.Active.Policy", null)
},
Passive = new PassiveHealthCheckOptions
{
Enabled = GetLabel(labels, "YARP.Backend.HealthCheck.Passive.Enabled", false),
Policy = GetLabel<string>(labels, "YARP.Backend.HealthCheck.Passive.Policy", null),
ReactivationPeriod = ToNullableTimeSpan(GetLabel<double?>(labels, "YARP.Backend.HealthCheck.Passive.ReactivationPeriod", null))
}
},
Metadata = clusterMetadata,
};
return cluster;
}

private static TimeSpan? ToNullableTimeSpan(double? seconds)
{
return seconds.HasValue ? (TimeSpan?)TimeSpan.FromSeconds(seconds.Value) : null;
}

private static string GetClusterId(Uri serviceName, Dictionary<string, string> labels)
{
if (!labels.TryGetValue("YARP.Backend.BackendId", out var backendId) ||
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ public interface IAffinityFailurePolicy
/// <param name="options">Session affinity options set for the cluster.</param>
/// <param name="affinityStatus">Affinity resolution status.</param>
/// <returns>
/// <see cref="true"/> if the failure is considered recoverable and the request processing can proceed.
/// Otherwise, <see cref="false"/> indicating that an error response has been generated and the request's processing must be terminated.
/// 'true' if the failure is considered recoverable and the request processing can proceed.
/// Otherwise, 'false' indicating that an error response has been generated and the request's processing must be terminated.
/// </returns>
public Task<bool> Handle(HttpContext context, ClusterSessionAffinityOptions options, AffinityStatus affinityStatus);
}
Expand Down
Loading

0 comments on commit dd9ac52

Please sign in to comment.