Multi-region, active-active deployment automation

[davidthor]

Hi everyone – our team is testing out Github Discussions as a means of sharing our feature ideas and getting feedback from peer developers. We'd love to hear what you think of some of our wilder ideas, so please let us know if you have any questions/comments/concerns that will help us design the most helpful features in the best ways.

Overview

Larger scale organizations often find themselves needing a higher degree of reliability and availability than can be offered by a single datacenter or container platform. Software fails after all, and the cloud providers and datacenters are no exception. In order to defend themselves from this kind of failure, software companies will often instrument an active-active deployment architecture.

This is a tactic that includes deploying your entire application to multiple different datacenters and allowing each datacenter to re-route to each other in the event they're experiencing outages. When instrumented correctly, this tactic has the added benefit of allowing you to host your software in datacenters closest to your users to ensure they receive faster response times and an better experience. A typical active-active deployment architecture looks something like this:

While multi-region, active-active deployment strategies are popular for large organizations, it often requires a large scale, collaborative effort between DevOps and SREs in order to instrument it. The SREs need to setup the clusters, build secure gateways connecting the clusters for failover, and instrument the edge gateways that will receive external requests and forward them to. DevOps needs to setup deployment pipelines to deploy to each region and needs to configure the ingress and edge gateways to expose only the relevant apps and services.

In our efforts to make environment provisioning and deployment as easy as possible, we recently asked ourselves if there's any support we can provide to ease the burdens associated with deploying and managing multi-region environments. Right now teams can register multiple regions as individual environments and deploy to them each separately, but cannot treat multi-region or multi-cloud environments as a single deploy target. This forces developers to know which regions are available explicitly and ensure they can deploy to each in their own pipelines, and also forces SREs to figure out how to configure the ingress and mesh gateways respectively.

Feature proposal

What we want to do is abstract away the fact that an environment may be backed by more than one region and allow developers to deploy to a single environment regardless of how many regions the code gets shipped to. This would allow SREs to expand or contract the number of regions without developers needing to change their workflows for deployment.

Current experience	Proposed experience

In order to make this a reality, we're looking to allow environment operators to have a few extra features available when they setup their environment:

1. The ability to select more than one container platform to back the environment – this would allow Architect to deploy components to all platforms simultaneously, and would also allow Architect to automatically instrument the cross-region, secure gateways that fulfill cluster failover.
2. The ability to register and select an edge gateway instance that will relay traffic to each cluster – whenever a new interface is exposed in the environment, Architect can configure the edge gateway to respond to the new host and forward to terminate traffic respectively.

What we're looking for

We'd love to hear from any developers or SREs interested in or currently implementing multi-region architectures. We want to make sure that we have a firm handle of the desired architecture as well as a good experience for creating and managing it. Please feel free to reply with free-form thoughts, but if you're looking for inspiration or structure then we'd love to know your answers to the following:

• What metrics did/will your team use to determine they need a multi-cloud/region strategy? New customer SLAs? Regional distribution of customers? Revenue per region? Something else?
• How much more likely would you be to instrument a multi-region strategy if it was automated using the approach above?
• What is missing (if anything) from our assumptions about an active-active architecture and deployment model?
• If you're team already has a multi-region strategy:
  • How much money/time/resources was used to setup your multi-region architecture and processes? What things did you miss in the first attempt(s) (if any) that forced you to re-design the implementation?
  • What mesh or federation tools do you use, and what made you choose it? Would it matter to you which tool we used in our deployments?

Thanks in advance for your help!

[tjhiggins]

Chiming in - love this concept. I'm curious how much automation would be desired. I see a couple options:

1. Semi-Automated

Architect deploys to multiple clusters and surfaces the list of load balancer ips. Then its up to the user to set up the networking.

Pros:

• User has full control of the routing/rules
• Flexible integration with any third party tool that support load balancer pools

Cons:

• Manual steps

2. Automated

Architect deploys to multiple clusters and auto configures the load balancer pools with a service like Cloudflare.
https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/resources/load_balancer_pool

Pros:

• Completes vision of one click multi-cloud

Cons:

• Not all users will want the load balancing managed by Architect

3. Automated + Private Cloudflare (lb providers)

Architect allows you to register your Cloudflare zones on an account. Then automates the creation of lbs/pools in that zone.

Pros:

• Best of both worlds

Cons:

• TBD

Obviously we could do all three, but would love feedback!