Deploying Azure Red Hat OpenShift (ARO) is a fairly straightforward process, by following the official documentation, creating the required Azure networking infrastructure, and running the deployment command, a highly available OpenShift cluster will become available and ready to run containerised workloads in approximately 30-40 minutes.
But ARO is just one piece of the secure workload and cloud puzzle. How does ARO integrate with an organisation's existing Azure environment, especially infrastructure that has been architected and deployed with security in mind?
This repo aims to provide insight into this question. Using a Github actions workflow this repo will deploy a functioning Azure Red Hat OpenShift Landing Zone, and configure ARO with some essential day 2 Azure integrations.
ARO is a first party service on Azure which means that it's part of the Azure API, as such I have purposely created this repo using tooling that is familiar to Azure customers to prove that ARO can be integrated into Azure environments the same way that they can deploy their other Azure resources. It is designed to take customers from nothing to an private environment with ARO as the destination for containerised workloads. Supporting this environment are all the recommended secure Azure resources including managed identities, keyvault, and private endpoints.
Let's start by looking at the High level architecture. Aligning with Azure's landing zone implementation the deployment creates three resource groups;
- A Network Hub resource group
- Contains resources to provide secure access into the private network such as Azure firewall and a Bastion for a Jumpbox virtual machine. This hub is capable of servicing multiple spoke resource groups and therefore multiple ARO clusters.
- A Spoke resource group
- Contains Azure Red Hat OpenShift (ARO) and an Azure Front Door load balancer which will expose an application on ARO to the internet.
- A Services resource group
- This resource group logically separates essential Azure resources used in the secure deployment and can be re-used for applications running on ARO. Resources such as keyvault, storage and log analytics workspace. Access to these resources is provided to the ARO and the private network via Azure private endpoints.
Click Here for a description of the Azure Red Hat OpenShift Landing Zone Resources.
Click Here for a description of the Azure Red Hat OpenShift Landing Zone deployment Github actions workflow jobs.
Configuring resources which are restricted to private networks can be quite problematic using public tooling such as Github actions as there is no direct connectivity. The solution to this problem was to initially use bicep to deploy the container registry as a public resource, build a container from the provided Dockerfile and then run bicep over the container registry again but this time modifying it so that it was locked down to private network access using a private endpoint only.
- Copy this repo to a private repo of your own
- Create a Github personal access token
- Create an Azure Application Registration/Service principal
- Create 3 resource groups and grant the Azure Service Principal
Contributor
andUser Access Administration
permissions scoped to these resource groups. - Create Github repository secrets and variables (use the helper script to make it easy)
- Run the Github actions workflow
- Log into Azure Red Hat OpenShift using your Azure EntraID Directory account
- az command line
- jq
- curl
- gh command line
Make a copy of this repo to your Github account using the template feature, ensure that it is a private repo, this wil prevent accidental leaking of secrets and avoid anyone from the public running the Github actions workflow.
Warning Please be careful about how you store secrets. It is advised to use a private repo to ensure that there is a less chance of private data exposure.
The Github Personal access token is injected as a secure environment variable into the Github runner container which is used in the workflow so that it can register itself to your Github repository as a runner and poll for jobs.
- Create a personal access token
- Github > User Settings > Developer Settings > Personal access tokens (or https://Github.com/settings/tokens)
- Set an expiry and add a note 'used for ARO deployment'
- Grant permissions for "repo", "workflow", "admin:org", "admin:public_key"
- Save the token
- Run the following command
gh auth login
and paste in the token - Github.com, SSH, <Select SSH Key>, Paste an authentication token
- Insert the PAT as a Github secret as per the "secrets" table below "PAT_Github"
Create an EntraID Security Group which will contain the users that will be administrators for Azure Red Hat OpenShift (ARO). You will need to place the
export SG_NAME="<insert name of security group for openshift administrators>"
az ad group create --display-name "$SG_NAME" --mail-nickname "mysecuritygroup" --security-enabled true
az ad group list --filter "displayName eq 'SG_NAME'" --query "[].objectId" --output tsv
For an Azure Red Hat OpenShift (ARO) private cluster deployment, a Red Hat pull secret is still needed to download and use Red Hat images for Operators found in the Operator Hub. A pull secret is essentially an authentication token that provides access to Red Hat's registry, which contains container images and other software packages. If you do not have a pull-secret from Red Hat already then please browse to cloud.redhat.com to create one. The Red Hat pull secret will be used as a Github repository secret.
Place the Red Hat pull secret in a file called pull-secret.json
at the root of this repo to ensure that the helper scripts can read it and write it to Github repository secrets for use in the Github actions workflow.
For this repo an Azure app registration/service principal is required. It has a few uses;
-
Deploying ARO - An app registration/service principal is needed to provide secure access to Azure resources that are required to deploy and manage an ARO cluster. A app registration/service principal is essentially an identity for your ARO cluster that allows it to access and manage Azure resources such as virtual networks, storage accounts, and other resources required by the cluster. By creating an app registration/service principal, you can grant your ARO cluster the necessary permissions to access and manage Azure resources without giving it full access to your entire Azure subscription.
-
Deploy Resources to Azure using Github actions - The Azure service principal is used by Github Actions to authenticate and authorize the deployment of your code to Azure resources. This allows Github Actions to perform actions such as creating or updating resources, deploying code to virtual machines, and more. The permissions that are granted to the service principal are
Contributor
andUser access admin
scoped to the three resource groups that are created.
Contributor
permits resource creationUser access admin
permits adding the Azure Red Hat OpenShift Resource Provider the appropriate permissions during the Azure installation.
-
OIDC Login for GitHub Actions With OpenID Connect (OIDC), you can configure your workflow to request a short-lived access token directly from the cloud provider. Your cloud provider also needs to support OIDC on their end, and you must configure a trust relationship that controls which workflows are able to request the access tokens. Providers that currently support OIDC include Amazon Web Services, Azure, Google Cloud Platform, and HashiCorp Vault, among others.
-
Credentials for ARO to use Entra as identity provider - During the configuration of the OpenShift identity provider the Azure service principal is used.
Set the name for the EntraID App Registration/Service Principal
export DISPLAYNAME="<Desired Entra App Registration/Service Principal name>"
Now create the app registration and the service principal. We will also start to configure this object for EntraID Group claims on OpenShift.
az ad app create --display-name=$DISPLAYNAME --sign-in-audience AzureADMyOrg --optional-claims "{\"idToken\":[{\"name\":\"preferred_username\",\"source\": null,\"essential\": false,\"additionalProperties\":[]},{\"name\":\"email\",\"source\": null,\"essential\": false,\"additionalProperties\": []}]}"
export AAD_APP_CLIENT_ID=$(az ad app list --filter "displayname eq '$DISPLAYNAME'" --query '[].appId' -o tsv)
echo $AAD_APP_CLIENT_ID
az ad sp create --id $AAD_APP_CLIENT_ID
- Now create a password for the App Registration / Service Principal
export AAD_CLIENT_SECRET=$(az ad app credential reset --id $AAD_APP_CLIENT_ID --query password -o tsv)
- Now run gh_secrets_create.sh to populate github values (YOU WILL NEED TO READ/CREATE GITHUB PERSONAL ACCESS TOKEN FIRST)
gh api user -q .login
gh auth logout
gh auth login
gh api user -q .login
Select the following options
- Github.com
- HTTPS
- Paste your authentication token
You should see output like the following
- gh config set -h github.com git_protocol https
✓ Configured git protocol
✓ Logged in as grantomation
Set the following variables in the file helper_vars.sh
to ensure that the helper scripts can run sucessfully.
Helper Var Name | Description |
---|---|
DISPLAYNAME | The name for the App registration/service principal that you created above. |
GH_REPO | Your copy of this github repository (private) where you are running this code from. |
GH_ORGANISATION | Your github username (or organisation). |
GH_BRANCH | Set the branch where you will run the github actions from (default master ) |
PAT_GITHUB | The Personal Access Token for Github that you created above |
LOCATION | Azure Region where cloud resources will be deployed |
HUB_RG | The name of the resource group which will store Network Hub services such as Azure Firewall |
SPOKE_RG | The name of the resource group which will store Azure Red Hat OpenShift (ARO) components |
SERVICES_RG | The name of the resource group which will store services such as keyvault and ACR |
AAD_ADMIN_GROUP_ID | The ID of the Azure EntraId Security Group. Get ID by running the following command az ad group show -g <AAD GROUP NAME> --query id -o tsv |
JUMPBOX_ADMIN_USER | The name of the Jumpbox Virtual Machine Admin User |
JUMPBOX_ADMIN_PWD | The password for the Jumpbox Virtual Machine Admin User |
CONTAINER_BUILD_NAME | The name and tag for the Github runner container that is build default aro-github-runner:1 |
All other variables in this file will be determined via lookup commands using az
cli
- Modify the parameters found in
./action_params/*.json
to suit your Azure environment.
The following parameter names require unique names - so please be mindful of this;
Cloud Resource | Parameter Name | Location in Code |
---|---|---|
Storage Account | storageAccountName |
action_params/storage.parameters.json |
Azure Container Registry | acrName |
action_params/acr.parameters.json |
Azure Keyvault | keyVaultName |
action_params/keyvault.parameters.json |
Azure Red Hat OpenShift (ARO) Domain | domain |
action_params/aro.parameters.json |
The following Azure services need to be registered prior to deployment;
az provider register -n Microsoft.RedHatOpenShift
az provider register -n Microsoft.Storage
az provider register -n Microsoft.KeyVault
az provider register -n Microsoft.Compute
az provider register -n Microsoft.ContainerRegistry
az provider register -n Microsoft.Cdn
az provider register -n Microsoft.ContainerInstance
az provider register -n Microsoft.OperationalInsights
az provider register -n Microsoft.Insights
By default ARO uses a minimum 40 cores and this will need to be accounted for in the Azure Quota.
Default virtual machine types for ARO are "Standard D4s v3" machines.
⚠️ Remember to log in to thegh
cli tool prior to thegh_secrets_create.sh
helper script.
Github repository secrets are encrypted environment variables that you can store in a repository on Github. These secrets are used to store sensitive information that is required by your Github Actions workflows, such as API keys, access tokens, and other secrets. Github Actions repository variables are environment variables that you can define for a specific repository in Github. These variables can be used in Github Actions workflows to store values that are specific to that repository.
Ensure that you are logged in to the Azure CLI with a user that has appropriate permissions and execute the following commands;
chmod +x helper_vars.sh
chmod +x gh_secrets_create.sh
./gh_secrets_create.sh
Prior to running the workflow a user with the appropriate permissions must create the initial resource groups with correct permissions for the App registration/service principal.
Run ./lz_rg_create.sh
to perform these actions for you.
chmod +x lz_rg_create.sh
./lz_rg_create.sh
To run the Github actions to deploy the environment select the following;
To complete a full cleanup of Azure resources execute the delete helper script ./lz_rg_delete.sh
which will run resource group delete commands for you. You will need to change the variables at the top of the file to suit your environment.
An azure keyvault stays in a deleted state for approximately 90 days after deletion. This script will also purge the keyvault to ensure that there are no failures on the next ARO deployment.
These values can be set in the action_params/aro.parameters.json
file in the controlPlaneVmSize
and computeVmSize
parameters.