Name	Name	Last commit message	Last commit date
parent directory ..
Git-Repo-Files	Git-Repo-Files
README.md	README.md

Provision Azure AKS using Terraform & Azure DevOps

Step-01: Introduction

Create Azure DevOps Pipeline to create AKS cluster using Terraform
We are going to create two environments Dev and QA using single pipeline.
Terraform Manifests Validate
Provision Dev AKS Cluster
Provision QA AKS Cluster

Step-02: Install Azure Market Place Plugins in Azure DevOps

Install below listed two plugins in your respective Azure DevOps Organization
Discuss about which plugin to use
We are going to use plugin-2 as on today it is very actively managed and good reviews and good features
Plugin-1: Terraform by Microsoft Devlabs
Plugin-2: Terraform Build & Release Tasks

Step-03: Review Terraform Manifests

01-main.tf

Comment Terraform Backend, because we are going to configure that in Azure DevOps

02-variables.tf

Two variables we will define in Azure DevOps and use it
- Environment
- SSH Public Key
Just comment the default values here (ideally not needed but we will do that)

03-resource-group.tf

We are going to create resource groups for each environment with terraform-aks-envname
Example Name:
- terraform-aks-dev
- terraform-aks-qa

04-aks-versions-datasource.tf

We will get the latest version of AKS using this datasource.
include_preview = false will ensure that preview versions are not listed

05-log-analytics-workspace.tf

Log Analytics workspace will be created per environment.
Example Name:
- dev-logs-some-random-petname
- qa-logs-some-random-petname

06-aks-administrators-azure-ad.tf

We are going to create Azure AD Group per environment for AKS Admins
To create this group we need to ensure Azure AD Directory Write permission is there for our Service Principal (Service Connection) created in Azure DevOps
We will see that in detail in upcoming steps.
VERY VERY IMPORTANT FIX TO MAKE THIS WORK

07-aks-cluster.tf

Name of the AKS Cluster going to be ResourceGroupName-Cluster
Example Names:
- terraform-aks-dev-cluster
- terraform-aks-qa-cluster
Node Lables and Tags will have a environment with respective environment name

08-outputs.tf

We will put out output values very simple
Resource Group
- Location
- Name
- ID
AKS Cluster
- AKS Versions
- AKS Latest Version
- AKS Cluster ID
- AKS Cluster Name
- AKS Cluster Kubernetes Version
AD Group
- ID
- Object ID

09-aks-cluster-linux-user-nodepools.tf

We will comment this file and leave it that way.
If you need to provision the new nodepool , uncomment all lines except first line and check-in code and new nodepool will be created
Node Lables and Tags will have a environment with respective environment name

10-aks-cluster-windows-user-nodepools.tf

We will comment this file and leave it that way.
If you need to provision the new nodepool windows, uncomment all lines except first line and check-in code and new nodepool will be created
Node Lables and Tags will have a environment with respective environment name

Step-04: Create Github Repository

Create Github Repository in Github

Create Repository in your github
Name: azure-devops-aks-kubernetes-terraform-pipeline
Descritpion: Provision AKS Cluster using Azure DevOps Pipelines
Repository Type: Public or Private (Your Choice)
Click on Create Repository

Copy files, Initialize Local Repo, Push to Remote Git Repo

# Create folder in local deskop
cd azure-devops-aks-demo-repos
mkdir azure-devops-aks-kubernetes-terraform-pipeline
cd azure-devops-aks-kubernetes-terraform-pipeline

# Copy folders from Git-Repo-Files folder to new folder created in local desktop
kube-manifests
terraform-manifests
pipeline-backups


# Initialize Git Repo
cd azure-devops-aks-kubernetes-terraform-pipeline
git init

# Add Files & Commit to Local Repo
git add .
git commit -am "First Commit"

# Add Remote Origin and Push to Remote Repo
git remote add origin https://github.com/stacksimplify/azure-devops-aks-kubernetes-terraform-pipeline.git
git push --set-upstream origin master 

# Verify the same on Github Repository
Refersh browser for Repo you have created
Example: https://github.com/stacksimplify/azure-devops-aks-kubernetes-terraform-pipeline.git

Step-05: Create New Azure DevOps Project for IAC

Go to -> Azure DevOps -> Select Organization -> aksdemo2 -> Create New Project
Project Name: terraform-azure-aks
Project Descritpion: Provision Azure AKS Cluster using Azure DevOps & Terraform
Visibility: Private
Click on Create

Step-07: Create Azure RM Service Connection for Terraform Commands

This is a pre-requisite step required during Azure Pipelines
We can create from Azure Pipelines -> Terraform commands screen but just to be in a orderly manner we are creating early.
Go to -> Azure DevOps -> Select Organization -> Select project terraform-azure-aks
Go to Project Settings
Go to Pipelines -> Service Connections -> Create Service Connection
Choose a Service Connection type: Azure Resource Manager
Authentication Method: Service Princiapl (automatic)
Scope Level: Subscription
Subscription: Pay-As-You-Go
Resource Group: LEAVE EMPTY
Service Connection Name: terraform-aks-azurerm-svc-con
Description: Azure RM Service Connection for provisioning AKS Cluster using Terraform on Azure DevOps
Security: Grant access permissions to all pipelines (check it - leave to default)
Click on SAVE

Step-08: VERY IMPORTANT FIX: Provide Permission to create Azure AD Groups

Provide permission for Service connection created in previous step to create Azure AD Groups
Go to -> Azure DevOps -> Select Organization -> Select project terraform-azure-aks
Go to Project Settings -> Pipelines -> Service Connections
Open terraform-aks-azurerm-svc-con
Click on Manage Service Principal, new tab will be opened
Click on View API Permissions
Click on Add Permission
Select an API: Microsoft APIs
Commonly used Microsoft APIs: Supported legacy APIs: Azure Active Directory Graph
Click on Application Permissions
Check Directory.ReadWrite.All and click on Add Permission
Click on Grant Admin consent for Default Directory

Step-09: Create SSH Public Key for Linux VMs

Create this out of your git repository
Important Note: We should not have these files in our git repos for security Reasons

# Create Folder
mkdir $HOME/ssh-keys-teerraform-aks-devops

# Create SSH Keys
ssh-keygen \
    -m PEM \
    -t rsa \
    -b 4096 \
    -C "azureuser@myserver" \
    -f ~/ssh-keys-teerraform-aks-devops/aks-terraform-devops-ssh-key-ububtu \

Note: We will have passphrase as : empty when asked

# List Files
ls -lrt $HOME/ssh-keys-teerraform-aks-devops
Private File: aks-terraform-devops-ssh-key-ububtu (To be stored safe with us)
Public File: aks-terraform-devops-ssh-key-ububtu.pub (To be uploaded to Azure DevOps)

Step-10: Upload file to Azure DevOps as Secure File

Go to Azure DevOps -> aksdemo2 -> terraform-azure-aks -> Pipelines -> Library
Secure File -> Upload file named aks-terraform-devops-ssh-key-ububtu.pub
Open the file and click on Pipeline permissions -> Authorize for use in all pipelines
Click on SAVE

Step-11: Create Azure Pipeline to Provision AKS Cluster

Go to -> Azure DevOps -> Select Organization -> Select project terraform-azure-aks
Go to Pipelines -> Pipelines -> Create Pipeline

Where is your Code?

Github
Select a Repository: stacksimplify/azure-devops-aks-kubernetes-terraform-pipeline
Provide your github password
Click on Approve and Install on Github

Configure your Pipeline

Select Pipeline: Starter Pipeline
Design your Pipeline
Pipeline Name: 01-terraform-provision-aks-cluster-pipeline.yml

Stage-1: Validate Stage

Stage-1: Terraform Validate Stage
- Step-1: Publish Artifacts to Pipeline (Pipeline artifacts provide a way to share files between stages in a pipeline or between different pipelines. )
- Step-2: Install Latest Terraform (0.13.5) (Ideally not needed if we use default Agents)
- Step-3: Validate Terraform Manifests

trigger:
- main

pool:
  vmImage: 'ubuntu-latest'

# Stage-1: Terraform Validate Stage
## Step-1: Publish Artifacts to Pipeline (Pipeline artifacts provide a way to share files between stages in a pipeline or between different pipelines. )
## Step-2: Install Latest Terraform (0.13.5) (Ideally not needed if we use default Ubuntu Agents)
## Step-3: Validate Terraform Manifests (terraform init, terraform validate)

stages:
- stage: TerraformValidate
  jobs:
    - job: TerraformValidateJob
      continueOnError: false
      steps:
      - task: PublishPipelineArtifact@1
        displayName: Publish Artifacts
        inputs:
          targetPath: '$(System.DefaultWorkingDirectory)/terraform-manifests'
          artifact: 'terraform-manifests-out'
          publishLocation: 'pipeline'
      - task: TerraformInstaller@0
        displayName: Terraform Install
        inputs:
          terraformVersion: 'latest'
      - task: TerraformCLI@0
        displayName: Terraform Init
        inputs:
          command: 'init'
          workingDirectory: '$(System.DefaultWorkingDirectory)/terraform-manifests'
          backendType: 'azurerm'
          backendServiceArm: 'terraform-aks-azurerm-svc-con'
          backendAzureRmResourceGroupName: 'terraform-storage-rg'
          backendAzureRmStorageAccountName: 'terraformstatexlrwdrzs'
          backendAzureRmContainerName: 'tfstatefiles'
          backendAzureRmKey: 'aks-base.tfstate'
          allowTelemetryCollection: false
      - task: TerraformCLI@0
        displayName: Terraform Validate
        inputs:
          command: 'validate'
          workingDirectory: '$(System.DefaultWorkingDirectory)/terraform-manifests'
          allowTelemetryCollection: false

Pipeline Save and Run

Click on Save and Run
Commit Message: First Pipeline Commit - Validate terraform manifests
Click on Job and Verify Pipeline

Stage-12: Deploy Dev AKS Cluster

Stage-2: Deploy Stages for Dev & QA
- Deployment-1: Deploy Dev AKS Cluster
  - Step-1: Define Variables for environments
  - Step-2: Download SSH Secure File
  - Step-3: Terraform Initialize (State Storage to store in Azure Storage Account for Dev AKS Cluster)
  - Step-4: Terraform Plan (Create Plan)
  - Step-5: Terraform Apply (Use the plan created in previous step)
Azure DevOps Pipelines - Deployment Jobs
Azure DevOps Pipelines - Environments

Stage-2: Deployment-1: Deploy Dev AKS Cluster

# Stage-2: Deploy Stages for Dev & QA
# Deployment-1: Deploy Dev AKS Cluster
## Step-1: Define Variables for environments
## Step-2: Download SSH Secure File
## Step-3: Terraform Initialize (State Storage to store in Azure Storage Account for Dev AKS Cluster)
## Step-4: Terraform Plan (Create Plan)
## Step-5: Terraform Apply (Use the plan created in previous step)

# Define Variables
variables:
- name: DEV_ENVIRONMENT
  value: dev 
- name: QA_ENVIRONMENT
  value: qa 
# Stage-2: Deploy Stages for Dev & QA
# Deployment-1: Deploy Dev AKS Cluster
## Step-1: Define Variables for environments
## Step-2: Download SSH Secure File
## Step-3: Terraform Initialize (State Storage to store in Azure Storage Account for Dev AKS Cluster)
## Step-4: Terraform Plan (Create Plan)
## Step-5: Terraform Apply (Use the plan created in previous step)

- stage: DeployAKSClusters
  jobs:
    - deployment: DeployDevAKSCluster
      displayName: DeployDevAKSCluster
      pool:
        vmImage: 'ubuntu-latest'
      environment: $(DEV_ENVIRONMENT)      
      strategy:
        runOnce:
          deploy:
            steps:            
            - task: DownloadSecureFile@1
              displayName: Download SSH Key
              name: sshkey
              inputs:
                secureFile: 'aks-terraform-devops-ssh-key-ububtu.pub'
            - task: TerraformCLI@0
              displayName: Terraform Init
              inputs:
                command: 'init'
                workingDirectory: '$(Pipeline.Workspace)/terraform-manifests-out'
                backendType: 'azurerm'
                backendServiceArm: 'terraform-aks-azurerm-svc-con'
                backendAzureRmResourceGroupName: 'terraform-storage-rg'
                backendAzureRmStorageAccountName: 'terraformstatexlrwdrzs'
                backendAzureRmContainerName: 'tfstatefiles'
                backendAzureRmKey: 'aks-$(DEV_ENVIRONMENT).tfstate'
                allowTelemetryCollection: false
            - task: TerraformCLI@0
              displayName: Terraform Plan
              inputs:
                command: 'plan'
                workingDirectory: '$(Pipeline.Workspace)/terraform-manifests-out'
                environmentServiceName: 'terraform-aks-azurerm-svc-con'
                commandOptions: '-var ssh_public_key=$(sshkey.secureFilePath) -var environment=$(DEV_ENVIRONMENT) -out $(Pipeline.Workspace)/terraform-manifests-out/$(DEV_ENVIRONMENT)-$(Build.BuildId).out'
                allowTelemetryCollection: false
            - task: TerraformCLI@0
              displayName: Terraform Apply
              inputs:
                command: 'apply'
                workingDirectory: '$(Pipeline.Workspace)/terraform-manifests-out'
                environmentServiceName: 'terraform-aks-azurerm-svc-con'
                commandOptions: '$(Pipeline.Workspace)/terraform-manifests-out/$(DEV_ENVIRONMENT)-$(Build.BuildId).out'
                allowTelemetryCollection: false

Pipeline Save and Run

Click on Save and Run
Commit Message: Second Commit - Dev AKS Provision via terraform
Click on Job and Verify Pipeline

Step-13: Verify all the resources created

Verify Pipeline logs

Verify Pipeline logs for all the tasks

Verify new Storage Account in Azure Mgmt Console

Verify if terraform init command ran successfully from Azure Pipelines
Verify Storage Account
Verify Storage Container
Verify tfstate file got created in storage container

Verify new AKS Cluster in Azure Mgmt Console

Verify Resource Group
Verify AKS Cluster
Verify AD Group
Verify Tags for a nodepool

Connect to AKS Cluster

# Setup kubeconfig
az aks get-credentials --resource-group <Resource-Group-Name>  --name <AKS-Cluster-Name>
az aks get-credentials --resource-group terraform-aks-dev  --name terraform-aks-dev-cluster --admin

# View Cluster Info
kubectl cluster-info

# List Kubernetes Worker Nodes
kubectl get nodes

Step-14: Deploy QA AKS Cluster

Stage-2: Deployment-1: Deploy Dev AKS Cluster

# Stage-2: Deploy Stages for Dev & QA
# Deployment-2: Deploy QA AKS Cluster
## Step-1: Download Secure File
## Step-2: Terraform Initialize (State Storage to store in Azure Storage Account)
## Step-3: Terraform Plan 
## Step-4: Terraform Apply

    - deployment: DeployQAAKSCluster
      dependsOn: DeployDevAKSCluster
      displayName: DeployQAAKSCluster
      pool:
        vmImage: 'ubuntu-latest'
      environment: $(QA_ENVIRONMENT)      
      strategy:
        runOnce:
          deploy:
            steps:            
            - task: DownloadSecureFile@1
              displayName: Download SSH Key
              name: sshkey
              inputs:
                secureFile: 'aks-terraform-devops-ssh-key-ububtu.pub'
            - task: TerraformCLI@0
              displayName: Terraform Init
              inputs:
                command: 'init'
                workingDirectory: '$(Pipeline.Workspace)/terraform-manifests-out'
                backendType: 'azurerm'
                backendServiceArm: 'terraform-aks-azurerm-svc-con'
                backendAzureRmResourceGroupName: 'terraform-storage-rg'
                backendAzureRmStorageAccountName: 'terraformstatexlrwdrzs'
                backendAzureRmContainerName: 'tfstatefiles'
                backendAzureRmKey: 'aks-$(QA_ENVIRONMENT).tfstate'
                allowTelemetryCollection: false
            - task: TerraformCLI@0
              displayName: Terraform Plan
              inputs:
                command: 'plan'
                workingDirectory: '$(Pipeline.Workspace)/terraform-manifests-out'
                environmentServiceName: 'terraform-aks-azurerm-svc-con'
                commandOptions: '-var ssh_public_key=$(sshkey.secureFilePath) -var environment=$(QA_ENVIRONMENT) -out $(Pipeline.Workspace)/terraform-manifests-out/$(QA_ENVIRONMENT)-$(Build.BuildId).out'
                allowTelemetryCollection: false
            - task: TerraformCLI@0
              displayName: Terraform Apply
              inputs:
                command: 'apply'
                workingDirectory: '$(Pipeline.Workspace)/terraform-manifests-out'
                environmentServiceName: 'terraform-aks-azurerm-svc-con'
                commandOptions: '$(Pipeline.Workspace)/terraform-manifests-out/$(QA_ENVIRONMENT)-$(Build.BuildId).out'
                allowTelemetryCollection: false

Pipeline Save and Run

Click on Save and Run
Commit Message: Third Commit - QA AKS Cluster Provision via terraform
Click on Job and Verify Pipeline

Step-15: Verify all the resources created

Verify Pipeline logs

Verify Pipeline logs for all the tasks

Verify new Storage Account in Azure Mgmt Console

Verify if terraform init command ran successfully from Azure Pipelines
Verify Storage Account
Verify Storage Container
Verify tfstate file got created in storage container

Verify new AKS Cluster in Azure Mgmt Console

Verify Resource Group
Verify AKS Cluster
Verify AD Group
Verify Tags for a nodepool

Connect to AKS Cluster

# Setup kubeconfig
az aks get-credentials --resource-group <Resource-Group-Name>  --name <AKS-Cluster-Name>
az aks get-credentials --resource-group terraform-aks-qa  --name terraform-aks-qa-cluster --admin

# View Cluster Info
kubectl cluster-info

# List Kubernetes Worker Nodes
kubectl get nodes

Step-16: Make Changes to Infrastructure and Push Code

Add new nodepool named linux102
Create file 11-aks-cluster-linux102-user-nodepools.tf

# Create Linux Azure AKS Node Pool

resource "azurerm_kubernetes_cluster_node_pool" "linux102" {
  availability_zones    = [1, 2, 3]
  enable_auto_scaling   = true
  kubernetes_cluster_id = azurerm_kubernetes_cluster.aks_cluster.id
  max_count             = 3
  min_count             = 1
  mode                  = "User"
  name                  = "linux102"
  orchestrator_version  = data.azurerm_kubernetes_service_versions.current.latest_version
  os_disk_size_gb       = 30
  os_type               = "Linux" # Default is Linux, we can change to Windows
  vm_size               = "Standard_DS2_v2"
  priority              = "Regular"  # Default is Regular, we can change to Spot with additional settings like eviction_policy, spot_max_price, node_labels and node_taints
  node_labels = {
    "nodepool-type" = "user"
    "environment"   = var.environment
    "nodepoolos"    = "linux"
    "ui-app"        = "reactjs-apps"
  }
  tags = {
    "nodepool-type" = "user"
    "environment"   = var.environment
    "nodepoolos"    = "linux"
    "ui-app"        = "reactjs-apps" 
  }
}

Commit Code

# First sync Remote repo with local repo
git pull

# Commit
git add .
git commit -am "Added New Node-Pool linux102"
git push

Verify the pipeline

Connect to Dev AKS Cluster & verify

# List Nodepools
az aks nodepool list --cluster-name terraform-aks-dev-cluster --resource-group terraform-aks-dev -o table

# Setup kubeconfig
az aks get-credentials --resource-group <Resource-Group-Name>  --name <AKS-Cluster-Name>
az aks get-credentials --resource-group terraform-aks-dev  --name terraform-aks-dev-cluster --admin

# View Cluster Info
kubectl cluster-info

# List Kubernetes Worker Nodes
kubectl get nodes

Connect to QA AKS Cluster & Verify

# List Nodepools
az aks nodepool list --cluster-name terraform-aks-qa-cluster --resource-group terraform-aks-qa -o table

# Setup kubeconfig
az aks get-credentials --resource-group <Resource-Group-Name>  --name <AKS-Cluster-Name>
az aks get-credentials --resource-group terraform-aks-qa  --name terraform-aks-qa-cluster --admin


# View Cluster Info
kubectl cluster-info

# List Kubernetes Worker Nodes
kubectl get nodes

Step-17: Clean-Up

Delete the Resource groups which will delete all resources
- terraform-aks-dev
- terraform-aks-qa
Delete AD Groups

Files

25-Azure-DevOps-Terraform-Azure-AKS

Directory actions

More options

Directory actions

More options

Latest commit

History

25-Azure-DevOps-Terraform-Azure-AKS

Folders and files

parent directory

README.md

Provision Azure AKS using Terraform & Azure DevOps

Step-01: Introduction

Step-02: Install Azure Market Place Plugins in Azure DevOps

Step-03: Review Terraform Manifests

01-main.tf

02-variables.tf

03-resource-group.tf

04-aks-versions-datasource.tf

05-log-analytics-workspace.tf

06-aks-administrators-azure-ad.tf

07-aks-cluster.tf

08-outputs.tf

09-aks-cluster-linux-user-nodepools.tf

10-aks-cluster-windows-user-nodepools.tf

Step-04: Create Github Repository

Create Github Repository in Github

Copy files, Initialize Local Repo, Push to Remote Git Repo

Step-05: Create New Azure DevOps Project for IAC

Step-07: Create Azure RM Service Connection for Terraform Commands

Step-08: VERY IMPORTANT FIX: Provide Permission to create Azure AD Groups

Step-09: Create SSH Public Key for Linux VMs

Step-10: Upload file to Azure DevOps as Secure File

Step-11: Create Azure Pipeline to Provision AKS Cluster

Where is your Code?

Configure your Pipeline

Stage-1: Validate Stage

Pipeline Save and Run

Stage-12: Deploy Dev AKS Cluster

Stage-2: Deployment-1: Deploy Dev AKS Cluster

Pipeline Save and Run

Step-13: Verify all the resources created

Verify Pipeline logs

Verify new Storage Account in Azure Mgmt Console

Verify new AKS Cluster in Azure Mgmt Console

Connect to AKS Cluster

Step-14: Deploy QA AKS Cluster

Stage-2: Deployment-1: Deploy Dev AKS Cluster

Pipeline Save and Run

Step-15: Verify all the resources created

Verify Pipeline logs

Verify new Storage Account in Azure Mgmt Console

Verify new AKS Cluster in Azure Mgmt Console

Connect to AKS Cluster

Step-16: Make Changes to Infrastructure and Push Code

Connect to Dev AKS Cluster & verify

Connect to QA AKS Cluster & Verify

Step-17: Clean-Up

References