terraform-multienv

A template for maintaining a multiple environments infrastructure with Terraform. This template includes a CI/CD process, that applies the infrastructure in an AWS account.

environment	drone.io	GitHub Actions	Circle Ci	Travis CI
dev
stg
prd

Assumptions

• Branches names are aligned with environments names, for example dev, stg and prd

• The CI/CD tool supports the variable ${BRANCH_NAME}, for example ${DRONE_BRANCH}

• The directory ./live contains infrastructure-as-code files - *.tf, *.tpl, *.json

• Multiple Environments

  • All environments are maintained in the same git repository
  • Hosting environments in different AWS account is supported (and recommended)

• Variables

  • ${app_name} = tfmultienv
  • ${environment} = dev or stg or prd

Getting Started

1. We're going to create

   • AWS VPC, Subnets and Routing Tables per environment (all free)
   • Terraform remote backend - S3 bucket and DynamoDB table

2. Create a new GitHub repository by clicking - Use this template

3. Edit ./.drone.yml - Find and Replace tfmultienv and eu-west-1

4. CI/CD setup

   1. AWS Console > Create an IAM User for CI/CD per environment

      • Name: cicd-${environment}
      • Permissions: Allow Programmatic Access and attach the IAM policy AdministratorAccess (See Recommendations)
      • Create AWS Access Keys and save them in a safe place, we'll use them in the next step

   2. drone.io > Create repository secrets for AWS Access Keys per environment

      • Sign in with your GitHub account to drone.io and activate your newly created git repository
      • Create secrets per environment for each AWS Access Keys pair, for example
        1. aws_access_key_id_dev
        2. aws_secret_access_key_dev
           IMPORTANT: The names of the secrets are not arbitrary, make sure you set them as shown in the example above

5. Deploy infrastructure - Commit and push the changes to your repository

   git checkout dev
   git add .
   git commit -m "deploy dev"
   git push --set-upstream origin dev

6. Check out your CI/CD logs in Drone Cloud and the newly created resources in AWS Console > VPC.
   To watch the CI/CD logs of this repository - unfor19/terraform-multienv

7. Promote dev environment to stg

   git checkout stg
   git merge dev
   git push

8. That's it, you've just deployed two identical environments, go ahead and do the same with prd

Recommendations

Generic

• Naming Convention should be consistent across your application and infrastructure. Avoid using master for production. A recommended set of names: dev, tst (qa), stg and prd. Using shorter names is preferred, since some AWS resources' names have a character limit
• Resources Names should contain the environment name, for example tfmultienv-natgateway-prd
• Terraform remote backend costs are negligible (less than 1$ per month)
• Using Multiple AWS Accounts for hosting different environments is recommended.
  The way I implement it - dev and stg in the same account and prd in a different account

Terraform

• backend.tf.tpl - Terraform Remote Backend settings per environment. The script prepare-files-folders.sh replaces APP_NAME with TF_VARS_app_name and ENVIRONMENT with BRANCH_NAME
• Remote Backend is deployed with a CloudFormation template to avoid the chicken and the egg situation
• Locked Terraform tfstate occurs when a CI/CD process is running per environment. Stopping and restarting, or running multiple deployments to the same environment will result in an error. This is the expected behavior, we don't want multiple entities (CI/CD or Users) to deploy to the same environment at the same time
• Unlock Terraform tfstate by deleting the md5 item from the state's DynamoDB table, for example
  • Table Name: ${app_name}-state-lock-${environment}
  • Item Name: ${app_name}-state-${environment}/terraform.tfstate-md5

Security

• AdministratorAccess Permission for CI/CD should be used only in early dev stages. After running a few successful deployments, make sure you restrict the permissions per environment and follow the least-previleged best practice. Use CloudTrail to figure out which IAM policies the CI/CD user needs, a great tool for that - trailscraper
• IAM Roles for self-hosted CI/CD runners (nodes) are preferred over AWS key/secret

Git

• Default Branch is dev since this is the branch that is mostly used
• Branches Names per environment makes the whole CI/CD process simpler
• Feature Branch per environment complicates the whole process, since creating an environment per feature-branch means creating a Terraform Backend per feature-branch. Though it is possible, it's not recommended
• Updating Environment Infrastructure is done with git push and git merge, this way we can audit the changes

Repository Structure

• Modules should be stored in a different repository
• Infrastructure Repository should separated from the Frontend and Backend Respositories

References

• To get started with Terraform, watch this webinar - Getting started with Terraform in AWS
• Terraform Best Practices - ozbillwang/terraform-best-practices

Authors

Created and maintained by Meir Gabay

License

This project is licensed under the MIT License - see the LICENSE file for details