buildbot-ci

Steps needed in order to build a Continuous Integration on cloud as well all the configuration specific configuration.

⚠️	The code and instructions fits to my purposes and requirements. However this repository could be useful if you are looking for how to do things using the tools that I used.

Table of Contents

• Motivation
• Tools
• Use case
• Configuration
• Steps
  • Use ansible-vault to manage sensitive files
  • Create the bootstrap AWS infrastructure
  • Create AMI images using Packer
  • Destroy the bootstrap AWS infrastructure
  • Create buildbot AWS infrastructure
  • Configure buildbot master
  • Teardown buildbot master
  • Destroy AWS infrastructure
• References

Motivation

A long time ago I heard about buildbot project as a framework to build Continuous Integration. In that time I thought that it could be a waste of time because there are many powerful and friendly tools (like Jenkins, Bamboo, Gitlab) as well SaaS solutions like travisCI, circleCI. So why anyone would like to spend time configuring a buildbot project ?

The quick answer is: flexibility and resource control when you need to build projects which use high CPU and disk resources.

After a while trying to use SaaS tools to get work with a huge time consuming build, I gave up and change my mind to give a try and test buildbot.

ℹ️	CI SaaS tools are great. I use to use in my other projects. But for this particular case those tools don’t fit.

To sum up, my aim is to provide a guide about how to use buildbot and tools that helped me when I was building the environment. There are many details captured by Packer, Terraform and Ansible configuration files and worth reading them in case of trying this out.

Tools

Ansible

is the tool to configure buildbot master server and any additional service needed. Like applying github webhook integration

Terraform

creates and destroys cloud infrastructure. The tool allows describing the cloud environment and versioning it using any source code control system

Packer

creates machine images installing and configuring based on a template that can be easily reused

AWS Command Line Interface

AWS default command line tool

Also three additional playbooks do all the buildbot configuration steps:

ansible-role-buildbot

a multi OS role that configures and installs buildbot master

ansible-role-worker

also a multi OS role that does all the steps need to configure a buildbot worker

ansible-role-awscli

a basic role install and setup AWS Command Line Interface

Use case

The main use case that this project solves is as follow:

1. A developer pushes code or open a Pull Request on Github repository

2. Github calls the buildbot master webhook, previously configured

   1. buildbot master receives all the context to initiate a new worker

   2. a new latent buildbot worker is spawned to handle the specific build context

   3. after finish the worker, the buildbot master sends the status and pull request back to Github

   4. the latent buildbot worker is finished after 10 minutes of inactivity

3. the developer can see the build results on Github or buildbot web view.

ℹ️	Maybe you are thinking that it is hard to buy the idea. Any CI SaaS does the same tasks. Sure and I agree. However, think about the possibilites to enhance your build CI using AWS services for example. E.g.: cache layers, increasing machine power, coordinate large scale tests.

Configuration

This repository has some steps in order to prepare the tools to work together.

Ansible external roles need to be installed

ansible-galaxy install -r ansible/requirements.yml

Creates ssh key pair

ssh-keygen -t rsa -b 4096 -C "buildbot" -f ~/.ssh/buildbot.key

Add buildbot key to keychain

ssh-add ~/.ssh/buildboot.key

Setup AWS CLI

the Quickly Configuring the AWS CLI is a good guide about how to get start with AWS CLI.

Create the Ansible vault file password

The file buildbot-ci/ansible/.vault should be create with a valid password to decrypt sensitive variables from ansible

echo "MY PASS" > ansible/.vault

Steps

ℹ️	All commands below are suppose to be running in buildbot-ci directory.

Use ansible-vault to manage sensitive files

The file group_vars/tag_Type_master/vault.yml has sensitive data and should be protect using ansible-vault.

ansible-vault [edit | view] \
  --vault-password-file .vault \
  group_vars/tag_Type_master/vault.yml

This file should be updated when changed any AWS or Github access parameters.

Create the bootstrap AWS infrastructure

The directory networkTerraform has a Terraform code that creates a basic networking in AWS, just the necessary elements to run :packer: later.

cd networkTerraform
terraform apply

Create AMI images using Packer

After running the initial Terraform environment, Terraform will output a variable called public_subnets. Packer will use that subnet to create the initial images there.

Calling Packer for each image template to create the images. When creating a buildbot worker AMI, Packer will need some extra variables from previous Terraform execution.

cd packer
packer build -var 'subnet_id=<subnet output by terraform>' bb-master.json
packer build -var 'subnet_id=<subnet output by terraform>' \
    -var 'aws_profile=<aws buildbot profile>' \
    -var 'aws_access_key_id=<aws buildpot access key>' \
    -var 'aws_secret_access_key=<aws buildbot secret key>' \
    bb-worker.json

Packer creates two private AMI with all the software installed and configured by Ansible.

Destroy the bootstrap AWS infrastructure

After running the Packer templates, The follow procedure destroys the bootstrap environment.

cd networkTerraform
terraform destroy

Create buildbot AWS infrastructure

Running Terraform to create the buildbot infrastructure.

cd terraform
terraform apply

Configure buildbot master

In the previously step, Terraform output some important variables that Ansible need when running the playbook to setup the correct buildbot master configuration values.

The follow variables should get from Terraform:

• url-bb-master

• public-subnets

• security-groups

And update the follow variables in the file ansible/group_vars/tag_Type_master/vars.yml:

• buildbot_aws_subnet_id, subnet where the buildmaster worker will live

• buildbot_aws_security_group_id: security group to attach each worker

• buildbot_aws_url: public Elastic IP Address where buildbot master is receiving requests

Also, any other buildbot parameter should be updated before running the ansible/bb-master-configure.yml playbook.

cd ansible
ansible-playbook \
  --vault-password-file .vault \
  -u ubuntu \
  -i config_aws_ec2.yml \
  -T 300 \
  bb-master-configure.yml

Teardown buildbot master

The playbook ansible/bb-master-teardown.yml deal with all the operations when removing any configuration done in external services, like Github.

cd ansible
ansible-playbook \
  --vault-password-file .vault \
  -u ubuntu \
  -i config_aws_ec2.yml \
  -T 300 \
  bb-master-teardown.yml

Destroy AWS infrastructure

The infrastructure can be destroyed using Terraform

cd terraform
terraform destroy

References

• [buildbot-tale] Buildbot: a tale with examples of one more continuous integration system, https://sudonull.com/post/977-Buildbot-a-tale-with-examples-of-one-more-continuous-integration-system

• [buildbot] Buildbot Manual, http://docs.buildbot.net/current/index.html

• Autobuilder2 presentation at the Yocto Project summit 2019, https://koansoftware.com/autobuilder2-talk-yocto-project/