An opinionated way to roll out BUCC on vsphere.
BUCC ensures a seamlessly integrated BOSH, UAA, Concourse and CredHub. The integration between those component is also thoroughly tested upstream by BUCC so we dont have to worry about versioning and compatibility issues.
By default, BUCC runs with all jobs (i.e. processes) co-located on one VM created by the command create-env. This project extends BUCC and puts the default Concourse worker job on its own VM. This makes it easy to recreate the VM in case of problems via the bosh CLI.
A similar approach can be applied to the other bosh jobs by following the same pattern if required. e.g. if you are worried about downtime on Concourse Web during BUCC upgrades.
The project also supports the following integrated components managed by the BUCC bosh director.
- minio (integrated to provide creds to Concourse pipelines so they can access an S3 bucket out of the box)
- prometheus (integrated to monitor Concourse)
Some opinions that affect if this project will work out of the box for you include (some of these can be modified fairly easily but there is no documentation around this)
- You use resource pools
- You will only run one BUCC on your Tools VM (i.e. where you will clone this repo to and run all the commands)
- The bosh cli alias and the fly cli alias are both
bucc - You treat the environment as ephemeral. See "State Caveat" section below.
The state repo for BOSH described in detail further below is your responsibility to manage and a deep understanding of BOSH is recommended for Day 2 Ops maintenance. Failing that, to avoid digging into the weeds when a need for troubleshooting arises, one option is to just blow away the state directory and start again. You may still have some clean up to do manually though... which is where your vcenter skills become important.
Just make sure you have scripted up any population of CredHub (see bin/credhub_populate_vcenter.sh for an example of this) and have everything else in git. There are various things you can do in addition to this. You could run back ups on the CredHub via export/import and your Concourse config/history could be saved by running Postgres back ups (bucc does support BBR for back ups).
NOTE: Use the Ansible playbooks as described in prereqs/README.md to check and set up your vcenter objects as required.
- You need to use a vcenter admin account or check your user has the permissions outlined in the docs here.
GOVC_env vars chosen for your installation. i.e. you have determined your choice of cluster, resource pool, vm folder etc. See env_bucc_template which will be used in the set up steps below.- The DRS config needs to be set to "Partially Automated" or "Fully Automated". If set to "Manual" bosh VM creation will fail. Go to the vcenter UI, click on your cluster, go to the Configure tab, and under Services > vSphere DRS to check. The prereq Ansible playbook mentioned above will check this and fail if not set up as required.
- Deploy the Tools VM that has been specifically configured to work with this solution.
The tools VM above automates the clone of this repo, but if you are going solo with your own approach then you will need to remember to include the submodules
git clone --recurse-submodules [email protected]:matthewcosgrove/lab-ops.git
# or with https
git clone --recurse-submodules https://github.com/matthewcosgrove/lab-ops.git
and also, if you are not using the associated Tools VM, ensure your local system has the correct bosh dependencies
After that you are on your own as the rest of this README assumes you are using the associated Tools VM.
This project is essentially a wrapper around BUCC. Just running bucc by itself creates a state dir within the bucc repo. We override the state location with the env var BBL_STATE_DIR which is what bucc uses. See the implementation here. We use another git repo outside of this one to manage your state and the specific configuration of the BUCC instance you are going to manage.
Next we need a way to tell our scripts and bucc where your state repo is..
In your ~/.profile the Tools VM automation has already put the lines
export BUCC_WRAPPER_ROOT_DIR="/home/ubuntu/lab-ops"
state_repo_root_dir="/home/ubuntu/lab-ops-state"
export BBL_STATE_DIR="${state_repo_root_dir}/state" # BBL_STATE_DIR is the convention use by BUCC https://github.com/starkandwayne/bucc/blob/2af7a2b47a151007b4db089f2349aa58bce8d1fc/bin/bucc#L8
IMPORTANT: Check the Tools VM automation created a .gitignore file at the root of your new state repo with the entry director-vars-*.yml. Without this you may accidently commit sensitive data to git.
IMPORTANT: The /home/ubuntu/lab-ops-state/state dir is ephemeral and is wiped out on teardown. Do NOT put your own bosh operator files in there unless they are copies!!
At the root of that repo there needs to be a file called env_bucc which we will generate in the next section. Anything you want to keep can be in the root of the repo just like the env_bucc which will not be wiped out in between deployments. i.e.
/home/ubuntu/lab-ops-state ---> safe
/home/ubuntu/lab-ops-state/state ---> NOT safe
- Set up the required environment variables.
a) Create the file mkdir -p /home/ubuntu/lab-ops-state; cp /home/ubuntu/lab-ops/env_bucc_template /home/ubuntu/lab-ops-state/env_bucc
b) Populate the env vars vim /home/ubuntu/lab-ops-state/env_bucc. Do NOT put your govc credentials in here as we will version control this file.
c) Source the env vars source /home/ubuntu/lab-ops-state/env_bucc
d) In your current shell export GOVC_USERNAME='changeme' GOVC_PASSWORD='changeme'
- Check the vcenter creds are configured as expected for govc by running
govc about; govc ls.
Either
- Deploy just lab-ops core functionality (bucc with an external Concourse worker VM)
/home/ubuntu/lab-ops/bin/deploy_bucc.sh
- OR, to deploy lab-ops core functionality plus Minio all in one go
/home/ubuntu/lab-ops/bin/deploy_full_stack.sh
Once the deployment is complete commit and push your state repo!
To interact with BUCC going forwards and have all the CLIs configured to work out the box
- Prep Env Vars
init-govc
# Nothing else to do here as BUCC env vars are sourced on login to the Tools VM
- Useful Commands
govc ls
bucc test
bucc info
bucc fly
fly -t bucc pipelines
bucc bosh
bosh vms
bucc credhub
credhub find
# etc
see BUCC README docs for more out of the box capabilities
- Look after your BUCC
You should learn the bucc cli and note that all the bucc commands that rely on state have to be run through the bucc wrapper script. This project has symlinked the bucc command to force it to go through the bin/bucc_wrapper.sh so that aspect is taken care of for you.
Assuming you understand bosh (if not see this tutorial and this explanation), any customizations should be put in the $BBL_STATE_DIR/state/operators directory for BUCC to find and integrate via the normal bosh operator file mechanism. But as discussed in the section on managing state, make sure your bosh operator files are backed up elsewhere.
Want to integrate additional bosh releases? See this list sorted by updated date https://github.com/search?o=desc&p=2&q=bosh+release&s=updated&type=Repositories
To extend the cloud-config.yml copy the infra/cloud-config.yml into your lab-ops-state root directory and start editing according to your needs. Additional vars should then be placed in the yaml bucc-extra-vars.yml within the root of the lab-ops-state dir. You can run bin/bosh_update_cloud_config.sh to update your cloud-config with the bosh director prior to running any deployment. That script looks in the lab-ops-state first so your copy will take precedence rather than the provided one.