Docker-Stack-Deploy

This repo is the home of a small but powerful utility that helps you to do gitops and maintain your docker deployments by checking in the corresponding configuration files into a private git repo and then running git push for them to take effect.

Architecture

There are a couple of pieces:

• A private git repo that holds your infrastructure definition, essentially a set of directories with docker compose.yml files, and an encrypted keepass database file that holds any secrets that might be required by those containers. Each of these directories if referred to as a "stack".
• One or more hosts on which you are running docker
• The docker-stack-deploy container that runs persistently on each of those docker hosts

Getting Started

Infrastructure Repo

If you don't already have a repo suitable for this purpose, create a new private one on GitHub.

docker-stack-deploy is fairly un-opinionated about directory layout, but for the sake of getting started:

• You will need to create a KeePass database to hold your encrypted secrets. On macOS you might want to look at Strongbox, which can be used for free for this purpose, but you can also use KeePassXC for free on macOS or any other OS.

  • Clone your infra repo locally
  • Create a new vault/database in the root of your infra repo, and name it .secrets.kdbx. If you have the option to select the file format version, select Keepass password database 2.x KDBX in order to be compatible with the rust keepass crate
  • Use a passphrase, rather than a key file, when you configure this database.
  • You will need that passphrase when you edit the database later, and also to bootstrap a docker host.

• You can now create a directory for each of your stack(s). They can be anywhere in the repo; they will be located based on the stack-deploy.toml file. I personally have some stacks deployed under hosts/HOSTNAME/STACKNAME and some under services/STACKNAME. You can pick whatever organization makes sense to you, but each individual stack must live in its own directory.

• In the stack directory you will need at least two files:

  • compose.yml - the docker stack definition
  • stack-deploy.toml - the definition for stack-deploy

Example Stack

In your infra repo, create a minecraft directory, and populate it:

Put this in minecraft/compose.yml:

services:
 minecraft:
   image: itzg/minecraft-server
   ports:
     - "25565:25565"
   environment:
     EULA: "TRUE"
   deploy:
     resources:
       limits:
         memory: 1.5G
   volumes:
     - minecraft_data:/data

volumes:
  minecraft_data:

Important

Avoid using local directories for mutable state, as you want to avoid dirtying your infrastructure repo checkout with files created by docker and potentially cause permission problems and potentially causing conflicts with future changes in your Git repo. I recommend using docker volumes, such as the minecraft_data volume in the example above, to hold the mutable state. Read-only mounts using config files in your repo are fine, and I used those often.

And this in minecraft/stack-deploy.toml:

# The name of the stack. Used for dependency purposes
name = "minecraft"

# Lists the hosts on which this stack should run.
# It should match the hostname of your docker host.
runs_on = ["mydockerhostname"]

Add and commit that to your infra repo and push it to github.

Bootstrapping

You need to deploy docker-stack-deploy to your docker host. This is done via a one-time bootstrap procedure.

You will need:

• The infra repo URL
• A PAT that grants access to read from the infra repo
• The passphrase for your infra keepass secrets db
• Access to the docker host

First, login to the docker host, and run the bootstrap command. You must run this as a user that has privileges to talk to docker:

$ docker run --rm -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /var/lib/docker-stack-deploy:/var/lib/docker-stack-deploy \
    ghcr.io/wez/docker-stack-deploy \
    docker-stack-deploy bootstrap \
    --project-dir /var/lib/docker-stack-deploy \
    --git-url https://github.com/YOURNAME/REPO.git
Github Token:
KeePass Passphrase:

This will pull the deploy image and run it, and it will then prompt you for your github token and keepass passphrase.

With that done, you can now see what is happening with the deployment:

$ docker logs docker-stack-deploy --tail 100 --follow

after a few moments, it should have pulled and launched the minecraft container.

docker-stack-deploy will pull your infra repo every 5 minutes to look for changes. If any files have changed, it will run through and deploy each stack.

The deploy command that gets run for each stack is:

docker compose up --detach --wait --remove-orphans

with the environment populated as described in the Secrets section below.

Secrets

The standard easy way to manage secrets with docker compose is to put them into an .env file in the stack directory. While you can do that here, it isn't ideal to check in clear-text secrets. This is where the KeePass database comes into play.

You can record the relevant secrets in this database and it will be stored encrypted on disk. With a sufficiently strong passphrase this is a significant upgrade over clear text .env files.

Secrets are selectively exposed to a stack based on the instructions in your stack-deploy.toml file for that stack.

For example, if you have this in gitea/compose.yml:

services:
  gitea:
    image: gitea/gitea:latest
    environment:
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=gitea
      - DB_USER=gitea
      - DB_PASSWD=${DB_PASSWD}
    restart: always
    volumes:
      - git_data:/data
    ports:
      - 3000:3000
  db:
    image: postgres:alpine
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=${DB_PASSWD}
    restart: always
    volumes:
      - db_data:/var/lib/postgresql/data
    expose:
      - 5432
volumes:
  db_data:
  git_data:

and this in gitead/stack-deploy.toml:

name = "gitea"

[secret_env]
DB_PASSWD = 'Database/Gitea Postgres DB/password'

Then create an entry in your secrets DB called Gitea Postgres DB, this stack now securely holds the relevant credential in the secrets database. At deploy time only the credentials listed in the secret_env section will be decrypted and set in the environment when docker compose is run.

docker-stack-deploy doesn't create or modify a .env file; those environment variables are set only in the context of the docker invocation.

Stack Dependencies

You can express dependencies between stacks on the same host. For example:

# This is the homepage stack
name = "homepage"
# It runs on the docker1 host
runs_on = ["docker1"]
# It requires that the traefik stack on docker1 be deployed first
depends_on = ["traefik"]

The stacks are topologically sorted based on their dependencies and then started in that order.

It is not possible to depend on stacks that are running on other hosts.

Stopping and removing a Stack

This is a two phase process:

• First you must edit the compose.yml and add scale: 0 to each service in the compose file, then commit and push that and wait 5 minutes or so for the change to take effect. It tells docker to scale down to 0 and stop the service.

• Once the service has stopped on all hosts, you can then git rm the stack directory, commit and push.

How do I force deployment to run?

If you don't want to wait 5 minutes for it to happen naturally, you can ssh into your docker host and run docker restart docker-stack-deploy. That will cause it to pull the repo immediately and run through the deploy commands.

Troubleshooting

You can use docker compose ls to review the stacks that are running. It might look something like this:

$ docker compose ls
NAME                  STATUS              CONFIG FILES
docker-stack-deploy   running(1)          /var/lib/docker-stack-deploy/compose.yml
dockerproxy           running(1)          /var/lib/docker-stack-deploy/repo/services/dockerproxy/compose.yml
frigate               running(1)          /var/lib/docker-stack-deploy/repo/hosts/huge/frigate/compose.yml
immich                running(4)          /var/lib/docker-stack-deploy/repo/hosts/huge/immich/compose.yml
jellyfin              running(2)          /var/lib/docker-stack-deploy/repo/hosts/huge/jellyfin/compose.yml

The /var/lib/docker-stack-deploy directory is the location where docker-stack-deploy maintains its state.

In that directory:

• The compose.yml file was created from the compose.yml file present in this repository when the docker-stack-deploy image was build by my CI.
• There is a .env file that captures the secrets from your bootstrap invocation.
• The repo directory is where your infrastructure repo is checked out

To stop a stack

If I wanted to stop frigate:

$ cd /var/lib/docker-stack-deploy/repo/hosts/huge/frigate/
$ docker compose down

To bring it back up again, docker restart docker-stack-deploy.