Skip to content

Latest commit

 

History

History

005_gitlab_ci

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 

Driving GitLab CI/CD pipelines with CUE

by Jonathan Matthews

This guide explains how to convert a GitLab CI/CD pipeline file from YAML to CUE, check its contents are valid, and then use CUE's tooling layer to regenerate YAML.

This is useful because it allows you to switch to CUE as a source of truth for GitLab pipelines and perform client-side validation, without GitLab needing to know you're managing your pipelines with CUE.

❗ WARNING ❗
This guide requires that you use cue version v0.11.0-alpha.4 or later. The process described below won't work with earlier versions. Check the version of your cue command by running cue version, and upgrade it if needed.

Prerequisites

  • You have a GitLab pipeline file.
    • The example shown throughout this guide uses the pipeline file from a specific commit in the gitlab-org/gitlab repository on gitlab.com, as linked from GitLab's CI documentation pages, but you don't need to use that repository in any way. It's used as the example in this guide only because it's a reasonably complex GitLab pipeline file.
  • You have cue installed.
    • You must have version v0.11.0-alpha.4 or later installed. Using an earlier version will cause certain commands in this guide to fail.
  • You have git installed.
  • You have curl installed, or can fetch a remote file some other way.

Steps

Convert YAML pipeline to CUE

➡️ Begin with a clean git state

Change directory into the root of the repository that contains your GitLab pipeline file, and ensure you start this process with a clean git state, with no modified files. For example:

💻 terminal

cd gitlab # our example repository
git status # should report "working tree clean"

➡️ Initialise a CUE module

Initialise a CUE module named after the organisation and repository you're working with, but containing only lowercase letters and numbers. For example:

💻 terminal

cue mod init gitlab.com/gitlab-org/gitlab

➡️ Import YAML pipeline

Use cue to import your YAML pipeline file:

💻 terminal

cue import .gitlab-ci.yml --with-context -p gitlab -f -l pipelines: \
  -l 'strings.TrimSuffix(path.Base(filename),path.Ext(filename))' -o gitlab-ci.cue

If your project uses a different name for your pipeline file then use that name in the above command, and throughout this guide.

Check that a CUE file has been created from your pipeline file. For example:

💻 terminal

ls {,.}*gitlab-ci*

Your output should look similar to this, with a matching YAML and CUE file:

.gitlab-ci.yml
gitlab-ci.cue

Observe that your file has been imported into the pipelines struct at a location derived from its original file name, by running:

💻 terminal

head -9 gitlab-ci.cue

The output should reflect your pipeline. In our example:

package gitlab

pipelines: ".gitlab-ci": {
	stages: [
		"sync",
		"preflight",
		"prepare",
		"build-images",
		"fixtures",

➡️ Store CUE pipelines in a dedicated directory

Create a directory called gitlab to hold your CUE-based GitLab pipeline files. For example:

💻 terminal

mkdir -p internal/ci/gitlab

You may change the hierarchy and naming of gitlab's parent directories to suit your repository layout. If you do so, you will need to adapt some commands and CUE code as you follow this guide.

Move the newly-created CUE pipeline file into its dedicated directory. For example:

💻 terminal

mv gitlab-ci.cue internal/ci/gitlab

Validate pipeline

➡️ Create a pipeline schema

Fetch a schema for GitLab pipelines, as defined by the GitLab project, and place it in the internal/ci/gitlab directory:

💻 terminal

curl -sSo internal/ci/gitlab/gitlab.cicd.pipeline.schema.json https://gitlab.com/gitlab-org/gitlab/-/raw/277c9f6b643c92d00101aca0f2b4b874a144f7c5/app/assets/javascripts/editor/schema/ci.json

We use a specific commit from the upstream repository to make sure that this process is reproducible.

Convert the GitLab schema from JSON Schema to CUE:

💻 terminal

cue import -p gitlab -l '#Pipeline:' \
  internal/ci/gitlab/gitlab.cicd.pipeline.schema.json

This command will create the file internal/ci/gitlab/gitlab.cicd.pipeline.schema.cue in the gitlab package, with the contents of the upstream schema placed in the field #Pipeline.

➡️ Apply the schema

We need to tell CUE to apply the schema to the pipeline.

To do this we'll create a file at internal/ci/gitlab/pipelines.cue in our example. However, if your earlier pipeline import already created a file with that same path and name, then simply select a different CUE filename that doesn't already exist.

Create the file in the internal/ci/gitlab/ directory and add this CUE:

💾 internal/ci/gitlab/pipelines.cue

package gitlab

// each member of the pipelines struct must be a valid #Pipeline
pipelines: [_]: #Pipeline

➡️ Validate your pipelines

💻 terminal

cue vet ./internal/ci/gitlab

If this command fails and produces any output, then CUE believes that at least one of your pipelines isn't valid. You'll need to resolve this before continuing, by updating your pipelines inside your new CUE files. If you're having difficulty fixing them, please come and ask for help in the friendly CUE Slack workspace or Discord server!

Generate YAML from CUE

➡️ Create a CUE workflow command

Create a CUE file in internal/ci/gitlab/ containing the following workflow command. Adapt the element commented with TODO:

💾 internal/ci/gitlab/ci_tool.cue

package gitlab

import (
	"path"
	"encoding/yaml"
	"tool/file"
)

_goos: string @tag(os,var=os)

// Regenerate pipeline files
command: regenerate: {
	pipeline_files: {
		// TODO: update _toolFile to reflect the directory hierarchy containing this file.
		// TODO: update _pipelineDir to reflect the directory containing your pipeline file.
		let _toolFile = "internal/ci/gitlab/ci_tool.cue"
		let _pipelineDir = path.FromSlash(".", path.Unix)
		let _donotedit = "Code generated by \(_toolFile); DO NOT EDIT."

		for _pipelineName, _pipelineConfig in pipelines
		let _pipelineFile = _pipelineName + ".yml"
		let _pipelinePath = path.Join([_pipelineDir, _pipelineFile]) {
			let delete = {
				"Delete \(_pipelinePath)": file.RemoveAll & {path: _pipelinePath}
			}
			delete
			create: file.Create & {
				$after:   delete
				filename: _pipelinePath
				contents: "# \(_donotedit)\n\n\(yaml.Marshal(_pipelineConfig))"
			}
		}
	}
}

Make the modifications indicated by the TODO comments.

The regenerate workflow command will export your CUE-based pipeline back into its required YAML file, on demand.

➡️ Test the CUE workflow command

With the modified ci_tool.cue file in place, check that the regenerate workflow command is available from a shell sitting at the repository root. For example:

💻 terminal

cd $(git rev-parse --show-toplevel) # make sure we're sitting at the repository root
cue help cmd regenerate ./internal/ci/gitlab   # the "./" prefix is required

The output of the cue help command must begin with the following:

Regenerate pipeline files

Usage:
  cue cmd regenerate [flags]
❗ WARNING ❗
If you don't see the usage explanation for the regenerate workflow command (or if you receive an error message) then either your workflow command isn't set up as CUE requires, or you're running a CUE version older than v0.11.0-alpha.4. If you've upgraded to at least that version but the usage explanation still isn't being displayed then: (1) double check the contents of the ci_tool.cue file and the modifications you made to it; (2) make sure its location in the repository is precisely as given in this guide; (3) ensure the filename is exactly ci_tool.cue; (4) run cue vet ./internal/ci/gitlab and check that your pipelines actually validate successfully - in other words: were they truly valid before you even started this process? Lastly, make sure you've followed all the steps in this guide, and that you invoked the cue help command from the repository's root directory. If you get really stuck, please come and join the CUE community and ask for some help!

➡️ Regenerate the YAML pipeline file

Run the regenerate workflow command to produce a YAML pipeline file from CUE. For example:

💻 terminal

cue cmd regenerate ./internal/ci/gitlab # the "./" prefix is required

➡️ Audit changes to the YAML pipeline file

Check that your YAML pipeline file has a single material change from the original:

💻 terminal

git diff .gitlab-ci.yml

Your output should look similar to the following example:

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,3 +1,5 @@
+# Code generated by internal/ci/gitlab/ci_tool.cue; DO NOT EDIT.
+
 stages:
   - sync
   - preflight

The main change in each YAML file is the addition of a header that warns the reader not to edit the file directly.

Your diff might also contain some YAML reformatting (with the number of leading spaces having been changed in nested structures) but this won't make a difference to the underlying meaning of the file.

Additionally, any comments in the original YAML file will now be found only in the CUE source file - which is important as that's the only file that you'll be manually changing, from now on.

➡️ Add and commit files to git

Add your files to git. For example:

💻 terminal

git add .gitlab-ci.yml internal/ci/gitlab/ cue.mod/module.cue

Make sure to include your slightly modified YAML pipeline file, wherever you store it, along with all the new files in internal/ci/gitlab/ and your cue.mod/module.cue file.

Commit your files to git, with an appropriate commit message:

💻 terminal

git commit -m "ci: create CUE sources for GitLab CI/CD pipelines"

Conclusion

Well done - your GitLab CI/CD pipeline file has been imported into CUE!

It can now be managed using CUE, leading to safer and more predictable changes. The use of a schema to check your pipeline means that you will catch and fix certain types of mistake earlier than before, without waiting for the slow "git add/commit/push; check if CI fails" cycle.

From now on, each time you make a change to a CUE pipeline file, immediately regenerate the YAML files required by GitLab CI/CD, and commit your changes to all the CUE and YAML files. For example:

💻 terminal

cue cmd regenerate ./internal/ci/gitlab/ # the "./" prefix is required
git add .gitlab-ci.yml internal/ci/gitlab/
git commit -m "ci: added new release pipeline" # example message