The Geode Native Concourse pipeline is actually two Concourse pipelines. The primary release pipeline builds the release artifacts for a given branch, like develop, release or support branch. The secondary pull request (pr) pipeline builds the same artifacts as the release pipeline but for pull requests and without actually releasing or publishing anything.
The pipeline is fully self updating and can easily be bootstrapped into a properly configured Concourse deployment.
Concourse configuration requires TBD. Changes to the ci
source directory will results in auto updates to the
pipelines.
Because Concourse workers aren't available on all platforms and have issues with resource sharing this pipeline utilizes external builders. These builders are currently Google Compute VMs that are launched on demand for each build.
The pipeline can be installed or reconfigured via the set-pipelin.sh
shell script.
./set-pipeline.sh --help
Given the local repository looks like the following:
$ git remote get-url origin
[email protected]:some-user/geode-native.git
$ git branch --show-current
wip/some-branch
The most typical usage should require nothing more than the Concourse target, unless that happens to be named "default". Everything else has reasonable defaults based on the currently checked out branch.
$ ./set-pipeline.sh --target=some-concourse
Executes fly
from the path setting pipeline to target some-concourse
for GitHub owner some-user
repository geode-native.git
. Pipeline names will be some-user-wip-something
and some-user-wip-something-pr
.
Sometimes you will have to support multiple versions of Concourse fly
or need to fetch sources via https.
$ ./set-pipeline.sh \
--fly=/path/to/fly \
--target=some-concourse \
--github-owner=other-user
Executes fly at /path/to/fly
setting pipeline to target some-concourse
for GitHub owner other-user
repository geode-native.git
. Pipelines name will be some-user-wip-something
and some-user-wip-something-pr
.
- Detects new version or source
- Build for each platform and configuration
- Creates VM instances
- Waits for VM instance to be accessible
- Builds and packages
- Runs all tests
- If anything fails it downloads the build directory for later analysis
- Deletes the VM instances
- Uploads artifacts to GCS
- Publishes to GitHub release (TODO)
- Detects changes to pipeline sources and auto updates
- Detects new PR
- Build for each platform and configuration
- Creates VM instances
- Waits for VM instance to be accessible
- Builds and packages
- Runs all tests
- If anything fails it downloads the build directory for later analysis
- Deletes the VM instances
- Updates PR status
- Detects changes to pipeline sources and auto updates
This Concourse pipeline YAML is rendered using ytt
. Depends on output from git
and gcloud
.
- Google Cloud
- Concourse v6.5.0+
ytt
v0.28.0+git
v2.25.2+yq
v4.6.0+ (optional for set-pipeline script)gcloud
SDK
- base - Defines all common tasks across both pipelines.
- release - Defines tasks tasks for release pipeline only.
- pr - Defines tasks for pr pipeline only.
- lib - ytt functions used by all templates.
- docker/task - Minimal image required to communicate with builders.
- Resolve chicken/egg problem with external API address.
helm install concourse concourse/concourse
helm upgrade concourse concourse/concourse --set web.service.api.type=LoadBalancer,concourse.web.externalUrl=http://1.2.3.4:8080
- Task for getting secrets into k8s.
kubectl create secret generic gcr-json-key --from-literal "value=$(cat XXX.json)" --namespace=concourse-main
kubectl create secret generic github-access-token --from-literal "value=XXX" --namespace=concourse-main
kubectl create secret generic github-private-key --from-literal "value=$(cat XXX)" --namespace=concourse-main
- Use docker locally for initial pipeline deployment to avoid
gcloud
,ytt
, andfly
version issues.