This package contains integration tests.
The terraform
folder has provisioning code to spin up a Nomad cluster on AWS.
The tests work with the NOMAD_ADDR
environment variable which can be set
either to a local dev Nomad agent or a Nomad client on AWS.
The workflow when developing end to end tests locally is to run the provisioning step described below once, and then run the tests as described below.
When making local changes, use ./bin/update $(which nomad) /usr/local/bin/nomad
and ./bin/run sudo systemctl restart nomad
to destructively modify the
provisioned cluster.
You'll need Terraform and AWS credentials (AWS_ACCESS_KEY_ID
and
AWS_SECRET_ACCESS_KEY
) to setup AWS instances on which e2e tests
will run. See the README
for details. The number of servers and clients is configurable, as is
the configuration file for each client and server.
You can use the Terraform output from the previous step to generate a provisioning configuration file for the e2e framework.
# from the ./e2e/terraform directory
terraform output provisioning | jq . > ../provisioning.json
By default the provisioning.json
will not include the Nomad version
that will be deployed to each node. You can pass the following flags
to go test
to set the version for all nodes:
-nomad.local_file=string
: provision this specific local binary of Nomad. This is a path to a Nomad binary on your own host. Ex.-nomad.local_file=/home/me/nomad
-nomad.sha=string
: provision this specific sha from S3. This is a Nomad binary identified by its full commit SHA that's stored in a shared s3 bucket that Nomad team developers can access. That commit SHA can be from any branch that's pushed to remote. Ex.-nomad.sha=0b6b475e7da77fed25727ea9f01f155a58481b6c
-nomad.version=string
: provision this version from releases.hashicorp.com. Ex.-nomad.version=0.10.2
Then deploy Nomad to the cluster by passing -provision.terraform
without a Nomad version flag:
go test -v . -nomad.version=0.10.2 -provision.terraform ./provisioning.json -skipTests
Because it can take a little while for the cluster to settle, it's
recommended to run this provisioning step (with -skipTests
) first,
and then run tests as separate step.
After completing the provisioning step above, you can set the client
environment for NOMAD_ADDR
and run the tests as shown below:
# from the ./e2e/terraform directory, set your client environment
# if you haven't already
$(terraform output environment)
cd ..
go test -v .
If you want to run a specific suite, you can specify the -suite
flag as
shown below. Only the suite with a matching Framework.TestSuite.Component
will be run, and all others will be skipped.
go test -v -suite=Consul .
If you want to run a specific test, you'll need to regex-escape some of the test's name so that the test runner doesn't skip over framework struct method names in the full name of the tests:
go test -v . -run 'TestE2E/Consul/\*consul\.ScriptChecksE2ETest/TestGroup'
You can use the Terraform output to find the IP address. The keys will
in the ./terraform/keys/
directory.
ssh -i keys/nomad-e2e-*.pem ubuntu@${EC2_IP_ADDR}
The provisioning.json
file output by Terraform has a blank field for
nomad_sha
for each node of the cluster (server and client). You can
manually edit the file to replace this value with a nomad_sha
,
nomad_local_binary
, or nomad_version
for each node to create a
cluster of mixed versions. The provisioning framework accepts any of
the following options for those fields:
nomad_sha
: This is a Nomad binary identified by its full commit SHA that's stored in a shared s3 bucket that Nomad team developers can access. That commit SHA can be from any branch that's pushed to remote. (Ex."nomad_sha": "0b6b475e7da77fed25727ea9f01f155a58481b6c"
)nomad_local_binary
: This is a path to a Nomad binary on your own host. (Ex."nomad_local_binary": "/home/me/nomad"
)nomad_version
: This is a version number of Nomad that's been released to HashiCorp. (Ex."nomad_version": "0.10.2"
)
Then deploy Nomad to the cluster by passing -provision.terraform
without a Nomad version flag:
go test -v . -provision.terraform ./provisioning.json -skipTests
The provisioning.json
file includes a bundles
section for each
node of the cluster (server and client). You can manually edit this
file to add, remove, or replace
"bundles": [
{
"destination": "/ops/shared/nomad/base.hcl",
"source": "/home/me/custom.hcl"
}
]
Right now the framework doesn't support this out-of-the-box because of
the way the provisioning script adds specific client configurations to
each client node (for constraint testing). You'll need to add
additional configuration files to
./e2e/terraform/shared/nomad/indexed
.