diff --git a/terraform/validator/INSTALL.md b/terraform/validator/INSTALL.md new file mode 100644 index 0000000000000..4144d91c12d59 --- /dev/null +++ b/terraform/validator/INSTALL.md @@ -0,0 +1,271 @@ +Diem Single Validator Network Deployment +======================================== + +These instructions are for deploying a Diem blockchain network with one single validator node from scratch, this would be good for training new operators. + +Deployment Package +------------------ + +The deployment package is in this directory. Clone the repo and checkout the branch corresponding to the version you want to deploy: + + $ git clone https://github.com/diem/diem + $ cd diem/terraform/validator + $ git checkout + +The release branch is usually named as `release-xx`, you can checkout to the latest one (e.g. `release-1.4`) + +Once you have the correct validator directory version follow the instructions below. + +Working with Diem Source Code +----------------------------- +If you haven't yet, it's highly recommended that you finish the "My first transaction" training: https://developers.diem.com/docs/my-first-transaction. + +The training will guide you through the steps to setup environment and build Rust code, which we will use to build our operational tools. + + +Terraform with Vault +-------------------- + +1. Install pre-requisites if needed: + + * Terraform 1.0: https://www.terraform.io/downloads.html + * Vault: https://www.vaultproject.io/downloads + * Docker: https://www.docker.com/products/docker-desktop + +2. Create a directory for your configuration: + + * Choose a workspace name e.g. testnet. Note: this defines terraform workspace name, which in turn is used to form resource names. If you are using a shared account please use a unique value here, e.g. `youralias-testnet` + + $ export WORKSPACE=testnet + + * Create a directory for the workspace + + $ mkdir -p ~/$WORKSPACE + +3. Change into the appropriate directory for your cloud provider, + make sure you setup credentials to access the cloud: + * [aws][]: Amazon Web Services + * [azure][]: Microsoft Azure + * [gcp][]: Google Cloud Platform + + $ cd validator/ + +4. Create a storage bucket for the Terraform state: + * AWS: S3 bucket + * GCP: Cloud Storage bucket + * Azure: Blob storage container + +5. Copy `backend.tfvars` to `~/$WORKSPACE/backend.tfvars` and edit to fill in your storage bucket details. For more detail on remote state see the Terraform documentation: https://www.terraform.io/docs/backends/index.html + + $ cp backend.tfvars ~/$WORKSPACE/backend.tfvars + $ vi ~/$WORKSPACE/backend.tfvars + +6. Initialise Terraform, providing your backend storage configuration: + + $ terraform init -backend-config ~/$WORKSPACE/backend.tfvars + +7. Create a new Terraform workspace to isolate your environments: + + * Check the existing workspace, make sure the workspace name is not taken: + + $ terraform workspace list + + * Create the new workspace: + + $ terraform workspace new $WORKSPACE + +8. Copy `terraform.tfvars` to `~/$WORKSPACE/validator.tfvars` and edit to set your validator name, SSH public key: + + $ cp terraform.tfvars ~/$WORKSPACE/validator.tfvars + $ vi ~/$WORKSPACE/validator.tfvars + + +9. Apply the configuration, enabling the bastion host, which you will use later to access your vault. This will also create a `kubernetes.json` file with information about the Kubernetes cluster. + + $ terraform apply -var-file ~/$WORKSPACE/validator.tfvars -var bastion_enable=1 + +10. Configure your Kubernetes client: + * aws: + + $ aws eks update-kubeconfig --name diem-$WORKSPACE + + * azure: + + $ az aks get-credentials --resource-group diem-$WORKSPACE --name diem-$WORKSPACE + + * gcp: + + $ gcloud container clusters get-credentials diem-$WORKSPACE --zone us-central1-a --project diem + +11. Find the private IP of one of the `vault` instances, and the public IP of the `bastion` instance. + * aws: + + $ terraform state show 'aws_instance.bastion[0]' | grep public_dns + $ aws ec2 describe-instances --filters "Name=tag:Name,Values=diem-$WORKSPACE/vault" --query "Reservations[*].Instances[*].PrivateDnsName" --output=text + + * gcp: + + $ terraform state show 'google_compute_instance.bastion[0]' | grep nat_ip + $ gcloud --project compute instances list + + * azure: + + $ terraform state show 'azurerm_linux_virtual_machine.bastion[0]' | grep public_ip + $ az vmss nic list-vm-nics -g diem-$WORKSPACE --vmss-name diem-$WORKSPACE-vault --instance-id 1 | grep privateIp + + * alternatively, cut-n-past this: + +```sh + public_ip=$(terraform state show 'azurerm_linux_virtual_machine.bastion[0]' | egrep 'public_ip_address\W' | awk -F'"' '{print $2}') + echo public_ip=$public_ip + private_ip=$(az vmss nic list -g diem-$WORKSPACE --vmss-name diem-$WORKSPACE-vault --query '[].ipConfigurations[].privateIpAddress' --output tsv) + echo private_ip=$private_ip +``` + +12. SSH to the vault host via the bastion host, setting up a tunnel to connect to the Vault API. + +* **Keep this running in another window.** +* `~/.ssh/ec2_rsa` is the private key you created in step 8. +* For Azure replace "ec2-user" with "az-user" +* for GCP replace with the username you configured in `ssh_keys`. + + $ ssh -o ProxyCommand="ssh -i ~/.ssh/ec2_rsa ec2-user@ proxy %h %p" -L 8200:localhost:8200 -i ~/.ssh/ec2_rsa ec2-user@ + + +13. Initialise Vault **from the vault host**. + + [ec2-user@vault ~]$ VAULT_CACERT=/etc/vault/vault.ca vault operator init + +14. Switch back to your local host, save the Initial Root Token in a file, you'll need it to manage Vault: + + $ vi ~/$WORKSPACE/vault.token + +15. Change to the [vault-init][] directory: + + $ cd ../vault-init + +16. Copy `backend.tf.example` to `backend.tf` and edit to uncomment the appropriate backend for your cloud provider. + + $ cp backend.tf.example backend.tf + $ vi backend.tf + +17. Copy `backend.tfvars` to `~/$WORKSPACE/backend-vault.tfvars` and edit to fill in your storage bucket details. + + $ cp backend.tfvars ~/$WORKSPACE/backend-vault.tfvars + $ vi ~/$WORKSPACE/backend-vault.tfvars + +18. Initialise Terraform, providing your backend storage configuration: + + $ terraform init -backend-config ~/$WORKSPACE/backend-vault.tfvars + +19. Specify your Vault server information (replace "aws" with "gcp" or "azure" as necessary): + + $ export VAULT_ADDR=https://localhost:8200 + $ export VAULT_CACERT=$PWD/..//$WORKSPACE-vault.ca + $ export VAULT_TOKEN= + + Check your `VAULT_CACERT` env (`echo $VAULT_CERT`), make sure it's the full path to the vault CA certificate file. + `` is the token you generated in step 13. + +20. Rename `diem-root.tf.example` to `diem-root.tf`, this will add `diem-root` key and `treasury_compliance` key in your vault for this test. + + $ cp diem-root.tf.example diem-root.tf + +21. Apply the configuration, providing your Kubernetes cluster information: + + $ terraform apply -var-file ..//$WORKSPACE-kubernetes.json + +22. Verify you have access to the vault: + + $ vault kv list transit/keys + + You should be able to see a list of keys, e.g. `diem__root`, `diem__operator` + +23. Change into your configuration directory: + + $ cd ~/$WORKSPACE + +24. Set the operator name: + + $ export OPERATOR= + +25. Collect the endpoints of the `-validator-lb` and `-fullnode-lb` services: + + * If using the included DNS setup: + + $ cd ~/diem/terraform/validator/ + $ export VALIDATOR_ADDRESS="$(terraform output validator_endpoint)" + $ export FULLNODE_ADDRESS="$(terraform output fullnode_endpoint)" + $ cd - + + * Otherwise on AWS: + + $ export VALIDATOR_ADDRESS="$(kubectl get svc ${WORKSPACE}-diem-validator-validator-lb -o go-template='/dns4/{{(index .status.loadBalancer.ingress 0).hostname}}/tcp/{{(index .spec.ports 0).port}}')" + $ export FULLNODE_ADDRESS="$(kubectl get svc ${WORKSPACE}-diem-validator-fullnode-lb -o go-template='/dns4/{{(index .status.loadBalancer.ingress 0).hostname}}/tcp/{{(index .spec.ports 0).port}}')" + + * Otherwise on GCP and Azure: + + $ export VALIDATOR_ADDRESS="$(kubectl get svc ${WORKSPACE}-diem-validator-validator-lb -o go-template='/ip4/{{(index .status.loadBalancer.ingress 0).ip}}/tcp/{{(index .spec.ports 0).port}}')" + $ export FULLNODE_ADDRESS="$(kubectl get svc ${WORKSPACE}-diem-validator-fullnode-lb -o go-template='/ip4/{{(index .status.loadBalancer.ingress 0).ip}}/tcp/{{(index .spec.ports 0).port}}')" + +26. Build the Diem genesis tool binary: + + $ git clone https://github.com/diem/diem + $ cd diem + $ git checkout + $ cargo build -p diem-genesis-tool + $ cargo run -p diem-genesis-tool -- help + + Follow [My first transaction][] to setup the environment for building Rust code. + +27. Use the genesis script to generate keys, create waypoints, and compile the genesis blob for your blockchain: + + $ cd ~/$WORKSPACE + $ export GENESIS=~/diem/target/debug/diem-genesis-tool + $ export BACKEND="backend=vault;server=https://localhost:8200;ca_certificate=;token=vault.token;namespace=diem" + $ ~/diem/scripts/genesis-single.sh $GENESIS $BACKEND $VALIDATOR_ADDRESS $FULLNODE_ADDRESS + +28. Insert the genesis blob into the Kubernetes configmap for the chain era (defined in [helm/values.yaml][]): + + $ kubectl create configmap $WORKSPACE-diem-validator-genesis-e --from-file=genesis.blob=genesis.blob + + change `` with current chain era number defined in [helm/values.yaml][]. + +29. Check that your pods are now running (this may take a few minutes): + + $ kubectl get pods + +30. To access the monitoring dashboard run the following: + + $ kubectl port-forward $WORKSPACE-diem-validator-monitoring-0 3000 + + And then load http://localhost:3000/d/validator + + +Destroy existing deployment +-------------------- + +1. Make sure you select the terraform workspace you want to destroy: + + $ cd ~/diem/terraform/validator/ + $ export WORKSPACE=testnet + $ terraform workspace select $WORKSPACE + +2. Enable destroying of protected resources in this deployment. + + $ sed -i 's/prevent_destroy = true/prevent_destroy = false/' cluster.tf vault.tf + +3. Destroy the deployment + + $ terraform destroy -var-file ~/$WORKSPACE/validator.tfvars + +4. Restore the code protecting resources. + + $ sed -i 's/prevent_destroy = false/prevent_destroy = true/' cluster.tf vault.tf + +[aws]: aws/ +[gcp]: gcp/ +[azure]: azure/ +[vault-init]: vault-init/ +[helm/values.yaml]: helm/values.yaml +[My first transaction]: https://developers.diem.com/docs/core/my-first-transaction diff --git a/terraform/validator/README.md b/terraform/validator/README.md new file mode 100644 index 0000000000000..72d30d86ab54e --- /dev/null +++ b/terraform/validator/README.md @@ -0,0 +1,11 @@ +Diem Validator Deployment +========================= + +This directory provides terraforms to deploy a Diem Validator node. The cloud-specific Terraform configs will +create a Kubernetes cluster from scratch and then install the Kubernetes configs. They also deploy Hashicorp +Vault outside of Kubernetes for storing validator keys and sensitive data. + +* If you are deploying a brand new validator please see [INSTALL.md][]. + + +[INSTALL.md]: INSTALL.md diff --git a/terraform/vault-init/.terraform.lock.hcl b/terraform/validator/vault-init/.terraform.lock.hcl similarity index 100% rename from terraform/vault-init/.terraform.lock.hcl rename to terraform/validator/vault-init/.terraform.lock.hcl diff --git a/terraform/vault-init/README.md b/terraform/validator/vault-init/README.md similarity index 100% rename from terraform/vault-init/README.md rename to terraform/validator/vault-init/README.md diff --git a/terraform/vault-init/backend.tf b/terraform/validator/vault-init/backend.tf similarity index 100% rename from terraform/vault-init/backend.tf rename to terraform/validator/vault-init/backend.tf diff --git a/terraform/vault-init/backend.tf.example b/terraform/validator/vault-init/backend.tf.example similarity index 100% rename from terraform/vault-init/backend.tf.example rename to terraform/validator/vault-init/backend.tf.example diff --git a/terraform/vault-init/backend.tfvars b/terraform/validator/vault-init/backend.tfvars similarity index 100% rename from terraform/vault-init/backend.tfvars rename to terraform/validator/vault-init/backend.tfvars diff --git a/terraform/vault-init/diem-root.tf.example b/terraform/validator/vault-init/diem-root.tf.example similarity index 100% rename from terraform/vault-init/diem-root.tf.example rename to terraform/validator/vault-init/diem-root.tf.example diff --git a/terraform/vault-init/kubernetes.tf b/terraform/validator/vault-init/kubernetes.tf similarity index 100% rename from terraform/vault-init/kubernetes.tf rename to terraform/validator/vault-init/kubernetes.tf diff --git a/terraform/vault-init/main.tf b/terraform/validator/vault-init/main.tf similarity index 100% rename from terraform/vault-init/main.tf rename to terraform/validator/vault-init/main.tf diff --git a/terraform/vault-init/policy.tf b/terraform/validator/vault-init/policy.tf similarity index 100% rename from terraform/vault-init/policy.tf rename to terraform/validator/vault-init/policy.tf diff --git a/terraform/vault-init/versions.tf b/terraform/validator/vault-init/versions.tf similarity index 100% rename from terraform/vault-init/versions.tf rename to terraform/validator/vault-init/versions.tf