Terraform Enterprise HVD on GCP GCE

Terraform module aligned with HashiCorp Validated Designs (HVD) to deploy Terraform Enterprise (TFE) on Google Cloud Platform (GCP) using Compute Engine instances with a container runtime. This module defaults to deploying TFE in the active-active operational mode, but external is also supported. Docker and Podman are the supported container runtimes.

Prerequisites

General

• TFE license file (e.g. terraform.hclic)
• Terraform CLI >= 1.9 installed on clients/workstations that will be used to deploy TFE
• General understanding of how to use Terraform (Community Edition)
• General understanding of how to use GCP
• git CLI and Visual Studio Code editor installed on workstations are strongly recommended
• GCP projeect that TFE will be deployed in with permissions to provision these resources via Terraform CLI
• (Optional) GCP GCS bucket for GCS remote state backend that will be used to manage the Terraform state file for this TFE deployment (out-of-band from the TFE application) via Terraform CLI (Community Edition)

Networking

• GCP VPC network with the following:
  • VM subnet for TFE GCE instances to reside with Private Google Access enabled (refer to the prereqs reference for more details)
  • (Optional) Load balancer subnet (can be the same as VM subnet if desired; only used when lb_is_internal is true)
  • Private Service Access (PSA) configured in VPC network for service servicenetworking.googleapis.com (refer to the prereqs reference for more details)
• Chosen fully qualified domain name (FQDN) for your TFE instance (e.g. tfe.gcp.example.com)
• (Optional) Google Cloud DNS zone for optional TFE DNS record creation

Firewall rules

This module will automatically create the necessary firewall rules within the existing VPC network that you provide.

• Identify CIDR range(s) that will need to access the TFE application (managed via cidr_allow_ingress_tfe_443 input variable)
• (Optional) Identity CIDR range(s) of monitoring tools that will need to access TFE metrics endpoint (managed via cidr_allow_ingress_tfe_metrics input variable)
• Be familiar with the TFE ingress requirements
• Be familiar with the TFE egress requirements

Secrets management

The following bootstrap secrets stored in Google Secret Manager in order to boostrap the TFE deployment and installation:

• TFE license file - raw contents of license file (e.g. cat terraform.hclic)
• TFE encryption password - random characters (used to protect TFE's internally-managed Vault unseal key and root token)
• TFE (PostgreSQL) database password - random characters between 8 and 99 characters in length; must contain at least one uppercase letter, one lowercase letter, and one digit or special character
• TFE TLS certificate - certificate file in PEM format, base64-encoded into a string, and stored as a secret
• TFE TLS certificate private key - private key file in PEM format, base64-encoded into a string, and stored as a secret
• TFE TLS CA bundle - Ca bundle file in PEM format, base64-encoded into a string, and stored as a secret

Refer to the prereqs reference for more details on how the secrets should be created and stored.

Compute

One of the following mechanisms for shell access to TFE GCE VM instances:

• Ability to enable IAP TCP forwarding (enabled by default - managed via allow_ingress_vm_ssh_from_iap)
• GCE SSH key pair (use this if you do not want to SSH via IAP TCP forwarding)

Log forwarding (optional)

One of the following logging destinations:

• Stackdriver (no action needed)
• A custom Fluent Bit configuration that will forward logs to your custom log destination

Usage

1. Create/configure/validate the applicable prerequisites.

2. Nested within the examples directory are subdirectories containing ready-made Terraform configurations for example scenarios on how to call and deploy this module. To get started, choose the example scenario that most closely matches your requirements. You can customize your deployment later by adding additional module inputs as you see fit (see the Deployment-Customizations doc for more details).

3. Copy all of the Terraform files from your example scenario of choice into a new destination directory to create your Terraform configuration that will manage your TFE deployment. This is a common directory structure for managing multiple TFE deployments:

   .
   └── environments
       ├── production
       │   ├── backend.tf
       │   ├── main.tf
       │   ├── outputs.tf
       │   ├── terraform.tfvars
       │   └── variables.tf
       └── sandbox
           ├── backend.tf
           ├── main.tf
           ├── outputs.tf
           ├── terraform.tfvars
           └── variables.tf
   

   📝 Note: In this example, the user will have two separate TFE deployments; one for their sandbox environment, and one for their production environment. This is recommended, but not required.

4. (Optional) Uncomment and update the GCS remote state backend configuration provided in the backend.tf file with your own custom values. While this step is highly recommended, it is technically not required to use a remote backend config for your TFE deployment.

5. Populate your own custom values into the terraform.tfvars.example file that was provided (in particular, values enclosed in the <> characters). Then, remove the .example file extension such that the file is now named terraform.tfvars.

6. Navigate to the directory of your newly created Terraform configuration for your TFE deployment, and run terraform init, terraform plan, and terraform apply.

7. After your terraform apply finishes successfully, you can monitor the installation progress by connecting to your TFE gcp instance shell via SSH or GCP IAP and observing the meta data script(user_data) logs:
   

   Higher-level logs:

   tail -f /var/log/tfe-cloud-init.log
   

   Lower-level logs:

   journalctl -xu google-startup-scripts -f
   

   📝 Note: The -f argument is to follow the logs as they append in real-time, and is optional. You may remove the -f for a static view.

   The log files should display the following message after the startup script (tfe_startup_script.sh) finishes successfully:

   [INFO] - tfe_startup_script finished successfully!
   

8. After the startup script (tfe_startup_script.sh) finishes successfully, while still connected to the TFE GCE instance shell, you can check the health status of TFE:

   cd /etc/tfe
   sudo docker compose exec tfe tfe-health-check-status
   

9. Follow the steps here to create the TFE initial admin user.

Docs

Below are links to various docs related to the customization and management of your TFE deployment:

• Deployment Customizations
• Prereqs Reference
• TFE TLS Certificate Rotation
• TFE Configuration Settings
• TFE Version Upgrades

Module support

This open source software is maintained by the HashiCorp Technical Field Organization, independently of our enterprise products. While our Support Engineering team provides dedicated support for our enterprise offerings, this open source software is not included.

• For help using this open source software, please engage your account team.
• To report bugs/issues with this open source software, please open them directly against this code repository using the GitHub issues feature.

Please note that there is no official Service Level Agreement (SLA) for support of this software as a HashiCorp customer. This software falls under the definition of Community Software/Versions in your Agreement. We appreciate your understanding and collaboration in improving our open source projects.

Requirements

Name	Version
terraform	>= 1.9
google	~> 6.6
google-beta	~> 6.6
random	~> 3.6

Providers

Name	Version
google	~> 6.6
google-beta	~> 6.6
random	~> 3.6

Resources

Name	Type
google-beta_google_project_service_identity.gcp_project_cloud_sql_sa	resource
google_compute_address.tfe_frontend_lb	resource
google_compute_firewall.vm_allow_ingress_ssh_from_cidr	resource
google_compute_firewall.vm_allow_ingress_ssh_from_iap	resource
google_compute_firewall.vm_allow_lb_health_checks_443	resource
google_compute_firewall.vm_allow_tfe_443	resource
google_compute_firewall.vm_allow_tfe_metrics_from_cidr	resource
google_compute_firewall.vm_tfe_self_allow	resource
google_compute_forwarding_rule.tfe_frontend_lb	resource
google_compute_health_check.tfe_auto_healing	resource
google_compute_instance_template.tfe	resource
google_compute_region_backend_service.tfe_backend_lb	resource
google_compute_region_health_check.tfe_backend_lb	resource
google_compute_region_instance_group_manager.tfe	resource
google_dns_record_set.tfe	resource
google_kms_crypto_key_iam_member.gcp_project_gcs_sa_cmek	resource
google_kms_crypto_key_iam_member.gcp_project_redis_sa_cmek	resource
google_kms_crypto_key_iam_member.postgres_cmek	resource
google_project_iam_member.tfe_logging_stackdriver	resource
google_redis_instance.tfe	resource
google_secret_manager_secret_iam_member.tfe_ca_bundle	resource
google_secret_manager_secret_iam_member.tfe_cert	resource
google_secret_manager_secret_iam_member.tfe_encryption_password	resource
google_secret_manager_secret_iam_member.tfe_license	resource
google_secret_manager_secret_iam_member.tfe_privkey	resource
google_service_account.tfe	resource
google_service_account_key.tfe	resource
google_sql_database.tfe	resource
google_sql_database_instance.tfe	resource
google_sql_user.tfe	resource
google_storage_bucket.tfe	resource
google_storage_bucket_iam_member.tfe_bucket_object_admin	resource
google_storage_bucket_iam_member.tfe_bucket_reader	resource
random_id.gcs_bucket_suffix	resource
random_id.postgres_instance_suffix	resource
google_client_config.current	data source
google_compute_image.tfe	data source
google_compute_network.vpc	data source
google_compute_subnetwork.lb_subnet	data source
google_compute_subnetwork.vm_subnet	data source
google_compute_zones.up	data source
google_dns_managed_zone.tfe	data source
google_kms_crypto_key.gcs_cmek	data source
google_kms_crypto_key.postgres_cmek	data source
google_kms_crypto_key.redis_cmek	data source
google_kms_key_ring.gcs_cmek	data source
google_kms_key_ring.postgres_cmek	data source
google_kms_key_ring.redis_cmek	data source
google_project.current	data source
google_secret_manager_secret_version.tfe_database_password	data source
google_storage_project_service_account.gcp_project_gcs_sa	data source

Inputs

Name	Description	Type	Default	Required
friendly_name_prefix	Friendly name prefix used for uniquely naming all GCP resources for this deployment. Most commonly set to either an environment (e.g. 'sandbox', 'prod'), a team name, or a project name.	string	n/a	yes
project_id	ID of GCP project to deploy TFE in.	string	n/a	yes
region	GCP region (location) to deploy TFE in.	string	n/a	yes
tfe_database_password_secret_id	Name of PostgreSQL database password secret to retrieve from Google Secret Manager.	string	n/a	yes
tfe_encryption_password_secret_id	Name of Google Secret Manager secret for TFE encryption password.	string	n/a	yes
tfe_fqdn	Fully qualified domain name (FQDN) of TFE instance. This name should resolve to the TFE load balancer IP address and will be what users/clients use to access TFE.	string	n/a	yes
tfe_license_secret_id	Name of Google Secret Manager secret for TFE license file.	string	n/a	yes
tfe_tls_ca_bundle_secret_id	Name of Google Secret Manager secret for private/custom TLS Certificate Authority (CA) bundle in PEM format. Secret must be stored as a base64-encoded string.	string	n/a	yes
tfe_tls_cert_secret_id	Name of Google Secret Manager secret for TFE TLS certificate in PEM format. Secret must be stored as a base64-encoded string.	string	n/a	yes
tfe_tls_privkey_secret_id	Name of Google Secret Manager secret for TFE TLS private key in PEM format. Secret must be stored as a base64-encoded string.	string	n/a	yes
vm_subnet_name	Name of VPC subnet to deploy TFE GCE VM instances in.	string	n/a	yes
vpc_network_name	Name of VPC network to deploy TFE in.	string	n/a	yes
allow_ingress_vm_ssh_from_iap	Boolean to create firewall rule to allow TCP/22 (SSH) inbound to TFE GCE instances from Google Cloud IAP CIDR block.	bool	true	no
cidr_allow_ingress_tfe_443	List of CIDR ranges to allow TCP/443 (HTTPS) inbound to TFE load balancer.	list(string)	null	no
cidr_allow_ingress_tfe_metrics	List of CIDR ranges to allow TCP/9090 (HTTP) and TCP/9091 (HTTPS) inbound to TFE metrics collection endpoint.	list(string)	null	no
cidr_allow_ingress_vm_ssh	List of CIDR ranges to allow TCP/22 (SSH) inbound to TFE GCE instances.	list(string)	null	no
cloud_dns_managed_zone_name	Name of Google Cloud DNS managed zone to create TFE DNS record in. Required when create_tfe_cloud_dns_record is true.	string	null	no
common_labels	Map of common labels to apply to all GCP resources.	map(string)	{}	no
container_runtime	Container runtime to use for TFE deployment. Supported values are docker or podman.	string	"docker"	no
create_tfe_cloud_dns_record	Boolean to create Google Cloud DNS record for TFE using the value of tfe_fqdn as the record name, resolving to the load balancer IP. cloud_dns_managed_zone_name is required when true.	bool	false	no
custom_fluent_bit_config	Custom Fluent Bit configuration for log forwarding. Only valid when tfe_log_forwarding_enabled is true and log_fwd_destination_type is custom.	string	null	no
custom_tfe_startup_script_template	Name of custom TFE startup script template file. File must exist within a directory named ./templates within your current working directory.	string	null	no
docker_version	Version of Docker to install on TFE GCE VM instances.	string	"26.1.4-1"	no
gce_disk_size_gb	Size in gigabytes of root disk of TFE GCE VM instances.	number	50	no
gce_image_name	VM image for TFE GCE instances.	string	"ubuntu-pro-2404-noble-amd64-v20241004"	no
gce_image_project	ID of project in which the TFE GCE VM image belongs.	string	"ubuntu-os-cloud"	no
gce_machine_type	Machine type (size) of TFE GCE VM instances.	string	"n2-standard-4"	no
gce_ssh_public_key	SSH public key to add to TFE GCE VM instances for SSH access. Generally not needed if using Google IAP for SSH.	string	null	no
gcs_force_destroy	Boolean indicating whether to allow force destroying the TFE GCS bucket. GCS bucket can be destroyed if it is not empty when true.	bool	false	no
gcs_kms_cmek_name	Name of Cloud KMS customer managed encryption key (CMEK) to use for TFE GCS bucket encryption.	string	null	no
gcs_kms_keyring_name	Name of Cloud KMS key ring that contains KMS customer managed encryption key (CMEK) to use for TFE GCS bucket encryption. Geographic location (region) of the key ring must match the location of the TFE GCS bucket.	string	null	no
gcs_location	Location of TFE GCS bucket to create.	string	"US"	no
gcs_storage_class	Storage class of TFE GCS bucket.	string	"MULTI_REGIONAL"	no
gcs_uniform_bucket_level_access	Boolean to enable uniform bucket level access on TFE GCS bucket.	bool	true	no
gcs_versioning_enabled	Boolean to enable versioning on TFE GCS bucket.	bool	true	no
is_secondary_region	Boolean indicating whether this TFE deployment is in your primary region or secondary (disaster recovery) region.	bool	false	no
lb_is_internal	Boolean to create an internal GCP load balancer for TFE.	bool	true	no
lb_static_ip_address	Static IP address to assign to TFE load balancer forwarding rule (front end) when lb_is_internal is true. Must be a valid IP address within vm_subnet_name. If not set, an available IP address will automatically be selected.	string	null	no
lb_subnet_name	Name of VPC subnet to deploy TFE load balancer in. This can be the same subnet as the VM subnet if you do not wish to provide a separate subnet for the load balancer. Only applicable when lb_is_internal is true. Must be null when lb_is_internal is false.	string	null	no
log_fwd_destination_type	Type of log forwarding destination for Fluent Bit. Valid values are stackdriver or custom.	string	"stackdriver"	no
mig_initial_delay_sec	Number of seconds for managed instance group to wait before applying autohealing policies to new GCE instances in managed instance group.	number	900	no
mig_instance_count	Desired number of TFE GCE instances to run in managed instance group. Must be 1 when tfe_operational_mode is external.	number	1	no
postgres_availability_type	Availability type of Cloud SQL for PostgreSQL instance.	string	"REGIONAL"	no
postgres_backup_start_time	HH:MM time format indicating when daily automatic backups of Cloud SQL for PostgreSQL should run. Defaults to 12 AM (midnight) UTC.	string	"00:00"	no
postgres_deletetion_protection	Boolean to enable deletion protection for Cloud SQL for PostgreSQL instance.	bool	false	no
postgres_disk_size	Size in GB of PostgreSQL disk.	number	50	no
postgres_insights_config	Configuration settings for Cloud SQL for PostgreSQL insights.	object({ query_insights_enabled = bool query_plans_per_minute = number query_string_length = number record_application_tags = bool record_client_address = bool })	{ "query_insights_enabled": false, "query_plans_per_minute": 5, "query_string_length": 1024, "record_application_tags": false, "record_client_address": false }	no
postgres_kms_cmek_name	Name of Cloud KMS customer managed encryption key (CMEK) to use for Cloud SQL for PostgreSQL database instance.	string	null	no
postgres_kms_keyring_name	Name of Cloud KMS Key Ring that contains KMS key to use for Cloud SQL for PostgreSQL. Geographic location (region) of key ring must match the location of the TFE Cloud SQL for PostgreSQL database instance.	string	null	no
postgres_machine_type	Machine size of Cloud SQL for PostgreSQL instance.	string	"db-custom-4-16384"	no
postgres_maintenance_window	Optional maintenance window settings for the Cloud SQL for PostgreSQL instance.	object({ day = number hour = number update_track = string })	{ "day": 7, "hour": 0, "update_track": "stable" }	no
postgres_ssl_mode	Indicates whether to enforce TLS/SSL connections to the Cloud SQL for PostgreSQL instance.	string	"ENCRYPTED_ONLY"	no
postgres_version	PostgreSQL version to use.	string	"POSTGRES_16"	no
redis_auth_enabled	Boolean to enable authentication on Redis instance.	bool	true	no
redis_connect_mode	Network connection mode for Redis instance.	string	"PRIVATE_SERVICE_ACCESS"	no
redis_kms_cmek_name	Name of Cloud KMS customer managed encryption key (CMEK) to use for TFE Redis instance.	string	null	no
redis_kms_keyring_name	Name of Cloud KMS key ring that contains KMS customer managed encryption key (CMEK) to use for TFE Redis instance. Geographic location (region) of key ring must match the location of the TFE Redis instance.	string	null	no
redis_memory_size_gb	The size of the Redis instance in GiB.	number	6	no
redis_tier	The service tier of the Redis instance. Set to STANDARD_HA for high availability.	string	"STANDARD_HA"	no
redis_transit_encryption_mode	Determines transit encryption (TLS) mode for Redis instance.	string	"DISABLED"	no
redis_version	The version of Redis software.	string	"REDIS_7_2"	no
tfe_capacity_concurrency	Maximum number of concurrent Terraform runs to allow on a TFE node.	number	10	no
tfe_capacity_cpu	Maxium number of CPU cores that a Terraform run is allowed to consume on a TFE node. Defaults to 0 which is no limit.	number	0	no
tfe_capacity_memory	Maximum amount of memory (in MiB) that a Terraform run is allowed to consume on a TFE node.	number	2048	no
tfe_database_name	Name of TFE PostgreSQL database to create.	string	"tfe"	no
tfe_database_parameters	Additional parameters to pass into the TFE database settings for the PostgreSQL connection URI.	string	"sslmode=require"	no
tfe_database_reconnect_enabled	Boolean to enable database reconnection in the event of a TFE PostgreSQL database cluster failover.	bool	true	no
tfe_database_user	Name of TFE PostgreSQL database user to create.	string	"tfe"	no
tfe_hairpin_addressing	Boolean to enable hairpin addressing within TFE container networking for loopback prevention with a layer 4 internal load balancer.	bool	true	no
tfe_http_port	HTTP port for TFE application containers to listen on.	number	8080	no
tfe_https_port	HTTPS port for TFE application containers to listen on.	number	8443	no
tfe_iact_subnets	Comma-separated list of subnets in CIDR notation that are allowed to retrieve the TFE initial admin creation token via the API or web browser.	string	null	no
tfe_iact_time_limit	Number of minutes that the TFE initial admin creation token can be retrieved via the API after the application starts.	number	60	no
tfe_iact_trusted_proxies	Comma-separated list of proxy IP addresses that are allowed to retrieve the TFE initial admin creation token via the API or web browser.	string	null	no
tfe_image_name	Name of the TFE container image. Only set this away from the default if you are hosting the TFE container image in your own custom registry.	string	"hashicorp/terraform-enterprise"	no
tfe_image_repository_password	Pasword for container registry where TFE container image is hosted. Leave as null if using the default TFE registry, as the default password is your TFE license file.	string	null	no
tfe_image_repository_url	URL of container registry where the TFE container image is hosted. Only set this away from the default if you are hosting the TFE container image in your own custom registry.	string	"images.releases.hashicorp.com"	no
tfe_image_repository_username	Username for container registry where TFE container image is hosted.	string	"terraform"	no
tfe_image_tag	Tag (release) for the TFE container image. This represents which version (release) of TFE to deploy.	string	"v202409-3"	no
tfe_license_reporting_opt_out	Boolean to opt out of TFE license reporting.	bool	false	no
tfe_log_forwarding_enabled	Boolean to enable TFE log forwarding configuration via Fluent Bit.	bool	false	no
tfe_metrics_enable	Boolean to enable TFE metrics collection endpoints.	bool	false	no
tfe_metrics_http_port	HTTP port for TFE metrics collection endpoint to listen on.	number	9090	no
tfe_metrics_https_port	HTTPS port for TFE metrics collection endpoint to listen on.	number	9091	no
tfe_operational_mode	Operational mode for TFE. Valid values are active-active or external.	string	"active-active"	no
tfe_run_pipeline_docker_network	Name of Docker network where the containers that execute Terraform runs (agents) will be created. The network must already exist, it will not be created automatically. Leave as null to use the default network created during the TFE installation.	string	null	no
tfe_run_pipeline_image	Name of container image used to execute Terraform runs on a TFE node. Leave as null to use the default agent that ships with TFE.	string	null	no
tfe_tls_enforce	Boolean to enforce TLS, Strict-Transport-Security headers, and secure cookies within TFE.	bool	false	no
tfe_usage_reporting_opt_out	Boolean to opt out of TFE usage reporting.	bool	false	no
tfe_vault_disable_mlock	Boolean to disable mlock for internal (embedded) Vault within TFE.	bool	false	no
vpc_network_project_id	ID of GCP project where the existing VPC network resides, if it is different than the project_id where TFE will be deployed.	string	null	no

Outputs

Name	Description
tfe_create_initial_admin_user_url	URL to create TFE initial admin user.
tfe_database_host	Private IP address and port of TFE Cloud SQL for PostgreSQL database instance.
tfe_database_instance_id	ID of TFE Cloud SQL for PostgreSQL database instance.
tfe_database_name	Name of TFE database.
tfe_database_password	Password of TFE database user.
tfe_database_user	Username of TFE database.
tfe_gcs_bucket_location	Location of TFE GCS bucket.
tfe_lb_ip_address	IP Address of TFE front end load balancer (forwarding rule).
tfe_load_balancing_scheme	Load balancing scheme of TFE front end load balancer (forwarding rule).
tfe_object_storage_google_bucket	Name of TFE GCS bucket.
tfe_redis_host	Hostname/IP address (and port if non-default) of TFE Redis instance.
tfe_redis_password	Password of TFE Redis instance.
tfe_redis_use_auth	Whether TFE should use authentication to connect to Redis instance.
tfe_redis_use_tls	Whether TFE should use TLS to connect to Redis instance.
tfe_redis_user	Username of TFE Redis instance.
tfe_retrieve_iact_url	URL to retrieve TFE initial admin creation token.
tfe_url	URL of TFE application based on tfe_fqdn input value.