AWS DMS Terraform module

Terraform module which creates AWS DMS (Database Migration Service) resources.

Usage

See examples directory for working examples to reference:

module "database_migration_service" {
  source  = "clowdhaus/dms/aws"
  version = "~> 1.0"

  # Subnet group
  repl_subnet_group_name        = "example"
  repl_subnet_group_description = "DMS Subnet group"
  repl_subnet_group_subnet_ids  = ["subnet-1fe3d837", "subnet-129d66ab", "subnet-1211eef5"]

  # Instance
  repl_instance_allocated_storage            = 64
  repl_instance_auto_minor_version_upgrade   = true
  repl_instance_allow_major_version_upgrade  = true
  repl_instance_apply_immediately            = true
  repl_instance_engine_version               = "3.4.5"
  repl_instance_multi_az                     = true
  repl_instance_preferred_maintenance_window = "sun:10:30-sun:14:30"
  repl_instance_publicly_accessible          = false
  repl_instance_class                        = "dms.t3.large"
  repl_instance_id                           = "example"
  repl_instance_vpc_security_group_ids       = ["sg-12345678"]

  endpoints = {
    source = {
      database_name               = "example"
      endpoint_id                 = "example-source"
      endpoint_type               = "source"
      engine_name                 = "aurora-postgresql"
      extra_connection_attributes = "heartbeatFrequency=1;"
      username                    = "postgresqlUser"
      password                    = "youShouldPickABetterPassword123!"
      port                        = 5432
      server_name                 = "dms-ex-src.cluster-abcdefghijkl.us-east-1.rds.amazonaws.com"
      ssl_mode                    = "none"
      tags                        = { EndpointType = "source" }
    }

    destination = {
      database_name = "example"
      endpoint_id   = "example-destination"
      endpoint_type = "target"
      engine_name   = "aurora"
      username      = "mysqlUser"
      password      = "passwordsDoNotNeedToMatch789?"
      port          = 3306
      server_name   = "dms-ex-dest.cluster-abcdefghijkl.us-east-1.rds.amazonaws.com"
      ssl_mode      = "none"
      tags          = { EndpointType = "destination" }
    }
  }

  replication_tasks = {
    cdc_ex = {
      replication_task_id       = "example-cdc"
      migration_type            = "cdc"
      replication_task_settings = file("task_settings.json")
      table_mappings            = file("table_mappings.json")
      source_endpoint_key       = "source"
      target_endpoint_key       = "destination"
      tags                      = { Task = "PostgreSQL-to-MySQL" }
    }
  }

  event_subscriptions = {
    instance = {
      name                             = "instance-events"
      enabled                          = true
      instance_event_subscription_keys = ["example"]
      source_type                      = "replication-instance"
      sns_topic_arn                    = "arn:aws:sns:us-east-1:012345678910:example-topic"
      event_categories                 = [
        "failure",
        "creation",
        "deletion",
        "maintenance",
        "failover",
        "low storage",
        "configuration change"
      ]
    }
    task = {
      name                         = "task-events"
      enabled                      = true
      task_event_subscription_keys = ["cdc_ex"]
      source_type                  = "replication-task"
      sns_topic_arn                = "arn:aws:sns:us-east-1:012345678910:example-topic"
      event_categories             = [
        "failure",
        "state change",
        "creation",
        "deletion",
        "configuration change"
      ]
    }
  }

  tags = {
    Terraform   = "true"
    Environment = "dev"
  }
}

Combinations

Within DMS you can have multiple combinations of resources depending on your use case. For example (not an exhaustive list of possible combinations):

Simple

• One source endpoint
• One target/destination endpoint
• One replication task
• Two event subscriptions
  • Replication instance event subscription
  • Replication task event subscriptions

Multiple endpoints, multiple tasks

• Two source endpoints
• Three target/destination endpoints
• Three replication tasks (source1 -> target1, source2 -> target2, source1 -> targe3)
• Four event subscriptions
  • Replication instance event subscription
  • Replication task event subscription for each task listed above

In order to accommodate a flexible, multi-resource combinatorial module, keys and maps are used for cross-referencing resources created within the module.

Given the following example (not complete, only showing the relevant attributes):

module "database_migration_service" {
  source  = "clowdhaus/dms/aws"
  version = "~> 1.0"

  # This value is used when subscribing to instance event notifications
  repl_instance_id = "readme-example"

  endpoints = {
    # These keys are used to map endpoints within task definitions by this key `source1`
    source1 = {
      endpoint_type = "source"
      ...
    }

    destination1 = {
      endpoint_type = "target"
      ...
    }

    destination2 = {
      endpoint_type = "target"
      ...
    }
  }

To create replication tasks, you simply reference the relevant keys from the endpoints map in the source_endpoint_key/target_endpoint_key fields:

  ...

  replication_tasks = {
    src1_dest1 = {
      ...
      source_endpoint_key = "source1"
      target_endpoint_key = "destination1"
    }
    src1_dest2 = {
      ...
      source_endpoint_key = "source1"
      target_endpoint_key = "destination2"
    }
  }

  ...

Continuing the same lookup patter, to create event subscriptions, you simply reference the replication instance ID in the instance_event_subscription_keysfield when subscribing to instance notifications, or the replication_tasks keys in the task_event_subscription_keys to subscribe to the tasks notifications (all or only select keys for select tasks):

  ...

  event_subscriptions = {
    instance = {
      instance_event_subscription_keys = ["readme-example"]
      ...
    }
    task = {
      task_event_subscription_keys = ["src1_dest1", "src1_dest2]
      ...
    }
  }

  ...

Tasks

Tasks are the "jobs" that perform the necessary actions of migrating from source to target, including any transformations and/or mappings of the data in transit. Tasks are largely controlled by task settings that are defined in a JSON document.

Example task settings JSON document:

{
  "TargetMetadata": {
    "TargetSchema": "",
    "SupportLobs": true,
    "FullLobMode": false,
    "LobChunkSize": 64,
    "LimitedSizeLobMode": true,
    "LobMaxSize": 32,
    "BatchApplyEnabled": true
  },
  "FullLoadSettings": {
    "TargetTablePrepMode": "DO_NOTHING",
    "CreatePkAfterFullLoad": false,
    "StopTaskCachedChangesApplied": false,
    "StopTaskCachedChangesNotApplied": false,
    "MaxFullLoadSubTasks": 8,
    "TransactionConsistencyTimeout": 600,
    "CommitRate": 10000
  },
  "Logging": {
    "EnableLogging": false
  },
  "ControlTablesSettings": {
    "ControlSchema":"",
    "HistoryTimeslotInMinutes":5,
    "HistoryTableEnabled": false,
    "SuspendedTablesTableEnabled": false,
    "StatusTableEnabled": false
  },
  "StreamBufferSettings": {
    "StreamBufferCount": 3,
    "StreamBufferSizeInMB": 8
  },
  "ChangeProcessingTuning": {
    "BatchApplyPreserveTransaction": true,
    "BatchApplyTimeoutMin": 1,
    "BatchApplyTimeoutMax": 30,
    "BatchApplyMemoryLimit": 500,
    "BatchSplitSize": 0,
    "MinTransactionSize": 1000,
    "CommitTimeout": 1,
    "MemoryLimitTotal": 1024,
    "MemoryKeepTime": 60,
    "StatementCacheSize": 50
  },
  "ChangeProcessingDdlHandlingPolicy": {
    "HandleSourceTableDropped": true,
    "HandleSourceTableTruncated": true,
    "HandleSourceTableAltered": true
  },
  "ErrorBehavior": {
    "DataErrorPolicy": "LOG_ERROR",
    "DataTruncationErrorPolicy":"LOG_ERROR",
    "DataErrorEscalationPolicy":"SUSPEND_TABLE",
    "DataErrorEscalationCount": 50,
    "TableErrorPolicy":"SUSPEND_TABLE",
    "TableErrorEscalationPolicy":"STOP_TASK",
    "TableErrorEscalationCount": 50,
    "RecoverableErrorCount": 0,
    "RecoverableErrorInterval": 5,
    "RecoverableErrorThrottling": true,
    "RecoverableErrorThrottlingMax": 1800,
    "ApplyErrorDeletePolicy":"IGNORE_RECORD",
    "ApplyErrorInsertPolicy":"LOG_ERROR",
    "ApplyErrorUpdatePolicy":"LOG_ERROR",
    "ApplyErrorEscalationPolicy":"LOG_ERROR",
    "ApplyErrorEscalationCount": 0,
    "FullLoadIgnoreConflicts": true
  }
}

Examples

Examples codified under the examples are intended to give users references for how to use the module(s) as well as testing/validating changes to the source code of the module. If contributing to the project, please be sure to make any appropriate updates to the relevant examples to allow maintainers to test your changes and to keep the examples up to date for users. Thank you!

• Complete

Security & Compliance

Security scanning results provided by Bridgecrew. Bridgecrew is the leading fully hosted, cloud-native solution providing continuous Terraform security and compliance.

Benchmark	Description
	Infrastructure Security Compliance
	Center for Internet Security, AWS Compliance
	Payment Card Industry Data Security Standards Compliance
	National Institute of Standards and Technology Compliance
	Information Security Management System, ISO/IEC 27001 Compliance
	Service Organization Control 2 Compliance
	Health Insurance Portability and Accountability Compliance
	FedRAMP Moderate Impact Level

Requirements

Name	Version
terraform	>= 0.13.1
aws	>= 4.6

Providers

Name	Version
aws	>= 4.6

Modules

No modules.

Resources

Name	Type
aws_dms_certificate.this	resource
aws_dms_endpoint.this	resource
aws_dms_event_subscription.this	resource
aws_dms_replication_instance.this	resource
aws_dms_replication_subnet_group.this	resource
aws_dms_replication_task.this	resource
aws_iam_role.dms_access_for_endpoint	resource
aws_iam_role.dms_cloudwatch_logs_role	resource
aws_iam_role.dms_vpc_role	resource
aws_iam_policy_document.dms_assume_role	data source
aws_iam_policy_document.dms_assume_role_redshift	data source
aws_partition.current	data source

Inputs

Name	Description	Type	Default	Required
certificates	Map of objects that define the certificates to be created	map(any)	{}	no
create	Determines whether resources will be created	bool	true	no
create_iam_roles	Determines whether the required DMS IAM resources will be created	bool	true	no
create_repl_subnet_group	Determines whether the replication subnet group will be created	bool	true	no
enable_redshift_target_permissions	Determines whether redshift.amazonaws.com is permitted access to assume the dms-access-for-endpoint role	bool	false	no
endpoints	Map of objects that define the endpoints to be created	any	{}	no
event_subscription_timeouts	A map of timeouts for event subscription create/update/delete operations	map(string)	{}	no
event_subscriptions	Map of objects that define the event subscriptions to be created	any	{}	no
iam_role_permissions_boundary	ARN of the policy that is used to set the permissions boundary for the role	string	null	no
iam_role_tags	A map of additional tags to apply to the DMS IAM roles	map(string)	{}	no
repl_instance_allocated_storage	The amount of storage (in gigabytes) to be initially allocated for the replication instance. Min: 5, Max: 6144, Default: 50	number	null	no
repl_instance_allow_major_version_upgrade	Indicates that major version upgrades are allowed	bool	null	no
repl_instance_apply_immediately	Indicates whether the changes should be applied immediately or during the next maintenance window	bool	null	no
repl_instance_auto_minor_version_upgrade	Indicates that minor engine upgrades will be applied automatically to the replication instance during the maintenance window	bool	null	no
repl_instance_availability_zone	The EC2 Availability Zone that the replication instance will be created in	string	null	no
repl_instance_class	The compute and memory capacity of the replication instance as specified by the replication instance class	string	null	no
repl_instance_engine_version	The engine version number of the replication instance	string	null	no
repl_instance_id	The replication instance identifier. This parameter is stored as a lowercase string	string	null	no
repl_instance_kms_key_arn	The Amazon Resource Name (ARN) for the KMS key that will be used to encrypt the connection parameters	string	null	no
repl_instance_multi_az	Specifies if the replication instance is a multi-az deployment. You cannot set the availability_zone parameter if the multi_az parameter is set to true	bool	null	no
repl_instance_preferred_maintenance_window	The weekly time range during which system maintenance can occur, in Universal Coordinated Time (UTC)	string	null	no
repl_instance_publicly_accessible	Specifies the accessibility options for the replication instance	bool	null	no
repl_instance_subnet_group_id	An existing subnet group to associate with the replication instance	string	null	no
repl_instance_tags	A map of additional tags to apply to the replication instance	map(string)	{}	no
repl_instance_timeouts	A map of timeouts for replication instance create/update/delete operations	map(string)	{}	no
repl_instance_vpc_security_group_ids	A list of VPC security group IDs to be used with the replication instance	list(string)	null	no
repl_subnet_group_description	The description for the subnet group	string	null	no
repl_subnet_group_name	The name for the replication subnet group. Stored as a lowercase string, must contain no more than 255 alphanumeric characters, periods, spaces, underscores, or hyphens	string	null	no
repl_subnet_group_subnet_ids	A list of the EC2 subnet IDs for the subnet group	list(string)	[]	no
repl_subnet_group_tags	A map of additional tags to apply to the replication subnet group	map(string)	{}	no
replication_tasks	Map of objects that define the replication tasks to be created	any	{}	no
tags	A map of tags to use on all resources	map(string)	{}	no

Outputs

Name	Description
certificates	A map of maps containing the certificates created and their full output of attributes and values
dms_access_for_endpoint_iam_role_arn	Amazon Resource Name (ARN) specifying the role
dms_access_for_endpoint_iam_role_id	Name of the IAM role
dms_access_for_endpoint_iam_role_unique_id	Stable and unique string identifying the role
dms_cloudwatch_logs_iam_role_arn	Amazon Resource Name (ARN) specifying the role
dms_cloudwatch_logs_iam_role_id	Name of the IAM role
dms_cloudwatch_logs_iam_role_unique_id	Stable and unique string identifying the role
dms_vpc_iam_role_arn	Amazon Resource Name (ARN) specifying the role
dms_vpc_iam_role_id	Name of the IAM role
dms_vpc_iam_role_unique_id	Stable and unique string identifying the role
endpoints	A map of maps containing the endpoints created and their full output of attributes and values
event_subscriptions	A map of maps containing the event subscriptions created and their full output of attributes and values
replication_instance_arn	The Amazon Resource Name (ARN) of the replication instance
replication_instance_private_ips	A list of the private IP addresses of the replication instance
replication_instance_public_ips	A list of the public IP addresses of the replication instance
replication_instance_tags_all	A map of tags assigned to the resource, including those inherited from the provider default_tags configuration block
replication_subnet_group_id	The ID of the subnet group
replication_tasks	A map of maps containing the replication tasks created and their full output of attributes and values

License

Apache-2.0 Licensed. See LICENSE.