Awesome
AWS DMS Terraform module
Terraform module which creates AWS DMS (Database Migration Service) resources.
Usage
See examples
directory for working examples to reference:
module "database_migration_service" {
source = "terraform-aws-modules/dms/aws"
version = "~> 2.0"
# Subnet group
repl_subnet_group_name = "example"
repl_subnet_group_description = "DMS Subnet group"
repl_subnet_group_subnet_ids = ["subnet-1fe3d837", "subnet-129d66ab", "subnet-1211eef5"]
# Instance
repl_instance_allocated_storage = 64
repl_instance_auto_minor_version_upgrade = true
repl_instance_allow_major_version_upgrade = true
repl_instance_apply_immediately = true
repl_instance_engine_version = "3.5.2"
repl_instance_multi_az = true
repl_instance_preferred_maintenance_window = "sun:10:30-sun:14:30"
repl_instance_publicly_accessible = false
repl_instance_class = "dms.t3.large"
repl_instance_id = "example"
repl_instance_vpc_security_group_ids = ["sg-12345678"]
endpoints = {
source = {
database_name = "example"
endpoint_id = "example-source"
endpoint_type = "source"
engine_name = "aurora-postgresql"
extra_connection_attributes = "heartbeatFrequency=1;"
username = "postgresqlUser"
password = "youShouldPickABetterPassword123!"
port = 5432
server_name = "dms-ex-src.cluster-abcdefghijkl.us-east-1.rds.amazonaws.com"
ssl_mode = "none"
tags = { EndpointType = "source" }
}
destination = {
database_name = "example"
endpoint_id = "example-destination"
endpoint_type = "target"
engine_name = "aurora"
username = "mysqlUser"
password = "passwordsDoNotNeedToMatch789?"
port = 3306
server_name = "dms-ex-dest.cluster-abcdefghijkl.us-east-1.rds.amazonaws.com"
ssl_mode = "none"
tags = { EndpointType = "destination" }
}
}
replication_tasks = {
cdc_ex = {
replication_task_id = "example-cdc"
migration_type = "cdc"
replication_task_settings = file("task_settings.json")
table_mappings = file("table_mappings.json")
source_endpoint_key = "source"
target_endpoint_key = "destination"
tags = { Task = "PostgreSQL-to-MySQL" }
}
}
event_subscriptions = {
instance = {
name = "instance-events"
enabled = true
instance_event_subscription_keys = ["example"]
source_type = "replication-instance"
sns_topic_arn = "arn:aws:sns:us-east-1:012345678910:example-topic"
event_categories = [
"failure",
"creation",
"deletion",
"maintenance",
"failover",
"low storage",
"configuration change"
]
}
task = {
name = "task-events"
enabled = true
task_event_subscription_keys = ["cdc_ex"]
source_type = "replication-task"
sns_topic_arn = "arn:aws:sns:us-east-1:012345678910:example-topic"
event_categories = [
"failure",
"state change",
"creation",
"deletion",
"configuration change"
]
}
}
tags = {
Terraform = "true"
Environment = "dev"
}
}
Combinations
Within DMS you can have multiple combinations of resources depending on your use case. For example (not an exhaustive list of possible combinations):
Simple
- One source endpoint
- One target/destination endpoint
- One replication task
- Two event subscriptions
- Replication instance event subscription
- Replication task event subscriptions
Multiple endpoints, multiple tasks
- Two source endpoints
- Three target/destination endpoints
- Three replication tasks (source1 -> target1, source2 -> target2, source1 -> targe3)
- Four event subscriptions
- Replication instance event subscription
- Replication task event subscription for each task listed above
In order to accommodate a flexible, multi-resource combinatorial module, keys and maps are used for cross-referencing resources created within the module.
Given the following example (not complete, only showing the relevant attributes):
module "database_migration_service" {
source = "terraform-aws-modules/dms/aws"
version = "~> 2.0"
# This value is used when subscribing to instance event notifications
repl_instance_id = "readme-example"
endpoints = {
# These keys are used to map endpoints within task definitions by this key `source1`
source1 = {
endpoint_type = "source"
...
}
destination1 = {
endpoint_type = "target"
...
}
destination2 = {
endpoint_type = "target"
...
}
}
To create replication tasks, you simply reference the relevant keys from the endpoints
map in the source_endpoint_key
/target_endpoint_key
fields:
...
replication_tasks = {
src1_dest1 = {
...
source_endpoint_key = "source1"
target_endpoint_key = "destination1"
}
src1_dest2 = {
...
source_endpoint_key = "source1"
target_endpoint_key = "destination2"
}
}
...
Continuing the same lookup pattern, to create event subscriptions, you simply reference the replication instance ID in the instance_event_subscription_keys
field when subscribing to instance notifications, or the replication_tasks
keys in the task_event_subscription_keys
to subscribe to the tasks notifications (all or only select keys for select tasks):
...
event_subscriptions = {
instance = {
instance_event_subscription_keys = ["readme-example"]
...
}
task = {
task_event_subscription_keys = ["src1_dest1", "src1_dest2]
...
}
}
...
Tasks
Tasks are the "jobs" that perform the necessary actions of migrating from source
to target
, including any transformations and/or mappings of the data in transit. Tasks are largely controlled by task settings that are defined in a JSON document.
Example task settings JSON document:
{
"TargetMetadata": {
"TargetSchema": "",
"SupportLobs": true,
"FullLobMode": false,
"LobChunkSize": 64,
"LimitedSizeLobMode": true,
"LobMaxSize": 32,
"BatchApplyEnabled": true
},
"FullLoadSettings": {
"TargetTablePrepMode": "DO_NOTHING",
"CreatePkAfterFullLoad": false,
"StopTaskCachedChangesApplied": false,
"StopTaskCachedChangesNotApplied": false,
"MaxFullLoadSubTasks": 8,
"TransactionConsistencyTimeout": 600,
"CommitRate": 10000
},
"Logging": {
"EnableLogging": false
},
"ControlTablesSettings": {
"ControlSchema":"",
"HistoryTimeslotInMinutes":5,
"HistoryTableEnabled": false,
"SuspendedTablesTableEnabled": false,
"StatusTableEnabled": false
},
"StreamBufferSettings": {
"StreamBufferCount": 3,
"StreamBufferSizeInMB": 8
},
"ChangeProcessingTuning": {
"BatchApplyPreserveTransaction": true,
"BatchApplyTimeoutMin": 1,
"BatchApplyTimeoutMax": 30,
"BatchApplyMemoryLimit": 500,
"BatchSplitSize": 0,
"MinTransactionSize": 1000,
"CommitTimeout": 1,
"MemoryLimitTotal": 1024,
"MemoryKeepTime": 60,
"StatementCacheSize": 50
},
"ChangeProcessingDdlHandlingPolicy": {
"HandleSourceTableDropped": true,
"HandleSourceTableTruncated": true,
"HandleSourceTableAltered": true
},
"ErrorBehavior": {
"DataErrorPolicy": "LOG_ERROR",
"DataTruncationErrorPolicy":"LOG_ERROR",
"DataErrorEscalationPolicy":"SUSPEND_TABLE",
"DataErrorEscalationCount": 50,
"TableErrorPolicy":"SUSPEND_TABLE",
"TableErrorEscalationPolicy":"STOP_TASK",
"TableErrorEscalationCount": 50,
"RecoverableErrorCount": 0,
"RecoverableErrorInterval": 5,
"RecoverableErrorThrottling": true,
"RecoverableErrorThrottlingMax": 1800,
"ApplyErrorDeletePolicy":"IGNORE_RECORD",
"ApplyErrorInsertPolicy":"LOG_ERROR",
"ApplyErrorUpdatePolicy":"LOG_ERROR",
"ApplyErrorEscalationPolicy":"LOG_ERROR",
"ApplyErrorEscalationCount": 0,
"FullLoadIgnoreConflicts": true
}
}
Examples
Examples codified under the examples
are intended to give users references for how to use the module(s) as well as testing/validating changes to the source code of the module. If contributing to the project, please be sure to make any appropriate updates to the relevant examples to allow maintainers to test your changes and to keep the examples up to date for users. Thank you!
Requirements
Name | Version |
---|---|
<a name="requirement_terraform"></a> terraform | >= 1.0 |
<a name="requirement_aws"></a> aws | >= 5.32 |
<a name="requirement_time"></a> time | >= 0.9 |
Providers
Name | Version |
---|---|
<a name="provider_aws"></a> aws | >= 5.32 |
<a name="provider_time"></a> time | >= 0.9 |
Modules
No modules.
Resources
Name | Type |
---|---|
aws_dms_certificate.this | resource |
aws_dms_endpoint.this | resource |
aws_dms_event_subscription.this | resource |
aws_dms_replication_config.this | resource |
aws_dms_replication_instance.this | resource |
aws_dms_replication_subnet_group.this | resource |
aws_dms_replication_task.this | resource |
aws_dms_s3_endpoint.this | resource |
aws_iam_policy.access | resource |
aws_iam_role.access | resource |
aws_iam_role.dms_access_for_endpoint | resource |
aws_iam_role.dms_cloudwatch_logs_role | resource |
aws_iam_role.dms_vpc_role | resource |
aws_iam_role_policy_attachment.access | resource |
aws_iam_role_policy_attachment.access_additional | resource |
time_sleep.wait_for_dependency_resources | resource |
aws_caller_identity.current | data source |
aws_iam_policy_document.access | data source |
aws_iam_policy_document.access_assume | data source |
aws_iam_policy_document.dms_assume_role | data source |
aws_iam_policy_document.dms_assume_role_redshift | data source |
aws_partition.current | data source |
aws_region.current | data source |
Inputs
Name | Description | Type | Default | Required |
---|---|---|---|---|
<a name="input_access_iam_role_description"></a> access_iam_role_description | Description of the role | string | null | no |
<a name="input_access_iam_role_name"></a> access_iam_role_name | Name to use on IAM role created | string | null | no |
<a name="input_access_iam_role_path"></a> access_iam_role_path | IAM role path | string | null | no |
<a name="input_access_iam_role_permissions_boundary"></a> access_iam_role_permissions_boundary | ARN of the policy that is used to set the permissions boundary for the IAM role | string | null | no |
<a name="input_access_iam_role_policies"></a> access_iam_role_policies | Map of IAM role policy ARNs to attach to the IAM role | map(string) | {} | no |
<a name="input_access_iam_role_tags"></a> access_iam_role_tags | A map of additional tags to add to the IAM role created | map(string) | {} | no |
<a name="input_access_iam_role_use_name_prefix"></a> access_iam_role_use_name_prefix | Determines whether the IAM role name (access_iam_role_name ) is used as a prefix | bool | true | no |
<a name="input_access_iam_statements"></a> access_iam_statements | A map of IAM policy statements for custom permission usage | any | {} | no |
<a name="input_access_kms_key_arns"></a> access_kms_key_arns | A list of KMS key ARNs the access IAM role is permitted to decrypt | list(string) | [] | no |
<a name="input_access_secret_arns"></a> access_secret_arns | A list of SecretManager secret ARNs the access IAM role is permitted to access | list(string) | [] | no |
<a name="input_access_source_s3_bucket_arns"></a> access_source_s3_bucket_arns | A list of S3 bucket ARNs the access IAM role is permitted to access | list(string) | [] | no |
<a name="input_access_target_dynamodb_table_arns"></a> access_target_dynamodb_table_arns | A list of DynamoDB table ARNs the access IAM role is permitted to access | list(string) | [] | no |
<a name="input_access_target_elasticsearch_arns"></a> access_target_elasticsearch_arns | A list of Elasticsearch ARNs the access IAM role is permitted to access | list(string) | [] | no |
<a name="input_access_target_kinesis_arns"></a> access_target_kinesis_arns | A list of Kinesis ARNs the access IAM role is permitted to access | list(string) | [] | no |
<a name="input_access_target_s3_bucket_arns"></a> access_target_s3_bucket_arns | A list of S3 bucket ARNs the access IAM role is permitted to access | list(string) | [] | no |
<a name="input_certificates"></a> certificates | Map of objects that define the certificates to be created | map(any) | {} | no |
<a name="input_create"></a> create | Determines whether resources will be created | bool | true | no |
<a name="input_create_access_iam_role"></a> create_access_iam_role | Determines whether the ECS task definition IAM role should be created | bool | true | no |
<a name="input_create_access_policy"></a> create_access_policy | Determines whether the IAM policy should be created | bool | true | no |
<a name="input_create_iam_roles"></a> create_iam_roles | Determines whether the required DMS IAM resources will be created | bool | true | no |
<a name="input_create_repl_instance"></a> create_repl_instance | Indicates whether a replication instace should be created | bool | true | no |
<a name="input_create_repl_subnet_group"></a> create_repl_subnet_group | Determines whether the replication subnet group will be created | bool | true | no |
<a name="input_enable_redshift_target_permissions"></a> enable_redshift_target_permissions | Determines whether redshift.amazonaws.com is permitted access to assume the dms-access-for-endpoint role | bool | false | no |
<a name="input_endpoints"></a> endpoints | Map of objects that define the endpoints to be created | any | {} | no |
<a name="input_event_subscription_timeouts"></a> event_subscription_timeouts | A map of timeouts for event subscription create/update/delete operations | map(string) | {} | no |
<a name="input_event_subscriptions"></a> event_subscriptions | Map of objects that define the event subscriptions to be created | any | {} | no |
<a name="input_iam_role_permissions_boundary"></a> iam_role_permissions_boundary | ARN of the policy that is used to set the permissions boundary for the role | string | null | no |
<a name="input_iam_role_tags"></a> iam_role_tags | A map of additional tags to apply to the DMS IAM roles | map(string) | {} | no |
<a name="input_repl_instance_allocated_storage"></a> repl_instance_allocated_storage | The amount of storage (in gigabytes) to be initially allocated for the replication instance. Min: 5, Max: 6144, Default: 50 | number | null | no |
<a name="input_repl_instance_allow_major_version_upgrade"></a> repl_instance_allow_major_version_upgrade | Indicates that major version upgrades are allowed | bool | true | no |
<a name="input_repl_instance_apply_immediately"></a> repl_instance_apply_immediately | Indicates whether the changes should be applied immediately or during the next maintenance window | bool | null | no |
<a name="input_repl_instance_auto_minor_version_upgrade"></a> repl_instance_auto_minor_version_upgrade | Indicates that minor engine upgrades will be applied automatically to the replication instance during the maintenance window | bool | true | no |
<a name="input_repl_instance_availability_zone"></a> repl_instance_availability_zone | The EC2 Availability Zone that the replication instance will be created in | string | null | no |
<a name="input_repl_instance_class"></a> repl_instance_class | The compute and memory capacity of the replication instance as specified by the replication instance class | string | null | no |
<a name="input_repl_instance_engine_version"></a> repl_instance_engine_version | The engine version number of the replication instance | string | null | no |
<a name="input_repl_instance_id"></a> repl_instance_id | The replication instance identifier. This parameter is stored as a lowercase string | string | null | no |
<a name="input_repl_instance_kms_key_arn"></a> repl_instance_kms_key_arn | The Amazon Resource Name (ARN) for the KMS key that will be used to encrypt the connection parameters | string | null | no |
<a name="input_repl_instance_multi_az"></a> repl_instance_multi_az | Specifies if the replication instance is a multi-az deployment. You cannot set the availability_zone parameter if the multi_az parameter is set to true | bool | null | no |
<a name="input_repl_instance_network_type"></a> repl_instance_network_type | The type of IP address protocol used by a replication instance. Valid values: IPV4, DUAL | string | null | no |
<a name="input_repl_instance_preferred_maintenance_window"></a> repl_instance_preferred_maintenance_window | The weekly time range during which system maintenance can occur, in Universal Coordinated Time (UTC) | string | null | no |
<a name="input_repl_instance_publicly_accessible"></a> repl_instance_publicly_accessible | Specifies the accessibility options for the replication instance | bool | null | no |
<a name="input_repl_instance_subnet_group_id"></a> repl_instance_subnet_group_id | An existing subnet group to associate with the replication instance | string | null | no |
<a name="input_repl_instance_tags"></a> repl_instance_tags | A map of additional tags to apply to the replication instance | map(string) | {} | no |
<a name="input_repl_instance_timeouts"></a> repl_instance_timeouts | A map of timeouts for replication instance create/update/delete operations | map(string) | {} | no |
<a name="input_repl_instance_vpc_security_group_ids"></a> repl_instance_vpc_security_group_ids | A list of VPC security group IDs to be used with the replication instance | list(string) | null | no |
<a name="input_repl_subnet_group_description"></a> repl_subnet_group_description | The description for the subnet group | string | null | no |
<a name="input_repl_subnet_group_name"></a> repl_subnet_group_name | The name for the replication subnet group. Stored as a lowercase string, must contain no more than 255 alphanumeric characters, periods, spaces, underscores, or hyphens | string | null | no |
<a name="input_repl_subnet_group_subnet_ids"></a> repl_subnet_group_subnet_ids | A list of the EC2 subnet IDs for the subnet group | list(string) | [] | no |
<a name="input_repl_subnet_group_tags"></a> repl_subnet_group_tags | A map of additional tags to apply to the replication subnet group | map(string) | {} | no |
<a name="input_replication_tasks"></a> replication_tasks | Map of objects that define the replication tasks to be created | any | {} | no |
<a name="input_s3_endpoints"></a> s3_endpoints | Map of objects that define the S3 endpoints to be created | any | {} | no |
<a name="input_tags"></a> tags | A map of tags to use on all resources | map(string) | {} | no |
Outputs
Name | Description |
---|---|
<a name="output_access_iam_role_arn"></a> access_iam_role_arn | Access IAM role ARN |
<a name="output_access_iam_role_name"></a> access_iam_role_name | Access IAM role name |
<a name="output_access_iam_role_unique_id"></a> access_iam_role_unique_id | Stable and unique string identifying the access IAM role |
<a name="output_certificates"></a> certificates | A map of maps containing the certificates created and their full output of attributes and values |
<a name="output_dms_access_for_endpoint_iam_role_arn"></a> dms_access_for_endpoint_iam_role_arn | Amazon Resource Name (ARN) specifying the role |
<a name="output_dms_access_for_endpoint_iam_role_id"></a> dms_access_for_endpoint_iam_role_id | Name of the IAM role |
<a name="output_dms_access_for_endpoint_iam_role_unique_id"></a> dms_access_for_endpoint_iam_role_unique_id | Stable and unique string identifying the role |
<a name="output_dms_cloudwatch_logs_iam_role_arn"></a> dms_cloudwatch_logs_iam_role_arn | Amazon Resource Name (ARN) specifying the role |
<a name="output_dms_cloudwatch_logs_iam_role_id"></a> dms_cloudwatch_logs_iam_role_id | Name of the IAM role |
<a name="output_dms_cloudwatch_logs_iam_role_unique_id"></a> dms_cloudwatch_logs_iam_role_unique_id | Stable and unique string identifying the role |
<a name="output_dms_vpc_iam_role_arn"></a> dms_vpc_iam_role_arn | Amazon Resource Name (ARN) specifying the role |
<a name="output_dms_vpc_iam_role_id"></a> dms_vpc_iam_role_id | Name of the IAM role |
<a name="output_dms_vpc_iam_role_unique_id"></a> dms_vpc_iam_role_unique_id | Stable and unique string identifying the role |
<a name="output_endpoints"></a> endpoints | A map of maps containing the endpoints created and their full output of attributes and values |
<a name="output_event_subscriptions"></a> event_subscriptions | A map of maps containing the event subscriptions created and their full output of attributes and values |
<a name="output_replication_instance_arn"></a> replication_instance_arn | The Amazon Resource Name (ARN) of the replication instance |
<a name="output_replication_instance_private_ips"></a> replication_instance_private_ips | A list of the private IP addresses of the replication instance |
<a name="output_replication_instance_public_ips"></a> replication_instance_public_ips | A list of the public IP addresses of the replication instance |
<a name="output_replication_instance_tags_all"></a> replication_instance_tags_all | A map of tags assigned to the resource, including those inherited from the provider default_tags configuration block |
<a name="output_replication_subnet_group_id"></a> replication_subnet_group_id | The ID of the subnet group |
<a name="output_replication_tasks"></a> replication_tasks | A map of maps containing the replication tasks created and their full output of attributes and values |
<a name="output_s3_endpoints"></a> s3_endpoints | A map of maps containing the S3 endpoints created and their full output of attributes and values |
<a name="output_serverless_replication_tasks"></a> serverless_replication_tasks | A map of maps containing the serverless replication tasks (replication_config) created and their full output of attributes and values |
License
Apache-2.0 Licensed. See LICENSE.