CloudFormation ECS Fargate advanced template

This article details how to install the Matillion Streaming agent on Amazon ECS Fargate using a CloudFormation template.

Note

It's recommended you follow the quick guide for deploying your Streaming agent using AWS templates. For more information, read Quick guide for deploying a Streaming agent in AWS.

There are three options to create a CloudFormation stack in the AWS console for agent deployment:

Quick-create, which will pre-fill values into the template for you. This is the recommended option.
Quick-create link from the documentation, it will provide the template but no values will be included.
Download the template yourself and define custom values for the parameters.

Ensure you have: - Permissions to provision cloud resources in the AWS environment that the source database is running. - Enabled CDC in your source database. - Allowed the IP address. - Set up the template variables as referenced in the documentation.

It is highly recommended you read the documentation and prerequisites before beginning this process. It's likely you will require input from your organization's cloud administrator for access and permissions.

Created resources

This template will create the following resources in your AWS account:

ECS Task Definition.
ECS Fargate Cluster.
S3 Bucket.
CloudWatch Log Group.
IAM Roles with permissions for the following:
Task Roles to your S3 bucket, ECS, EC2, and Secrets Manager (see "Task IAM Role" section).
Task Execution Roles to pull from ECR Public repositories (see "Task Execution IAM Role" section).

Visualization

Prerequisites

Edit the template

Users should inspect the template in a text editor and ensure the values are as expected before proceeding. In particular, the PLATFORM_WEBSOCKET_ENDPOINT environment variable should be edited to match the expected endpoint and region. Read Environment Variables for more information.

Resources

The advanced template assumes you have certain resources already set up in your AWS stack. You need to provide details on these resources such as names, paths, and ARNs.

Subnets within a customer private cloud with outbound access to the relevant data sources and destinations, as well as to ECR and to the Matillion CDC websocket endpoint.
Security group controlling ingress/egress within the subnets.
Secrets Manager entries for your Platform Key and database passwords.

User access

You as a user are also expected to have access to certain details and permissions:

Access to the Hub account and Data Loader.
Streaming agent environment variables (generated in Data Loader when creating a new agent).
Data Loader platform key (generated once per Data Loader account the first time you make an agent).
Access to AWS with the ability to create a stack on a billable account. You may require an administrator from your organization to either give access or perform this process with you.

AWS Permissions

cloudformation:CreateStack: Allows the user to create CloudFormation stacks.
s3:GetObject: Grants the ability to get objects (CloudFormation templates) from an S3 bucket.
s3:ListBucket: Permits listing objects within an S3 bucket.
iam:CreateRole: Allows the creation of IAM roles.
iam:AttachRolePolicy: Enables attaching policies to an IAM role.
iam:PassRole: Necessary if the CloudFormation stack involves passing an IAM role to an AWS service.
Permissions for actions such as ec2:CreateVpc, ec2:CreateSubnet, etc., depend on the networking configuration.

In most cases, you don't need to manually create a service-linked role. When you create a cluster or create/update a service in the AWS Management Console, the AWS CLI, or the AWS API, Amazon ECS creates the service-linked role for you.

Downloads

This article covers the advanced template installation manually by downloading the template below.

matillion-cdc-agent-template

And an accompanying parameter file for those installing via CLI:

parameters-additional.json

Next steps

When your stack has been fully created (this may take a few minutes) you can return to Matillion CDC and view the Agents list. Your new agent should appear in the list and have the Connected status if all has gone well. If not, please consult our troubleshooting documentation.

If you are creating a pipeline and have completed this step, consult the CDC Pipelines documentation to continue.