Fargate and ECS

Using the Streaming agent within AWS means using ECS to launch containers. This means creating in AWS:

• ECS Clusters
• Task Definitions and included container definitions
• Cluster Services

---

Prerequisites

This should be one of the final steps for a manual installation of the agent, requiring the below resources to be provisioned and their details (such as ARNs) known. Please consult your cloud administrator before performing these actions.

• Secrets Manager secrets
• CloudWatch log groups
• IAM Roles
• S3 Buckets
• Subnets and Security groups

Note

This step is only required for manual installations. It is highly recommended to instead use templated installations which, both Basic and Advanced, will create an ECS Cluster, Task Definitions and Cluster Services on your behalf.

---

Create a Cluster

1. Log in to your AWS account. This should be the same account that you will be using the Matillion Streaming agent in.
2. Browse to the Elastic Container Service.
3. Click Create Cluster.
4. Choose Networking only and Next Step .
5. Give your Cluster a Cluster name and click Create.

Create Task Definition

1. While still in the Elastic Container Service.
2. Click Create new Task Definition.
3. Select FARGATE and click Next step.
4. Enter details for Configure task and container definitions:
   • Name: An arbitrary name for the Task Definition.
   • Task role: Select the Task Role created for the Streaming agent. See IAM Roles for more information.
   • Operating system family: Linux.
   • Task execution role: Select the Task Execution Role created for the Streaming agent. See IAM Roles for more information.
   • Task memory (GB): 8 GB.
   • Task CPU: 4vCPU.
5. Click Add Container.

Container details

1. In the Add Container dialog, give your container an arbitrary Container name.
2. Set the Image to public.ecr.aws/matillion/cdc-agent:2.
3. Add the following environment variables:

Environment Variable	Description
ID_AGENT	This value is provided when creating a new Streaming agent in Data Loader.
ID_ORGANIZATION	This value is provided when creating a new Streaming agent in Data Loader.
PLATFORM_WEBSOCKET_ENDPOINT	This value must be set to wss://ws-<region>.matillion-cdc-prod.matillion.com:443/ws where <region> is either eu or us depending on the Data Loader region you are building the pipeline in.
PLATFORM_KEY_PROVIDER	aws-secrets-manager
PLATFORM_KEY_NAME	The name of the secret containing your Platform Key. By default this is agent-rsa
SECRET_PROVIDERS	aws-secrets-manager:1

1. In Log configuration, ensure the Log driver is awslogs and enter the following Log options:

Key	Value
awslogs-group	The name of the CloudWatch log group you wish to use. See CloudWatch Logs
awslogs-region	The AWS region code that your log group belongs in
awslogs-steam-prefix	An arbitrary prefix for the name of your CloudWatch log streams

1. Click Add.

Completing the Task Definition

1. Now back on the Task Definition page with your Container added, scroll to the bottom and click Create.

Creating a Cluster Service

At this point we have created an ECS Cluster and a Task Definition (with a defined Container). We now have to create a service within our Cluster.

1. Return to the Elastic Container Service and navigate to the Cluster previously set up.
2. On the Services tab, click Create

3. Enter the following details:

   Field	Value
   Launch Type	FARGATE
   Operating system family	Linux
   Task Definition	Select the Task Definition created in the previous steps.
   Revision	Select your Latest revision.
   Platform Version	LATEST version
   Cluster	Select the cluster.
   Service Name	Enter an arbitrary name for this Service. Default value is the name of your Cluster.
   Service type	REPLICA
   Number of tasks	1
   Minimum healthy percent	100
   Maximum percent	200
   Deployment circuit breaker	Disabled
   Deployment type	Rolling update
   Enable ECS managed tags	Checked
   Propagate tags from	Do not propagate

4. Click Next step.

5. Select the customer private cloud, subnets and Security groups you wish this service to use. For more information, see Subnets and Security Groups.
6. For Load balancer type:
   • Our recommended installation uses a private subnet that requires no inbound traffic. Thus, we recommend selecting None.
   • If you plan to use inbound traffic, consult your cloud administrator.
7. Click Next step.
8. On the Set Auto Scaling page, click Next step.
9. Finally, click Create Service.