Skip to content

Fargate and ECS

Using the CDC agent within AWS means using ECS to launch containers. This means creating in AWS:

  • ECS Clusters
  • Task Definitions and included container definitions
  • Cluster Services

Prerequisites

This should be one of the final steps for a manual installation of the agent, requiring the below resources to be provisioned and their details (such as ARNs) known. Please consult your cloud administrator before performing these actions.

  • Secrets Manager secrets
  • CloudWatch log groups
  • IAM Roles
  • S3 Buckets
  • Subnets and Security groups

:::warning{title='Warning'} This step is only required for manual installations. It is highly recommended to instead use templated installations which, both Basic and Advanced, will create an ECS Cluster, Task Definitions and Cluster Services on your behalf. :::

Create a Cluster

  1. Log in to your AWS account. This should be the same account that you will be using the Matillion CDC agent in.
  2. Browse to the Elastic Container Service.
  3. Click Create Cluster.
  4. Choose Networking only and Next Step .
  5. Give your Cluster a Cluster name and click Create.

Create Task Definition

  1. While still in the Elastic Container Service.
  2. Click Create new Task Definition.
  3. Select FARGATE and click Next step.
  4. Enter details for Configure task and container definitions:
    • Name: An arbitrary name for the Task Definition.
    • Task role: Select the Task Role created for the cdc agent. See IAM Roles for more information.
    • Operating system family: Linux.
    • Task execution role: Select the Task Execution Role created for the cdc agent. See IAM Roles for more information.
    • Task memory (GB): 8 GB.
    • Task CPU: 4vCPU.
  5. Click Add Container.

Container details

  1. In the Add Container dialog, give your container an arbitrary Container name.
  2. Set the Image to public.ecr.aws/matillion/cdc-agent:2.
  3. Add the following environment variables:
Environment Variable Description
ID_AGENT This value is provided when creating a new CDC agent in Data Loader.
ID_ORGANIZATION This value is provided when creating a new CDC agent in Data Loader.
PLATFORM_WEBSOCKET_ENDPOINT This value must be set to wss://ws-<region>.matillion-cdc-prod.matillion.com:443/ws where <region> is either eu or us depending on the Data Loader region you are building the pipeline in.
PLATFORM_KEY_PROVIDER aws-secrets-manager
PLATFORM_KEY_NAME The name of the secret containing your Platform Key. By default this is agent-rsa
SECRET_PROVIDERS aws-secrets-manager:1
  1. In Log configuration, ensure the Log driver is awslogs and enter the following Log options:
Key Value
awslogs-group The name of the CloudWatch log group you wish to use. See CloudWatch Logs
awslogs-region The AWS region code that your log group belongs in
awslogs-steam-prefix An arbitrary prefix for the name of your CloudWatch log streams
  1. Click Add.

Completing the Task Definition

  1. Now back on the Task Definition page with your Container added, scroll to the bottom and click Create.

Creating a Cluster Service

At this point we have created an ECS Cluster and a Task Definition (with a defined Container). We now have to create a service within our Cluster.

  1. Return to the Elastic Container Service and navigate to the Cluster previously set up.
  2. On the Services tab, click Create

  3. Enter the following details:

    Field Value
    Launch Type FARGATE
    Operating system family Linux
    Task Definition Select the Task Definition created in the previous steps.
    Revision Select your Latest revision.
    Platform Version LATEST version
    Cluster Select the cluster.
    Service Name Enter an arbitrary name for this Service. Default value is the name of your Cluster.
    Service type REPLICA
    Number of tasks 1
    Minimum healthy percent 100
    Maximum percent 200
    Deployment circuit breaker Disabled
    Deployment type Rolling update
    Enable ECS managed tags Checked
    Propagate tags from Do not propagate

  4. Click Next step.

  5. Select the customer private cloud, subnets and Security groups you wish this service to use. For more information, see Subnets and Security Groups.
  6. For Load balancer type:
    • Our recommended installation uses a private subnet that requires no inbound traffic. Thus, we recommend selecting None.
    • If you plan to use inbound traffic, consult your cloud administrator.
  7. Click Next step.
  8. On the Set Auto Scaling page, click Next step.
  9. Finally, click Create Service.