Skip to content

CloudFormation EC2 ECS proxy templates

This document outlines the prerequisites and steps for the deployment of the Matillion CDC agent on EC2 ECS and behind a web proxy server.

Prerequisites

  • An active Amazon Web Services (AWS) account with the requisite permissions to add inbound rules and to set network and security settings.
  • Access to AWS VPC.
  • Two or more private subnets within the target VPC that require access to a NAT gateway.
  • One public subnet within a NAT gateway and routing tables configured.
  • URLs for HTTP and HTTPS proxy server.

Contact your cloud network engineer for the required permissions and network settings.

Required parameters

The list of required parameters from AWS/Data Loader:

Name Location Owner Example
VpcId AWS Cloud Portal > VPC AWS vpc-xxxxxxxxxx
PrivateSubnetId AWS Cloud Portal > Subnets AWS subnet-xxxxxx
SecurityGroupId AWS Cloud Portal > EC2 > Security Groups AWS sg-xxxxxxx
SSHKeyName AWS Cloud Portal > EC2 > Key Pair AWS Name of key pair
AgentId Hub > MDL > Agents > Add Agent (Prerequisite page) Data Loader 111111111-2222-3333-3333-44444444444
OrganizationId Hub > MDL > Agents > Add Agent(Prerequisite page) Data Loader 111111111-2222-3333-3333-44444444444
PlatformKeyName AWS Cloud Portal > Secret Manager AWS Certificate value stored within Secret Manager in AWS
PlatformWebsocketEndpoint Hub > MDL > Agents > Add Agent(Prerequisite page) Data Loader wss://ws-eu.matillion-cdc-prod.matillion.com:443/wsorwss://ws-us.matillion-cdc-prod.matillion.com:443/ws

If using the proxy template, the following additional parameters are required:

Name Location Owner Example
Http Proxy Url Networking Team Client http://:
Https Proxy Url Networking Team Client https://:

If proxy requires certificate authentication:

Name Location Owner Example
Certificate Download Url Networking Team Client http:// /
Certificate Zip Filename Networking Team Client http://

Create a CDC agent in Data Loader

  1. Log in to the Hub.

  2. The My Accounts page lists any accounts you have already created or joined. At the bottom of this list, click Add new account. Read Create an Account to learn more about this topic.

    Each Hub account can generate its own unique platform key that your CDC agent will use to communicate with Data Loader. With this in mind, create the CDC agent in the account that matches the platform key you will be using.

  3. Choose Data Loader as the service on the Select your service page.

  4. On the Data Loader dashboard, scroll to the lower-right of the UI and choose your region.
  5. Select Agents in the left sidebar and click Add agent.
  6. Give your agent a sensible Agent name and Description. Click Continue.
  7. Since this guide is for AWS, select AWS as your cloud provider.
  8. Choose CloudFormation as the service to provision and deploy your cloud resources from for the CDC agent installation.
  9. In the Prerequisites for agent setup, note the following values:
    • ID_ORGANIZATION: This value is used when deploying the CDC agent in AWS. The value is unique per agent.
    • ID_AGENT: Also used when deploying the CDC agent. The value is unique per agent.
    • PLATFORM_WEBSOCKET_ENDPOINT: Also used when deploying the agent. The value is unique for the Data Loader region (US or EU).
  10. Public/Private key pair: This is a generated value. If you haven't generated a platform secret for your account yet, Data Loader will prompt you to do so when creating a CDC pipeline. You need to store this value in AWS Secrets Manager where your CDC agent can access it. For security reasons, this key pair can only be generated and shown once per account, so make sure to copy and save it for future use.
  11. Check the I have saved the private key in AWS Secrets Manager and made a note of the secret name checkbox.
  12. Click Submit key pair.

Deploy the CDC agent in AWS with a proxy

  1. Download the required template from the Download Templates section at the bottom of this page.
  2. Log in to the AWS console.

  3. In the AWS console, navigate to the region drop-down and select the region in which you wish to deploy the CDC agent.

    Make sure the region is either eu or us, depending on the Data Loader region you are building the pipeline within. In the AWS console, you must choose the same region.

  4. Navigate to CloudFormation and click Create stack > With new resources (standard).

  5. Under Specify template, select Upload a template file. Select Choose file and upload the template from step 1. Click Next.
  6. Provide the following information for the stack details. Note that the template you choose will autofill part of the information.
    • Stack Name: A unique name for the stack.
    • AgentID: The value you copied from the Prerequisites for agent setup dialog, (step 8) while Creating your CDC agent in Data Loader.
    • Assign PublicIp: Set to Enabled.
    • BucketName: The name of the S3 bucket for staging. The template should autofill this value.
    • ClusterName: The name for the ECS Fargate cluster to be created for hosting your agent. The template should autofill this value.
    • ExecutionRoleName: The role name for the role that ECS will use to initialize the task.
    • PrivateSubnets: The subnet of the private network.
    • HttpProxyUrl: URL of the HTTP proxy.
    • HttpsProxyUrl: URL of the HTTPS proxy.
    • ImageUrl: The URL of the repository you are pulling the CDC agent image from. The template should autofill this value.
    • LogGroupName: Name of the CloudWatch Log Group for agent logging. The template should autofill this value.
    • OrganizationID: This is the value you copied from the Prerequisites for agent setup dialog, (step 8) while Creating your CDC agent in Data Loader.
    • Owner: The owner of the resources created.
    • PlatformKeyName: The name of the AWS Secrets Manager secret in which your generated private key is stored. Consult the AWS Secrets Manager documentation for more information about creating a secrets manager in AWS.
    • PlatformWebSocketEndpoint: The value for PLATFORM_WEBSOCKET_ENDPOINT, copied from the Prerequisites for agent setup dialog, (step 8) while Creating your CDC agent in Data Loader.
    • RoleName: The role name for the role that the task will run as. The template should autofill this value.
    • SSHKeyName: The SSH key required for the server access.
    • SecurityGroups: If there are any existing security groups, select them from the drop-down menu. For more detail see here
    • ServiceName: The name of the ECS Service you want these tasks to run under.
    • TaskDefinitionName: The name of the task definition for agent tasks.
    • VpcId: Select at least two subnets in your selected VPC.
  7. Click Next and then Next.
  8. Tick the statement under Capabilities.
  9. Review the information you've entered, click the required checkboxes and click Create Stack.
  10. The stack creation will then begin and should complete in approximately five minutes.
  11. When the stack creation is complete, the agent container will be deployed as an AWS Elastic Container Service (ECS) Cluster. The CloudFormation template also creates resources in IAM, S3, and CloudWatch Logs.

In Data Loader, your created CDC agent's status should display as Connected and offer the Add Pipeline button.

Download templates

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement