Skip to content

CloudFormation ECS Fargate basic template

This article details how to install the Matillion Streaming agent on Amazon ECS Fargate using a CloudFormation template.

These steps can only be taken after completing the preliminary steps found in AWS Installation: Templated Installations and assumes you have done so.

It is highly recommended you read the documentation and prerequisites before beginning this process. It is likely you will require input from your organization's cloud administrator for access and permissions.

The template provides a blueprint for installation that you may use verbatim, but you may need to modify it to suit your own needs and rules governing your cloud infrastructure.

Created Resources

This template will create the following resources in your AWS account: - ECS Task Definition - ECS Fargate Cluster and Service

Visualization

image.png

Prerequisites

Edit the template

Users should inspect the template in a text editor and ensure the values are as-expected before proceeding. In particular, the PLATFORM_WEBSOCKET_ENDPOINT environment variable should be edited to match the expected endpoint and region. See Environment Variables for more information.

Resources

The basic template assumes you have certain resources already set up in your AWS stack. You will also be required to provide details on these resources such as names, paths and ARN.

  • Subnet and security group within a customer private cloud with outbound access.
  • CloudWatch log group for container log ingestion.
  • S3 bucket for data ingestion.
  • Secrets Manager entries for your Platform Key and database passwords.
  • IAM Roles with permissions for the following:
    • Task Roles to your S3 bucket, ECS, EC2 and Secrets Manager (see "Task IAM Role" section)
    • Task Execution Roles to pull from ECR Public repositories (see "Task Execution IAM Role" section)

User Access

You are also expected to have access to certain details and permissions:

  • Access to a Hub account and Data Loader.
  • Streaming agent Environment Variables (generated in Data Loader when creating a new agent).
  • Data Loader platform key (generated once per Data Loader account the first time you make an agent).
  • Access to AWS with the ability to create a stack on a billable account. You may require an administrator from your organization to either give access or perform this process with you.
  • Before deploying an agent, confirm the existence of a service-linked role for Amazon ECS in your AWS account. Typically, Amazon ECS generates this role automatically. If not, follow the manual creation documentation for setup instructions.

AWS Permissions

  • cloudformation:CreateStack: Allows the user to create CloudFormation stacks.
  • s3:GetObject: Grants the ability to get objects (CloudFormation templates) from an S3 bucket.
  • s3:ListBucket: Permits listing objects within an S3 bucket.
  • iam:CreateRole: Allows the creation of IAM roles.
  • iam:AttachRolePolicy: Enables attaching policies to an IAM role.
  • iam:PassRole: Necessary if the CloudFormation stack involves passing an IAM role to an AWS service.
  • Permissions for actions such as ec2:CreateVpc, ec2:CreateSubnet, etc., depend on the networking configuration.

In most cases, you don't need to manually create a service-linked role. When you create a cluster or create/update a service in the AWS Management Console, the AWS CLI, or the AWS API, Amazon ECS creates the service-linked role for you.

Create Agent Stack

You should have your Platform Key and Agent Identity Environment Variables ready before beginning this process. If you do not have these, please first consult the Creating Agents documentation.

You need permissions to create and manage resources in your AWS or Azure account. Contact your administrator to complete this task. Matillion's provided templates will not work if you have insufficient permissions.

Using the CloudFormation Template

  1. Download the ECS Fargate (CloudFormation) Basic template files in the Downloads section at the bottom of this article.
  2. Log in to your AWS account and navigate to the CloudFormation service.
  3. Choose Create Stack.
  4. Select Template is ready.
  5. Select Upload a template file and then Choose File.
  6. Upload your template .json and click Next to move to the next screen.

Specify stack details

Complete the form with the required details:

Field Description
Stack name An arbitrary name for your new stack.
AgentID The ID_AGENT attained during agent creation.
ClusterName Name the ECS Fargate cluster that will be created from the template. This is where your agent is hosted.
ExecutionRoleArn The ARN of the IAM Role that Task Execution will use. See the Task Execution Role in IAM Roles for more information.
ImageURL The agent image location. Do not edit this field.
LogGroupPath Name of an existing CloudWatch log group to be used for agent logging.
OrganizationID ID_ORGANIZATION attained during agent creation.
PlatformKeyName The name of the secret generated to hold your Platform Key. If you are following our recommended install, this will be agent-rsa. Note that your Task Role policy must include the ARN for this secret.
Region The name of the region you want to create these resources in.
RoleArn The ARN of the IAM Role the Task will use. See the Task Role in IAM Roles for more information.
SecurityGroups Select the Security Group that your service will create log streams in.
ServiceName A unique name for the new ECS Service you want to run tasks under.
SubnetIDs The subnet ID for your intended subnet.
TaskDefinitionName A unique name for agent tasks to run under.

Next Steps

  1. When your details are entered, click Next.
  2. Configure Stack Options: No action is required on this page. Click Next.
  3. Review: Select Acknowledge the template has changed and click Create.

When your stack has been fully created (will take some time) you can return to Matillion CDC and view the Agents list. Your new agent should appear on the list and have the "Connected" status if all has gone well. If not, please consult the Troubleshooting documentation.

If you are creating a Pipeline and have completed this step, consult the CDC Pipelines documentation to review the process and find your next steps.

Downloads

This article covers the Basic Template installation. The template itself can be downloaded here:

matillion-cdc-agent-basic.json

And an accompanying parameters file for those installing via CLI:

parameters-basic.json

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement