Skip to content

AWS Fargate manual agent setup

This page is a guide to manually installing an agent into AWS Fargate.

Note

This documentation refers to the New ECS Experience for Elastic Container Service.

Note

Each time you click Create in AWS, a colored banner will appear at the top of the AWS UI.

  • Blue indicates the creation of a resource is in-progress.
  • Green indicates the creation of a resource has completed.
  • Red indicates the creation of a resource has failed.

Prerequisites

  • First you must create an agent from the Hub, as described in Create an agent. This will provide you with the parameters and values needed to manually complete the agent setup in AWS Fargate.
  • Before deploying an agent in AWS, confirm the existence of a service-linked role for Amazon ECS in your AWS account. Typically, Amazon ECS generates this role automatically. If not, follow the manual creation documentation for setup instructions.

Security groups

  1. Log in to the AWS Console.
  2. Once logged in, type security groups in the search bar.
  3. In the Features list, select Security Groups|EC2 feature.
  4. Choose Create security group.

Complete the following fields:

Property Description
Security group name Add the following: matillion-agent-<agentID> Your agent ID is available at Agent environment variables in Hub → Manage Agents.
Description Add the following description: agent security group for agent ID <agentID>. Alternatively, add a description of your choice.
VPC Select your VPC.
Inbound rules Add any inbound rules. None are required by default.
Outbound rules 0.0.0.0/0
Tags You may want to add a tag where Key = matillion-agent-id and Value = <agentID>.

Click Create security group to confirm creation. On successful creation of the security group, you will be redirected to the new security group's dashboard. Please make a note of your Security group ID.

Log groups (optional)

This part may be omitted if you want to use the default that can be created as part of the Task definition.

  1. In the AWS console, type CloudWatch in the search bar and navigate to AWS CloudWatch.
  2. In the left-hand menu, choose Logs → Log groups.
  3. Click Create log group.

Complete the following fields:

Property Description
Log group name Add the following: /ecs/matillion-agent/<agentID>.
Retention setting Select a retention setting. It's recommended to choose 1 month (30 days) from the dropdown.
KMS key ARN - optional Not required.
Tags You may want to add a tag where Key = matillion-agent-id and Value = <agentID>.

Click Create to confirm creation of the log group. On successful creation of the log group, you will be redirected to Log groups.

Cluster

  1. In the AWS console, type Elastic Container Service in the search bar and choose that service.
  2. Click Get started or Clusters if this is your first time using this service.
  3. Click Create Cluster.

Cluster configuration

Property Description
Cluster name Add the following: matillion-agent-<agentID>
VPC Select your VPC.
Subnets Optional. Choose subnets where your tasks will run.
Use Container Insights Optional, but recommended. Toggle this to On.
Tags You may want to add a tag where Key = matillion-agent-id and Value = <agentID>.

Click Create to confirm creation of the cluster. On successful creation of the cluster, you will be redirected to All Clusters.

Task definitions

  1. In the AWS console, type Elastic Container Service in the search bar and choose that service.
  2. In the left-hand menu, click Task definitions.
  3. Click Create new task definition.

Complete the following fields:

Task definition configuration

Property Description
Task definition family Add the following: matillion-agent-<agentID>. This must be unique—two task definitions can't share a name.

Container - n

Property Description
Name matillion-agent
Image URI Copy and paste your Agent Image URI from Agent details.
Port mappings Remove any existing entries.

In Environment Variables under Add individually, add the following environment variables using the values in your Agent details page in Hub:

Key Value type Value
ACCOUNT_ID Value Your ACCOUNT_ID value.
AGENT_ID Value Your AGENT_ID value.
MATILLION_REGION Value Your MATILLION_REGION value.
OAUTH_CLIENT_ID ValueFrom The Amazon resource number (ARN) of the AWS Secrets Manager secret appended with the key name that contains the agent client_id in this format {your-arn}:{key}::. For example: arn:aws:secretsmanager:eu-west-1:112233445566:secret:my_secret_name-9wxygh:client_id:: To learn more about this process, read Add agent credentials to AWS Secrets Manager.
OAUTH_CLIENT_SECRET ValueFrom The Amazon resource number (ARN) of the AWS Secrets Manager secret appended with the key name that contains the agent client_secret in this format {your-arn}:{key}::. For example: arn:aws:secretsmanager:eu-west-1:112233445566:secret:my_secret_name-9wxygh:client_secret::
EXTENSION_LIBRARY_LOCATION* (optional) Value (Optional) The location for your additional Python libraries, specified without the protocol. For example: agent-python-libs or mybucket/agent-python-libs. Currently, only S3 is supported.
EXTENSION_LIBRARY_PROTOCOL* (optional) Value (Optional) Specified in conjunction with EXTENSION_LIBRARY_LOCATION, for example S3.

Click Next.

Configure environment, storage, monitoring, and tags

Property Description
App environment Choose AWS Fargate (serverless)
CPU Choose 1 vCPU.
Memory Choose 4 GB.
Task role Use an IAM role that has similar permissions to your role. If you're using the same IAM role, you might need to add ecs-tasks.amazonaws.com to the Service section in the Trust Relationship section of your IAM role to allow the role to work with the ECS service. An example trusted entity can be found beneath this table. To learn more, read AWS IAM Roles.
Task execution role Choose escTaskExecutionRole unless instructed otherwise.

Example trusted entity:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ecs-tasks.amazonaws.com",
                    "ec2.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Scroll down to Monitoring and logging - optional.

The awslogs-group value should have automatically suggested a log group name. If you don't wish to use another log group, leave the aws-create-group entry and a log group will be created with the awslogs-group value.

If the naming of components has been consistent so far, the awslogs-group value should equal the log group that was previously created (note the use of the / preceding the ID in the original Log group name). Otherwise, select the log group created earlier. Delete the aws-create-group entry by clicking Remove.

Once again, add a tag where Key = matillion-agent-id and Value = <agentID>.

Click Next. Review your settings and if you're happy, click Create. You will receive a blue banner at the top of the page explaining that AWS is currently creating the task definition. This banner will turn green upon completion.

Service

  1. In your newly created task definition, click the Deploy drop-down button and then click Create service.
  2. In the Choose a cluster field, choose the cluster you created earlier.

In Deployment configuration, complete the following fields:

Property Description
Service name Add the following: matillion-agent-<agentID> or another suitable descriptor.
Desired tasks Choose your desired number of AWS tasks to use. A minimum of 2 tasks are required. More tasks reduce the execution time of multi-threaded pipelines. If you use only 1 task, you run the risk of service outages during agent upgrades.

In Deployment options, set both Min running tasks and Max running tasks. The max must be greater than the min.

In Networking, under Security group name, locate the security group you created earlier and select it. Remove any other security groups if they are not required.

Ensure that Public IP is toggled to Turned on, unless you are using Network Address Translation.

As in previous sections, you may want to add a tag where Key = matillion-agent-id and Value = <agentID>.

Click Create.

Once the deployment has finished, your agent start up and when connected back to Matillion, you should see a running status in your Manage Agents screen.

Running agent