AWS Fargate manual Streaming agent setup

This page is a guide to manually installing a Streaming agent into AWS Fargate.

Note

This documentation refers to the New ECS Experience for Elastic Container Service.

Each time you click Create in AWS, a colored banner will appear at the top of the AWS UI.

Blue indicates the creation of a resource is in-progress.
Green indicates the creation of a resource has completed.
Red indicates the creation of a resource has failed.

Prerequisites

First you must create an agent from the Hub, as described in Create an agent. This will provide you with the parameters and values needed to manually complete the agent setup in AWS Fargate.
Before deploying an agent in AWS, confirm the existence of a service-linked role for Amazon ECS in your AWS account. Typically, Amazon ECS generates this role automatically. If not, follow the manual creation documentation for setup instructions.

Security groups

Log in to the AWS Console.
Once logged in, type security groups in the search bar.
In the Features list, select Security Groups|EC2 feature.
Choose Create security group.
Complete the following fields:
- Security group name: Enter matillion-agent-<agentID>, where <agentID> is the AGENT_ID found on the Agent details screen.
- Description: Add the following description: agent security group for agent ID <agentID>, where <agentID> is the AGENT_ID found on the Agent details screen. Alternatively, add a description of your choice.
- VPC: Select your VPC.
- Inbound rules: Add any inbound rules. None are required by default.
- Outbound rules: Add 0.0.0.0/0.
- Tags: You may want to add a tag where Key = matillion-agent-id and Value = <agentID>.
Click Create security group to confirm creation. On successful creation of the security group, you will be redirected to the new security group's dashboard.
Make a note of your Security group ID.

Log groups (optional)

This part may be omitted if you want to use the default that can be created as part of the Task definition.

In the AWS console, type CloudWatch in the search bar and navigate to AWS CloudWatch.
In the left-hand menu, choose Logs → Log groups.
Click Create log group.
Complete the following fields:
- Log group name: Add the following: /ecs/matillion-agent/<agentID>, where <agentID> is the AGENT_ID found on the Agent details screen.
- Retention setting: Select a retention setting from the dropdown. 1 month (30 days) is recommended.
- KMS key ARN - optional: Not required.
- Tags: You may want to add a tag where Key = matillion-agent-id and Value = <agentID>.
Click Create to confirm creation of the log group. On successful creation of the log group, you will be redirected to Log groups.

Cluster

In the AWS console, type Elastic Container Service in the search bar and choose that service.
If this is your first time using this service, click Get started or Clusters.
Click Create Cluster.
Complete the following fields:
- Cluster name: Add the following: matillion-agent-<agentID>, where <agentID> is the AGENT_ID found on the Agent details screen.
- Monitoring: Optional, but recommended that you choose Container Insights with enhanced observability.
- Tags: Optional, but you may want to add a tag where Key = matillion-agent-id and Value = <agentID>, where <agentID> is the AGENT_ID found on the Agent details screen.
Leave all other settings on this screen as default values.
Click Create to confirm creation of the cluster. On successful creation of the cluster, you will be redirected to the Clusters list. Your cluster should now be listed, and show No tasks running. The next step is to add task definitions.

Task definitions

In the AWS console, type Elastic Container Service in the search bar and choose that service.
In the left-hand menu, click Task definitions.
Click Create new task definition and select Create new task definition (not Create new task definition with JSON) from the drop-down.
For Task definition family, enter matillion-agent-<agentID>, where <agentID> is the AGENT_ID found on the Agent details screen. This must be unique—two task definitions can't share a name.
Under Infrastructure requirements, configure the following:
- Launch type: Choose AWS Fargate.
- CPU: Choose 1 vCPU.
- Memory: Choose 4 GB.
- Task role: Select an IAM role that has similar permissions to your role. If you're using the same IAM role, you might need to add ecs-tasks.amazonaws.com to the Service section in the Trust Relationship section of your IAM role to allow the role to work with the ECS service. An example trusted entity is as follows:
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ecs-tasks.amazonaws.com",
                    "ec2.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
```
  To learn more, read AWS IAM roles for Streaming agents.
- Task execution role: Choose escTaskExecutionRole.
Under the Container-1 heading, complete the following:
- Name: Enter matillion-agent.
- Image URI: Copy and paste your Agent Image URI from Agent details.
- Port mappings: Remove any existing entries.

Under Environment variables - optional, click Add environment variable and add the following key-value pairs, using the values from the Agent details page:

Key	Value type	Value
ACCOUNT_ID	Value	Copy and paste the ACCOUNT_ID value from the Agent Details page.
AGENT_ID	Value	Copy and paste the AGENT_ID value from the Agent Details page.
MATILLION_REGION	Value	Copy and paste the MATILLION_REGION value from the Agent Details page.
OAUTH_CLIENT_ID	Value	Copy and paste the client_id value from the Agent Details page. Click Reveal to make the value visible before copying.
OAUTH_CLIENT_SECRET	Value	Copy and paste the client_secret value from the Agent Details page. Click Reveal to make the value visible before copying.

Under Logging - optional, the awslogs-group value should have been automatically populated. If you don't wish to use another log group, leave the aws-create-group entry and a log group will be created with the awslogs-group value.
If the naming of components has been consistent so far, the awslogs-group value should equal the log group that was previously created (note the use of the / preceding the ID in the original Log group name). Otherwise, select the log group created earlier, and delete the aws-create-group entry by clicking Remove.
Under Tags - optional, you may want to add a tag where Key = matillion-agent-id and Value = <agentID>, where <agentID> is the AGENT_ID found on the Agent details screen.
Click Create to confirm creation of the task definition.

Service

On the **Task definitions screen, choose your newly-created task definition.
Click Deploy then Create service from the drop-down.
In the existing cluster drop-down, choose the cluster you created earlier.
Complete the following fields:
- Service name: Add the following: matillion-agent-<agentID>, where <agentID> is the AGENT_ID found on the Agent details screen, or another suitable descriptor.
- Desired tasks: Set this to 1.
Under Networking, complete the following:
- Security group name: Use the drop-down to select the security group you created earlier.
- Public IP: Ensure that this is toggled to Turned on, unless you are using Network Address Translation.
Under Tags - optional, you may want to add a tag where Key = matillion-agent-id and Value = <agentID>, where <agentID> is the AGENT_ID found on the Agent details screen.
Click Create.

The agent will then start. You can return to the agent dashboard and view the agents list. Your new agent should be on the list and have a Running status.