Skip to content

AWS Fargate manual agent setup

This page is a guide to manually installing an agent into AWS Fargate.

Note

This documentation refers to the New ECS Experience for Elastic Container Service.

Each time you click Create in AWS, a colored banner will appear at the top of the AWS UI.

  • Blue indicates the creation of a resource is in-progress.
  • Green indicates the creation of a resource has completed.
  • Red indicates the creation of a resource has failed.

Prerequisites

  • First you must create an agent from the Hub, as described in Create an agent. This will provide you with the parameters and values needed to manually complete the agent setup in AWS Fargate.
  • Before deploying an agent in AWS, confirm the existence of a service-linked role for Amazon ECS in your AWS account. Typically, Amazon ECS generates this role automatically. If not, follow the manual creation documentation for setup instructions.

Security groups

  1. Log in to the AWS Console.
  2. Once logged in, type security groups in the search bar.
  3. In the Features list, select Security Groups|EC2 feature.
  4. Choose Create security group.

  5. Complete the following fields:

    • Security group name: Add the following: matillion-agent-<agentID> Your agent ID is available at Agent environment variables in Hub → Manage Agents.
    • Description: Add the following description: agent security group for agent ID <agentID>. Alternatively, add a description of your choice.
    • VPC: Select your VPC.
    • Inbound rules: Add any inbound rules. None are required by default.
    • Outbound rules: Add 0.0.0.0/0.
    • Tags: You may want to add a tag where Key = matillion-agent-id and Value = <agentID>.|

  6. Click Create security group to confirm creation. On successful creation of the security group, you will be redirected to the new security group's dashboard. 1.1.

  7. Make a note of your Security group ID.

Log groups (optional)

This part may be omitted if you want to use the default that can be created as part of the Task definition.

  1. In the AWS console, type CloudWatch in the search bar and navigate to AWS CloudWatch.
  2. In the left-hand menu, choose Logs → Log groups.
  3. Click Create log group.

  4. Complete the following fields:

    • Log group name: Add the following: /ecs/matillion-agent/<agentID>.
    • Retention setting: Select a retention setting from the dropdown. 1 month (30 days) is recommended.
    • KMS key ARN - optional: Not required.
    • Tags: You may want to add a tag where Key = matillion-agent-id and Value = <agentID>.

  5. Click Create to confirm creation of the log group. On successful creation of the log group, you will be redirected to Log groups.

Cluster

  1. In the AWS console, type Elastic Container Service in the search bar and choose that service.
  2. Click Get started or Clusters if this is your first time using this service.
  3. Click Create Cluster.

  4. Complete the following fields:

    • Cluster name: Add the following: matillion-agent-<agentID>
    • VPC: Select your VPC.
    • Subnets: Optional. Choose subnets where your tasks will run.
    • Use Container Insights: Optional, but recommended. Toggle this to On.
    • Tags: You may want to add a tag where Key = matillion-agent-id and Value = <agentID>.

  5. Click Create to confirm creation of the cluster. On successful creation of the cluster, you will be redirected to All Clusters.

Task definitions

  1. In the AWS console, type Elastic Container Service in the search bar and choose that service.
  2. In the left-hand menu, click Task definitions.
  3. Click Create new task definition.

  4. Complete the following fields:

    • Task definition family: Add the following: matillion-agent-<agentID>. This must be unique—two task definitions can't share a name.
    • Name: matillion-agent.
    • Image URI: Copy and paste your Agent Image URI from Agent details.
    • Port mappings: Remove any existing entries.

  5. In Environment Variables under Add individually, add the following environment variables using the values in your Agent details page in Hub:

    Key Value type Value
    ACCOUNT_ID Value Your ACCOUNT_ID value.
    AGENT_ID Value Your AGENT_ID value.
    MATILLION_REGION Value Your MATILLION_REGION value.
    OAUTH_CLIENT_ID ValueFrom The Amazon resource number (ARN) of the AWS Secrets Manager secret appended with the key name that contains the agent client_id in this format: {your-arn}:{key}::. For example: arn:aws:secretsmanager:eu-west-1:112233445566:secret:my_secret_name-9wxygh:client_id:: To learn more about this process, read Add agent credentials to AWS Secrets Manager.
    OAUTH_CLIENT_SECRET ValueFrom The Amazon resource number (ARN) of the AWS Secrets Manager secret appended with the key name that contains the agent client_secret in this format: {your-arn}:{key}::. For example: arn:aws:secretsmanager:eu-west-1:112233445566:secret:my_secret_name-9wxygh:client_secret::
    EXTENSION_LIBRARY_LOCATION* (optional) Value (Optional) The location for your additional Python libraries, specified without the protocol. For example: agent-python-libs or mybucket/agent-python-libs. Currently, Amazon S3 and Azure Blob Storage are supported.
    EXTENSION_LIBRARY_PROTOCOL* (optional) Value (Optional) Specified in conjunction with EXTENSION_LIBRARY_LOCATION, for example S3.

    For a full description of optional environment variables, read Optional agent parameters.

Click Next.

Configure environment, storage, monitoring, and tags

  1. Complete the following configuration:

    • App environment: Choose AWS Fargate (serverless).
    • CPU: Choose 1 vCPU.
    • Memory: Choose 4 GB.
    • Task role: Use an IAM role that has similar permissions to your role. If you're using the same IAM role, you might need to add ecs-tasks.amazonaws.com to the Service section in the Trust Relationship section of your IAM role to allow the role to work with the ECS service. An example trusted entity can be found below. To learn more, read AWS IAM Roles.
    • Task execution role: Choose escTaskExecutionRole unless instructed otherwise.

    Example trusted entity:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ecs-tasks.amazonaws.com",
                    "ec2.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

  2. Scroll down to Monitoring and logging - optional.

  3. The awslogs-group value should have automatically suggested a log group name. If you don't wish to use another log group, leave the aws-create-group entry and a log group will be created with the awslogs-group value.
  4. If the naming of components has been consistent so far, the awslogs-group value should equal the log group that was previously created (note the use of the / preceding the ID in the original Log group name). Otherwise, select the log group created earlier. Delete the aws-create-group entry by clicking Remove.
  5. Once again, add a tag where Key = matillion-agent-id and Value = <agentID>.
  6. Click Next. Review your settings and if you're happy, click Create. You will receive a blue banner at the top of the page explaining that AWS is currently creating the task definition. This banner will turn green upon completion.

Service

  1. In your newly created task definition, click the Deploy drop-down button and then click Create service.
  2. In the Choose a cluster field, choose the cluster you created earlier.

  3. In Deployment configuration, complete the following fields:

    • Service name: Add the following: matillion-agent-<agentID> or another suitable descriptor.
    • Desired tasks: Set this to 0. This ensures the ECS service will not be linked to the same version of the container image whenever an ECS task starts, and is therefore allowed to use whatever version the latest tagged image is set to.

  4. In Networking, under Security group name, locate the security group you created earlier and select it. Remove any other security groups if they are not required.

  5. Ensure that Public IP is toggled to Turned on, unless you are using Network Address Translation.
  6. As in previous sections, you may want to add a tag where Key = matillion-agent-id and Value = <agentID>.
  7. Click Create.

Once the deployment has finished, you should see your ECS service running inside your ECS Cluster with 0 (zero) tasks. This zero-task deployment is deliberate, to prevent fixing the service to a specific container image version, which would prevent the automatic updating functionality from working. The next step is to increase the number of tasks to your requirement.

  1. In the AWS Console, locate the service and click Update service.

  2. On the Deployment configuration screen, set the Desired tasks count to your normal operational requirements. A minimum of two tasks are required. More tasks reduce the execution time of multi-threaded pipelines. If you use only one task, you run the risk of service outages during agent upgrades.

    Note

    Ensure that Force new deployment is not selected on this screen.

  3. Click Update.

The agent will then start. You can return to the agent dashboard and view the agents list. Your new agent will be on the list and will have a Running status.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement