AWS IAM roles

The following only applies to the default, ECS Fargate deployment prescribed by the CloudFormation templates Matillion provides.

To use your agent in AWS, two IAM roles may need to be configured to ensure the component parts of an agent installation have the access they require. The CloudFormation template creates and assigns these roles by default, but additional configuration may be required to access existing resources such as S3 buckets. If you are launching AWS agents by other means (manual, Terraform, or other IaC tool), ensure the roles are created correctly and are applied to the Fargate service.

The roles are:

Task role: The task role defines the permissions that the Matillion Data Productivity Cloud agent application has. For example, if your pipeline needs to access other AWS services such S3 or Redshift, you would define these permissions in the task role.
Task execution role: The task execution role is associated with the ECS container instances and grants permission to the ECS service to make API calls on your behalf. For the Matillion Data Productivity Cloud agent application, the task execution role needs permissions to start and manage tasks, pull images, and inject secrets, and permissions should be limited to only these operations.

Configuration of these roles is done in the AWS console. The following sections describe what's needed to configure the roles.

Task role

This is the role that allows your agent instance to use other AWS services, such as Amazon S3.

The policy shown here corresponds to the trust relationship required to allow the task role to be assumed by the running tasks and containers. This is a required configuration.

You will need the following trusted entity:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ecs-tasks.amazonaws.com",
                    "ec2.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

You will also need a permissions policy that:

Allows secretsmanager:ListSecrets for all secrets.
Allows secretsmanager:GetSecretValue only for specific secret ARNs.

The following permissions policy will allow this:

{
    "Version": "2012-10-17",
    "Statement": [
    {
        "Effect": "Allow",
        "Action": "secretsmanager:ListSecrets",
        "Resource": "*"
    },
    {
        "Effect": "Allow",
        "Action": "secretsmanager:GetSecretValue",
        "Resource": [
            "arn:aws:secretsmanager:<region>:<account-id>:secret:<secret-id-1>",
            "arn:aws:secretsmanager:<region>:<account-id>:secret:<secret-id-2>",
            ...
        ]
    }
    ]
}

The <region> and <account-id> values should be replaced with the appropriate values for your AWS account. The <secret-id> is the name of the secret in Secrets Manager. You can add as many individual secrets as you need.

Note

For secretsmanager:GetSecretValue, we strongly suggest you follow the principle of least privilege and limit the scope of the policy to only the secrets that are required for your agent, using named secrets as shown in the example above. However, secretsmanager:ListSecrets needs "Resource": "*" because it's a list-type action and not tied to individual ARNs.

For S3 Bucket access for staging files, you will also need to specify appropriate S3 permissions like s3:ListBucket, s3:GetObject, and s3:PutObject. These should be scoped to specific staging buckets, again following the principle of least privilege.

The above role should be considered a baseline, and does not include other Data Productivity Cloud pipeline integrations with specific databases or services. Any such integrations will require their own additional permissions in the task role.

Task execution role

The task execution role grants the ECS Fargate tasks permission to make AWS API calls. If you don't have an existing ECS task execution role, you should create one by selecting Create new role when creating a task definition.

For the agent credentials to be accessed when they are stored in AWS Secrets Manager, the ECS task execution role will need to be given permission to AWS Secrets Manager.

Note

The task execution role's purpose here is to allow the container to retrieve Matillion's platform credentials so it can establish the connection to the Matillion agent gateway. For managing the Data Productivity Cloud's secret definitions, cloud credentials, and OAuths, the permissions for those actions must be assigned to the task role.

To configure the task execution role:

Navigate to Roles in the AWS console, and locate your ECS task execution role.
Go to Add permissions and select Create inline policy

Choose the JSON tab, and replace the JSON displayed there with the following, substituting your secret ARNs in the <resource-scope>:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue"
            ],
            "Resource": "<resource-scope>"
        }
    ]
}

If secrets will use customer-managed KMS keys (CMKs), you require the kms:Decrypt permission in addition to the secretsmanager:GetSecretValue permission, so you will need this policy instead:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "kms:Decrypt"
            ],
            "Resource": "<resource-scope>"
        }
    ]
}

Note

The <resource-scope> value can be a specific secret ARN, or the wildcard ARN *. We strongly suggest you follow the principle of least privilege and limit the scope of the policy to only the secrets that are required for your agent:

"Resource": "arn:aws:kms:<region>:<account-id>:key/<key-id>"

or, to specify multiple secrets:

"Resource": [
    "arn:aws:kms:<region>:<account-id>:key/<key-id-1>",
    "arn:aws:kms:<region>:<account-id>:key/<key-id-2>",
    ...
]

Where the <region> and <account-id> values should be replaced with the appropriate values for your AWS account. The <key-id> is the KMS key ID or ARN. You can add as many individual secrets as you need.

Using a <resource-scope> of * would give access to all secrets in your AWS account, which is not recommended for production use.

Click Review policy to check the results.
Give your inline policy a name.
Click on Create policy to complete the creation and have it added to your role.