Skip to content

IAM roles & permissions (AWS)

Overview

You manage access in AWS by creating policies and attaching them to IAM identities (users, groups of users, or roles) or AWS resources. A policy is an object in AWS that, when associated with an identity or resource, defines their permissions. AWS evaluates these policies when an IAM principal (user or role) makes a request. Permissions in the policies determine whether the request is allowed or denied. Most policies are stored in AWS as JSON documents.

AWS credentials are required for Matillion ETL instance to access various services such as discovering S3 buckets and using KMS. Appropriate permissions must be given via your AWS admin console and details of your GCP account must be entered into the Matillion ETL instance via Project, then Manage Credentials where credentials for other platforms may also be entered.

:::info{title='Note'} For more information on Manage Credentials, please visit the document here. :::

Adding an AWS access policy

There are two ways you can add a policy, and give access to Matillion ETL:

  1. With instance credentials: This is done by specifying an IAM Role for the EC2 instance at launch time. Once in place, you can add and remove permissions from the chosen IAM role and the changes take effect immediately. This gives you fine grained control over exactly which AWS services Matillion ETL is allowed to use. This method is the easiest to manage if you attach the role to the instance while it is being launched. However it is possible to add an IAM to a running instance, using the AWS command line.

  2. With existing user defined credentials: These can be defined via the Matillion ETL user interface at any time. The credentials can be created via an IAM User, and Matillion will be able to access what the IAM user has access to. You can simply enter an Access Key ID and a Secret Access Key.

Depending on your company's security policy, you may choose to use either:

  • Coarse-grained access control: easier to administer, but typically grants many more privileges than are necessary.
  • Fine-grained access control: requires you to specify a more complex policy but follows the principle of least privilege. To see the template for this policy type, refer to AWS Policy.

AWS coarse-grained access control

With this approach, you simply attach the following Managed Policies to the roles you are using.

Managed Policy Managed Policy
AmazonRedshiftReadOnlyAccess This policy allows Matillion to read metadata about the Redshift clusters, such as endpoint and port number. This makes it easier to set up new projects and environments. If you don't allow this role, you will need to type in all the information manually. Note: It does not allow Matillion to read data from Redshift (that privilege is granted by you providing a username and password, and by you permitting network connectivity between Matillion and Redshift).
AmazonS3FullAccess This policy is used to read data to and from S3 buckets. It can be used in almost every load component, since they almost all either read directly from S3 or else use an S3 Staging area.
AmazonSNSFullAccess This policy is used when creating SNS topics and publishing messages to them. This is the purpose of the SNS Message Component.
AmazonSQSFullAccess This policy enables Matillion to read from an SQS queue, or to write an SQS message using the SQS Message Component.
CloudWatchFullAccess This policy is required to publish metrics to CloudWatch Publish component.
AmazonRDSReadOnlyAccess This policy is used by the RDS Bulk Output components to discover information about the available RDS servers in the region. Without this policy, Matillion may still be able to connect to RDS provided the network path is set up and a username and password have been provided.
AmazonEC2FullAccess This policy is used when performing automated backups.
SecretsManagerReadWrite Provides access to Secrets Manager operations. The policy doesn't allow the identity to configure rotation because rotation requires IAM permissions to create roles. To learn more, read AWS managed policies available for use with AWS Secrets Manager.

:::info{title='Note'} There are no single Managed Policy which can be used to authorize KMS access in the Manage Passwords dialog through Project, then Manage Password. Please use the KMS table in the fine-grained access control section below. :::

AWS fine-grained access control

This section lists all the IAM privileges which Matillion ETL can require during normal operation. Privileges are shown using the AWS policy action metamodel version "2012-10-17". It doesn't take into account any privileges required by custom AWS requests made in Bash or Python scripts.

There are no mandatory IAM Actions which Matillion requires. However the recommended ones listed in the next section are the minimum requirements for productivity and ease of use.

Recommended Actions Description
redshift:DescribeClusters During first login, allows Matillion to find Redshift clusters. Also used when creating a new environment. Note: this privilege is not required to use a particular Redshift cluster: it's just useful to make setup faster and less error prone.
s3:ListAllMyBuckets Enables every S3 Staging Area drop-down list to work. Used in every Orchestration data load component. Note: Matillion ETL can run without this if you wish to keep your buckets private. However, exploration and discovery of buckets from within the client will no longer work.
s3:ListBucket This permission allows the user to access files in the folder (and any sub-folders). Can be used in every Orchestration data load component.
s3:GetObject Enables Matillion to read S3 files. Used in every Orchestration data load component.
s3:PutObject Used by every Orchestration data load component to create temporary files in a nominated staging bucket. The files are used temporarily to take advantage of Redshift's bulk loader and transfer the data as quickly as possible into Redshift. This Action could be restricted to specific staging buckets.
s3:DeleteObject Used for the same purpose as s3:PutObject, to remove temporary S3 files created during data loads.
s3:GetBucketLocation Uses the subresource location to return a bucket's AWS region.
secretsmanager:ListSecrets Allows permission to populate the secret name drop-down menu when configuring an external password located in a secret manager.
secretsmanager:GetSecretValue Allows permission to resolve the value associated with a secret, for example when a component is configured to use a secret from the secret manager as a password.

:::info{title='Note'} For more information about permission policy examples for AWS Secrets Manager, see here. :::

Automated backups can be enabled from the Admin, then the Manage Backups menu to the top right of the page. Backups require the EC2 actions detailed below.

EC2 Actions Description
ec2:CreateSnapshot Used during creation of the EC2 snapshot.
ec2:CreateTags Used to tag the generated snapshot with a name.
ec2:DescribeInstances Required to retrieve instance metadata.
ec2:DescribeVolumes Required to retrieve instance metadata.

This section lists the actions required to use the KMS option in the Manage Passwords dialog. In addition to the below, the chosen KMS key must be:

  • In the same region as Matillion
  • Enabled

The following tables outline actions for KMS, RDS, SQS, SNS and Cloudwatch, and are required by Matillion to perform certain functions.

KMS Action Description
kms:ListAliases Enables Matillion to populate the "Master Key" dropdown by listing all the KMS aliases which are associated with a Key.
kms:Encrypt Enables Matillion to store an encrypted password.
kms:Decrypt Enables Matillion to retrieve and use an encrypted password.
RDS Actions Description
rds:DescribeDBInstances Enables Matillion to list the endpoints of the chosen RDS database type. Note this privilege isn't required to query data from any RDS instance.
SQS Actions Description
sqs:ListQueues Enables the operator to select an SQS queue in the main SQS Configuration window.
sqs:GetQueueUrl Required to monitor queues.
sqs:ReceiveMessage Enables Matillion to receive messages from SQS (e.g. using the SQS Message orchestration component).
sqs:DeleteMessage Enables Matillion to receive messages from SQS and remove them from the queue.
sqs:SendMessage Enables Matillion to send messages to SQS.
SNS Actions Description
sns:Publish Required to publish messages with SNS (e.g. Using the SNS Message component).
sns:ListTopics Enables the operator to select from a list of existing SNS topics (such as when using the SNS Message component.
sns:CreateTopic If the user manually types in a topic name which doesn't already exist in that region, Matillion tries to create the new topic, which requires this privilege.
CloudWatch Actions Description
cloudwatch:ListMetrics Required to enable the CloudWatch Publish component to list existing metrics.
cloudwatch:PutMetricData Required to publish metrics to Cloudwatch using Matillion's CloudWatch Publish component. This Action also includes the authorization to create new Custom Namespaces and new Metrics.

To see the template for this policy type, refer to AWS policy.

Credentials for CDC

Below is a full list of the permissions a Matillion ETL instance requires to use CDC as described in the above sections. We provide a brief outline of what each permission allows Matillion ETL to do and which section of the CDC setup it pertains to.

Lambda Action Description Client Section
lambda:CreateFunction Create Lambda Functions. Manage
lambda:DeleteFunction Delete Lambda Functions. Manage
lambda:GetFunction Get information about the Lambda Function. Manage
lambda:TagResource Add tags to a Lambda Function. Manage
lambda:UpdateFunctionCode Update a Lambda Function's code. Manage
lambda:UpdateFunctionConfiguration Modify the version-specifc settings of a Lambda function. Manage
lambda:AddPermission Get information about the Lambda Function. Manage
lambda:RemovePermission Revokes Lambda Function permission from an AWS Service. Manage
lambda:GetPolicy Get the resource-based IAM policy for a Lambda Function. CDC Monitor
SQS Action Description
sqs:DeleteMessage Delete a message from a specified queue. Manage
sqs:ReceiveMessage Retrieve up to 10 messages from a specified queue. Manage
sqs:DeleteQueue Delete a queue specified by the QueueUrl, regardless of the queue's contents. Manage
sqs:ListQueues (Optional) Allows discovery of available queues. Manage
sqs:ChangeMessageVisibility Change the visibility timeout of a message in a queue. Manage
sqs:GetQueueUrl Get the URL of an Amazon SQS queue. Manage
DMS Action Description
dms:CreateEndpoint Create a DMS endpoint. CDC Task Config
dms:DeleteEndpoint Delete a DMS endpoint. CDC Task Config
dms:StartReplicationTask Start a replication task. CDC Monitor
dms:CreateReplicationTask Create a replication task. CDC Task Config
dms:ModifyEndpoint Modify a DMS endpoint CDC Monitor
dms:DescribeConnections Get the status of the connections between a replication instance and an endpoint. CDC Task Config
dms:DescribeReplicationInstances Get information on replication instances on your account (region-specific). CDC Task Config
dms:DescribeReplicationTasks Get information on replication tasks on your account (region-specific). CDC Task Config
dms:StopReplicationTask Stop a replication task. CDC Monitor
dms:DeleteReplicationTask Delete a replication task. CDC Monitor
dms:DescribeEndpoints Get information on DMS endpoints on your account (region specific). CDC Task Config
dms:TestConnection Tests the connection between a replication instance and a DMS endpoint. CDC Task Config
S3 Permissions Description
s3:GetBucketNotification Get the notification configuration of a bucket. CDC Task Config
s3:ListBucket View folders inside S3 buckets
s3:PutBucketNotification Set notification configuration of an S3 bucket. CDC Task Config
s3:ListAllMyBuckets (Optional) Get a list of all owned S3 buckets. CDC Task Config
IAM Permissions Description
iam:ListRoles (Optional) Get a list of IAM Roles. Manage
iam:PassRole Allows passing a role to an AWS service. Manage
EC2 Permissions Description
ec2:DescribeRegions (Optional) Get the regions currently available to the instance. Manage

DynamoDB

The DynamoDB data staging components require some credentials to function correctly.

DynamoDB Actions Description
dynamodb:ListTables Return information about the table, including the current status of the table, when it was created, the primary key schema, and any indexes on the table.
dynamodb:DescribeTable Return an array of table names associated with the current account and endpoint.
dynamodb:Scan Scan a table or index.

AWS policy

The following AWS policy template grants fine-grained access control to your Matillion ETL instance:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "StmtMinRedshift",
            "Effect": "Allow",
            "Action": [
                "redshift:DescribeClusters"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinS3",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinSQS",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:ListQueues",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:GetQueueUrl"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinRDS",
            "Effect": "Allow",
            "Action": [
                "rds:DescribeDBInstances"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinEC2",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSnapshot",
                "ec2:CreateTags",
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinSNS",
            "Effect": "Allow",
            "Action": [
                "sns:ListTopics",
                "sns:CreateTopic",
                "sns:Publish"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinCloudwatch",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "cloudwatch:ListMetrics"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinKMS",
            "Effect": "Allow",
            "Action": [
                "kms:ListAliases",
                "kms:Encrypt",
                "kms:Decrypt"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinDDB",
            "Effect": "Allow",
            "Action": [
                "dynamodb:ListTables",
                "dynamodb:DescribeTable",
                "dynamodb:Scan"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:CreateSecret",
                "secretsmanager:ListSecrets"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}