IAM roles & permissions (AWS)

You manage access in AWS by creating policies and attaching them to IAM identities (users, groups of users, or roles) or AWS resources. A policy is an object in AWS that, when associated with an identity or resource, defines their permissions. AWS evaluates these policies when an IAM principal (user or role) makes a request. Permissions in the policies determine whether the request is allowed or denied. Most policies are stored in AWS as JSON documents.

AWS credentials are required for Matillion ETL instance to access various services such as discovering S3 buckets and using KMS. Appropriate permissions must be given via your AWS admin console and details of your GCP account must be entered into the Matillion ETL instance via Project, then Manage Credentials where credentials for other platforms may also be entered.

For more information on Manage Credentials, please visit the document here.

---

Adding an AWS access policy

There are two ways you can add a policy, and give access to Matillion ETL:

1. With instance credentials: This is done by specifying an IAM Role for the EC2 instance at launch time. Once in place, you can add and remove permissions from the chosen IAM role and the changes take effect immediately. This gives you fine grained control over exactly which AWS services Matillion ETL is allowed to use. This method is the easiest to manage if you attach the role to the instance while it is being launched. However it is possible to add an IAM to a running instance, using the AWS command line.

2. With existing user defined credentials: These can be defined via the Matillion ETL user interface at any time. The credentials can be created via an IAM User, and Matillion will be able to access what the IAM user has access to. You can simply enter an Access Key ID and a Secret Access Key.

Depending on your company's security policy, you may choose to use either:

• Coarse-grained access control: easier to administer, but typically grants many more privileges than are necessary.
• Fine-grained access control: requires you to specify a more complex policy but follows the principle of least privilege. To see the template for this policy type, refer to AWS Policy.

---

AWS coarse-grained access control

With this approach, you simply attach the following Managed Policies to the roles you are using.

Managed Policy	Managed Policy
AmazonRedshiftReadOnlyAccess	This policy allows Matillion to read metadata about the Redshift clusters, such as endpoint and port number. This makes it easier to set up new projects and environments. If you don't allow this role, you will need to type in all the information manually. Note: It does not allow Matillion to read data from Redshift (that privilege is granted by you providing a username and password, and by you permitting network connectivity between Matillion and Redshift).
AmazonS3FullAccess	This policy is used to read data to and from S3 buckets. It can be used in almost every load component, since they almost all either read directly from S3 or else use an S3 Staging area.
AmazonSNSFullAccess	This policy is used when creating SNS topics and publishing messages to them. This is the purpose of the SNS Message Component.
AmazonSQSFullAccess	This policy enables Matillion to read from an SQS queue, or to write an SQS message using the SQS Message Component.
CloudWatchFullAccess	This policy is required to publish metrics to CloudWatch Publish component.
AmazonRDSReadOnlyAccess	This policy is used by the RDS Bulk Output components to discover information about the available RDS servers in the region. Without this policy, Matillion may still be able to connect to RDS provided the network path is set up and a username and password have been provided.
AmazonEC2FullAccess	This policy is used when performing automated backups.
SecretsManagerReadWrite	Provides access to Secrets Manager operations. The policy doesn't allow the identity to configure rotation because rotation requires IAM permissions to create roles. To learn more, read AWS managed policies available for use with AWS Secrets Manager.

There is no single managed policy that grants KMS access through the Project → Manage Password dialog. Instead, refer to the KMS table in the fine-grained access control section below for appropriate permissions.

---

AWS fine-grained access control

This section lists all the IAM privileges which Matillion ETL can require during normal operation. Privileges are shown using the AWS policy action metamodel version "2012-10-17". It doesn't take into account any privileges required by custom AWS requests made in Bash or Python scripts.

There are no mandatory IAM Actions which Matillion requires. However the recommended ones listed in the next section are the minimum requirements for productivity and ease of use.

Recommended Actions	Description
redshift:DescribeClusters	During first login, allows Matillion to find Redshift clusters. Also used when creating a new environment. Note: this privilege is not required to use a particular Redshift cluster: it's just useful to make setup faster and less error prone.
s3:ListAllMyBuckets	Enables every S3 Staging Area drop-down list to work. Used in every Orchestration data load component. Note: Matillion ETL can run without this if you wish to keep your buckets private. However, exploration and discovery of buckets from within the client will no longer work.
s3:ListBucket	This permission allows the user to access files in the folder (and any sub-folders). Can be used in every Orchestration data load component.
s3:GetObject	Enables Matillion to read S3 files. Used in every Orchestration data load component.
s3:PutObject	Used by every Orchestration data load component to create temporary files in a nominated staging bucket. The files are used temporarily to take advantage of Redshift's bulk loader and transfer the data as quickly as possible into Redshift. This Action could be restricted to specific staging buckets.
s3:DeleteObject	Used for the same purpose as s3:PutObject, to remove temporary S3 files created during data loads.
s3:GetBucketLocation	Uses the subresource location to return a bucket's AWS region.
secretsmanager:ListSecrets	Allows permission to populate the secret name drop-down menu when configuring an external password located in a secret manager.
secretsmanager:GetSecretValue	Allows permission to resolve the value associated with a secret, for example when a component is configured to use a secret from the secret manager as a password.

For more information about permission policy examples for AWS Secrets Manager, see here.

Automated backups can be enabled from the Admin, then the Manage Backups menu to the top right of the page. Backups require the EC2 actions detailed below.

EC2 Actions	Description
ec2:CreateSnapshot	Used during creation of the EC2 snapshot.
ec2:CreateTags	Used to tag the generated snapshot with a name.
ec2:DescribeInstances	Required to retrieve instance metadata.
ec2:DescribeVolumes	Required to retrieve instance metadata.

This section lists the actions required to use the KMS option in the Manage Passwords dialog. In addition to the below, the chosen KMS key must be:

• In the same region as Matillion
• Enabled

The following tables outline actions for KMS, RDS, SQS, SNS and Cloudwatch, and are required by Matillion to perform certain functions.

KMS Action	Description
kms:ListAliases	Enables Matillion to populate the "Master Key" dropdown by listing all the KMS aliases which are associated with a Key.
kms:Encrypt	Enables Matillion to store an encrypted password.
kms:Decrypt	Enables Matillion to retrieve and use an encrypted password.
RDS Actions	Description
rds:DescribeDBInstances	Enables Matillion to list the endpoints of the chosen RDS database type. Note this privilege isn't required to query data from any RDS instance.
SQS Actions	Description
sqs:ListQueues	Enables the operator to select an SQS queue in the main SQS Configuration window.
sqs:GetQueueUrl	Required to monitor queues.
sqs:ReceiveMessage	Enables Matillion to receive messages from SQS (e.g. using the SQS Message orchestration component).
sqs:DeleteMessage	Enables Matillion to receive messages from SQS and remove them from the queue.
sqs:SendMessage	Enables Matillion to send messages to SQS.
SNS Actions	Description
sns:Publish	Required to publish messages with SNS (e.g. Using the SNS Message component).
sns:ListTopics	Enables the operator to select from a list of existing SNS topics (such as when using the SNS Message component.
sns:CreateTopic	If the user manually types in a topic name which doesn't already exist in that region, Matillion tries to create the new topic, which requires this privilege.
CloudWatch Actions	Description
cloudwatch:ListMetrics	Required to enable the CloudWatch Publish component to list existing metrics.
cloudwatch:PutMetricData	Required to publish metrics to Cloudwatch using Matillion's CloudWatch Publish component. This Action also includes the authorization to create new Custom Namespaces and new Metrics.

To see the template for this policy type, refer to AWS policy.

---

Credentials for CDC

Below is a full list of the permissions a Matillion ETL instance requires to use CDC as described in the above sections. We provide a brief outline of what each permission allows Matillion ETL to do and which section of the CDC setup it pertains to.

Lambda Action	Description	Client Section
lambda:CreateFunction	Create Lambda Functions.	Manage
lambda:DeleteFunction	Delete Lambda Functions.	Manage
lambda:GetFunction	Get information about the Lambda Function.	Manage
lambda:TagResource	Add tags to a Lambda Function.	Manage
lambda:UpdateFunctionCode	Update a Lambda Function's code.	Manage
lambda:UpdateFunctionConfiguration	Modify the version-specifc settings of a Lambda function.	Manage
lambda:AddPermission	Get information about the Lambda Function.	Manage
lambda:RemovePermission	Revokes Lambda Function permission from an AWS Service.	Manage
lambda:GetPolicy	Get the resource-based IAM policy for a Lambda Function.	CDC Monitor
SQS Action	Description
sqs:DeleteMessage	Delete a message from a specified queue.	Manage
sqs:ReceiveMessage	Retrieve up to 10 messages from a specified queue.	Manage
sqs:DeleteQueue	Delete a queue specified by the QueueUrl, regardless of the queue's contents.	Manage
sqs:ListQueues (Optional)	Allows discovery of available queues.	Manage
sqs:ChangeMessageVisibility	Change the visibility timeout of a message in a queue.	Manage
sqs:GetQueueUrl	Get the URL of an Amazon SQS queue.	Manage
DMS Action	Description
dms:CreateEndpoint	Create a DMS endpoint.	CDC Task Config
dms:DeleteEndpoint	Delete a DMS endpoint.	CDC Task Config
dms:StartReplicationTask	Start a replication task.	CDC Monitor
dms:CreateReplicationTask	Create a replication task.	CDC Task Config
dms:ModifyEndpoint	Modify a DMS endpoint	CDC Monitor
dms:DescribeConnections	Get the status of the connections between a replication instance and an endpoint.	CDC Task Config
dms:DescribeReplicationInstances	Get information on replication instances on your account (region-specific).	CDC Task Config
dms:DescribeReplicationTasks	Get information on replication tasks on your account (region-specific).	CDC Task Config
dms:StopReplicationTask	Stop a replication task.	CDC Monitor
dms:DeleteReplicationTask	Delete a replication task.	CDC Monitor
dms:DescribeEndpoints	Get information on DMS endpoints on your account (region specific).	CDC Task Config
dms:TestConnection	Tests the connection between a replication instance and a DMS endpoint.	CDC Task Config
S3 Permissions	Description
s3:GetBucketNotification	Get the notification configuration of a bucket.	CDC Task Config
s3:ListBucket	View folders inside S3 buckets
s3:PutBucketNotification	Set notification configuration of an S3 bucket.	CDC Task Config
s3:ListAllMyBuckets (Optional)	Get a list of all owned S3 buckets.	CDC Task Config
IAM Permissions	Description
iam:ListRoles (Optional)	Get a list of IAM Roles.	Manage
iam:PassRole	Allows passing a role to an AWS service.	Manage
EC2 Permissions	Description
ec2:DescribeRegions (Optional)	Get the regions currently available to the instance.	Manage

---

DynamoDB

The DynamoDB data staging components require some credentials to function correctly.

DynamoDB Actions	Description
dynamodb:ListTables	Return information about the table, including the current status of the table, when it was created, the primary key schema, and any indexes on the table.
dynamodb:DescribeTable	Return an array of table names associated with the current account and endpoint.
dynamodb:Scan	Scan a table or index.

---

AWS policy

The following AWS policy template grants fine-grained access control to your Matillion ETL instance:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "StmtMinRedshift",
            "Effect": "Allow",
            "Action": [
                "redshift:DescribeClusters"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinS3",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinSQS",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:ListQueues",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:GetQueueUrl"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinRDS",
            "Effect": "Allow",
            "Action": [
                "rds:DescribeDBInstances"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinEC2",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSnapshot",
                "ec2:CreateTags",
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinSNS",
            "Effect": "Allow",
            "Action": [
                "sns:ListTopics",
                "sns:CreateTopic",
                "sns:Publish"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinCloudwatch",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "cloudwatch:ListMetrics"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinKMS",
            "Effect": "Allow",
            "Action": [
                "kms:ListAliases",
                "kms:Encrypt",
                "kms:Decrypt"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "StmtMinDDB",
            "Effect": "Allow",
            "Action": [
                "dynamodb:ListTables",
                "dynamodb:DescribeTable",
                "dynamodb:Scan"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:CreateSecret",
                "secretsmanager:ListSecrets"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}