Connect to Amazon S3
Matillion CDC can load data from your pipelines into an Amazon S3 bucket for storage. Follow the steps on this page to configure Amazon S3 as a destination.
Amazon S3 prerequisites
To start using Amazon S3 as a destination, some requirements need to be met. These prerequisites ensure that a working connection can be set up to transfer data to your Amazon S3 bucket.
- An Amazon Web Services (AWS) account. Signing up is free - click here to create an account if you don't have one already.
- Permissions to create and manage S3 buckets in AWS. Your AWS user must be able to create a bucket if one doesn't already exist, add/modify bucket policies, and upload files to the bucket.
- The IAM role used by the Agent container has
putObjectpermissions for the S3 bucket and its prefix to be used as the destination by the pipeline. - An active Amazon S3 bucket.
- A unique prefix name for each pipeline.
Connecting to Amazon S3
Select Destination
- After you configure the source during a CDC pipeline creation, you will be directed to choose a destination you would like to load your data into.
- In the Choose destination page, select Amazon S3.
Configure Amazon S3 Connection Settings
Specify the following settings in the Connect to AmazonS3 Destination page:
| Property | Description |
|---|---|
| Bucket | This refers to the name of the Amazon S3 bucket you want to use as a destination. Find your bucket name in the AWS Management Console under Services → S3. |
| Prefix | Prefix is the name of the 'folder' or a location within the S3 bucket that all CDC data for this pipeline should be saved to. You can have multiple agents using the same bucket with different prefixes. |
:::info{title='Note'} A pipeline prefix must be unique. :::
Test Connection
You can test your connection by clicking Test connection. If the test is successful, click Test and Continue.