Streaming agent installation
The Matillion Streaming agent is provided as a container image. This image is required to be deployed and configured within your organization's private cloud and/or on-premises technology stack, and will require access to the source database, the target cloud data lake, and secrets management application of either AWS Secrets Manager, Azure Key Vault, or Google Secret Manager.
Agent installation process
We recommend using the following quick guides to add a Streaming agent in Data Loader, and deploy that agent in your chosen cloud platform:
- Quick guide for deploying a Streaming agent in AWS. There are two types of templates AWS offers, Basic and Advanced. Basic templates include less resources and are for users who have pre-existing AWS resources that you want to use in your CDC pipelines. Advanced templates give you a more robust setup and are the recommended method for installing the Streaming agent.
- Quick guide for deploying a Streaming agent in Azure. Use the Azure ARM template to deploy your Streaming agent in Azure.
- Quick guide for deploying a Streaming agent in GCP. Use the Terraform Compute Engine Advanced template to deploy your Streaming agent in GCP.
Note
Your source database will also require configuration to work with CDC. This is independent from the agent installation process. More information can be found in the CDC Sources category. Your choice of agent installation does not limit the available data sources.
Manual installations
Manual installations are complex and are documented for advanced users. We have supporting articles, such as the Cloud Resources sections, and individual template articles for each of the cloud providers. However, we highly recommend following the quick guides for Streaming agent creation and deployment. The quick guides document how to create and configure almost every cloud resource on your behalf, and is by far the best way to install the Matillion Streaming agent. For more information, read:
The Agent
- The Matillion Streaming agent can run only one CDC pipeline. Each CDC pipeline requires a new agent installation.
- An agent entry must be added in Data Loader, first. Then the agent itself must be installed on your cloud platform.
- CDC supports multiple schema and you can select any schemas and tables that you wish to include in the CDC pipeline.
Location
Container images can't be accessed directly via a browser. To pull a container image from the public repository specified, please reference the AWS documentation. The agent container image is available at the following public repository:
public.ecr.aws/matillion/cdc-agent:2
Access
The agent requires direct access to the Matillion CDC servers for control and management. To enable this route, the agent will require outbound access to the following public IP addresses:
EU region:
13.248.238.58
76.223.106.122
US region:
75.2.56.46
99.83.216.99
Technical Requirements
Matillion recommends providing the Streaming agent with 4 CPU Cores and 8 GB RAM. The agent is intended for use with x86 Linux systems.