Getting started

Learn how to get started with CDC pipelines using Data Loader.

Prerequisites

To get started with Data Loader, you will need:

To register for a Hub account.
A source application or database where your data resides.
A destination where you want to load the data to.
Credentials to allow Data Loader to connect with the CDC pipeline source and destination systems.

Streaming agent setup

Every CDC pipeline requires a Data Loader Streaming agent to orchestrate the data loading tasks.

Agents must be installed within the cloud service provider of your choice before a CDC pipeline can be created.
The Streaming agent requires access to the source database, the target cloud data platform, and a secrets management application. You can use any of AWS Secrets Manager, Azure Key Vault, or Google Secret Manager. A platform key is generated (if not already done for this account) and stored in your chosen secrets management application.
Cloud resources are created for use by the agent, such as storage and logs. Read specific installation guides for more information:
Amazon S3
Azure Blob
Google Cloud Storage
The agent must be deployed to the appropriate environment with the correct environment variables.
Agent status must be Connected for a CDC pipeline to succeed.
Once the Streaming agent is connected, you can create CDC pipelines.
Read agent installation documentation for more information and consult your cloud administrator for help and permissions where required.

Pipelines

CDC pipeline sources require significant configuration to enable CDC. Consult the Sources section of documentation to learn how you can configure your source for CDC.

A source application or database where your data resides.
A database or data warehouse destination to which the data must be replicated.
Access for Data Loader to connect to the source and the destination systems.

Destinations

A cloud storage area is used to store your data. Matillion currently supports the following CDC destinations:

You'll need access for Data Loader to connect to the your storage resources.