Getting started🔗
Learn how to get started with CDC pipelines using Data Loader.
Prerequisites🔗
To get started with Data Loader, you will need:
- To register for a Data Productivity Cloud account.
- A source application or database where your data resides.
- A destination where you want to load the data to.
- Credentials to allow Data Loader to connect with the CDC pipeline source and destination systems.
Streaming agent setup🔗
Every CDC pipeline requires a Data Loader Streaming agent to orchestrate the data loading tasks.
- Agents must be installed within the cloud service provider of your choice before a CDC pipeline can be created.
- The Streaming agent requires access to the source database, the target cloud data platform, and a secrets management application. You can use any of AWS Secrets Manager, Azure Key Vault, or Google Secret Manager. A platform key is generated (if not already done for this account) and stored in your chosen secrets management application.
- Cloud resources are created for use by the agent, such as storage and logs. Read specific installation guides for more information:
- Amazon S3
- Azure Blob
- Google Cloud Storage
- The agent must be deployed to the appropriate environment with the correct environment variables.
- Agent status must be Connected for a CDC pipeline to succeed.
- Once the Streaming agent is connected, you can create CDC pipelines.
- Read agent installation documentation for more information and consult your cloud administrator for help and permissions where required.
Pipelines🔗
CDC pipeline sources require significant configuration to enable CDC. Consult the Sources section of documentation to learn how you can configure your source for CDC.
- A source application or database where your data resides.
- A database or data warehouse destination to which the data must be replicated.
- Access for Data Loader to connect to the source and the destination systems.
Destinations🔗
A cloud storage area is used to store your data. Matillion currently supports the following CDC destinations:
You'll need access for Data Loader to connect to the your storage resources.