Skip to content

Getting started

Learn how to get started with CDC pipelines using Data Loader.

Prerequisites

To get started with Data Loader, you will need:

  • To register for a Hub account.
  • A source application or database where your data resides.
  • A destination where you want to load the data to.
  • Credentials to allow Data Loader to connect with the CDC pipeline source and destination systems.

CDC agent setup

Every CDC pipeline requires a Data Loader CDC agent to orchestrate the data loading tasks.

  • Agents must be installed within the cloud service provider of your choice before a CDC pipeline can be created.
  • The CDC agent requires access to the source database, the target cloud data platform, and a secrets management application. You can use any of AWS Secrets Manager, Azure Key Vault, or Google Secret Manager. A platform key is generated (if not already done for this account) and stored in your chosen secrets management application.
  • Cloud resources are created for use by the agent, such as storage and logs. Read specific installation guides for more information:
  • Amazon S3
  • Azure Blob
  • Google Cloud Storage
  • The agent must be deployed to the appropriate environment with the correct environment variables.
  • Agent status must be Connected for a CDC pipeline to succeed.
  • Once the CDC agent is connected, you can create CDC pipelines.
  • Read agent installation documentation for more information and consult your cloud administrator for help and permissions where required.

Pipelines

CDC pipeline sources require significant configuration to enable CDC. Consult the Sources section of documentation to learn how you can configure your source for CDC.

  • A source application or database where your data resides.
  • A database or data warehouse destination to which the data must be replicated.
  • Access for Data Loader to connect to the source and the destination systems.

Destinations

A cloud storage area is used to store your data. Matillion currently supports the following CDC destinations:

You'll need access for Data Loader to connect to the your storage resources.