Using the documentation
Data Loader is an application for the bulk import or export of data using two functionalities: Batch pipelines and Change Data Capture (CDC) pipelines.
This article is concerned with Batch pipelines. Batch Load Replication provides a full SaaS incremental data loading experience to extract, transform, and load data at user-specified time intervals from chosen sources, to cloud data warehouse/platform destinations, such as Snowflake, Amazon Redshift, Delta Lake on Databricks, and Google BigQuery.
Batch pipelines
A pipeline moves your data from a source system to a destination database. Pipelines help you to replicate raw data from your source application or database to a destination database or data warehouse. You can customize the data replication process to suit the type of source, and the destination.
A pipeline creation includes the following components:
- Source configuration: A source can be a database, a SaaS-based application (an API endpoint), or a file storage location that has the data that you want to analyze. Matillion CDC integrates with a variety of sources.
- Table selection: Select the table you would like to extract the data from, and load it to the destination.
- Destination configuration: Destination is a data warehouse or database where the data is extracted from a source. For more information, see the Destinations section.
- Settings: Apply these to your connection details and the pipeline you're creating.