Connectors overview

The Data Productivity Cloud supports several types of connector, with each performing different processes for different purposes. A connector's key function is to extract data from a data source and provide it to other Matillion processes.

When creating data pipelines, you can connect to any data source that supports REST API using one of Matillion's pre-built connectors, or by creating your own custom connector. Visit Connect to any source to review all the connectors Matillion offers.

Connector components

Connector components query data from a specific source and load that data into a table in a data warehouse. They are available in Designer's orchestration pipelines and so can be used as part of larger data workflows including complex transformations.

Components overview introduces Designer components. Connectors are listed in the Orchestration components section.
Salesforce Query is an example of a connector in Designer.

Custom connectors

You can create your own connectors based on any REST API that returns JSON data by using the Custom Connector interface. These custom connectors can then be brought into Designer or Data Loader to build pipelines. You will require access and knowledge of the target API to create these.

Custom Connector overview provides an overview of the process.

Flex connectors

Flex connectors are pre-configured custom connectors available directly from Data Loader's Batch data sources list. Once used, these will appear in the Flex connectors tab of the Custom Connector interface and can be edited for further customization, such as adding additional endpoints, if required.

Pendo is an example of a Flex connector.
Flex connector example provides a full walkthrough to create a pipeline using the Snapchat Flex connector.

Batch connectors

These connectors are used in Data Loader and are pre-configured to batch load data from a service. These connectors are typically very quick and easy to use, only requiring database and authentication details. Data Loader will form a pipeline around your Batch connector to schedule data loads.

Batch loads introduction introduces Batch pipelines in Data Loader. Each connector is listed in the Sources section.
Google Analytics is an example of a Batch data source.

CDC connectors

These connectors facilitate near real-time data replication from a source database. CDC connectors require a specialized agent to be installed within your cloud infrastructure along with other resources such as a secrets manager, storage location, and replication database.

Matillion Change Data Capture (CDC) introduces Matillion CDC. Each connector is listed in the Sources section.
Streaming agent installation explains Streaming agents.
Db2 for IBM i connector is an example of a CDC data source.