Skip to content

Connectors overview

The Data Productivity Cloud supports several types of connector, with each performing different processes for different purposes. A connector's key function is to extract data from a data source and provide it to other Matillion processes.

When creating data pipelines, you can connect to any data source that supports REST API using one of Matillion's pre-built connectors, or by creating your own custom connector. Visit Connect to any source to review all the connectors Matillion offers.

Connector components

Connector components query data from a specific source and load that data into a table in a data warehouse. They are available in Designer's orchestration pipelines and so can be used as part of larger data workflows including complex transformations.

  • Components overview introduces Designer components. Connectors are listed in the Orchestration components section.
  • Salesforce Query is an example of a connector in Designer.

Custom connectors

You can create your own connectors based on any REST API that returns JSON data by using the Custom Connector interface. These custom connectors can then be brought into Designer or Data Loader to build pipelines. You will require access and knowledge of the target API to create these.

Flex connectors

Flex connectors are pre-configured connectors available directly in orchestration pipelines in Designer, and from Data Loader's Batch data sources list. Flex connectors can be edited in the Custom Connector interface for further customization, such as adding additional endpoints, if required.

  • Pendo is an example of a Flex connector.
  • Flex connector example provides a full walkthrough to create a pipeline using the Snapchat Flex connector.

Query connectors

Query connectors are used to retrieve data from your chosen source using an SQL-like query. When configuring these connectors, you can choose between Basic mode and Advanced mode. Basic mode will build a query for you based on the options you choose in the component properties, while Advanced mode lets you write the query yourself.

Streaming connectors

Streaming connectors are built for use in streaming pipelines. They allow you to capture data from a source database so you can then write it to cloud storage or a data warehouse.

Batch connectors

Batch connectors are used in Data Loader and are pre-configured to batch load data from a service. These connectors are typically very quick and easy to use, only requiring database and authentication details. Data Loader will form a pipeline around your Batch connector to schedule data loads.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement