Custom Connector Data Loader pipeline
Matillion Custom Connector allows users to create connectors for any REST API that can then be used to construct data pipelines in Data Loader.
Matillion Custom Connector provides the configuration details as a JSON object to allow the Data Loader batch integration to extract and load data from the chosen source, as defined and stored in the connector profile, and the endpoints therein. Authentication is not stored as part of the connector profile, but the method of authentication—such as an API key, OAuth entry, or basic authentication—is stored, to assist with configuring the Data Loader batch pipeline.
Architecture
- Create or manage custom connector profiles on the dashboard, by selecting the appropriate endpoint options after logging into the Hub.
- Configure a connector to any RESTful API, setting the API call, various parameters (URI, Query, Header), and pagination in the user interface. The response to the call can be viewed, and the structure it returns can be altered.
- Your custom connector can be used either in Data Loader Batch or Matillion ETL, you just need the correct authorization credentials, configuration of the pipeline, and any dynamic parameters necessary for the use case.
Matillion Custom Connector supports RESTful APIs that return data in JSON format.
Useful links
- For an overview of Matillion Custom Connector, read Matillion Custom Connector Overview.
- For creating a Data Loader pipeline with Custom Connector, read Creating a pipeline with custom connectors.