Skip to content

Custom connector setup

In Matillion Custom Connector, you can create custom data source connectors to build data pipelines in Data Loader.

This guide refers to standard custom connectors. Flex connectors are also available, which are preconfigured custom connectors tailored to a specific service. They are accessed from the list of Sources when creating a pipeline. After you have created a Flex connector, it will appear in the list of Custom Connectors when choosing a data source, and can be used in the same way as a standard custom connector at that point. Read Flex connector setup for more information.

Prerequisites

  • You need to know how to authenticate with your third-party service.
  • You need the API documentation for the third-party service, including the main URL.

:::info{title='Note'} Matillion Custom Connector only works with REST APIs that return data in JSON format. XML data is not supported. :::

Add connector

  1. On the Custom Connector dashboard, click Add connector.
  2. Your new connector will begin with a placeholder name such as Untitled Connector X. Click the edit button and rename your connector to something unique and descriptive.

Request

  1. Set the request method to either GET or POST, depending on the requirements of the third-party API.
  2. Paste or add the endpoint URL of your API.

Authentication

:::info{title='Note'} The authentication details you enter here are only used for making test requests, and aren't saved as part of the connector—Custom Connector only saves the authentication type. You'll be asked to enter authentication details when you set up a pipeline with this connector. :::

Set the authentication type by clicking the drop-down menu.

  • Basic Auth Requires a username and password to access the third-party service.
  • Bearer Token Requires a single bearer token from your third-party service.
  • API Key Requires a key name and the value of the key. Additionally, you can set the API key as a header or query parameter.
  • OAuth Requires an OAuth connection entry. Select one from the drop-down, or click Add new OAuth. You can also choose Manage OAuths

Parameters

In the Parameters tab, you can specify any query or URI parameters needed to connect to your endpoint.

To add a query or URI parameter:

  1. Click Add a Query parameter or Add a URI parameter.
  2. Set the parameter name and value in the corresponding columns.
  3. In the Configurability column, toggle to either Constant (the default) or Configurable. Constant parameters retain the value you specify in this setup, whereas configurable parameters are set when building the pipeline, and their value here is for testing only.
  4. If you wish to discard the parameter, click Remove (trashcan).

:::info{title='Note'} You can also add query parameters by typing them at the end of the URL. :::

Headers

In the Headers tab you can specify any request headers that define the HTTP(S) interaction.

:::warning{title='Warning'} Don't add authentication-specific headers here, use the Authentication tab instead to ensure they're managed with the appropriate level of security. :::

To add a header parameter:

  1. Click Add a Header Parameter.
  2. Set the parameter name and value in the corresponding columns.
  3. In the Configurability column, toggle to either Constant (the default) or Configurable. Constant parameters retain the value you specify in this setup, whereas configurable parameters are set when building the pipeline, and their value here is for testing only.
  4. If you wish to discard the parameter, click Remove (trashcan).

Body

This field is only enabled for POST requests. Enter any data to be sent from the client to the API.

Send

Click Send to make a test request. Click the Response tab to view response body and headers, which you can inspect.

The UI also displays the status code of the request/response, for example, 200 (success).

Structure

This is where you can build and edit the schema that represents the shape of your API's response. It's used later when loading your data into your destination.

  1. After sending a request and receiving a response, click the settings icon located to the right of the response body. This will open the Configure the Structure tab.
  2. In the Configure the structure tab, you will see a list of column names and their corresponding data types representing the API's response.
  3. You can edit the column names or data types by clicking ... next to the column and selecting Edit element. Make the necessary changes and save them.
  4. If you wish to delete a column from the schema, click ... next to the column and choose Delete element. Confirm the deletion.
  5. To save any changes made to the structure, click Refresh structure.
  6. If the API's response contains arrays or nested elements, you can select which part of the schema the connector should retrieve data from. To do this, click the ellipsis . next to an array element and choose Set as selected data.

Ensure that the structure accurately represents the API's response to ensure proper loading of data into the destination.

Pagination

In this tab, you can set your pagination strategy. No pagination is the default setting.

Here is an overview of the different pagination options:

Relative Path paging

Property Description
Base URI The URI constant to exclude from the relative path.
Next Page URI The property in the response used to indicate the next page. The drop-down will display each column name as selectable.
Page Size Parameter The query parameter used to specify the page size.
Page Size The number of records to return per page.

Full Path paging

Property Description
Next Page URI The property in the response used to indicate the next page. Click the Choose from tree icon to the right of the drop-down field to select one of the columns.
Page Size Parameter The query parameter used to specify the page size.
Page Size The number of records to return per page.

Page Based paging

Property Description
Page Size Parameter The query parameter used to specify the page size.
Page Size The number of records to return per page.
Page Number Parameter The query parameter used to specify the page number.
Last Page The property in the response used to indicate the last page.
Property Description
Page Size Parameter The query parameter used to specify the page size.
Page Size The number of records to return per page.

Offset paging

Property Description
Limit The number of records to return per page.
Record Count Select a column from the tree to count the total records.

Cursor paging

Property Description
Cursor The property in the response for the cursor. Click the Choose from tree icon to the right of the drop-down field to select one of the columns.
Cursor Parameter The name of the cursor query parameter.
Page Size Parameter The query parameter used to specify the page size.
Page Size The number of records to return per page.

Click Save to save your connector's configuration.

Click Connectors in the left-hand navigation to return to the Connectors overview.

Find your connector in Data Loader

You can select your newly created connector in Data Loader. Click Choose source after clicking Add pipeline. Data Loader lists sources alphabetically. You can use the search field to filter.