Custom connector setup

Custom Connector empower users to create specialized connectors for interfacing with external data sources, thereby strengthening integration capabilities. They facilitate seamless data source integration, allowing custom data transformations, and ensuring secure authentication.

In Custom Connector, you can create custom data source connectors to build data pipelines in Data Loader.

This guide primarily focuses on standard custom connectors. However, there's an alternative option known as Flex connectors. These Flex connectors come preconfigured for specific services and can be easily accessed from the Sources list when setting up a data pipeline. Once you've created a Flex connector, it will become available in the Custom Connectors list when selecting a data source. You can use Flex connectors in the same manner as standard custom connectors from that point onwards. For more details, refer to the Flex connector setup documentation.

Prerequisites

You will need the base URL and endpoint information for the API you are connecting to. See the resource's API documentation for more information.

Note

Custom Connector only works with REST APIs that return data in JSON format. XML data is not supported.

Process overview

Manage Custom Connectors: Choose Manage Custom Connector from the Data Productivity Cloud Hub. This grants access to the Connectors dashboard for creating and managing custom connectors.
Connectors Dashboard: The dashboard lists your custom connectors alphabetically, simplifying management.
Contextual Actions: Utilize icons alongside each connector for specific actions, such as editing or deleting.
Sidebar Functions:
- Connectors: Return to the dashboard from any custom connector interface.
- OAuths: Create and store OAuth connections to authenticate data sources.

Add connector

On the Custom Connector dashboard, click Add connector.
Your new connector will begin with a placeholder name such as Untitled Connector X. Click the edit button and rename your connector to something unique and descriptive.
By default, a customer connector has a * symbol as its logo. You can change the connector logo by clicking it to open the *Update connector logo dialog. Input the URL of the image you'd prefer to use and click Apply.
You can add one or more new endpoints to the connector. Each new endpoint will begin with a placeholder name such as Untitled Endpoint X. Click the edit button and rename your endpoint.

Note

You can delete or clone an endpoint at any time by using the three dots next to the listed endpoint.

Request

Set the request method to either GET or POST, depending on the requirements of the third-party API.
Paste or add the endpoint URL of your API.

Authentication

Note

The authentication details you enter in the interface are only used for making test requests and aren't saved as part of the connector. The custom connector only saves the authentication type. You'll be asked to enter authentication details when you set up a pipeline with this connector.

Set the authentication type by clicking the drop-down menu.

Basic Auth: Requires a username and password to access the third-party service.
Bearer Token: Requires a single bearer token from your third-party service.
API Key: Requires a key name and the value of the key. Additionally, you can set the API key as a header or query parameter.
OAuth: Requires an OAuth connection entry. Select one from the drop-down, or click Add new OAuth. You can also choose Manage OAuths

For more details, refer to the Authentication documentation.

Parameters

In the Parameters tab, you can specify any query or URI parameters needed to connect to your endpoint.

To add a query or URI parameter:

Click Add a Query parameter or Add a URI parameter.
Set the parameter name and value in the corresponding columns.
In the Configurability column, toggle to either Constant (the default) or Configurable. Constant parameters retain the value you specify in this setup, whereas configurable parameters are set when building the pipeline, and their value here is for testing only.
If you wish to discard the parameter, click Remove (trashcan).

Note

You can also add query parameters by typing them at the end of the URL.

For more details, refer to the API Parameter documentation.

Headers

In the Headers tab you can specify any request headers that define the HTTP(S) interaction.

Warning

Don't add authentication-specific headers here, use the Authentication tab instead to ensure they're managed with the appropriate level of security.

To add a header parameter:

Click Add a Header Parameter.
Set the parameter name and value in the corresponding columns.
In the Configurability column, toggle to either Constant (the default) or Configurable. Constant parameters retain the value you specify in this setup, whereas configurable parameters are set when building the pipeline, and their value here is for testing only.
If you wish to discard the parameter, click Remove (trashcan).

Body

This field is only enabled for POST requests. Enter any data to be sent from the client to the API.

Format the Body as follows:

JSON:

{
    "key1": "value1", "key2": "value2"
}

x-www-form-urlencoded:

key1=value1&key2=value2

Send

Click Send to make a test request. Click the Response tab to view response body and headers, which you can inspect.

The UI also displays the status code of the request/response, for example, 200 (success).

Structure

This is where you can build and edit the schema that represents the shape of your API's response. It's used later when loading your data into your destination.

After sending a request and receiving a response, click the settings icon located to the right of the response body. This will open the Configure the Structure tab.
In the Configure the structure tab, you will see a list of column names and their corresponding data types representing the API's response.
You can edit the column names or data types by clicking ... next to the column and selecting Edit element. Make the necessary changes and save them.
If you wish to delete a column from the schema, click ... next to the column and choose Delete element. Confirm the deletion.
To save any changes made to the structure, click Refresh structure.
If the API's response contains arrays or nested elements, you can select which part of the schema the connector should retrieve data from. To do this, click the ellipsis ... next to an array element and choose Set as selected data.

Ensure that the structure accurately represents the API's response to ensure proper loading of data into the destination.

Pagination

In this tab, you can set your pagination strategy. Pagination is the strategy for how multiple pages of data are retrieved from the API. Custom Connectors provide some common pagination methods as standard, and an option for you to script your own custom pagination method.

No Pagination: This is the default, and should be used when no pagination is required.
Relative Path: Pages using the relative URI returned in the response.
Full Path: Similar to Relative Path, but uses the full URI returned in the response.
Page Based: Pages by incrementing a page parameter.
Link Header: Pages using the link header in the response headers.
Offset: Pages by incrementing a query parameter.
Cursor: Pages using a cursor parameter.
Script: Pages using a custom script.

Each of the standard methods requires some property configuration, described in the following sections.

No Pagination

This simply retrieves unpaginated data from the API. This is the default, and should be used when no pagination is required.

No additional configuration is required to use this option.

Relative Path

Pages using the relative URI returned in the response.

Property	Description
Base URI	The URI constant to exclude from the relative path.
Next Page URI	The property in the response used to indicate the next page. The drop-down will display the response schema for you to choose a property from.
Page Size Parameter	The query parameter used to specify the page size.
Page Size	The number of records to return per page.

Full Path

Similar to Relative Path paging, but uses the full URI returned in the response.

Property	Description
Next Page URI	The property in the response used to indicate the next page. The drop-down will display the response schema for you to choose a property from.
Page Size Parameter	The query parameter used to specify the page size.
Page Size	The number of records to return per page.

Page Based

Pages by incrementing a page parameter.

Property	Description
Page Size Parameter	The query parameter used to specify the page size.
Page Size	The number of records to return per page.
Page Number Parameter	The query parameter used to specify the page number.
Last Page	The property in the response used to indicate the last page. The drop-down will display the response schema for you to choose a property from.

Link Header

Pages using the link header in the response headers.

Property	Description
Page Size Parameter	The query parameter used to specify the page size.
Page Size	The number of records to return per page.

Offset

Pages by incrementing a query parameter.

Property	Description
Limit Parameter	The query parameter used for the page limit.
Limit	The number of records to return per page.
Record Count	The property in the response used for the page offset. The drop-down will display the response schema for you to choose a property from. You can also use a boolean to indicate whether there are any more records, such as `has_more` = `true` or `false`.
Offset Parameter	The query parameter used for the page offset.

Cursor

Pages using a cursor parameter extracted from the response. There are two possible ways of using the cursor, and the method used depends on the setting of the Pagination Method property:

Header: Extracts the cursor from a header in the response and sends it as a header in the subsequent request.
Query Parameter: Extracts the cursor from the JSON response and sends it as a query parameter in the subsequent request.

Property	Description
Pagination Method	Select either Header or Query Parameter.
Cursor Source Header Name	The name of the header parameter to extract the cursor from. Required if the Pagination Method is Header.
Cursor Header Name	The name of the cursor header to pass via the API request. Required if the Pagination Method is Header.
Page Size Header	The Header used to indicate the page. Required if the Pagination Method is Header.
Page Size	The number of records to return per page. Required if the Pagination Method is Header.
Cursor	The property in the response used for the cursor. The drop-down will display the response schema for you to choose a property from. Required if the Pagination Method is Query Parameter.
More Pages Indicator Key	The property in the response that indicates that there are more pages. Required if the Pagination Method is Query Parameter.
Cursor Parameter Name	The name of the cursor query parameter to pass via the API request. Required if the Pagination Method is Query Parameter.
Page Size Parameter	The query parameter used to specify the page size. Required if the Pagination Method is Query Parameter.
Page Size	The number of records to return per page. Required if the Pagination Method is Query Parameter.

Script

Pages using a custom script. You can use Matillion's Scripted Paging, a powerful scripting language that gives you control over pagination, rate limiting, and API request and response management. Read Script paging for details.

Save your connector

Click Save to save your connector's configuration.

Click Connectors in the left-hand navigation to return to the Connectors overview.

For more details, refer to the Pagination documentation.

Find your connector in Data Loader

You can select your newly created connector in Data Loader. Click Choose source after clicking Add pipeline. Data Loader lists sources alphabetically. You can use the search field to filter.