Skip to content

Custom connector setup

Custom Connector empower users to create specialized connectors for interfacing with external data sources, thereby strengthening integration capabilities. They facilitate seamless data source integration, allowing custom data transformations, and ensuring secure authentication.

In Custom Connector, you can create custom data source connectors to build data pipelines in Data Loader.

This guide primarily focuses on standard custom connectors. However, there's an alternative option known as Flex connectors. These Flex connectors come preconfigured for specific services and can be easily accessed from the Sources list when setting up a data pipeline. Once you've created a Flex connector, it will become available in the Custom Connectors list when selecting a data source. You can use Flex connectors in the same manner as standard custom connectors from that point onwards. For more details, refer to the Flex connector setup documentation.

Prerequisites

You will need the base URL and endpoint information for the API you are connecting to. See the resource's API documentation for more information.

:::info{title='Note'} Custom Connector only works with REST APIs that return data in JSON format. XML data is not supported. :::

Process Overview

  1. Manage Custom Connectors: Choose Manage Custom Connector from the Data Productivity Cloud Hub. This grants access to the Connectors dashboard for creating and managing custom connectors.
  2. Connectors Dashboard: The dashboard lists your custom connectors alphabetically, simplifying management.
  3. Contextual Actions: Utilize icons alongside each connector for specific actions, such as editing or deleting.
  4. Sidebar Functions:
    • Connectors: Return to the dashboard from any custom connector interface.
    • OAuths: Create and store OAuth connections to authenticate data sources.

Add connector

  1. On the Custom Connector dashboard, click Add connector.
  2. Your new connector will begin with a placeholder name such as Untitled Connector X. Click the edit button and rename your connector to something unique and descriptive.
  3. By default, a customer connector has a * symbol as its logo. You can change the connector logo by clicking it to open the *Update connector logo dialog. Input the URL of the image you'd prefer to use and click Apply.
  4. You can add one or more new endpoints to the connector. Each new endpoint will begin with a placeholder name such as Untitled Endpoint X. Click the edit button and rename your endpoint.

:::info{title='Note'} You can delete or clone an endpoint at any time by using the three dots next to the listed endpoint. :::

Request

  1. Set the request method to either GET or POST, depending on the requirements of the third-party API.
  2. Paste or add the endpoint URL of your API.

Authentication

Note

The authentication details you enter in the interface are only used for making test requests and aren't saved as part of the connector. The custom connector only saves the authentication type. You'll be asked to enter authentication details when you set up a pipeline with this connector.

Set the authentication type by clicking the drop-down menu.

  • Basic Auth: Requires a username and password to access the third-party service.
  • Bearer Token: Requires a single bearer token from your third-party service.
  • API Key: Requires a key name and the value of the key. Additionally, you can set the API key as a header or query parameter.
  • OAuth: Requires an OAuth connection entry. Select one from the drop-down, or click Add new OAuth. You can also choose Manage OAuths

For more details, refer to the Authentication documentation.

Parameters

In the Parameters tab, you can specify any query or URI parameters needed to connect to your endpoint.

To add a query or URI parameter:

  1. Click Add a Query parameter or Add a URI parameter.
  2. Set the parameter name and value in the corresponding columns.
  3. In the Configurability column, toggle to either Constant (the default) or Configurable. Constant parameters retain the value you specify in this setup, whereas configurable parameters are set when building the pipeline, and their value here is for testing only.
  4. If you wish to discard the parameter, click Remove (trashcan).

:::info{title='Note'} You can also add query parameters by typing them at the end of the URL. :::

For more details, refer to the API Parameter documentation.

Headers

In the Headers tab you can specify any request headers that define the HTTP(S) interaction.

:::warning{title='Warning'} Don't add authentication-specific headers here, use the Authentication tab instead to ensure they're managed with the appropriate level of security. :::

To add a header parameter:

  1. Click Add a Header Parameter.
  2. Set the parameter name and value in the corresponding columns.
  3. In the Configurability column, toggle to either Constant (the default) or Configurable. Constant parameters retain the value you specify in this setup, whereas configurable parameters are set when building the pipeline, and their value here is for testing only.
  4. If you wish to discard the parameter, click Remove (trashcan).

Body

This field is only enabled for POST requests. Enter any data to be sent from the client to the API.

Format the Body as follows:

JSON:

{
"key1": "value1", "key2": "value2"
}

x-www-form-urlencoded:

key1=value1&key2=value2

Send

Click Send to make a test request. Click the Response tab to view response body and headers, which you can inspect.

The UI also displays the status code of the request/response, for example, 200 (success).

Structure

This is where you can build and edit the schema that represents the shape of your API's response. It's used later when loading your data into your destination.

  1. After sending a request and receiving a response, click the settings icon located to the right of the response body. This will open the Configure the Structure tab.
  2. In the Configure the structure tab, you will see a list of column names and their corresponding data types representing the API's response.
  3. You can edit the column names or data types by clicking ... next to the column and selecting Edit element. Make the necessary changes and save them.
  4. If you wish to delete a column from the schema, click ... next to the column and choose Delete element. Confirm the deletion.
  5. To save any changes made to the structure, click Refresh structure.
  6. If the API's response contains arrays or nested elements, you can select which part of the schema the connector should retrieve data from. To do this, click the ellipsis ... next to an array element and choose Set as selected data.

Ensure that the structure accurately represents the API's response to ensure proper loading of data into the destination.

Configuring your custom connector

Pagination

In this tab, you can set your pagination strategy. No pagination is the default setting.

Here is an overview of the different pagination options:

Relative Path paging

Property Description
Base URI The URI constant to exclude from the relative path.
Next Page URI The property in the response used to indicate the next page. The drop-down will display each column name as selectable.
Page Size Parameter The query parameter used to specify the page size.
Page Size The number of records to return per page.

Full Path paging

Property Description
Next Page URI The property in the response used to indicate the next page. Click the Choose from tree icon to the right of the drop-down field to select one of the columns.
Page Size Parameter The query parameter used to specify the page size.
Page Size The number of records to return per page.

Page Based paging

Property Description
Page Size Parameter The query parameter used to specify the page size.
Page Size The number of records to return per page.
Page Number Parameter The query parameter used to specify the page number.
Last Page The property in the response used to indicate the last page.
Property Description
Page Size Parameter The query parameter used to specify the page size.
Page Size The number of records to return per page.

Offset paging

Property Description
Limit The number of records to return per page.
Record Count Select a column from the response to count the total records. You can also use a boolean to indicate whether there are any more records, such as has_more = true or false.

Cursor paging

Property Description
Cursor The property in the response for the cursor. Click the Choose from tree icon to the right of the drop-down field to select one of the columns.
Cursor Parameter The name of the cursor query parameter.
Page Size Parameter The query parameter used to specify the page size.
Page Size The number of records to return per page.

Click Save to save your connector's configuration.

Click Connectors in the left-hand navigation to return to the Connectors overview.

For more details, refer to the Pagination documentation.

Find your connector in Data Loader

You can select your newly created connector in Data Loader. Click Choose source after clicking Add pipeline. Data Loader lists sources alphabetically. You can use the search field to filter.