Skip to content

Custom connector worked example

The following guide describes how to create a custom connector in Matillion Data Loader. A custom connector is a completely un-configured connector that can extract data from any compatible REST API. Because the connector has no endpoints or authentication details, you will need to acquire and provide this information yourself. As such, custom connectors are ideally suited to connect to data sources that are not currently available in Data Productivity Cloud.

The example we are going to follow uses the Spotify API to perform a fairly basic data extraction call. The guide is written to effectively guide you through the process, including authentication steps.

Note

Matillion is not currently affiliated with Spotify, nor is Spotify a supported data source in Data Productivity Cloud. All API and authentication details are correct at the time of writing. Changes to Spotify's processes and/or documentation may mean this guide becomes inaccurate. Please notify us and the document will be corrected.

Prerequisites

Read the following documents to familiarise yourself with custom connectors and the Spotify API:

Authentication

You will require an access token in order to make calls to Spotify's API, which first requires a valid user account and the creation of an app. For reference, the process we are following is documented here, but we will go through it fully here for clarity.

  1. Log in to the Spotify developer dashboard using your Spotify account. If you don't have one, you will need to create one.
  2. Click Create App to begin creating a new app. Complete the fields, as required. We used the Data Loader URI as the Redirect URI but anything can be used. Click Save.
  3. You will return to the main dashboard for your app. Click Settings in the top-right corner to view the Basic Information for your app, including the Client ID and Client Secret. You will need to click View client secret to access the client secret.
  4. Make copies of both the Client ID and Client Secret.

You will now need to make a POST request to Spotify's API through your preferred API client. We will use Postman.

  1. Configure a POST request to the Spotify token request endpoint:

    https://accounts.spotify.com/api/token

  2. Add a header entry as follows:

    Key: Content-Type, Value: application/x-www-form-urlencoded

  3. Add the following HTTP body, replacing the placeholder values with your own Client ID and Client Secret. If you are using Postman, you can use the body as Raw.

    grant_type=client_credentials&client_id=<your-client-id>&client_secret=<your-client-secret>

  4. Send your request.

A successful request should return the access token formatted as follows:

{
    "access_token": "BQCRUpmOZF5XvXGKNsdsTBAVc4NpL_7y8DSZTv_BG56rJvRfS5wYR9HFzKphgRqMCDnTnFn97KBmtqtPRrtL0ANVcezyjFsz0B6JAU1kOK3wFUA19EE",
    "token_type": "Bearer",
    "expires_in": 3600
}

The token expires after one hour but this will be sufficient for our example. You can re-send the POST request to receive a new token as required. Make a copy of the access token, as we can now begin building our connector.

Compose the endpoint

We are going to use the Get Artist's Top Tracks endpoint. This is a relatively simple endpoint that will return the top tracks for a specified artist for a given geographic region. Refer to the linked document for full information regarding the endpoint and response data.

The endpoint URL is: https://api.spotify.com/v1/artists/{id}/top-tracks

As you can see, the endpoint first requires a single parameter: {id}. This is the artist_id of the artist you want to query, and is gathered as follows:

  1. Locate your chosen artist in the Spotify client.
  2. Click the three dots ... near the top of the artist's page, and click Share -> Copy link to artist.

  3. Paste the copied link into a text editor, and you should see the following:

    https://open.spotify.com/artist/06HL4z0CvFAxyc27GXpf02?si=r_2RdlprRwSsQhSM63VKFw

    The artist ID is the string after https://open.spotify.com/artist/ and before the question mark ?. For example:

    https://open.spotify.com/artist/06HL4z0CvFAxyc27GXpf02?si=r_2RdlprRwSsQhSM63VKFw

  4. Copy the artist ID.

Additionally, you will need to specify a geographic region by using a country code. The country code is the two-digit ISO 3166-1 alpha-2 country code for the relevant country. For example, a country in the United Kingdom would use the code GB. We will detail how to add this parameter later on, but for now make a note of the country code you want to use.

We now have everything we need to compose the GET request for our connector: the access token for authentication, the endpoint URL, and the artist ID and country code required for that endpoint.

Create a custom connector

  1. Access the Custom Connector dashboard and click Add connector.
  2. The connector will have a placeholder name such as "Untitled Connector X". Click the pencil icon next to the name to rename your connector to something unique and descriptive.
  3. Similarly, your first endpoint is listed on the left with a placeholder name. Do the same again to rename it to something descriptive, such as "GetArtistsTopTracks".

Next, we will configure a few details in the Request tab.

Configure the request

  1. If not already, set the request type to GET from the drop-down menu.

  2. Paste the endpoint URL from earlier into the address field:

    https://api.spotify.com/v1/artists/{id}/top-tracks

  3. Within the Authentication tab, select Bearer Token from the Authentication Type drop-down menu.

  4. Paste the access token you generated earlier into the Token text field. You can generate a new one if your original has expired.

    Note

    It is important to note here that the access token is only used to test the connector at this point. It is not saved to the connector and as such will need to be re-entered in Matillion Data Loader or Designer when used as a pipeline data source.

  5. Select the Parameters tab. Here you can see a parameter already in place for the {id} parameter in the endpoint. Paste your chosen artist ID into the Value field.

  6. Now we need to add the country code. Click Add a Query Parameter to add a new parameter. We will define our query for the United Kingdom, so configure it as follows:
    1. Name: market
    2. Value: GB

At this point our connector is configured. We can't save the endpoint to the connector until we test it. Click Send to make your GET request to the Get All Artist's Top Tracks endpoint.

A successful result should return a large body of text, part of which is shown below. Now that we know our endpoint is configured correctly, we can now Save it and complete the process. The connector is now available for pipelines in Matillion Data Loader and Designer. You can return to this connector at any point to add further endpoints and edit existing ones following the process above.

{
    "tracks": [
        {
            "album": 
                {
                    "album_type": "album",
                    "artists":
                                [
                                    {
                                        "external_urls": {
                                                           "spotify": "https://open.spotify.com/artist/06HL4z0CvFAxyc27GXpf02"
                                                        },
                                        "href": "https://api.spotify.com/v1/artists/06HL4z0CvFAxyc27GXpf02",
                                        "id": "06HL4z0CvFAxyc27GXpf02",
                                        "name": "Taylor Swift",
                                        "type": "artist",
                                        "uri": "spotify:artist:06HL4z0CvFAxyc27GXpf02"
                                    }
                                ]
                }
        }
]
}