Skip to content

Flex connector tutorial

This tutorial explains how to create a Flex connector for use in Data Loader. In the Data Productivity Cloud, Flex connectors let you connect to a curated set of endpoints to load data from a third party such as Salesforce or Google. Currently, you can use a Flex connector:

  • In Data Loader as part of a batch pipeline.
  • In Designer as part of an orchestration pipeline.
  • In Custom Connector to edit the Flex connector for both Data Loader and Designer.

When you edit a Flex connector in the Custom Connector UI, you can:

  • Add new endpoints.
  • Edit existing endpoints.
  • Delete existing endpoints.
  • Clone existing endpoints.
  • Rename existing endpoints.

The example in this tutorial uses the Snapchat Flex connector for Data Loader. While certain parameter names may be specific to Snapchat, the overall processes covered in this tutorial can be applied to all Flex connectors.

Note

The Snapchat Flex connector is not currently available for Designer.

Create pipeline

  1. Log in to the Hub.
  2. Choose your account.
  3. Select the Load data tile on the What do you want to do today? menu.
  4. Select Add pipeline on the Pipelines page.
  5. In the search bar, type "snapchat" to filter out other connectors, or scroll down the Sources list and select Snapchat. This will begin the Snapchat Flex connector creation wizard.

Step 1 - choose your endpoints

Step 1 of the wizard requires you to choose which Snapchat endpoints to connect to. Each Flex connector comes with a selection of endpoints out-of-the-box. Using the dual listbox, you can move endpoints from the All endpoints box to the Endpoints to extract and load box.

  • Click >> to move all endpoints to Endpoints to extract and load. This means all endpoints will be included in your batch pipeline.
  • Click a single endpoint and then click → to move that endpoint to Endpoints to extract and load. Hold SHIFT when clicking endpoints to select multiple.
  • Click a single endpoint and then click ← to move that endpoint to All endpoints, thus removing that endpoint from the extract and load operation. Hold SHIFT when clicking endpoints to select multiple.
  • Click << to move all endpoints back to All endpoints. This means no endpoints are selected to include in your batch pipeline and you can't move forward yet.
  • Click ↻ to reset the dual listbox and start over.

When you're happy with your endpoint selection, click Continue with X endpoints.

Step 2 - configure your endpoints

Step 2 of the wizard lists your selected endoints under the name of the connector on the left. In the centre of the page, you can configure the API call that the Flex connector is based on by completing General, Authentication, Behaviour, Parameter, and Keys tabs. Required elements in each tab feature a red ! symbol. You need to configure each required element for each endpoint you have selected.

General

In the General tab, you will see the composition of the endpoint URL. Parameters in { } brackets are added from your input.

https://adsapi.snapchat.com/{version}/adaccounts/{ad_account_id}/campaigns

The name of the selected endpoint appears in the field for the Data warehouse table name. You can edit this if you wish.

Use the Configuration Mode radio buttons to switch between Basic and Advanced. This tutorial will cover the Basic configuration mode.

Note

The Behaviour and Keys tabs are only accessible when Configuration Mode is set to Advanced.

Authentication

Move forward by clicking the Authentication tab.

Before you can load data from any Flex connector, you must authenticate with the service you're connecting to—in this case, Snapchat. Snapchat uses OAuth for its authentication mechanism, so you need a valid Snapchat OAuth profile to continue.

Other Flex connectors use authentication methods such as an API key, a bearer token, or basic authentication (username and password).

Read the Snapchat documentation to learn how to acquire Snapchat OAuth credentials.

If you have already created a Snapchat OAuth profile, click the drop-down menu under Authentication profile and select your OAuth profile. Otherwise, click Manage Custom OAuths, which will open the OAuths page in a new browser tab. To add a new OAuth, click Add OAuth and follow the steps in Add OAuth.

Parameters

Move forward by clicking the Parameters tab.

The Campaigns endpoint only requires you to configure URI details, so select the URI tab. From this tab, observe the two parameters that were in brackets in the endpoint URL above:

  • {version}: The version of the Snapchat API to connect to. Currently the value is v1.
  • {ad_account_id}: Because the Campaigns endpoint is specific to Snapchat ad accounts, the Snapchat API identifies your Snapchat account via an ad account ID. To learn how to find your ad account ID, read Find Your Organization and Ad Account ID. Make a copy of the ad account ID as you will need it again later.

Click Continue to move forward.

Next, you need to configure a destination and define the pipeline frequency.

Choose and connect to a destination

Select an existing Destination tile or click Add a new destination to begin the process of creating one. At this point, Snapchat can only use Snowflake as a destination in Data Loader. Read Connect to Snowflake to learn how to connect to Snowflake via Data Loader and configure it as your destination for batch-loading a pipeline.

Set frequency

On the Set frequency page, give your pipeline a name, such as "Snapchat Campaigns Endpoint Connector", and choose how often to sync the pipeline. When you are satisfied, click Create pipeline to complete the process and create your pipeline.

Pipeline creation

You will be redirected to the Pipelines dashboard, where you can observe your new pipeline. Initially it will have a status of Active and that will change to Setting up and then Syncing as the pipeline starts up and runs for the first time. The status will return to Active as the pipeline continues running.

If you see a red X alongside the pipeline status, that means the pipeline has failed and is not returning data. Click ... to the right of the pipeline and click Edit to address any issues preventing the pipeline from running correctly.

Edit your Flex connector and add a new endpoint

Now that your Snapchat Flex connector has been created, you can edit it. There are two methods available:

  1. From the Pipelines dashboard, click ... to the right of the pipeline and then click Edit to reopen the pipeline creation wizard and revisit each step.
  2. Via the Custom Connector UI. From here you can:
    • Add new endpoints.
    • Edit existing endpoints.
    • Delete existing endpoints.
    • Clone existing endpoints.
    • Rename existing endpoints.

The remainder of this tutorial will describe how to add a new endpoint to your Snapchat Flex connector.

  1. Click 𓃑 and then click Hub.

  2. From the What do you want to do today? menu, scroll down to Quick links and click Manage custom connectors.

    Here you can observe all of the custom connectors associated with your account, including Flex connectors.

  3. Locate your new Snapchat Flex connector and click the pencil button to edit it.

This tutorial won't explore every aspect of this page, but you should see a similar interface to the Configure your endpoints stage of the pipeline creation wizard—only now, it is fully expanded with every API configuration field made available. You can see a list of all the preset endpoints on the left, with the endpoints you selected when creating the pipeline at the top. Here you can make more detailed changes to each endpoint as you wish. If you click Add an endpoint at the very bottom of the list, you can add any other Snapchat endpoint that you wish to connect to.

  1. Click Add an endpoint at the bottom of the list of endpoints.
  2. Click the pencil button alongside the new endpoint to rename the endpoint. This tutorial is using the Get all Roles in Ad Account endpoint, so the ascribed name is "GetAllRoles".
  3. Scroll back to the top of the screen to begin configuring the endpoint.

  4. Leave the request type as GET and paste the URL into the text field:

    https://adsapi.snapchat.com/v1/adaccounts/{ad_account_id}/roles

  5. This endpoint uses the same authentication method as before—OAuth. Select the Authentication tab and choose OAuth from the Authentication Type drop-down menu.

  6. Since you have already created a Snapchat OAuth profile, you can select the same profile from the Authentication drop-down menu.
  7. Select the Parameters tab. You should be able to see that the ad_account_id parameter is already added as a URI parameter. If it isn't, then simply click Add a URI parameter and give it the name ad_account_id. Enter your ad account ID into the Value field.

You have now configured your new endpoint, but you can't save the changes to the connector until the endpoint has been tested. Therefore, click Send to make a test GET request to the roles endpoint. A successful test should return a response like the following:

{
  "request_status":"SUCCESS",
  "request_id":"07e9bc92-677e-47a3-8ae1-5bf62c6921ba",
  "paging": {},
  "roles": []
}

If the test fails, confirm that the endpoint URL, parameters, and authentication values are correct, and try again.

Click Save to add your new endpoint to your Snapchat Flex connector.

Return to Data Loader and edit your existing Snapchat pipeline. Observe that your newly added endpoint is available in the Choose endpoints dual listbox. Move this endpoint to the Endpoints to extract and load box and finish editing the pipeline to begin extracting data from your custom endpoint.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement