Authenticating Matillion rest API bearer token

Overview

This document describes a mechanism for authenticating calls for Matillion REST APIs, and the requirements to create a Matillion API call request using a bearer token.

We currently offer "basic" authentication on our v0 API, and both "basic" authentication and support for OpenID access tokens via an external OpenID provider on our v1 API.

Using and Configuring OpenID

You should be familiar with the following:

The URL for the /userinfo endpoint of your OpenID provider domain. E.g. https://yourdomain.provider.com/userinfo.
An identifying single user attribute. E.g. name, given_name, email, preferred_username, etc.
You must be able to generate an Access Token to include as a Bearer header in your API requests.
An existing connection between Matillion ETL and the /userinfo endpoint.

The OpenID Connect, in abstract, uses the following steps:

The user client sends a request to the OpenID Provider.

The OpenID Provider authenticates the user and obtains authorization.

The OpenID responds with a token and usually an Access Token.

Add configuration properties and set up a Matillion ETL user.

The user client then sends a call request with the Access Token to the Matillion REST API Endpoint.

The Matillion server sends a UserInfo call request with the bearer token to the OpenID provider.

OpenID provider validates the token and retrieves the UserInfo.

The Matillion server successfully presents the requested data for the REST API call.

These steps are illustrated in the following diagram:

Below is a step-by-step guide to accessing the Matillion ETL REST API response using a bearer token:

Configure the environment variables for Matillion ETL.
Create an internal user with API permissions in Matillion ETL.
Make an API call request to the Matillion server using a bearer token.

Likewise, this document assumes that you have already obtained the token from the OpenID provider. Please follow the instructions in this guide.

Please Note

You can make API call requests as many times as you like. You DO NOT have to configure environment variables and internal users every time you make API requests.
If you require a second user, then you need to create another internal user with API permissions.
If you want to change the OpenID provider—for example, from Okta to Microsoft-AD—then you just need to configure the environment variable API_PROVIDER_USER_URL.

Configuring the Environment Variables for Matillion ETL

Before you make a request to access a Matillion REST API call, you need to set up environment variables in a file using the CLI. These settings can be configured in the Emerald.properties file. To do this, edit the following file:

/usr/share/emerald/WEB-INF/classes/Emerald.properties

This guide assumes you have access to the Emerald.properties file. If not, please contact the Matillion Support team.

Two new configuration properties need to be added; one is required and the other is optional (depending on the user's configuration).

API_PROVIDER_USER_URL

This configuration property is required, as it will contain the provider's address for the /userinfo endpoint, which will be used to confirm token validity, and retrieve user details to map to a Matillion ETL internal user.

Example: GET http://yourdomain.provider.com/userinfo

API_PROVIDER_USER_ATTRIBUTE

This configuration property is optional, as it will automatically default to "email". This property is used to map the above Userinfo request to a Matillion ETL internal username.

Please Note

The user attribute you set API_PROVIDER_USER_ATTRIBUTE will relate to the name of the field in the JSON response back from the /userinfo endpoint. Example:

{
    "sub": "00u441k9tOpO0rIw05d6",
    "email": "firstName.lastName@companyName.com",
    "email_verified": true
}

This will then act as the Matillion ETL internal username. Example:

API_PROVIDER_USER_ATTRIBUTE = email
Internal username = firstName.lastName@companyName.com

Once you finish editing the file, please make sure to restart the container to pick up the set variables from the file. To do this, click Admin in the top-right of your Matillion ETL instance, and click Restart Server. For more information, please refer to Restart Server within Admin Menu.

Creating an Internal User with API permission in the Matillion ETL

1. You are required to set up a new user within your Matillion ETL instance. To do this, click Admin and then click User Configuration.

2. To add an internal user, click in the User Configuration dialog. Click OK.

For detailed instructions on creating an internal user, refer to User Configuration.

Please Note

As described in Configuring the Environment Variables for Matillion ETL, the "Username" must be the same as the value of the attribute, which you can set up using API_PROVIDER_USER_ATTRIBUTE.

Making an API request to the Matillion Server

1. Once you have created an internal user, and set up environment variables in your Matillion ETL instance, you can create a Matillion REST API request with your bearer token, as follows:

GET / http:///rest/v1/group/name//project/name//export
Authorization: Bearer eyJ0NiJ9eyJ1dfI6IjIifX0.DeWt4QuZXso
Content-Type: application/json

Please Note

For further information about the Matillion REST API Endpoint, refer to API v1 - Group/Project.

2. After successful validation of the bearer token, pass the API call. The Matillion server will respond with the HTTP status "200 OK".

Matillion Server Validation Process

UserInfo Request

The Matillion Server sends a request to the /userinfo endpoint to obtain end-user information using the bearer token obtained through OpenID connect authentication. The following is a non-normative example of a UserInfo Request:

GET /userinfo HTTP/1.1
Host: yourdomain.provider.com
Authorization: Bearer SlAV32hkKGkhrfierfeifsdfs

Successful UserInfo Response

The OpenID provider will then validate the token and return end-user information (/userinfo) to map to a Matillion ETL user. The UserInfo MUST be returned as the members of a JSON object.

In most cases, this will be an email, but a different user attribute can be defined in properties by setting, for example, API_PROVIDER_USER_ATTRIBUTE=name.

If the above OpenID provider Userinfo request returns a successful response and returns the defined user attribute (API_PROVIDER_USER_ATTRIBUTE) in the JSON, then the Matillion server will attempt to find a Matillion ETL user with the username of the attribute.

The following is a non-normative example of such a response:

{
   "sub": "XX123456789XXXX01234",
   "name": "FirstName Surname",
   "given_name": "FirstName",
   "family_name": "Surname",
   "preferred_username": "f.s",
   "email": "firstname.surname@company.com",
}

UserInfo Error Response

When an error condition occurs, the /userInfo endpoint returns an error response:

401 Unauthorized

Example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer

Possible reasons for an error like this to occur are as follows:

The request was denied due to an invalid or missing bearer access token.
The configured environment variable (API_PROVIDER_USER_URL) does not exist.
UserInfo endpoint is down. This could be for scheduled or unscheduled maintenance, as an example.

To assist with troubleshooting within your Matillion ETL instance, you can log error responses from the /userinfo endpoint in the "catalina.log" file, which can be accessed by clicking Admin and then clicking Download Server Log.