API v1 - Queue

Overview

The queue endpoint allows Queues to be managed externally, similarly to how they are managed within the Matillion ETL client via Project → Manage SQS Configuration or Manage Pub/Sub Configuration. Queue configuration can then be exported, imported and tested using the API v1

Important Information

This document is part of a series on API Profiles and the Matillion ETL v1 API
This process requires the Matillion ETL instance URL, and the username and password of a user with appropraite permissions.

Queue API Endpoints

Queue API is available on standard REST-based APIs that uses HTTP or HTTPS request to GET and POST data. The Queue API service is accessed through the Uniform Resource Identifier (URI). All following references in this document will assume the API Base URL has been specified. The available API endpoints are listed below:

Function	Method	Path	URL
Export Queue configurations	GET	/export	https://<InstanceURI>/rest/v1/queue/export
Import Queue configurations	POST	/import	https://<InstanceURI>/rest/v1/queue/import
Test Queue configurations	GET	/test	https://<InstanceURI>/rest/v1/queue/test

Please Note

The username and password of a user with access to the Queue within Matillion ETL instance will need to be used to authenticate the connection. These may need to be specified before the command when using a command line tool—for example: -u user:password.
When exporting or importing Queue configurations, the source file or content type and target file will also need to be specified.
Matillion ETL can currently only use Queue configurations formatted in JSON.

Error

While manually editing an imported Queue configuration file in JSON is possible, it is not recommended.

API Parameters and Description

Below is the list of endpoint parameters and their brief description:

Parameter	Description
<InstanceURI>	The server IP address or domain name of the Matillion ETL instance
<export>	This allows the API to export the Queue configuration to another Matillion ETL instance
<import>	This allows the API to import the Queue configuration from another Matillion ETL instance
<test>	This allows the API to test the validity of the Queue configuration settings

Endpoints and Server Response

These APIs provide a REST-based web service—offering ease of use and a flexible choice of programming language—and can be used to access and analyse data and service provider content. Below are a number of example endpoints along with the corresponding summary, parameters and server responses:

GET/export

This endpoint permits the API to export the Queue configurations from the Matillion ETL instance.

Base URL

https://<InstanceURI>/rest/v1/queue/export

Server Response

{
      "credentialsName": "Instance Credentials",
      "successEnabled": true,
      "compressSuccess": false,
      "failureEnabled": true,
      "compressFailure": false,
      "region": "eu-west-1",
      "sqsEnabled": true,
      "listenQueue": "LISTEN_QUEUE",
      "successQueue": "SUCCESS_QUEUE",
      "failureQueue": "FAIL_QUEUE"
    }

POST/import

This endpoint permits the API to import Queue configurations into the Matillion ETL instance. This will be a "POST" method API call as it will have to attach the project, in the body as a JSON file to import into the Matillion ETL instance.

Base URL

https://<InstanceURI>/rest/v1/queue/import

POST Body (JSON)

{
      "credentialsName": "Instance Credentials",
      "successEnabled": true,
      "compressSuccess": false,
      "failureEnabled": true,
      "compressFailure": false,
      "region": "eu-west-1",
      "sqsEnabled": true,
      "listenQueue": "LISTEN_QUEUE",
      "successQueue": "SUCCESS_QUEUE",
      "failureQueue": "FAIL_QUEUE"
    }

Below is the description of the fields included in the POST body:

Field name	Data type	Description
credentialsName	String	The credentials used to talk to the queue.
successEnabled	Boolean	A message will be placed on a queue when an orchestration job has completed if true.
compressSuccess	Boolean	The body of the "completed" message on the queue will be compressed if true.
failureEnabled	Boolean	A message will be placed on a queue when an orchestration job has failed if true.
compressFailure	Boolean	The body of the "failed" message on the queue will be compressed if true.
region	String	The region in which the Queue exists.
sqsEnabled	Boolean	The sub-system that listens to queues is turned on if true.
listenQueue	String	The name of the queue to listen for messages.
successQueue	String	The name of the Queue on which "completed" messages will be placed.
failQueue	String	The name of the Queue on which "failed" messages will be placed.

Server Response

{
      "success": true,
      "msg": "Successfully imported queue.",
      "id": 0
    }

GET/test

This endpoint permits the API to test the Queue configurations on the Matillion ETL instance.

Base URL

https://<InstanceURI>/rest/v1/queue/test

Server Response

{
        "success": true,
        "msg": "Queue test was successful.",
        "id": 0
      }