API v1 - Running Jobs

You can use the Matillion ETL API to run any orchestration job within a Matillion ETL instance. This enables you to run jobs programmatically, via a REST API GUI, or via or a command-line interface, as described in Matillion ETL API - v1.

This article describes how to run an orchestration job from the API.

The API can't be used to run transformation jobs.

The best-practice method for running jobs as part of a pipeline is to use a messaging queue such as:

• Azure Queue Storage. Read Azure Queue Message Configuration for details.
• GCP Pub/Sub. Read Manage Pub-Sub Configuration for details.
• AWS Simple Queue Service. Read Manage SQS Configuration for details.

This endpoint requires a POST method API call, as variables for the job to be queued will have to be sent in the body of the call in JSON format.

---

Prerequisites

To run a job via the API, gather the following information:

• The address of your Matillion ETL instance
• The Group name
• The Project name
• The Version name
• The Job name
• Login credentials of a user who has the API Role within the Matillion ETL instance

This document is part of a series on the Matillion ETL API - v1. To use the Matillion ETL API you will need access to the Matillion ETL instance and should be aware of how to make REST API calls using either a REST API GUI client such as Postman or a command-line interface such as cURL.

---

The API "run" URL

The API URL must fully describe the job to be run, as follows:

http://{instanceAddress}/rest/v1/group/name/{groupName}/project/name/{projectName}/version/name/{versionName}/job/name/{jobName}/run?environmentName={environmentName}

Parameter Name	Description
instanceAddress	This is the server IP address or domain name of the Matillion ETL instance.
groupName	The name of the project group within the Matillion ETL instance.
projectName	The name of the project that contains the job to be run.
versionName	The name of the project version to be used. See Manage Versions for an explanation of version naming. If you aren't using project versioning, this parameter will be the name of the default version, which is "default".
jobName	The name of the job that you want to run.
environmentName	The name of the environment that contains the job you want to run.

---

POST body

Any variables required by the job must be sent in the body of the API call in JSON format. Two types of variable can be sent: Job Variables and Grid Variables.

Job variables are called scalarVariables in the body of the API call, as shown in the following JSON example.

The JSON required to pass job (scalar) and grid variables has the following format:

{
    "scalarVariables": {
        "{variable-name-1}": "{value-1}", "{variable-name-2}": "{value-2}"
    },
    "gridVariables": {
        "{grid-variable-name}": [
            [
                "{column-value-1}",
                "{column-value-2}",
                "{column-value-3}"
            ]
        ]
    }
}

Where:

• {variable-name} is the name of a job variable.
• {value} is the value that will be assigned to the job variable when the job executes.
• {grid-variable-name} is the name of a grid variable.
• {column-value} is a value that will be assigned to a grid variable column when the job executes. Multiple values can be passed, separated by commas, and they will be assigned to columns in the order that the columns have been defined in the grid variable. So the first value listed in the JSON post body goes into the grid variable's first column, the second listed value goes into the second column, etc.

You can pass any number of job variables in the scalarVariables statement, separating each "{variable-name}": "{value}" pair with a comma. All job variables must be pre-defined in Matillion ETL before you can pass values via the API. If you attempt to pass a value to a variable which doesn't exist, the API call will execute and schedule the job, but the job will ignore the passed value. The grid variable and all of its columns must be pre-defined in Matillion ETL before you can pass values via the API. If you attempt to pass values to a grid variable which doesn't exist, the API call will execute and schedule the job, but the job will ignore the passed values. However, if the variable does exist then you must pass exactly one value for every column in the grid. If you pass too many or too few values, the job will fail to execute.

---

Server response

On completion of the API call, Matillion ETL will return a response of the following form:

{
    "success": true,
    "msg": "Successfully queued job apiTest",
    "id": 33060
}

The response doesn't indicate that the job has completed, just that it has been queued to run. You can verify that the job has run and completed successfully (or otherwise) by looking at the Tasks Panel or the Task History in the Matillion ETL instance. You can also obtain this information programmatically via the API using the Tasks API via the task id from the server response.