Skip to content

API v1 - Tasks

This is a guide to using the Task API in Matillion ETL. The Task API endpoint allows users to explore and manage the tasks created whenever an orchestration or transformation operation is performed in the Matillion ETL instance. This includes:

  • Tasks created by user operations, including almost all operations that the user performs that generate database queries, such as running a job, retrieving a sample, or a row count.
  • Tasks created by the scheduler.
  • Tasks run via the Matillion ETL API (such tasks will display as "API" in the Tasks tab of the client interface).

Tasks are individual operations that components use to perform their functions. A task is always called from a component and a component always sits inside a job. Components can call multiple tasks.

As an orchestration or transformation task is performed (such as executing an orchestration job), the overall task is broken down into more granular tasks as the execution continues. Task information is immediately available in the Tasks panel at the bottom-right of the client interface, giving concise information on recently run tasks and how they have progressed. A Task History is also available, through the Project menu.

The Task API provides the ability to list running tasks, retrieve the task history, and cancel tasks.

Prerequisites

  • Users responsible for experimenting with Matillion ETL API services require access to the Matillion ETL instance and ought to know how to make REST API calls either employing a REST API GUI client such as Postman or employing a command-line interface like cURL.
  • For more information about accessing the Matillion API, and general information on the Matillion API platform before using the API, read Matillion ETL API - v1.
  • Matillion ETL API endpoints require authorization to make any REST API call, so ensure a username and password for the Matillion ETL instance is configured before making any API call.
  • Read Task Management for information about how tasks are used in Matillion ETL.

Using the API

Return task information from a Matillion ETL instance by making a GET, POST, or DELETE call with the form:

<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/<endpoint>

Where:

  • <instanceAddress> is the URI of your Matillion ETL instance in the form http://host:port.
  • <groupName> is the name of the project group running the task.
  • <projectName> is the name of the project running the task.
  • <endpoint> is the name of the API endpoint that accomplishes the task you want to perform. These endpoints are documented in the following sections.

Get running

Returns the details of the tasks running when an orchestration or transformation operation is being performed within the Matillion ETL instance.

To return running task details, make the following GET call:

<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/running

A successful GET call to the endpoint will return a response from the server as described in Server response below.

Get history

This endpoint returns the history of tasks that have been running in the Matillion ETL instance since a specified date.

Task history results are subject to a limit of 1,000 records for a single task run.

To retrieve the task history, make the following GET call:

<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/history?since=<since>

Where:

  • <since> is the starting point of the returned history, in the form yyyy-mm-dd.

A successful GET call to the endpoint will return a response from the server as described in Server response below.

Get id

Retrieves task details of a task with a given ID.

To get the detail of the task using taskid, make the following GET call:

<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/id/<taskId>

Where:

  • <taskId> is the unique ID of the task whose details you want to retrieve.

A successful GET call to the endpoint will return a response from the server as described in Server response below.

Post cancel

This will cancel a running task with a specified ID.

To cancel a task, make the following POST call:

<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/id/<taskId>/cancel

Where:

  • <taskId> is the unique ID of the task you want to cancel.

A successful POST call to the endpoint will return the following response from the server, in JSON format:

{
    "success": true,
    "msg": "Cancel the task[12345].",
    "id": -1
}

Get instance

Retrieves the details of all tasks performed by a specified orchestration or transformation job.

To retrieve the task details, make the following GET call:

<InstanceAddress>/rest/v1/group/name/<groupname>/project/name/<projectName>/task/instance/<jobId>

Where:

  • <jobId> is the unique ID of the job you want to retrieve tasks for.

A successful GET call to the endpoint will return a response from the server as described in Server response below.

Get filter/by

Retrieves entries in the task history according to some filter criteria. Several filter/by endpoints are available, allowing you to use different types of filter, as described below.

To retrieve an unfiltered list (i.e. all entries in the task history), make the following GET call:

<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/none

To retrieve all entries in the task history which started on or after the given date, make the following GET call: 

<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/start/after/date/<date>

To retrieve all entries in the task history which started before the given date and time, make the following GET call: 

<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/start/before/date/<date>/time/<time>

To retrieve all entries in the task history which started after the given date and time, make the following GET call:

http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/start/after/date/<date>/time/<time>

To retrieve the task detail of a specific job (orchestration or transformation), make the following GET call:

http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/job/name/<jobName>

To retrieve all entries in the task history with the given task type (for example, where the task type is "Run Job"), make the following GET call:

http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/type/<type>

To retrieve all entries in the task history with a given state (for example, "success"), make the following GET call:

http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/state/<state>

To retrieve all entries in the task history which started within a specific time range, make the following GET call:

http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/start/range/date/<startDate>/time/<startTime>/to/date/<endDate>/time/<endTime>

To retrieve all entries in the task history that end after a specific date, make the following GET call:

http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/end/after/date/<date>

To retrieve all entries in the task history that end after a specific date and time, make the following GET call:

http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/end/after/date/<date>/time/<time>

To retrieve all entries in the task history that started within a given date/time range, make the following GET call:

http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/end/range/date/<startDate>/time/<startTime>/to/date/<endDate>/time/<endTime>

Where:

  • <date> is a date in the format yyyy-mm-dd, for example 2020-09-28.
  • <time> is a time in the format hh:mm, for example 11:43.
  • <jobName> is the name of the job you want to retrieve task details for.
  • <type> is the type of task you want to retrieve details for, for example Run Job.
  • <state> is the state of the task you want to retrieve details for, for example `success.
  • <startDate> is a date in the format yyyy-mm-dd, for example 2020-09-28.
  • <startTime> is a time in the format hh:mm, for example 11:43.
  • <endDate> is a date in the format yyyy-mm-dd, for example 2020-09-28.
  • <endTime> is a time in the format hh:mm, for example 11:43.

Server response

A successful GET call to an endpoint that retrieves task information will return a JSON response from the server. This will have the following form:

[
    {
        "id": 62737,
        "type": "RUN_ORCHESTRATION",
        "customerID": 953,
        "groupName": "Docs",
        "projectID": 955,
        "projectName": "Sush-test-Salesforce",
        "versionID": 956,
        "versionName": "default",
        "jobID": 960,
        "jobName": "dim_airport_setup",
        "environmentID": 959,
        "environmentName": "RS",
        "state": "SUCCESS",
        "enqueuedTime": 1601375591924,
        "startTime": 1601375591925,
        "endTime": 1601375592100,
        "message": null,
        "originatorID": "ws_61482_4",
        "rowCount": 0,
        "tasks": [
            {
                "taskID": 1,
                "parentID": -1,
                "type": "VALIDATE_ORCHESTRATION",
                "jobID": 960,
                "jobName": "dim_airport_setup",
                "jobRevision": 2,
                "jobTimestamp": 1601338169683,
                "componentID": 976,
                "componentName": "Start 0",
                "state": "SUCCESS",
                "rowCount": -1,
                "startTime": 1601375591926,
                "endTime": 1601375591927,
                "message": ""
            },
        ]
        "hasHistoricJobs": true,
        "jobNames": [
            "dim_airport_setup"
        ]
    }
]

The fields in this response are defined as follows.

  • id: The internal ID of the job. This does not change between runs of the job.
  • type: The type of job being run, either RUN_ORCHESTRATION or RUN_TRANSFORMATION.
  • customerID: The internal ID of the project group this job belongs to.
  • groupName: The name of the project group this job belongs to.
  • projectID: The internal ID of the project this job belongs to.
  • projectName: The name of the project this job belongs to.
  • versionID: The project version this job belongs to.
  • jobID: The internal ID of this job.
  • jobName: The name of this job.
  • environmentID: The internal ID of the environment this job has been run in.
  • environmentName: The name of the environment this job has been run in.
  • state: The current state of the job. In the task history, this will be SUCCESS or FAILED. For currently running jobs, the state is RUNNING.
  • enqueuedTime: The time this job was entered into the queue (Unix/Epoch timestamp).
  • startTime: The time this job was executed (Unix/Epoch timestamp).
  • endTime: The time this job ended (Unix/Epoch timestamp).
  • message: The message passed back from the job upon ending. This can include error messages or statements of successful runs, but is usually blank for jobs.
  • originatorID: The internal ID of the websocket (ws_) or scheduler (s_) that launched the job.
  • rowCount: The number of rows affected by this task. Jobs that do not act on rows of data will return 0 or -1.
  • tasks: An array of one or more tasks belong to the job. Each task contains the following fields:
    • taskID: The internal ID of this task.
    • parentID: The internal ID of the parent job for this task.
    • type: A descriptive typing for the current task. Types are generally VALIDATE_, CREATE_, or EXECUTE_, followed by the resource being acted upon. For example, CREATE_VIEW.
    • jobID: The internal ID of the job this task was called from.
    • jobName: The name of the job this task was called from.
    • jobRevision: The revision ID of the job that this task belongs to.
    • jobTimestamp: The timestamp for when this task's current job revision was created (Unix/Epoch timestamp).
    • componentID: The internal ID of the component this task was called from.
    • componentName: The name of the component this task was called from.
    • state: The current state of the task. In the task history, this will be SUCCESS or FAILED. For currently running tasks, the state is RUNNING.
    • rowCount: The number of rows affected by this task. Tasks that do not act on rows of data will return 0 or -1.
    • startTime: The time this task began (Unix/Epoch timestamp).
    • endTime: The time this task ended (Unix/Epoch timestamp).
    • message: The message passed back from the task upon ending. This usually includes error messages or statements of successful runs. Can be blank.
  • hasHistoricJob: true if this job has Historic Jobs available via the task history. false otherwise. For more information, read Historic jobs.
  • jobNames: If hasHistoricJob is true, the historic jobs' names are listed here