Executing and managing a pipeline

Try in Console

In this guide, you will learn how to execute a published pipeline, check its status, and terminate a running pipeline using Matillion's API. Additionally, you'll discover how to obtain the necessary Project ID and Environment Name for pipeline execution.

---

Prerequisites

Before you begin, make sure you have the following:

• A Hub account.
• Valid API credentials for Matillion API access.
• An access token with the required privileges (refer to Authentication and Get the access token).

---

Step 1: Get the Project ID

Retrieve the list of projects using the Project API.

Base URL: GET /v1/projects

Example response:

{
  "page": 0,
  "results": [
    {
      "description": "project-1",
      "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "name": "Test-project"
    }
  ],
  "size": 0,
  "total": 0
}

Note

Make a note of the projects and their respective IDs, as you'll need these IDs when executing pipelines.

Step 2: Get the environment details

Once you have the project ID, use the Environment API to obtain details about the environment in which the project is running.

Base URL: GET /v1/projects/{projectId}/environments

Example Response:

{
  "page": 0,
  "results": [
    {
      "defaultAgentId": "3fa85f64-5717-4562-b3fc-2c963f66afaf",
      "defaultAgentName": "Dev Agent",
      "name": "Environment-1"
    }
  ],
  "size": 0,
  "total": 0
}

This includes the environment name, as well as the associated agent's name and agent ID.

Step 3: Get the list of published pipelines

Retrieve the list of published pipelines using the Pipeline API.

Base URL: GET /v1/projects/{projectId}/published-pipelines

Example response:

{
  "page": 0,
  "results": [
    {
      "name": "My pipeline",
      "publishedTime": "2023-11-24T10:39:02.182Z"
    }
  ],
  "size": 0,
  "total": 0
}

Note

Review the available pipelines before execution.

Step 4: Execute published pipeline

To execute a published pipeline, send a POST request with the project ID, environment name, and pipeline name.

Base URL: POST /v1/projects/{projectId}/pipeline-executions

Example request body:

{
  "pipelineName": "Pipeline 1",
  "environmentName": "Dev Environment"
}

Upon successful execution, you'll receive a pipeline execution ID needed for subsequent steps.

Example response:

{
  "pipelineExecutionId": "1398aa31-af57-4a6a-9752-27c2e8556c3f"
}

Execute published pipeline while preventing concurrent executions

Matillion's API now supports an optional executionTag parameter, which helps prevent concurrent executions of the same API call.

If the executionTag parameter is supplied, the pipeline will run with this tag. If another request is made with the same tag while the previous execution is still running, the request will fail with a 409 conflict response. This feature is particularly useful for external schedulers to prevent duplicate executions.

Example request body with executionTag:

{
  "pipelineName": "Pipeline 1",
  "environmentName": "Dev Environment",
  "executionTag": "nightly-run"
}

Example 409 conflict response if a duplicate execution is attempted:

{
  "error": "A pipeline execution with this tag is already running.",
  "status": 409
}

Step 5: Check pipeline status

Optionally, check the pipeline status using the Pipeline status API with the project ID and pipeline execution ID.

Base URL: GET /v1/projects/{projectId}/pipeline-executions/{pipelineExecutionId}

Example response:

{
  "result": {
    "finishedAt": "2023-11-24T10:47:46.944Z",
    "message": "\"Rewrite Table\": Success",
    "startedAt": "2023-11-24T10:47:46.944Z",
    "status": "RUNNING"
  }
}

Step 6: Terminate pipeline

To cancel a running pipeline, use the PATCH endpoint. Provide a request body with the force update and status details.

Base URL: PATCH /v1/projects/{projectId}/pipeline-executions/{pipelineExecutionId}

Example request body:

{
  "forceUpdate": true,
  "status": "My pipeline"
}

Upon successful termination, you'll receive a response along with the pipeline execution ID.

{
  "pipelineExecutionId": "1398aa31-af57-4a6a-9752-27c2e8556c3f"
}