Executing and managing a pipeline

Try in Console

In this guide, you will learn how to execute a published pipeline, check its status, and terminate a running pipeline using Matillion's API. Additionally, you'll discover how to obtain the necessary Project ID and Environment Name for pipeline execution.

Prerequisites

Before you begin, make sure you have the following:

A Hub account.
Valid API credentials for Matillion API access.
An access token with the required privileges (refer to Authentication / Get the access token).

Step 1: Get the Project ID

Retrieve the list of projects using the Project API.

Base URL: : GET /v1/projects

Example response:

{
  "page": 0,
  "results": [
    {
      "description": "project-1",
      "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "name": "Test-project"
    }
  ],
  "size": 0,
  "total": 0
}

Note

Make a note of the projects and their respective IDs, as you'll need these IDs when executing pipelines.

Step 2: Get the environment details

Once you have the project ID, use the Environment API to obtain details about the environment in which the project is running.

Base URL: : GET /v1/projects/{projectId}/environments

Example Response:

{
  "page": 0,
  "results": [
    {
      "defaultAgentId": "3fa85f64-5717-4562-b3fc-2c963f66afaf",
      "defaultAgentName": "Dev Agent",
      "name": "Environment-1"
    }
  ],
  "size": 0,
  "total": 0
}

This includes the environment name, as well as the associated agent's name and agent ID.

Step 3: Get the list of published pipelines

Retrieve the list of published pipelines using the Pipeline API.

Base URL: GET /v1/projects/{projectId}/published-pipelines

Example response:

{
  "page": 0,
  "results": [
    {
      "name": "My pipeline",
      "publishedTime": "2023-11-24T10:39:02.182Z"
    }
  ],
  "size": 0,
  "total": 0
}

Note

Review the available pipelines before execution.

Step 4: Execute published pipeline

To execute a published pipeline, send a POST request with the project ID, environment name, and pipeline name.

Base URL: POST /v1/projects/{projectId}/pipeline-executions

Example request body:

{
  "pipelineName": "Pipeline 1",
  "environmentName": "Dev Environment"
}

Upon successful execution, you'll receive a pipeline execution ID needed for subsequent steps.

Example response:

{
  "pipelineExecutionId": "1398aa31-af57-4a6a-9752-27c2e8556c3f"
}

Step 5: Check pipeline status

Optionally, check the pipeline status using the Pipeline status API with the project ID and pipeline execution ID.

Base URL: GET /v1/projects/{projectId}/pipeline-executions/{pipelineExecutionId}

Example response:

{
  "result": {
    "finishedAt": "2023-11-24T10:47:46.944Z",
    "message": "\"Rewrite Table\": Success",
    "startedAt": "2023-11-24T10:47:46.944Z",
    "status": "RUNNING"
  }
}

Step 6: Terminate pipeline

To cancel a running pipeline, use the PATCH endpoint. Provide a request body with the force update and status details.

Base URL: PATCH /v1/projects/{projectId}/pipeline-executions/{pipelineExecutionId}

Example request body:

{
  "forceUpdate": true,
  "status": "My pipeline"
}

Upon successful termination, you'll receive a response along with the pipeline execution ID.

{
  "pipelineExecutionId": "1398aa31-af57-4a6a-9752-27c2e8556c3f"
}