Review pipeline quality using the API🔗

Try in Console

This guide explains how to use the Matillion API to review pipelines against configured pipeline quality rules. This allows you to programmatically validate that pipelines meet your organization's defined standards using the same rules as the Review feature in the Designer canvas. These rules are defined in the ./matillion/config/pipeline_quality_rules.yaml file.

The review checks pipeline files from a specific project branch and commit, and returns any rule violations based on your configuration. For more information about pipeline quality rules, see Reviewing pipeline quality.

Prerequisites🔗

Before you begin, make sure you have the following:

A Data Productivity Cloud account.
Valid API credentials with access to the target branch.
An access token with the required privileges. For more information, read Obtaining an API access token.
A configured pipeline_quality_rules.yaml file in the target branch.
The pipeline files to review and the pipeline_quality_rules.yaml file must be committed and pushed to the remote repository.

Get the project ID🔗

Retrieve the list of projects using the Project API, then copy the id value. This value is your project ID.

Base URL: GET /v1/projects

Example response:

{
  "page": 0,
  "results": [
    {
      "description": "project-1",
      "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "name": "Test-project"
    }
  ],
  "size": 0,
  "total": 0
}

Get the branch name and commit ID🔗

The branch name is the Data Productivity Cloud branch that contains the pipeline you want to review.
To find the commit ID:
- In the Designer canvas, open the Git drop-down menu and select View Commit History, then copy the commit ID, or
- Retrieve the commit ID directly from your external Git provider.

Request a pipeline review🔗

Use this API endpoint to review one or more pipeline files against the quality rules configured in the target branch and commit.

Base URL: POST /v1/projects/{projectId}/reviews

Configure the following variables:

projectId: Enter the target project ID.
Agent-Version (header): Enter the agent version, for example 11.0.0.

Example request body:

{
  "branchName": "main",
  "commitId": "3b93ffc3",
  "pipelines": [
    "orchestration/my_orchestration_pipeline.orch.yaml",
    "transformation/my_transformation_pipeline.tran.yaml"
  ]
}

Note

The request body must reference pipeline files that exist in the specified branch and commit on the remote repository.

Review response🔗

If the request succeeds, the API returns the results of all evaluated rules, including whether each rule passed or failed. The response also includes the overall review status, which is determined by whether any rules with error enforcement failed.

Any pipelines that could not be evaluated due to an unsupported file type will appear in a list of excludedFiles.

Example response:

{
  "reviewStatus": "FAILED",
  "results": {
    "misc/iterators.orch.yaml": [
      {
        "ruleName": "minimum-components",
        "status": "ERROR",
        "errorDetail": {
          "type": "designer/invalid-rule-configuration"
        },
        "violations": ["minComponents"]
      },
      {
        "ruleName": "maximum-components",
        "status": "ERROR",
        "errorDetail": {
          "type": "designer/invalid-enforcement-level"
        }
      },
      {
        "ruleName": "copied-variables",
        "status": "PASSED"
      }
    ]
  },
  "excludedFiles": [
    "python.py",
    ".matillion/config/pipeline_quality_rules.yaml"
  ]
}

Review error types🔗

Pipeline reviews evaluate each pipeline against the configured quality rules and return a status. In the API response, rules return either PASSED, FAILED, or ERROR. A rule returning ERROR indicates that there has been an error with evaluating the rule against the pipelines.

If a rule errors, it should also include an errorDetail object. The type field describes the category of evaluation error.

For example:

{
  "reviewStatus": "FAILED",
  "results": {
    "GreenWave Pipelines/GreenWave Technologies Demo.orch.yaml": [
      {
        "ruleName": "start-and-end-components",
        "status": "PASSED"
      },
      {
        "ruleName": "component-naming-convention",
        "status": "ERROR",
        "errorDetail": {
          "type": "designer/invalid-enforcement-level"
        }
      }
    ]
  }
}

In this example, the component-naming-convention rule returns an ERROR status, indicating the rule errored. It also includes an errorDetail object indicating an issue encountered while evaluating the rule.

The type field in errorDetail identifies the cause of the evaluation error. The following table lists the possible error types and their descriptions:

Error type	Description
`designer/rules-file-retrieval-error`	Unable to retrieve the rules file.
`designer/rules-file-parse-error`	The rules file could not be parsed.
`designer/document-parse-error`	The pipeline could not be parsed.
`designer/validator-not-found`	The specified rule does not exist.
`designer/invalid-enforcement-level`	The rule has an invalid enforcement level. The value must be either `warn` or `error`.
`designer/empty-regex-pattern`	The rule has an empty regex pattern.
`designer/invalid-regex-pattern`	The rule has an invalid regex pattern.
`designer/no-start-component`	The pipeline must have a start component for the rule to be successful.
`designer/invalid-rule-configuration`	The rule configuration is invalid. Check that all required configuration fields are present and correct.
`designer/duplicate-enabled-rule`	A duplicate rule was detected. Only one instance of the rule can be enabled. Disable or remove the others.

Note

For full API specification, see Data Productivity Cloud Public API.