Publishing shared pipeline artifacts with the API🔗

In Maia, a shared pipeline artifact is a versioned package containing one or more shared pipelines from a project. You can use the Matillion API to publish shared pipeline artifacts as part of your CI/CD workflow.

You can use the API to interact with shared pipeline artifacts in the following ways:

Publish a shared pipeline artifact
Verify the published artifact contents
Disable a broken version

For an overview of shared pipeline artifacts, terminology, and all available endpoints, read Understanding shared pipeline artifacts.

Prerequisites🔗

Before publishing, you can use the Matillion-provided Python script to streamline the process. This script handles file discovery, multipart upload, and post-publish verification.

Download the script and requirements here:

Set the following environment variables:

API_BASE_URL: The base URL of the Data Productivity Cloud public API. Read API overview for details.
AUTH_TOKEN: The authentication token (Admin or Super Admin account role required). Read Authentication for details.

Install dependencies:

pip install -r requirements.txt

Using the script🔗

API_BASE_URL="https://eu1.api.matillion.com/dpc/v1" \
AUTH_TOKEN="your-token" \
python publish-shared-pipelines.py \
  -S my-project \
  -V 1.0.0 \
  -c abc123 \
  -b main \
  -p /path/to/project

The script will:

Walk the project directory and collect all pipeline files.
Upload them as a multipart request to POST /v1/shared-pipelines/artifacts.
Check the ignoredResources list and warn if any pipelines were excluded.
Call the details endpoint to verify the artifact contents.

See python publish-shared-pipelines.py --help for all options, including --dry-run to preview which files would be included without publishing.

Publish a shared pipeline artifact🔗

Follow these steps to publish a shared pipeline artifact using the API. This POST call uploads your project files and creates a versioned artifact.

POST /v1/shared-pipelines/artifacts

The request must be sent as multipart/form-data.
Each file must be explicitly listed. Archive files are not supported.
The sharedPipelineIdPrefix and versionName combination must be unique. Re-publishing the same version returns 409 Conflict.
If a pipeline exists within a folder, the key should follow the format: folder/pipeline.orch.yaml.

Note

This request does not make any changes to the folder structure within your Maia project. The file path you provide is how your pipeline will be referenced. We recommend that the implied folder structure of your provided files matches the folder structure you are using in Designer.

Example request — publish🔗

curl --location 'https://eu1.api.matillion.com/dpc/v1/shared-pipelines/artifacts' \
--header 'sharedPipelineIdPrefix: my-project' \
--header 'versionName: 1.0.0' \
--header 'commitHash: abc123def456' \
--header 'branch: main' \
--header 'Authorization: Bearer ••••••' \
--form '.matillion/shared-pipelines.yaml=@"/path/to/project/.matillion/shared-pipelines.yaml"' \
--form 'pipelines/load-customers.orch.yaml=@"/path/to/project/pipelines/load-customers.orch.yaml"'

Replace my-project, 1.0.0, abc123def456, main, and the file paths with the appropriate values for your project.

Example response — publish🔗

{
  "versionName": "1.0.0",
  "ignoredResources": []
}

Note

Always check the ignoredResources list. If a pipeline you expected is listed there, the artifact may be incomplete.

Verify after publishing🔗

After publishing, call the details endpoint to confirm the artifact contains the expected pipelines:

curl --location 'https://eu1.api.matillion.com/dpc/v1/shared-pipelines/artifacts/details?sharedPipelineIdPrefix=my-project&versionName=1.0.0' \
--header 'Authorization: Bearer ••••••'

The response includes artifactDetails (version metadata) and sharedPipelines (the individual pipelines in the artifact). Read Understanding shared pipeline artifacts for the full response format.

Disable a shared pipeline artifact🔗

If an artifact version is no longer required or introduces breaking changes, it can be disabled. A disabled version can't be selected by consumers using [Latest].

PATCH /v1/shared-pipelines/artifacts

Example request — disable🔗

curl --location --request PATCH 'https://eu1.api.matillion.com/dpc/v1/shared-pipelines/artifacts' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ••••••' \
--data '{
  "sharedPipelineIdPrefix": "my-project",
  "versionName": "1.0.0",
  "enabled": false
}'

Replace my-project and 1.0.0 with the appropriate values.

CI/CD integration🔗

The publish endpoint is designed for automation. A typical workflow is:

Validate — Lint all YAML pipeline files.
Version — Calculate the next version (for example, from git tags).
Authenticate — Obtain an OAuth2 bearer token with an Admin or Super Admin account role. Read Authentication for details.
Publish — Use the publish script or curl to upload your project files.
Verify — Call GET /v1/shared-pipelines/artifacts/details to confirm the expected pipelines are present and the ignoredResources list is empty.

Note

If you already automate publishing standard project artifacts, the shared pipeline workflow follows the same pattern. Read Creating artifacts with the API for the standard artifact workflow.