Artifacts and CI/CD

In the Data Productivity Cloud, an artifact is an immutable collection of resources (such as data pipelines and script files) that is built as a compilation of executable assets and deployed to a specific environment on publication.

You can use the Matillion API to interact with artifacts in the following ways:

• Create an artifact
• Promote an artifact to environments
• Disable an artifact

---

Prerequisites

Before creating an artifact, you can use the Matillion-provided Python script to streamline the process. This script simplifies artifact creation by handling necessary configurations and file structuring. While using the script is optional, it is recommended for a more efficient experience.

Additionally, if your project includes custom or Flex connectors, the script helps ensure these connectors are correctly incorporated into the artifact. In such cases, using the script is advised to avoid manual configuration issues.

Download the script, requirements, and README here:

1. publish.py
2. requirements.txt
3. README.md

Alternatively, you can add connectors manually.

---

Create an artifact

Follow these steps to create an artifact in your project using the API. This POST call with generate a deployable artifact within a specified environment.

POST /v1/projects/{projectId}/artifacts

• The request must be sent as multipart/form-data.
• Each file should maintain the same folder structure as in your project.
• If a pipeline (such as example_pipeline) exists within a folder (such as example_folder), then the key should follow this format: example_pipeline/example_folder.orch.yaml.

Example request - create artifact

curl --location 'https://eu1.api.matillion.com/dpc/v1/projects/<your-project-id>/artifacts' \
--header 'versionName: 1.0.0.1' \
--header 'environmentName: <your-environment>' \
--header 'commitHash: <your-git-commit-hash>' \
--header 'Authorization: ••••••' \
--form 'example_pipeline/example_folder.orch.yaml=@"/Users/my-username/workspace/dpc-demo/example_folder/example_pipeline.orch.yaml"' \
--form 'example_folder/helloworld.py=@"/Users/my-username/workspace/dpc-demo/example_folder/hello-world.py"'

Replace <your-project-id>, <your-environment>, and <your-git-commit-hash> with the appropriate values.

---

Promote artifacts between environments

Once an artifact is created, it can be promoted (deployed) to other environments.

POST /v1/projects/{projectId}/artifacts/promotions

You'll need to obtain the artifact versionName using the following GET call: /v1/projects/{projectId}/artifacts/.

Once you have the artifact versionName, you can use the API to promote the artifact.

Example request - promote artifact

curl --location 'https://eu1.api.matillion.com/dpc/v1/projects/<your-project-id>/artifacts/promotions' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "sourceEnvironmentName": "your-source-environment",
    "targetEnvironmentName": "your-target-environment",
    "versionName":"1.2.1"
}'

Replace <your-project-id>, your-source-environment, your-target-environment, and the value of versionName with the appropriate values.

---

Disable an artifact

If an artifact is no longer required, it can be disabled. A disabled artifact cannot be executed and will not appear as an available artifact by default.

PATCH /v1/projects/{projectId}/artifacts/

Example request - disable artifact

curl --location --globoff --request PATCH 'https://eu1.api.matillion.com/dpc/v1/projects/<your-project-id>/artifacts' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
  "environmentName": "your-environment",
  "versionName": "2.1",
  "enabled": false
}'

Replace <your-project-id>, your-environment, and the value of versionName with the appropriate values.

---

Add connectors manually

You can manually retrieve and incorporate connectors. First, you'll need to use the following endpoints to get your connector objects.

• Custom connectors: GET /v1/custom-connectors
• Flex connectors: GET /v1/flex-connectors

Each API call returns a JSON array containing multiple connector objects. Iterate through the array and save each object as a separate file in the root directory of your repository.

Naming convention for connectors

Each file should be named as follows:

connector-profile:custom-{id}.json

Example:

connector-profile:custom-77e6b173-24f5-4bfd-9f8f-787dbdd74c6a.json

Naming convention for flex connectors

connector-profile:flex-{id}.json

Example:

connector-profile:flex-datadog.json

Warning

Do not add these files to your Git repository if the repository is linked to your Data Productivity Cloud project.