Artifacts and CI/CD

In the Data Productivity Cloud, an artifact is an immutable collection of resources (such as data pipelines and script files) that is built as a compilation of executable assets and deployed to a specific environment on publication.

You can use the Matillion API to interact with artifacts in the following ways:

• Create an artifact
• Promote an artifact to environments
• Disable an artifact

---

Prerequisites

Incorporate connectors

If your project does not require connectors, ignore this section.

If your project includes custom connectors or Flex connectors, these must be added to the Git repository before artifact creation. Matillion provides a script to automate this process. The script is available via these downloadable links:

1. README.md
2. Requirements.txt
3. Publish.py

You can also choose to manually add connectors, using the process documented further down the page.

---

Create an artifact

Follow these steps to create an artifact in your project using the API. This POST call with generate a deployable artifact within a specified environment.

POST /v1/projects/{projectId}/artifacts

• The request must be sent as multipart/form-data.
• Each file should maintain the same folder structure as in your project.
• If a pipeline (such as example_pipeline) exists within a folder (such as example_folder), then the key should follow this format: example_pipeline/example_folder.orch.yaml.

Example request - create artifact

curl --location 'https://eu1.api.matillion.com/dpc/v1/projects/<your-project-id>/artifacts' \
--header 'versionName: 1.0.0.1' \
--header 'environmentName: <your-environment>' \
--header 'commitHash: <your-git-commit-hash>' \
--header 'Authorization: ••••••' \
--form 'example_pipeline/example_folder.orch.yaml=@"/Users/my-username/workspace/dpc-demo/example_folder/example_pipeline.orch.yaml"' \
--form 'example_folder/helloworld.py=@"/Users/my-username/workspace/dpc-demo/example_folder/hello-world.py"'

Replace <your-project-id>, <your-environment>, and <your-git-commit-hash> with the appropriate values.

---

Promote artifacts between environments

Once an artifact is created, it can be promoted (deployed) to other environments.

POST /v1/projects/{projectId}/artifacts/promotions

You'll need to obtain the artifact versionName using the following GET call: /v1/projects/{projectId}/artifacts/.

Once you have the artifact versionName, you can use the API to promote the artifact.

Example request - promote artifact

curl --location 'https://eu1.api.matillion.com/dpc/v1/projects/<your-project-id>/artifacts/promotions' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "sourceEnvironmentName": "your-source-environment",
    "targetEnvironmentName": "your-target-environment",
    "versionName":"1.2.1"
}'

Replace <your-project-id>, your-source-environment, your-target-environment, and the value of versionName with the appropriate values.

---

Disable an artifact

If an artifact is no longer required, it can be disabled. A disabled artifact cannot be executed and will not appear as an available artifact by default.

PATCH /v1/projects/{projectId}/artifacts/

Example request - disable artifact

curl --location --globoff --request PATCH 'https://eu1.api.matillion.com/dpc/v1/projects/<your-project-id>/artifacts' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
  "environmentName": "your-environment",
  "versionName": "2.1",
  "enabled": false
}'

Replace <your-project-id>, your-environment, and the value of versionName with the appropriate values.

---

Add connectors manually

You can manually retrieve and incorporate connectors. First, you'll need to use the following endpoints to get your connector objects.

• Custom connectors: GET /v1/custom-connectors
• Flex connectors: GET /v1/flex-connectors

Each API call returns a JSON array containing multiple connector objects. Iterate through the array and save each object as a separate file in the root directory of your repository.

Naming convention for connectors

Each file should be named as follows:

connector-profile:custom-{id}.json

Example:

connector-profile:custom-77e6b173-24f5-4bfd-9f8f-787dbdd74c6a.json

Naming convention for flex connectors

connector-profile:flex-{id}.json

Example:

connector-profile:flex-datadog.json

Warning

Do not add these files to your Git repository if the repository is linked to your Data Productivity Cloud project.