Skip to content

API v1 - Versions

Overview

This is a guide to providing details on the Version API from the family of the Group API . The Version endpoint allows users to explore and manages the versions of the Matillion ETL Transformation and Orchestration Jobs.

Versioning supports many use cases, but perhaps the most salient is to be able to capture your development at a point in time to designate as "live" or "production", etc., and to then continue working on the Default Version.

All projects have a default version. This is the main working version. It is possible to make changes to a non-default version you have created via Project Menu → Manage Versions, but it must be in an unlocked state.

Managed Version

The Version API provides the resource details to run the version details within the project group in the Matillion ETL instance. By resources here, we refer to explore the version, export versions, delete version, and create a new version within the ETL Transformation and Orchestration Jobs.

Important Information

  • This document is part of a series on API v1 - Group and the Matillion ETL API - v1.
  • Users responsible for experimenting with Matillion ETL API services require access to the Matillion ETL instance and ought to know how to make REST API calls either employing a REST API GUI client such as Postman or employing a command-line interface like cURL.
  • Matillion ETL API endpoints require authorisation to make any REST API call, so ensure a username and password for the Matillion ETL instance is configured before making any API call.

Version API Endpoints

API Base URL

http(s)://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version

API Endpoints and Function

Version endpoint API is available on standard REST-based APIs that uses HTTP or HTTPS request to GET, POST, and DELETE data. The Version endpoint API service is accessed through the Uniform Resource Identifier (URI). The available API endpoints are listed below:

/tr>
Method Path URI Function
GET /name http://<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/name Get the name of the current version.
GET /id http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/id Get the id of the selected version.
GET /export http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/export To export the metadata of the current version within the project.
GET /orchestration http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/orchestration To retrieve the details of the Orchestration within the selected project.
GET /transformation http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/transformation To retrieve the details of the Transformation within the selected project.
POST /create http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/create To create a new version which is a copy of the existing version.
POST /delete http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/delete To delete the selected version using POST HTTP method.
DELETE /versionName http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName> To delete the selected version using DELETE HTTP method.
PATH/job
GET /export http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/export To export the data of the job within the current version.
POST /import http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/import To import the data of the job within the current version.
POST /delete http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/delete To delete the selected job using POST HTTP method.
POST /run http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/run?environmentName=<environmentName> WITH POST DATA arg0 Run an Orchestration Job, where variable overrides can be passed in the body.
POST /validate http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/validate?environmentName=<environmentName> To validate the Job by passing Environment name in the body.
DELETE /jobName http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName> To delete the selected job using DELETE HTTP method.
PATH/scm

Graphical Representation

To illustrate the Version API v1, endpoints and methods to the further, below is the graphical flow of the /version endpoint showing possible PATH, GET , POST, and DELETE options.

Version endpoint Flow

Please Note

For detailed description about accessing the SCM data of the project for PATH/scm part of Version API, with some general information on the SCM and Git commands, please refer to SCM Integration.



URL Parameters and Description

Below is the list of endpoint parameters and their brief description:

Parameters Name Description
<InstanceAddress> This is the server IP address or domain name.
<groupName> the name of the group.
<projectName> The name of the project within the group.
<versionName> The name of the version within the selected project.
<JobName> The name of the job within the selected version.
<name> To get the name of the resource.
<export> This allows to export the metadata of the resource.
<delete> Delete the selected resource.
<id> The queue id of the current resource.
<run> Run an Orchestration Job, where variable overrides can be passed in the body.
<validate> Validate the Orchestration Job by passing Environment name.

Endpoints and Server Response

This chapter describes the Version API endpoints and the Job API endpoints along with some server response examples. However, PATH/job is a part of PATH/version API as shown in the Overview section. These APIs offers REST-based web service, offering ease of use and a flexible choice of programming language. These APIs can be used to access and analyse the versions, jobs and scm within the project in the Matillion ETL instance. All the APIs listed in this chapter are available to use with GET/POST/DELETE methods to retrieve the data used to get, update, or delete the versions.

PATH/version

List of endpoints for the /version:

Below is the detailed description of these endpoints with the server response examples.

GET/name

This example is a GET method REST API request that will provide the name of the version within the project in the instance.

  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/name
  • Server Response 
    default

GET/id

This is to get the id for the selected version within the project using GET HTTP method.

  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/id
  • Server Response 
    796

GET/export

To export the selected version detail from the project within the Matillion instance, provide the versionName and use the /export endpoint. This example using GET method REST API call to export the jobs (Transformation/Orchestration) details available within the selected version.

  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/export
  • Server Response 
    {
    "objects": [
        {
            "name": "default",
            "description": "Default Version",
            "jobs": {
                "objects": [
                    {
                        "jobObject": {...
                        },
                        "info": {
                            "id": 800,
                            "name": "dim_airport_setup",
                            "type": "ORCHESTRATION",
                        },
                        "path": []
                    },
                    {
                        "jobObject": {...
                        },
                        "info": {
                            "id": 831,
                            "name": "dim_airports",
                            "type": "TRANSFORMATION",
                        },
                ],
                "version": "master",
                "environment": "redshift"
        }
    ],
    "version": "master",
    "environment": "redshift"
}

GET/orchestration

To retrieve the Orchestration detail from the project, provide the versionName and use the /orchestration endpoint. This example using GET method REST API call to get the Orchestration jobs details available within the selected version of the project.

  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/orchestration
  • Server Response 
    [
  {
    "id": 800,
    "name": "dim_airport_setup",
    "description": null,
    "type": "ORCHESTRATION",
    "tag": "02fc1dee-99c5-4fcf-928f-852536bfebeb"
  },
  {
    "id": 827,
    "name": "Orchestration-Job-1",
    "description": null,
    "type": "ORCHESTRATION",
    "tag": "bbc43d86-e130-46c0-9a6e-73099bff2b94"
  }
]

GET/transformation

To retrieve the Transformation detail from the project, provide the versionName and use the /transformation endpoint. This example using GET method REST API call to get the Transformation jobs details available within the selected version of the project.

  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/orchestration
  • Server Response 
    [
  {
    "id": 831,
    "name": "dim_airports",
    "description": null,
    "type": "TRANSFORMATION",
    "tag": "844e3ea8-af95-4b6e-8dfa-db9150a4cdfb"
  }
]

POST/create

This is to create a new which is a copy of the existing version by using /create endpoint. This API call is a POST method API call as we will have to attach the resource data, in the body as a JSON file to import into the Matillion ETL instance.

  • Base URL 
        
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/create
  • POST Body (JSON) 
    {
    "newVersionName": "name",
    "newVersionDescription" : "This is a new version",
    "newVersionIsLocked" : true
  }
  • Server Response 
    {
    "success": true,
    "msg": "",
    "id": -1
}

POST/delete

This will be a POST method API call. The /delete endpoint will allow to delete the selected version from the project.

  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/delete
  • Server Response 
    {
  "success": true,
  "msg": "Successfully deleted the version: default",
  "id": -1
}

DELETE/versionName

To remove any resource from the list, we will use DELETE API request. In this example, we will delete a version from the project.

Please Note

There are often multiple ways of achieving the same task within the API as resources and methods branch out and overlap.


  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>
  • Server Response 
    {
  "success": true,
  "msg": "Successfully deleted version [default]",
  "id": -1
}

PATH/job

PATH/job is the part of the PATH/version, which is further combined with HTTP methods GET, POST and DELETE.

List of endpoints associated with the PATH/job

Below is the detailed description of these endpoints with the server response examples.

GET/export

This is a GET HTTP method to export the details of the Jobs available within the selected version of the project. To export any "resource" data we use /export endpoint after the "resource name". For example: /jobName

  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/export
  • Server Response 
    {
    "objects": [
        {
            "jobObject": {
                "JobType": ".OrchestrationJob",
                "id": 827,
                "revision": 2,
                "created": 1594599191329,
                "components": {
                    "828": {
                        "id": 828,
                        ....
                    },
                    "829": {
                        "id": 829,
                        ....
                    },
            "info": {
                "id": 827,
                "name": "Example-Job",
                "type": "ORCHESTRATION",
                "tag": "bbc43d86-e130-46c0-9a6e-73099bff2b94"
            },
            "path": []
        }
    ],
    "version": "master",
    "environment": "redshift"
}

POST/import

This will import the resource data which is exported as shown in the above example using /import endpoint. If a resource of the same name already exists, you must delete the existing resource before importing the new or you can change the name of the imported resource (Job-Imported)in the POST BODY. This API call is a POST method API call as we will have to attach the resource data, in the body as a JSON file to import into the Matillion ETL instance.

  • Base URL 
        
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/import
  • POST Body (JSON) 
    {
    "objects": [
        {
            "jobObject": {
                "JobType": ".OrchestrationJob",
                "id": 827,
                "revision": 2,
                "created": 1594599191329,
                "components": {
                    "828": {
                        "id": 828,
                        ....
                    },
                    "829": {
                        "id": 829,
                        ....
                    },
            "info": {
                "id": 827,
                "name": "Example-Job",
                "type": "ORCHESTRATION",
                "tag": "bbc43d86-e130-46c0-9a6e-73099bff2b94"
            },
            "path": []
        }
    ],
    "version": "master",
    "environment": "redshift"
}
  • Server Response 
    {
  "name": "Version",
  "statusList": [
    {
      "success": true,
      "name": "Job-Imported"
    }
  ],
  "success": true
}
  • URL Parameters

    There is an optional parameter, onConflict, which determines what should happen if an import clashes with something that already exists. For an explanation of this parameter's use, read API Import conflicts - Explanation in Matillion ETL API - v1.

POST/delete

This will be a POST method API call. The /delete endpoint will allow to delete the selected job from the project.

  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/delete
  • Server Response 
    {
  "success": true,
  "msg": "Successfully deleted the job: Example-Job",
  "id": -1
}

POST/run

The /run endpoint will allow to run an Orchestration job within the version. This will be a POST method API call as we will have to attach the variable for the job to be queued, in the body as a JSON file to run into the Matillion ETL instance.

  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/run?environmentName=<environmentName> WITH POST DATA arg0
  • POST BODY(JSON) 
    { 
          "scalarVariables": {},
           "gridVariables": {}
}
  • Server Response 
    {
  "success": true,
  "msg": "Successfully queued job dim_airport_setup",
  "id": 65659
}

POST/validate

The /validate endpoint will allow to validate an Orchestration job within the version. This will be a POST method API call as we will have to pass the environment name for the job to be queued, in the body as a JSON file to run into the Matillion ETL instance.

  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>/validate?environmentName=<environmentName>
  • Server Response 
    {
    "success": true,
    "msg": "Successfully queued validation for Orchestration Job: [API query], for Environment: [test].  Task Id: [xxxxxxx]",
    "id": xxxxxxx
}

DELETE/jobName

To remove any resource from the list, we will use DELETE API request. In this example, we will delete a job from the project.

  • Base URL 
    http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/job/name/<jobName>
  • Server Response 
    {
  "success": true,
  "msg": "Successfully deleted job [dim_airport_setup]",
  "id": -1
}