API v1 - Git integration for projects

This article is a guide to using the source control management (SCM) API in Matillion ETL. Matillion ETL uses Git for its version control, allowing teams to track changes (both small and large) to a project or a shared job with speed and efficiency.

The API has a different set of endpoints for using Git with shared jobs, described in API v1 - Git integration for shared jobs.

Prerequisites

Consider whether Git integration is appropriate for your Matillion ETL practices with our When to Choose Git article.
For more information about Git integration, read Git integration and Git FAQ.
If you are new to Matillion ETL's API, read our Matillion API introduction and explore the v1 API map.
Git integration in Matillion ETL is an Enterprise Mode feature. Read about the benefits of upgrading your license to:
- Enterprise edition (Hub customers).
- Enterprise mode (marketplace customers).

Using the API

Git integration can be used with shared jobs by making a POST call with the form:

<instanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/<endpoint>

Where:

<instanceAddress> is the URI of your Matillion ETL instance in the form http://host:port.
<groupName> is the name of the Matillion ETL project group.
<projectName> is the name of the Matillion ETL project.
<versionName> is the name of the Matillion ETL project version.
<endpoint> is the name of the API endpoint that accomplishes the task you want to perform. These endpoints are documented in the following sections.

Initialize local repository

Initializes a new repository in this Matillion ETL instance. This request creates an initial Git commit within Matillion ETL.

To initialize a local repository, make the following POST call:

<instanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/init

The POST body should contain the following, in JSON format:

{
    "commitMessage": "<Your Commit Message>",
    "userIdentifier": {
        "username": "<username>",
        "emailAddress": "<emailaddress>"
    }
}

commitMessage: The message to include with your commit. This will be visible on the commit inside the Matillion ETL client.
username: The username to attach to the commit. Note this isn't necessarily your username for the remote repository, nor your Matillion ETL instance username, and doesn't affect anything else.
emailAddress: The email address to attach to the commit (in the format xxx@xxx.xxx). Note this isn't necessarily your email for the remote repository, nor your Matillion ETL instance email, and doesn't affect anything else.

A successful POST call to the endpoint will return the following response from the server:

{
   "success": true,
   "result": "Success"
}

Clone remote repository

Clone an existing remote repository in this Matillion ETL version. Successful requests will initialize from or clone your remote repository.

To clone a remote repository, make the following POST call:

<instanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/clone

The POST body should contain the following, in JSON format:

{
    "remoteURI":"<URL>",
    "auth": {
        "authType": "HTTPS",
        "username": "<username>",
        "password": "<password>"
    }
}

or:

{
    "remoteURI":"<URL>", 
    "auth": { 
        "authType": "SSH", 
        "privateKey": "<key>", 
        "passphrase": "<passphrase>"
    }
}

remoteURI: The URL of your remote Git repository.
authType: Matillion ETL supports HTTPS and SSH.
username: The username to authenticate with the remote repository.
password: The corresponding password for the above username.
privateKey: An SSH private key.
passphrase: The passphrase associated with the SSH private key.

Commit to local repository

To commit changes to a local repository, make the following POST call:

<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/commit

The POST body should contain the following, in JSON format:

{
    "branchName":"<branch>",
    "changes":[{
        "type":"ADDED","filePath":"<path>"
    }],
    "commitMetadata":{"commitMessage":"<Your Commit Message>",
    "userIdentifier":{
        "username":"<username>",
        "emailAddress":"<email address>"
    }
}

branchName: The repository branch that the changes will be committed to. E.g. "default".
filePath: A valid filepath to the changed objects being committed.
commitMessage: A message to include with your commit. This will be visible on the commit inside the Matillion ETL client.
username: The username to attach to the commit. Note this is not necessarily your username for the remote repository, nor your Matillion ETL instance username, and does not affect anything else.
emailAddress: The email address to attach to the commit (in the form xxx@xxx.xxx). Note this isn't necessarily your email for the remote repository, nor your Matillion ETL instance email, and doesn't affect anything else.

Create branch

Create a new branch on the current commit. The POST call will return the branch name along with the commit ID.

To create a new branch on the current commit, make the following POST call:

<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/createBranch

The POST body should contain the following, in JSON format:

{
    "branchName": "<name of new branch>"
}

branchName: The name of the branch to be created.

Switch commit

Switch to an existing commit by passing the commit ID. You can find commit IDs by using the getState endpoint detailed in the Get Commits section of this article.

To switch to another Git commit, make the following POST call:

<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/switchCommit

The POST body should contain the following, in JSON format:

{
    "commitID": "<ID>"
}

commitID: The commit ID of the commit to switch to.

A successful POST call to the endpoint will return the following response from the server:

{
   "success": true,
   "result": "Success"
}

A failed call will return a brief error description, for example:

{
    "success": false,
    "result": "Error while running command: An error occurred while switching commits\nCaused by: ab01418adceb9a6ec51e49ee7d7787dde74220e"
}

Fetch from remote repository

Fetch from a remote repository. A remote repository must be configured before fetch requests can be made.

To perform a fetch from a remote repository, make the following POST call:

<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/fetch

The POST body should contain the following, in JSON format:

{
    "auth": {
        "authType": "HTTPS",
        "username": "<username>",
        "password": "<password>"
    },
    "fetchOptions": {
        "removeDeletedRefs": "<true/false>",
        "thinFetch": "<true/false>"
    }
}

or:

{
    "auth": { 
        "authType": "SSH", 
        "privateKey": "<key>", 
        "passphrase": "<passphrase>"
    },
    "fetchOptions": {
        "removeDeletedRefs": "<true/false>",
        "thinFetch": "<true/false>"
    }
}

authType: Matillion ETL supports HTTPS and SSH.
username: The username to authenticate with the remote repository.
password: The corresponding password for the above username.
privateKey: An SSH private key.
passphrase: The passphrase associated with the SSH private key.
removeDeletedRefs: Choose to remove deleted references ("true" or "false").
thinFetch: Reduces the data sent when the sender and receiver share many of the objects ("true" or "false").

Push to a remote repository

Push to a remote repository.

To push to a remote repository, make the following POST call:

<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/push

The POST body should contain the following, in JSON format:

{
    "auth": {
        "authType": "HTTPS",
        "username": "<username>",
        "password": "<password>"
    },
    "pushOptions": {
        "atomic": "<true/false>",
        "forcePush": "<true/false>",
        "thinPush": "<true/false>"
    }
}

or

{
    "auth": {
        "authType": "SSH", 
        "privateKey": "<key>", 
        "passphrase": "<passphrase>"
    },
    "pushOptions": {
        "atomic": "<true/false>",
        "forcePush": "<true/false>",
        "thinPush": "<true/false>"
    }
}

authType: Matillion ETL supports HTTPS and SSH.
username: The username to authenticate with the remote repository.
password: The corresponding password for the above username.
privateKey: An SSH private key.
passphrase: The passphrase associated with the SSH private key.
atomic: Guarantees that either all references will be pushed on the remote, or none of them will ("true" or "false"). This option avoids partial pushes.
forcePush": Forces the local revision to be pushed into the remote repository ("true" or "false"). This action can cause the remote repository to lose commits, and should be used with caution.
thinPush": Reduces the data sent when the sender and receiver share many of the objects ("true" or "false").

Configure a remote repository

Configure a remote repository.

To configure a remote repository, make the following POST call:

<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/remote

The POST body should contain the following, in JSON format:

{
    "url": "<your_remote_uri>"
}

url: The URL of your remote Git repository.

Get remote

Returns the URL of your remote repository.

To return the URL of your remote repository, make the following GET call:

<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/remote

Get commits

Provides a list of commits on the local repository.

To retrieve a list of each Git commit on the local repository, make the following GET call:

<instance address>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/getState

Get differences

Returns the difference between the current Matillion ETL version (including uncommitted changes) and the head.

To get differences between the current version and the branch head, make the following GET call:

<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/differences

A successful GET call to the endpoint will return the following response from the server:

{
    "success": true,
    "result": {
        "branches": [
            "master"
        ],
        "changes": {
            "dotfileChanges": [],
            "nonDotfileChanges": [
                {
                    "type": "MODIFIED",
                    "filePath": "ROOT/o1.ORCHESTRATION"
                },
                {
                    "type": "MODIFIED",
                    "filePath": "version"
                }
            ]
        }
    }
}

The type response can be ADDED, REMOVED, or MODIFIED.

Contact us

If any help is needed navigating the Git API in Matillion ETL, please read Getting Support.