Skip to content

API v1 - Git integration for shared jobs

This article is a guide to using the source control management API in Matillion ETL with shared jobs. Matillion ETL uses Git for its version control, allowing teams to track changes (both small and large) to a shared job with speed and efficiency.

Note

The examples in this article are for use with shared jobs only. The API has a different set of endpoints for using Git with projects: API v1 - Git integration for projects.

Prerequisites

Using the API

Git integration can be used with shared jobs by making a POST call with the form:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/<endpoint>

Where:

  • <instanceAddress> is the URI of your Matillion ETL instance in the form http://host:port.
  • <endpoint> is the name of the API endpoint that accomplishes the task you want to perform. These endpoints are documented in the following sections.

Initialize local repository

Initializes a new shared jobs repository in this Matillion ETL instance. This request creates an initial Git commit within Matillion ETL.

To initialize a local repository, make the following POST call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/init

The POST body should contain the following, in JSON format:

{
    "commitMessage": "<Your Commit Message>",
    "userIdentifier": {
        "username": "<username>",
        "emailAddress": "<emailaddress>"
    }
}
  • commitMessage: The message to include with your commit. This will be visible on the commit inside the Matillion ETL client.
  • username: The username to attach to the commit. Note this isn't necessarily your username for the remote repository, nor your Matillion ETL instance username, and doesn't affect anything else.
  • emailAddress: The email address to attach to the commit (in the format xxx@xxx.xxx). Note this isn't necessarily your email for the remote repository, nor your Matillion ETL instance email, and doesn't affect anything else.

A successful POST call to the endpoint will return the following response from the server:

{
   "success": true,
   "result": "Success"
}

Clone remote repository

Clone an existing a remote repository for shared jobs in this Matillion ETL instance. Successful requests will initialize from or clone your remote repository.

To clone a remote repository, make the following POST call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/clone

The POST body should contain the following, in JSON format:

{
    "remoteURI":"<URL>",
    "auth": {
        "authType": "HTTPS",
        "username": "<username>",
        "password": "<password>"
    }
}

or:

{
    "remoteURI":"<URL>", 
    "auth": { 
        "authType": "SSH", 
        "privateKey": "<key>", 
        "passphrase": "<passphrase>"
    }
}
  • remoteURI: The URL of your remote Git repository for shared jobs.
  • authType: Matillion ETL supports HTTPS and SSH for authentication. The type specified here will determine which other parameters must be passed.
  • username: The username to authenticate with the remote repository.
  • password: The corresponding password for the above username.
  • privateKey: An SSH private key.
  • passphrase: The passphrase associated with the SSH private key.

Commit changes to local repository

Commit shared job changes to the local repository.

To commit changes to the local repository, make the following POST call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/commit

The POST body should contain the following, in JSON format:

{
    "branchName":"<branch>",
    "changes":[{
        "type":"ADDED",
        "filePath":"<path>"
    }],
    "commitMetadata":{
        "commitMessage":"<Your Commit Message>",
        "userIdentifier":{
            "username":"<username>",
            "emailAddress":"<email address>"
        }
    }
}
  • branchName: The repository branch that the changes will be committed to, for example default.
  • filePath: A valid filepath to the changed objects being committed.
  • commitMessage: A message to include with your commit. This will be visible on the commit inside the Matillion ETL client.
  • username: The username to attach to the commit. Note this is not necessarily your username for the remote repository, nor your Matillion ETL instance username, and doesn't affect anything else.
  • emailAddress: The email address to attach to the commit (in the format xxx@xxx.xxx). Note this isn't necessarily your email for the remote repository, nor your Matillion ETL instance email, and doesn't affect anything else.

Switch commit

Switch to an existing commit by passing the commit ID. You can find commit IDs by using the endpoint described in Get commits.

To switch to another Git commit, make the following POST call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/switchCommit

The POST body should contain the following, in JSON format:

{
    "commitID":"<ID>"
}
  • commitID: The commit ID of the commit to switch to.

A successful POST call to the endpoint will return the following response from the server:

{
   "success": true,
   "result": "Success"
}

A failed call will return a brief error description, for example:

{
   "success": false,
   "result": "Error while running command: An error occurred while switching commits\nCaused by: ab01418adceb9a6ec51e49ee7d7787dde74220e"
}

Get commits

Retrieve a list of each Git commit on the local repository.

To retrieve a list of commits, make the following GET call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/getState

Fetch from remote repository

Fetch from a remote repository. A remote repository must be configured before fetch requests can be made.

To perform a fetch from a remote repository, make the following POST call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/fetch

The POST body should contain the following, in JSON format:

{
    "auth": {
        "authType": "HTTPS",
        "username": "<username>",
        "password": "<password>"
    },
    "fetchOptions": {
        "removeDeletedRefs": "<true/false>",
        "thinFetch": "<true/false>"
    }
}

or

{
    "auth": { 
        "authType": "SSH", 
        "privateKey": "<key>", 
        "passphrase": "<passphrase>"
    },
    "fetchOptions": {
        "removeDeletedRefs": "<true/false>",
        "thinFetch": "<true/false>"
    }
}
  • authType: Matillion ETL supports HTTPS and SSH for authentication. The type specified here will determine which other parameters must be passed.
  • username: The username to authenticate with the remote repository.
  • password: The corresponding password for the above username.
  • privateKey: An SSH private key.
  • passphrase: The passphrase associated with the SSH private key.
  • removeDeletedRefs: Choose to remove deleted references ("true") or not ("false").
  • thinFetch: Reduce the data sent when the sender and receiver share many of the objects ("true"), or do not reduce the data ("false").

Push to a remote repository

Push commits to a remote repository.

To push to a remote repository, make the following POST call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/push

The POST body should contain the following, in JSON format:

{
    "auth": {
        "authType": "HTTPS",
        "username": "<username>",
        "password": "<password>"
    },
    "pushOptions": {
        "atomic": "<true/false>",
        "forcePush": "<true/false>",
        "thinPush": "<true/false>"
    }
}

or

{
    "auth": {
        "authType": "SSH", 
        "privateKey": "<key>", 
        "passphrase": "<passphrase>"
    },
    "pushOptions": {
        "atomic": "<true/false>",
        "forcePush": "<true/false>",
        "thinPush": "<true/false>"
    }
}
  • authType: Matillion ETL supports HTTPS and SSH for authentication. The type specified here will determine which other parameters must be passed.
  • username: The username to authenticate with the remote repository.
  • password: The corresponding password for the above username.
  • privateKey: An SSH private key.
  • passphrase: The passphrase associated with the SSH private key.
  • atomic: If "true", guarantees that either all references will be pushed on the remote, or none of them will. This avoids partial pushes. Set to "false" if partial pushes are to be allowed.
  • forcePush: Forces the local revision to be pushed into the remote repository ("true"). This action can cause the remote repository to lose commits, and should be used with caution. Set to "false" to prevent a forced push.
  • thinPush: Reduce the data sent when the sender and receiver share many of the objects ("true") or do not reduce the data ("false").

Get differences

Returns the difference between the current version (including uncommitted changes) and the head.

To get differences, make the following GET call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/differences

The following is an example of the server response to the GET call:

{
    "success": true,
    "result": {
        "branches": [
            "master"
        ],
        "changes": {
            "dotfileChanges": [],
            "nonDotfileChanges": [
                {
                    "type": "MODIFIED",
                    "filePath": "ROOT/o1.ORCHESTRATION"
                },
                {
                    "type": "MODIFIED",
                    "filePath": "version"
                }
            ]
        }
    }
}

The type response can be ADDED, REMOVED, or MODIFIED.

Set remote URL

Sets the remote URL for the shared jobs repository.

To set the remote repository URL, make the following POST call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/remote

The POST body should contain the following, in JSON format:

{
    "url": "<URL>"
}
  • url: The URL of the remote repository.

Create branch

Create a new branch on the current commit.

To create a new branch, make the following POST call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/branch

The POST body should contain the following, in JSON format:

{
    "branchName": "<branchName>"
}
  • branchName: The name of the branch to be created.

Get remote

Returns the URL of the remote repository.

To return the URL of your remote repository, make the following GET call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/remote

Return state

Return the history of commits in all branches of the repository.

To return the URL of your remote repository, make the following GET call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/state/<max_count>/<skip>

Where:

  • <max_count> is the number of commits to be returned.
  • <skip> is the number of commits to skip before returning the <max_count> commits.

Delete branch

Deletes a selected branch from the repository.

To delete a branch, make the following DELETE call:

<instanceAddress>/rest/v1/git/shared-jobs/branch/<branchName>

Where <branchName> is the name of the branch to be deleted.

Delete the repository

Deletes the entire repository.

To delete the repository, make the following DELETE call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs

Merge

Merge a branch, assuming there are no conflicts. If conflict occurs, the server returns a list of conflicts. If the request contains conflict resolutions in the conflictsResolution field of the POST body, then the resolutions are applied.

To merge a branch, make the following POST call:

<instanceAddress>/rest/v1/scm/repository/shared-jobs/merge

The POST body should contain the following, in JSON format:

{
    "commitMetadata": {
        "commitMessage": "<message>",
        "userIdentifier": {
            "username": "<name>",
            "emailAddress": "<email>"
        }
    },
    "checkoutAfterMerge": "<true | false>",
    "mergeFrom": "<source branch>",
    "mergeInTo": "<target branch>",
    "conflictsResolution": [
        {
            "package":"<package>",
            "name": "<name>",
            "revision": "<number>",
            "choice" : "<OURS | THEIRS>"
        }
    ]
}

The fields in this POST body match the fields used Matillion ETL MergeManager. See that article for more information.

Contact us

If any help is needed navigating the Git API in Matillion ETL, please read Getting Support.