Skip to content

Using Git to implement SDLC best practices

This page covers how to use Designer with a Git provider to supplement your DevOps processes.

Through this process we will be creating branches from the main branch to represent a "dev" branch and a "test" branch, respectively. You can repeat these steps to create as many new branches as required, for whatever purpose you need. We also demonstrate higher-level Git functionality, such as requesting reviews, approvals, and merging changes back to the main branch for other users to access.

The walkthrough uses various names for repositories, branches, projects, etc. Consider these values to be placeholders and feel free to replace them with your own, if required. Ensure you are consistent throughout, as certain steps in the process reference earlier ones. Read Git in Designer for more information.

Completing these steps will enable the following:

  • All branches created on the Data Productivity Cloud are automatically synced with your Github repository. The dev and test branches will be added to the list of branches on the selected GitHub repository.
  • When a Data engineer makes a changes and pushes local changes, the changes are automatically pushed to the exact branch name on the GitHub repository.
  • Reviewers and lead data engineers can review pull requests before merging them to the dev branch. This process can be extended to both the test and main branches.

Prerequisites

You will need:

Connect GitHub to the Data Productivity Cloud

  1. Read

Step 1 - Connect your Data Productivity Cloud to GitHub

  1. Log in to your GitHub account.
  2. Create the repository where you want to store your Data Productivity Cloud pipelines. This is the repository that you will connect your Data Productivity Cloud instance to. For the purposes of this example, we will create a repository called matillion-project-demo.
  3. Locate the Matillion app in the GitHub marketplace.
  4. Begin installation of the Matillion app to your selected GitHub org. This can only be performed by a user with admin privileges.
  5. Select All repositories or Only select repositories:
    1. If you select All repositories, you will be able to select any current or new repositories in your GitHub org when you create a new project in Data Productivity Cloud. The Matillion app acts on your behalf, therefore, you need to grant it access to the repositories you want it to have access to.
    2. If you select Only select repositories, choose the repository that you created earlier.
  6. Install Matillion app.

Step 2 - Create a Project linked to your GitHub repository

  1. Log in to the Data Productivity Cloud.
  2. In the left navigation, click the Projects icon .
  3. Click Add project. Complete the steps as required. For the purposes of this example we will use a project called my-project. Read Projects for information about creating projects.
  4. And select GitHub repository as the repository type.
  5. Choose the repository you created earlier, matillion-project-demo.
  6. Complete the remaining project creation steps.

Step 3 - Create dev and test branches

  1. Navigate to my-project in the Data Productivity Cloud.
  2. Select Add new branch.
  3. Give the branch the name test.
  4. Select main from the Branch from dropdown. This will create a new branch by copying files from the main branch.
  5. Select an environment.
  6. Repeat steps 2 through 5 to create the dev branch.

Step 4 - Creating a feature branch and making changes to pipelines

At this point we have now designated the main, test, and dev branches for production, testing, and development pipelines respectively. Data Engineers can create branches, make changes, then create a Pull request to request for reviews before a branch is merged to another branch, which is described as follows:

  1. Create a new branch named, for example, feature-test.
  2. Select dev from the Branch from dropdown menu.
  3. Open the feature-test branch. You will be redirected to Designer.
  4. Modify your pipelines as required.
  5. Click the name of your branch on the project bar.
  6. Commit changes. At this point, the changes you've made are still local and only accessible to you. Read Commit for more information.
  7. Select Push local changes from the name of your branch on the project bar. Pushing the changes stores them on the GitHub repository and makes the file available to your colleagues.

Step 5 - Requesting a review

To request a review of the changes you've made, you need to create a Pull Request on GitHub. Follow the steps below to create a Pull Request. This is an optional process that helps teams streamline their workflows and ensure that teammates can review work done by colleagues before adding changes to a production environment.

  1. Log in to your GitHub account.
  2. Navigate to the main page of the matillion-project-demo repository you created.
  3. In the Branch menu, choose the feature-branch which is the branch that contains the new changes.
  4. Above the list of files, in the yellow banner, click Compare & pull request to create a pull request for the associated branch.
  5. Use the base branch dropdown menu to select the dev branch, which is the branch you'd like to merge your changes into. Then use the Compare branch drop-down menu to choose the topic branch you made your changes in.
  6. Enter a Title and Description for your pull request.

Read GitHub's documentation for more information.

Step 6: Reviewing and approving changes

To review a proposed change by your colleague on GitHub, follow the official GitHub documentation page on reviewing pull requests.

Step 7: Testing the changes on the dev branch in the Data Productivity Cloud

When changes are approved and merged to the dev branch on GitHub, the changes are not automatically synced to your dev branch on the Data Productivity Cloud. To view the new changes on GitHub:

  1. Log in to the Data Productivity Cloud.
  2. From the Data Productivity Cloud homepage, navigate to my-project.
  3. Open the dev branch. Changes on GitHub are currently isolated from the dev branch because we have not yet performed a pull.
  4. Click the name of your branch on the project bar.
  5. Select Pull remote changes. This adds any new changes merged to the GitHub repository to your dev branch on the Data Productivity Cloud.
  6. Click validate and run to test the pipelines on the Data Productivity Cloud.

Repeat steps 3 – 6 to merge the dev branch to test and the test branch to main, respectively.

Got feedback or spotted something we can improve?

We'd love to hear from you. Join the conversation in the Documentation forum!

© Copyright 2025 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement