Skip to content

Projects

A project is a group of resources that Designer uses when running your orchestration and transformation pipelines. From the Your projects menu, select a project, and then navigate between resource tabs to manage resources including:

  • Branches: used for collaborative working and version control using Git. Your default branch is main.
  • Environments: for configuring your connection to your data warehouse, and setting environment permissions for users.
  • Streaming: build streaming pipelines to capture the continuous data flow from supported sources in near real-time and send it to your preferred cloud destination. To create and manage your streaming pipelines, read Create streaming pipeline and Manage streaming pipelines, respectively. You can also click View Features to access a dialog outlining the key capabilities of streaming pipelines in the Data Productivity Cloud. Additionally, you can reach out for support by filling out the Talk to an Expert form to contact sales.
  • Artifacts: view published versions of pipelines called 'artifacts' in each environment, and deploy artifacts to different environments.
  • Schedules: schedule your orchestration and transformation pipelines. All dates and times in the table use Greenwich Mean time (GMT) as its timezone. If you have set the time using a different timezone, view the individual schedule's details for the selected timezone's details.
  • Access: grant users access to your project.
  • Secret definitions: define secrets stored within your secret manager tool so you can reuse them across projects. Supports AWS Secrets Manager and Azure Key Vault.
  • Cloud credentials: set up cloud provider credentials to authenticate with your chosen cloud provider.
  • OAuth: create an OAuth entry to connect to your chosen third-party service.

Add a new project

  1. Log in to Matillion Data Productivity Cloud or register for an account.
  2. Select your account.
  3. From the dashboard, click ☰ → Designer.

  4. Click Add new project and complete the following parameters.

    Parameter Description
    Project name A unique name for the project. Max 255 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page.
    Description An optional, referential description for the project. Max 1000 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page.
    Data platform The cloud data platform to connect to. Select Snowflake, Databricks or Amazon Redshift.

    Note

    Some components may not be available, or may have limited functionality, on some data platforms. These limitations are noted in the relevant component documentation.

  5. Click Continue.

  6. Choose how you would like your project to be configured.

    • Matillion managed: A full software-as-a-service (SaaS) deployment model where Matillion handles the infrastructure, maintenance, and upgrades. You can access Matillion's Data Productivity Cloud through a web browser and focus on building data transformation and orchestration workflows without worrying about the underlying infrastructure. This deployment model is the default option for Enterprise and non-enterprise customers.
    • Advanced settings: A Hybrid SaaS deployment model that lets you manually deploy and manage Matillion agents, connect to third-party repositories (currently only GitHub is supported), and store your secrets within your own cloud infrastructure, whilst building data transformation and orchestration workflows within Matillion's Data Productivity Cloud. It combines the flexibility of cloud deployment with the control and security of managing agents, repositories, and secrets within your environment. This option is suitable if you have specific requirements or constraints related to data residency, security, or network connectivity.

    Note

    Choosing Advanced settings is considered a "custom" configuration option.

    You can choose to connect to a third-party repository but use a Matillion-hosted agent, and conversely choose the Hybrid SaaS deployment model of using the Matillion agent in your organization's cloud infrastructure while using a Matillion hosted Git repository.

    Warning

    Some components may have limitations if using a Full SaaS project. These limitations are noted in the relevant component documentation.

  7. Click Continue.

Select your Git provider

The following instructions only apply if you selected Advanced settings.

  1. Select your Git provider.

    1. Matillion-hosted Git: Matillion will create and host the repository for your project.

    2. GitHub: Connect to one of your organization's GitHub repositories. Read the following docs to get started:

      1. Prerequisites for connecting a GitHub repository to the Data Productivity Cloud
      2. Installing the Matillion Data Productivity Cloud app in GitHub

      Note

      If you log in to GitHub using SSO, you are required to click Authorize and not Continue to authenticate to matillion-apps. Clicking Continue won't complete the authorization properly, and you won't be able to perform remote actions. To reauthorize, follow the steps documented in Permission denied to access GitHub repository.

    3. Azure DevOps: Connect to one of your organization's Azure DevOps repositories. Read the following docs to get started:

      1. Prerequisites for connecting an Azure DevOps repository to the Data Productivity Cloud
      2. Installing the Matillion Data Productivity Cloud app in Azure
    4. GitLab: Connect to one of your organization's GitLab repositories. Read the following guide to get started:
      1. Prerequisites for connecting a GitLab repository to the Data Productivity Cloud
    5. Bitbucket: Connect to one of your organization's Bitbucket repositories. Read the following guide to get started:
      1. Prerequisites for connecting a Bitbucket repository to the Data Productivity Cloud

  2. Click Continue to trigger the authorization flow, if not already authorized. A new browser tab will open.

  3. Click Authorize Data Productivity Cloud. The browser tab will close and return you to the project flow.

    Note

    For GitHub only. On successful authorizaton, you will see a modal with a list of your authorized accounts. Click Continue.

  4. Once you've authorized with your Git provider, you'll be taken to the Select repository page, where you'll find the list of repositories you have granted access to the Data Productivity Cloud. You can connect either an existing repository with multiple branches and a large history, or a brand new repository.

    Note

    You can only choose a repository that already exists in your organization or project. The Data Productivity Cloud does not create a new repository in your account.

  5. Select one of the repositories from the list. You can use the search bar to refine your search—for example, if the list contains many repositories.

  6. Click Continue.

Select agent deployment

The following instructions only apply if you selected Advanced settings.

  1. Select how you want to manage your agent and secrets for this project.
    • Full SaaS: Use a Matillion-hosted agent.
    • Hybrid SaaS: Host a Matillion agent in your organization's cloud infrastructure. If you choose this option, read Create an agent in your infrastructure.

Create an environment

Parameter Description
Environment name A unique name for the environment.
Agent Use the drop-down menu to select a working agent. This option is only used in a Hybrid SaaS deployment model.
Default environment access Use the drop-down menu to select the default access for all new and existing users added to the project. For more information, read Environment access.

Click Continue.

Specify cloud data warehouse credentials

Use the reference tables below to set up your environment connection to your cloud data platform. If you're using a Full SaaS deployment, credentials such as passwords and private keys are stored directly as strings. However, if you're using a Hybrid SaaS deployment with your own AWS or Azure agent, credentials such as passwords and private keys are only retrieved via references to secrets created in either AWS Secrets Manager or Azure Key Vault.

Snowflake

Password

Refer to this table if you're using your Snowflake password to authenticate to Snowflake.

Parameter Description
Account Enter your Snowflake account name and region (part of the URL you use to log in to Snowflake). Uses the format [accountName].[region_id].
Credentials type Select Username and password.
Username Your Snowflake username.
Password Your Snowflake password. This field is only available if using a Full SaaS deployment; otherwise, you will specify your password as a secret.
Vault name For Hybrid SaaS on Azure deployment models only. Select the Azure Key Vault vault that this project will use to store secrets. Select [Default] to use the default key vault specified in the agent environment variables.
Secret name A named entry created in AWS Secrets Manager or Azure Key Vault for holding your Snowflake password. Field only available if using a Hybrid SaaS deployment model.
Secret key A named secret key tied to your secret name. Field only available if using a Hybrid SaaS on AWS deployment model.

Key-pair

Refer to this table if you're using Snowflake key-pair authentication.

Parameter Description
Account Enter your Snowflake account name and region (part of the URL you use to log in to Snowflake). Uses the format [accountName].[region_id].
Credentials type Select Key pair.
Username Your Snowflake username.
Private key Your Snowflake private key. To generate a key, read the Snowflake documentation for Generate the private key. The full content of the generated Snowflake private key file must be copied into this field, including the header and footer lines. Field only available if Credentials type is Key pair and when using a Full SaaS deployment model.
Passphrase An optional passphrase to use with your private key. Field only available if Credentials type is Key pair and when using a Full SaaS deployment model.
Vault name For Hybrid SaaS on Azure deployment models only. Select the Azure Key Vault vault that this project will use to store secrets. Select [Default] to use the default key vault specified in the agent environment variables.
Private key secret name A named entry created in AWS Secrets Manager or Azure Key Vault denoting the secret that holds your Snowflake private key. Read Using snowflake key-pair authentication to learn how to store the key as a secret. Field only available if using a Hybrid SaaS deployment model.
Passphrase secret name (optional) A named entry created in AWS Secrets Manager or Azure Key Vault denoting the secret that holds your Snowflake keypair passphrase. Field only available if using a Hybrid SaaS deployment model.
Passphrase secret key (optional) The secret key tied to your passphrase secret name. Field only available if using a Hybrid SaaS deployment model.

Databricks

Parameter Description
Instance name Your Databricks instance name. Read the Databricks documentation to learn how to determine your instance name.
Personal Access Token Your Databricks personal access token. Read the Databricks documentation to learn how to create a personal access token.
Vault name For Hybrid SaaS on Azure deployment models only. Select the Azure Key Vault vault that this project will use to store secrets. Select [Default] to use the default key vault specified in the agent environment variables.
Secret name A named entry created in AWS Secrets Manager or Azure Key Vault.
Secret key For Hybrid SaaS on AWS deployment model only. A named secret key tied to your secret name.

Amazon Redshift

Parameter Description
Endpoint The physical address of the leader node. This will be either a name or an IP address.
Port This is usually 5439 or 5432, but it can be configured differently when setting up your Amazon Redshift cluster.
Use SSL Select this to encrypt communications between Data Productivity Cloud and Amazon Redshift. Some Amazon Redshift clusters may be configured to require this.
Username The username for the environment connection.
Password For Full SaaS deployment model only. Your Redshift password.
Vault name For Hybrid SaaS on Azure deployment models only. Select the Azure Key Vault vault that this project will use to store secrets. Select [Default] to use the default key vault specified in the agent environment variables.
Secret name For Hybrid SaaS deployment model only. A named entry created in AWS Secrets Manager or Azure Key Vault.
Secret key For Hybrid SaaS on AWS deployment model only. A named secret key tied to your secret name.

Click Continue.

Select data warehouse defaults

Snowflake

Parameter Description
Default role The default Snowflake role for this environment connection. Read Overview of Access Control to learn more.
Default warehouse The default Snowflake warehouse for this environment connection. Read Overview of Warehouses to learn more.
Default database The default Snowflake database for this environment connection. Read Database, Schema, and Share DDL to learn more.
Default schema The default Snowflake schema for this environment connection. Read Database, Schema, and Share DDL to learn more.

Databricks

Parameter Description
Endpoint/Cluster The Databricks cluster that Data Productivity Cloud will connect to.
Catalog Choose a Databricks Unity Catalog to connect to.
Schema Choose a Databricks schema to connect to.

Amazon Redshift

Parameter Description
Default database The database you created when setting up your Amazon Redshift cluster. You may run with multiple database names—in which case, choose the one you want to use for this environment.
Default schema This is public by default, but if you have configured multiple schemas within your Amazon Redshift database, you should specify the schema you want to use.

Click Finish.

You will then redirect to your new project's menu. From here, you can cycle tabs for Branches, Environments, Secret definitions, Cloud credentials, OAuth, Schedules, and user Access.

You can edit the project name and description at any time by clicking the pencil buttons next to those items at the top of the screen.

Delete a project

Prior to deleting a project, make sure any active schedules are disabled. If you have any active schedules when you try to delete a project, you'll receive an error message.

To delete a project, follow these steps:

  1. In the Your projects page, click ... next to the intended project.
  2. Click Delete project.
  3. The Delete project dialog will be displayed. Type the name of the project you want to delete in the Enter project name field.
  4. To confirm, click Delete Project.

Note

If you create another project with the same name as a deleted project, make sure the new project's environment name is a different name to the name of the deleted project's environment.

If necessary, you can enable and disable sampling at the project level. This is useful if your data contains personal information that cannot be viewed outside your region. Only users with project admin permissions can change this setting.

To enable or disable sampling for a project:

  1. In the Your projects page, click ... next to the intended project.
  2. Click Enable sampling or Disable sampling as required.
© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement