Skip to content

Projects

A project is a group of resources that Designer uses when running your orchestration and transformation pipelines. From the Your projects menu, select a project, and then navigate between resource tabs to manage resources including:

  • Branches: used for collaborative working and version control using Git. Your default branch is main.
  • Environments: for configuring your connection to your data warehouse.
  • Secret definitions: define secrets stored within your secret manager tool so you can reuse them across projects. Supports AWS Secrets Manager and Azure Key Vault.
  • Cloud Provider Credentials: set up cloud provider credentials to authenticate with your chosen cloud provider.
  • OAuth: create an OAuth entry to connect to your chosen third-party service.
  • Schedules: schedule your orchestration and transformation pipelines. All dates and times in the table use Greenwich Mean time (GMT) as its timezone. If you have set the time using a different timezone, view the individual schedule's details for the selected timezone's details.
  • Access: grant users access to your project.

Add a new project

  1. Log in to Matillion Hub or register for an account.
  2. Select your account.
  3. From the dashboard, click ☰ → Designer.

  4. Click Add new project and complete the following parameters.

    Parameter Description
    Project name A unique name for the project. Max 255 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page.
    Description An optional, referential description for the project. Max 1000 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page.
    Data platform The cloud data platform to connect to. Select Snowflake, Databricks or Amazon Redshift.

    Note

    Some components may not be available, or may have limited functionality, on some data platforms. These limitations are noted in the relevant component documentation.

  5. Click Continue.

  6. Choose how you would like your project to be configured.

    • Matillion managed: A full software-as-a-service (SaaS) deployment model where Matillion handles the infrastructure, maintenance, and upgrades. You can access Matillion's Data Productivity Cloud through a web browser and focus on building data transformation and orchestration workflows without worrying about the underlying infrastructure. This deployment model is the default option for Enterprise and non-enterprise customers.
    • Advanced settings: A hybrid SaaS deployment model that lets you manually deploy and manage Matillion agents, connect to third-party repositories (currently only GitHub is supported), and store your secrets within your own cloud infrastructure, whilst building data transformation and orchestration workflows within Matillion's Data Productivity Cloud. It combines the flexibility of cloud deployment with the control and security of managing agents, repositories, and secrets within your environment. This option is suitable if you have specific requirements or constraints related to data residency, security, or network connectivity.

    Note

    Choosing Advanced settings is considered a "custom" configuration option.

    You can choose to connect to a third-party repository but use a Matillion hosted agent, and conversely choose the hybrid SaaS deployment model of using the Matillion agent in your organization's cloud infrastructure while using a Matillion hosted Git repository.

    Warning

    Some components may have limitations if using a Full SaaS project. These limitations are noted in the relevant component documentation.

  7. Click Continue.

Select your Git provider

The following instructions only apply if you selected Advanced settings.

  1. Select your Git provider.

    1. Matillion-hosted Git: Matillion will create and host the repository for your project.

    2. GitHub: Connect to one of your organization's GitHub repositories. Read the following docs to get started:

      1. Prerequisites for connecting a GitHub repository to the Data Productivity Cloud
      2. Installing the Matillion Data Productivity Cloud app in GitHub

      Note

      If you log in to GitHub using SSO, you are required to click Authorize and not Continue to authenticate to matillion-apps. Clicking Continue won't complete the authorization properly, and you won't be able to perform remote actions. To reauthorize, follow the steps documented in Permission denied to access GitHub repository.

    3. Azure DevOps: Connect to one of your organization's Azure DevOps repositories. Read the following docs to get started:

      1. Prerequisites for connecting an Azure DevOps repository to the Data Productivity Cloud
      2. Installing the Matillion Data Productivity Cloud app in Azure

  2. Click Continue to trigger the authorization flow, if not already authorized. A new browser tab will open.

  3. Click Authorize Data Productivity Cloud. The browser tab will close and return you to the project flow.

  4. Once you've authorized with your Git provider, you'll be taken to the Select repository page, where you'll find the list of repositories you have granted access to the Data Productivity Cloud. You can connect either an existing repository with multiple branches and a large history, or a brand new repository.

    Note

    You can only choose a repository that already exists in your organization or project. The Data Productivity Cloud does not create a new repository in your account.

  5. Select one of the repositories from the list. You can use the search bar to refine your search—for example, if the list contains many repositories.

  6. Click Continue.

Select agent deployment

The following instructions only apply if you selected Advanced settings.

  1. Select how you want to manage your agent and secrets for this project.
    • Full SaaS: Use a Matillion hosted agent.
    • Hybrid SaaS: Host a Matillion agent in your organization's cloud infrastructure. If you choose this option, read Create an agent in your infrastructure.

Create an environment

Parameter Description
Environment name A unique name for the environment.
ETL Agent Use the drop-down menu to select a working agent. This option is only used in a Hybrid SaaS deployment model.

Click Continue.

Specify data warehouse credentials

Parameter Description
Account Enter your Snowflake account name and region (part of the URL you use to log in to Snowflake). Uses the format [accountName].[region_id].
Username Your Snowflake username.
Password For Full SaaS deployment model only. Your Snowflake password.
Vault name For Hybrid SaaS on Azure deployment models only. Select the Azure Key Vault vault that this project will use to store secrets. Select [Default] to use the default key vault specified in the agent environment variables.
Secret name For Hybrid SaaS deployment model only. A named entry created in AWS Secrets Manager or Azure Key Vault.
Secret key For Hybrid SaaS on AWS deployment model only. A named secret key tied to your secret name.
Parameter Description
Instance name Your Databricks instance name. Read the Databricks documentation to learn how to determine your instance name.
Username Your Databricks username.
Password For Full SaaS deployment model only. Your Databricks password.
Vault name For Hybrid SaaS on Azure deployment models only. Select the Azure Key Vault vault that this project will use to store secrets. Select [Default] to use the default key vault specified in the agent environment variables.
Secret name For Hybrid SaaS deployment model only. A named entry created in AWS Secrets Manager or Azure Key Vault.
Secret key For Hybrid SaaS on AWS deployment model only. A named secret key tied to your secret name.
Parameter Description
Endpoint The physical address of the leader node. This will be either a name or an IP address.
Port This is usually 5439 or 5432, but it can be configured differently when setting up your Amazon Redshift cluster.
Use SSL Select this to encrypt communications between Data Productivity Cloud and Amazon Redshift. Some Amazon Redshift clusters may be configured to require this.
Username The username for the environment connection.
Password For Full SaaS deployment model only. Your Redshift password.
Vault name For Hybrid SaaS on Azure deployment models only. Select the Azure Key Vault vault that this project will use to store secrets. Select [Default] to use the default key vault specified in the agent environment variables.
Secret name For Hybrid SaaS deployment model only. A named entry created in AWS Secrets Manager or Azure Key Vault.
Secret key For Hybrid SaaS on AWS deployment model only. A named secret key tied to your secret name.

Click Continue.

Select data warehouse defaults

Property Description
Default role The default Snowflake role for this environment connection. Read Overview of Access Control to learn more.
Default warehouse The default Snowflake warehouse for this environment connection. Read Overview of Warehouses to learn more.
Default database The default Snowflake database for this environment connection. Read Database, Schema, and Share DDL to learn more.
Default schema The default Snowflake schema for this environment connection. Read Database, Schema, and Share DDL to learn more.
Property Description
Endpoint/Cluster The Databricks cluster that Data Productivity Cloud will connect to.
Catalog Choose a Databricks Unity Catalog to connect to.
Schema (Database) Choose a Databricks schema (database) to connect to.

Note

Databricks sometimes use the words Schema and Database interchangeably in their documentation. We always use the word Schema in component parameters.

Property Description
Default database The database you created when setting up your Amazon Redshift cluster. You may run with multiple database names—in which case, choose the one you want to use for this environment.
Default schema This is public by default, but if you have configured multiple schemas within your Amazon Redshift database, you should specify the schema you want to use.
S3 bucket Your default Amazon S3 bucket.

Click Finish.

You will then redirect to your new project's menu. From here, you can cycle tabs for Branches, Environments, Secret definitions, Cloud credentials, OAuth, Schedules, and user Access.

You can edit the project name and description at any time by clicking the pencil buttons next to those items at the top of the screen.

Delete a project

Prior to deleting a project, make sure any active schedules are disabled. If you have any active schedules when you try to delete a project, you'll receive an error message.

To delete a project, follow these steps:

  1. In the Your projects page, click ... next to the intended project.
  2. Click Delete project.
  3. The Delete project dialog will be displayed. Type the name of the project you want to delete in the Enter project name field.
  4. To confirm, click Delete Project.

Note

If you create another project with the same name as a deleted project, make sure the new project's environment name is a different name to the name of the deleted project's environment.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement