Projects
A project is a group of resources that Designer uses when running your orchestration and transformation pipelines. From the Your projects menu, select a project, and then navigate between resource tabs to manage resources including:
- Branches: used for collaborative working and version control using Git. Your default branch is
main. - Environments: for configuring your connection to your data warehouse.
- Secret definitions: define secrets stored within your secret manager tool so you can reuse them across projects. Presently, only supports AWS Secrets Manager.
- Cloud Provider Credentials: set up cloud provider credentials to authenticate with your chosen cloud provider.
- OAuth: create an OAuth entry to connect to your chosen third-party service.
- Schedules: schedule your orchestration and transformation pipelines. All dates and times in the table use Greenwich Mean time (GMT) as its timezone. If you have set the time using a different timezone, please view the individual schedule's details for the selected timezone's details.
- Access: grant users access to your project.
Add a new project
-
Click Add new project and complete the following parameters.
Parameter Description Project name A unique name for the project. Max 255 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page. Description An optional, referential description for the project. Max 1000 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page. Data platform The cloud data platform to connect to. Select Snowflake, Databricks or Amazon Redshift. Note
Some components may not be available, or may have limited functionality, on some data platforms. These limitations are noted in the relevant component documentation.
-
Click Continue.
-
Choose how you would like your project to be configured.
- Matillion managed: A full software-as-a-service (SaaS) deployment model where Matillion handles the infrastructure, maintenance, and upgrades. You can access Matillion's Data Productivity Cloud through a web browser and focus on building data transformation and orchestration workflows without worrying about the underlying infrastructure. This deployment model is the default option for Enterprise and non-enterprise customers.
- Advanced settings: A hybrid SaaS deployment model that lets you manually deploy and manage Matillion agents, connect to third-party repositories (currently only GitHub is supported), and store your secrets within your own cloud infrastructure, whilst building data transformation and orchestration workflows within Matillion's Data Productivity Cloud. It combines the flexibility of cloud deployment with the control and security of managing agents, repositories, and secrets within your environment. This option is suitable if you have specific requirements or constraints related to data residency, security, or network connectivity.
Note
Choosing Advanced settings is considered a "custom" configuration option.
You can choose to connect to a third-party repository but use a Matillion hosted agent, and conversely choose the hybrid SaaS deployment model of using the Matillion agent in your organization's cloud infrastructure while using a Matillion hosted Git repository.
Warning
Some components may have limitations if using a Full SaaS project. These limitations are noted in the relevant component documentation.
-
Click Continue.
Select your repository type
The following instructions only apply if you selected Advanced settings.
-
Select the type of repository you want to use.
- Matillion repository: Use a Matillion hosted Git repository.
- GitHub repository: Use a GitHub repository within your company's GitHub organization. Choosing this option will activate the Authorize with GitHub dialog. Click Authorize and a new browser tab will open. Click the green Authorize Data Productivity Cloud. The browser tab will close and return you to project flow. Click Continue.
If you choose to connect to a GitHub repository, read Prerequisites for connecting a GitHub repository to the Data Productivity Cloud followed by Installing the Matillion Data Productivity Cloud app in GitHub.
Note
If you log in to GitHub using SSO, you are required to click Authorize and not Continue to authenticate to matillion-apps. Clicking Continue won't complete the authorization properly, and you won't be able to perform remote actions. To reauthorize, follow the steps documented in Permission denied to access GitHub repository.
- If you have successfully authorized with GitHub, a drop-down menu will activate, listing any available GitHub repositories. You can connect either an existing repository with multiple branches and a large history, or a brand new repository.
- Select one of the repositories from the drop-down.
- Click Continue.
Select agent deployment
The following instructions only apply if you selected Advanced settings.
- Select how you want to manage your agent and secrets for this project.
- Full SaaS: Use a Matillion hosted agent.
- Hybrid SaaS: Host a Matillion agent in your organization's cloud infrastructure. If you choose this option, read Create an agent in your infrastructure.
Create an environment
| Parameter | Description |
|---|---|
| Environment name | A unique name for the environment. |
| ETL Agent | Use the drop-down menu to select a working agent. This option is only used in a Hybrid SaaS deployment model. |
Click Continue.
Specify data warehouse credentials
| Parameter | Description |
|---|---|
| Account | Enter your Snowflake account name and region (part of the URL you use to log in to Snowflake). Uses the format [accountName].[region_id]. |
| Username | Your Snowflake username. |
| Password | For Full SaaS deployment model only. Your Snowflake password. |
| Secret name | For Hybrid SaaS deployment model only. A named entry created in AWS Secrets Manager. |
| Secret key | For Hybrid SaaS deployment model only. A named secret key tied to your secret name. |
| Parameter | Description |
|---|---|
| Instance name | Your Databricks instance name. Read the Databricks documentation to learn how to determine your instance name. |
| Username | Your Delta Lake user name. |
| Password | For Full SaaS deployment model only. Your Databricks password. |
| Secret name | For Hybrid SaaS deployment model only. A named entry created in AWS Secrets Manager. |
| Secret key | For Hybrid SaaS deployment model only. A named secret key tied to your secret name. |
| Parameter | Description |
|---|---|
| Endpoint | The physical address of the leader node. This will be either a name or an IP address. |
| Port | This is usually 5439 or 5432, but it can be configured differently when setting up your Amazon Redshift cluster. |
| Use SSL | Select this to encrypt communications between Data Productivity Cloud and Amazon Redshift. Some Amazon Redshift clusters may be configured to require this. |
| Username | The username for the environment connection. |
| Password | For Full SaaS deployment model only. Your Redshift password. |
| Secret name | For Hybrid SaaS deployment model only. A named entry created in AWS Secrets Manager. |
| Secret key | For Hybrid SaaS deployment model only. A named secret key tied to your secret name. |
Click Continue.
Select data warehouse defaults
| Property | Description |
|---|---|
| Default role | The default Snowflake role for this environment connection. Read Overview of Access Control to learn more. |
| Default warehouse | The default Snowflake warehouse for this environment connection. Read Overview of Warehouses to learn more. |
| Default database | The default Snowflake database for this environment connection. Read Database, Schema, and Share DDL to learn more. |
| Default schema | The default Snowflake schema for this environment connection. Read Database, Schema, and Share DDL to learn more. |
| Property | Description |
|---|---|
| Endpoint/Cluster | The Databricks cluster that Data Productivity Cloud will connect to. |
| Catalog | Choose a Databricks Unity Catalog to connect to. |
| Database | Choose a Delta Lake database to connect to. |
| Property | Description |
|---|---|
| Default database | The database you created when setting up your Amazon Redshift cluster. You may run with multiple database names—in which case, choose the one you want to use for this environment. |
| Default schema | This is public by default, but if you have configured multiple schemas within your Amazon Redshift database, you should specify the schema you want to use. |
Click Finish.
You will then redirect to your new project's menu. From here, you can cycle tabs for Branches, Environments, Secret definitions, Cloud credentials, OAuth, Schedules, and user Access.
You can edit the project name and description at any time by clicking the pencil buttons next to those items at the top of the screen.