Projects
A project is a group of resources that Designer uses when running your orchestration and transformation pipelines. From the Your projects menu, select a project, and then navigate between resource tabs to manage resources including:
- Branches: your default branch is
main. - Environments: for configuring your connection to Snowflake.
- Secret definitions: define secrets stored within your secret manager tool so you can reuse them across projects. Presently, only supports AWS Secrets Manager.
- Cloud Provider Credentials: set up cloud provider credentials to authenticate with your chosen cloud provider.
- OAuth: create an OAuth entry to connect to your chosen third-party service.
- Schedules: schedule your orchestration and transformation pipelines. All dates and times in the table use Greenwich Mean time (GMT) as its timezone. If you have set the time using a different timezone, please view the individual schedule's details for the selected timezone's details.
- Access: grant users access to your project.
Add a new project
Click Add new project.
| Parameter | Description |
|---|---|
| Project name | A unique name for the project. Max 255 characters. |
| Description | An optional, referential description for the project. Max 1000 characters. |
| Data platform | The cloud data platform to connect to. Currently, only supports Snowflake. |
| Use Matillion infrastructure? | Toggle Yes or No. "Yes" is the default option, and means you want to use a Matillion Fully-Managed solution where the agent and secrets are securely hosted by Matillion. "No", means you want to host your own agent, which is known as the Hybrid Cloud deployment model. This is explained in more detail below. |
Deployment models: If you have a Hub account, and subscribe to the Enterprise edition, you can choose the deployment model that best suits your needs:
- Fully-Managed: This is a complete Software-as-a-Service (SaaS) deployment model where Matillion handles the infrastructure, maintenance, and upgrades. You can access Matillion's Data Productivity Cloud through a web browser and focus on building data transformation and orchestration workflows without worrying about the underlying infrastructure. This deployment model is the default option for Enterprise and non-enterprise customers.
- Hybrid Cloud: The Hybrid Cloud deployment model allows you to deploy and manage Matillion agents, and store your secrets within your own cloud infrastructure, whilst building data transformation and orchestration workflows within Matillion's Data Productivity Cloud. It combines the flexibility of cloud deployment with the control and security of managing agents and secrets within your environment. This option is suitable if you have specific requirements or constraints related to data residency, security, or network connectivity. For more information, read create an agent.
Note
Some components may have limitations if using a Fully-Managed project. These limitations are noted in the relevant component documentation.
Click Continue.
Create an environment
| Parameter | Description |
|---|---|
| Environment name | A unique name for the environment. |
| ETL Agent | Use the drop-down menu to select a working agent. This option will be available if you have toggled No in the previous step, when asked if you would like to use Matillion infrastructure. |
Click Continue.
Specify Snowflake credentials
| Parameter | Description |
|---|---|
| Account | Enter your Snowflake account name and region (part of the URL you use to log into Snowflake). Uses the format [accountName].[region_id]. |
| Username | Your Snowflake username. |
| Password | For Fully-Managed deployment model only. Your Snowflake password. |
| Secret name | For Hybrid Cloud deployment model only. A named entry created in AWS Secrets Manager. |
| Secret key | For Hybrid Cloud deployment model only. A named secret key tied to your secret name. |
Click Continue.
Select Snowflake defaults
| Property | Description |
|---|---|
| Default role | The default Snowflake role for this environment connection. Read Overview of Access Control to learn more. |
| Default warehouse | The default Snowflake warehouse for this environment connection. Read Overview of Warehouses to learn more. |
| Default database | The default Snowflake database for this environment connection. Read Database, Schema, and Share DDL to learn more. |
| Default schema | The default Snowflake schema for this environment connection. Read Database, Schema, and Share DDL to learn more. |
Click Finish.
You will then redirect to your new project's menu. From here, you can cycle tabs for Branches, Environments, Secret definitions, Cloud Provider Credentials, OAuth, Schedules, and user Access.