Skip to content

Designer overview

Matillion Designer lets you create your own complex ETL pipeline workflows that can be scheduled and run in a Full SaaS or Hybrid SaaS architecture.

  • Full SaaS: Use a Matillion hosted agent and secure secrets manager to execute your ETL tasks. This is the recommended starting solution.
  • Hybrid SaaS: Host your own agent and secrets in your AWS cloud. This requires additional setup. See Agent Setup for details.

Note

Some components may have limitations when using a Full SaaS solution. These limitations are noted in the relevant component documentation.

You can access Designer through Design data pipelines on the Hub, or through the Designer option on the app launcher menu in the top-left.

You can also open a pipeline in Designer by clicking its tile under Recent pipelines on the Hub.

Cloud data warehouses

Designer currently supports these cloud data warehouses:

You'll need your account credentials and connection details for the cloud data warehouse you use.

Note

Some components are limited or not available with specific data warehouses. These limitations are noted in the relevant component documentation.

Getting started

Getting started with Designer is simple. With Matillion's Full SaaS solution, an agent and secret store will be provided so you can skip setting up infrastructure and get right into designing pipelines.

  1. Log in to the Hub.
  2. Choose an account.
  3. Click Design data pipelines from the Hub, or Designer from the app launcher menu in the top-left.
  4. Provide some information about yourself as the administrator of this new account.
  5. You will receive notice that your agent is being created. A preset project name will be assigned to the project created for you.
  6. Provide a name for your account and region for the account to reside in.
  7. After a short wait for the setup to complete, select your cloud data warehouse and enter your data warehouse credentials.
  8. Select a default role, warehouse, database, and schema (Snowflake), Unity Catalog and schema (Databricks) or default database and schema (Amazon Redshift).
  9. Click Finish and wait for your project to be created. You will be taken straight to the Designer UI.

First time users will be allocated credits as part of the trial onboarding process. For more information, read Free trial.

Designer projects

Projects are groups of resources that work together to allow the creation and execution of pipelines. These resources are defined independently of one another so they can be mixed and matched later. For example, a defined secret can be used in multiple components, pipelines, and environments.

  • Environments: A configuration to a target cloud data warehouse, including the default database and schema to be used. Currently supports Snowflake and Amazon Redshift.
  • Schedules: A set of rules dictating when pipelines are automatically run. Schedules are configured to work with a chosen environment, agent, and pipeline, so these can be used in multiple combinations. For example, a single pipeline can be included in multiple schedules to run in different environments.
  • Secret definitions: Secrets are used throughout the Designer to securely store sensitive credentials. For example, once defined, these can be used for authentication in connectors and environments. Secrets are stored in your cloud provider's secret manager (AWS Secret Manager) as configured in your agent, and these are the secrets that the Designer has access to.
  • OAuths: Authorised connections to third party services using OAuth 2.0 can be stored and used with connectors in your Designer pipelines.
  • Branches: Designer is built with a focus on Git source control. Users can create multiple branches within a project to store and track pipelines. Changes to pipelines within the Designer should be committed and merged to make those changes available to other team members through the Designer. Clicking on a branch's name will take you to the Designer for that branch.
  • Designer: The Designer itself is a UI that allows for the easy creation of pipelines, incorporating many elements of the above described resources.

Designer pipelines

Pipelines are a logical sequence of components—task-specific building blocks that can be assembled into pipelines and connected to one another to make a complex workflow. This workflow is shown on the canvas. There are two types of pipelines:

  • Orchestration pipelines: Orchestration pipelines deal with the loading of data from source system to target data warehouse. Typical orchestration components are connectors, flow logic components, and scripting components.
  • Transformation pipelines: Transformation pipelines deal with transforming table data that exists in your target data warehouse, typically because of an orchestration pipeline. Transformation components are often analogs of SQL operations such as creating and deleting tables, joining data, or performing calculations.

Release cycle

The Data Productivity Cloud's SaaS architecture means that updates are automatically made available to all users—you don't need to manually update your version of Data Productivity Cloud to receive new features.

Some features may be released in private or public preview versions before becoming general availability (GA).

  • Private preview features are released to selected customers for early feedback. They will be in a state of development that makes them unsuitable for use in production workloads. Private preview features will usually be shown on the public roadmap; however, participation in private preview is by invitation only. Customers who are interested in participating in a private preview of a given feature should reach out to their Customer Success Manager at Matillion. Private preview features are clearly marked Preview in the user interface, and tagged as Private preview in the documentation.
  • Public preview features are made available to all users. Public preview features aren't recommended for use in production workloads, but we encourage user testing and welcome feedback on these features. Public preview features are clearly marked Preview in the user interface, and tagged as Public preview in the documentation.
  • Restricted Availability features are suitable for production use and have been made available to a limited number of customers, selected by Matillion. The feature is ready for production use, and covered by an SLA, where applicable.
  • General availability (GA) features have been approved for customer use in production workloads. All product features not otherwise marked should be considered general availability.