Designer UI basics
All work in Designer is accomplished by pipelines, and these pipelines are created, configured, and managed through the main Designer user interface (UI).
To open the Designer UI and create your first pipeline:
- Click Design data pipelines in the Services section of the Data Productivity Cloud home page.
- From Your projects, select your project.
- Select your branch. Designer will open.
- Get started by creating a pipeline. Click the Orchestration pipeline tile, which is the recommended first step, or the Transformation pipeline tile. For more information, read Pipelines.
The Designer UI has several different elements, illustrated below:
This image identifies the elements of the Designer UI, numbered as follows and described in detail in the following sections:
- Matillion logo: Click this to return to the Data Productivity Cloud, from where you can navigate to any other Matillion service.
- User ID: Displays your Data Productivity Cloud username and account you have joined. Click here to open a menu that allows you to view your personal details, switch to a different account, or log out of the Data Productivity Cloud.
- Project bar: Displays an icon that indicates the data warehouse being used in the current project, followed by the project name, the current environment, and the current branch. Click the project name to go to the project details screen.
- Branch menu: Click to open a drop-down menu with Designer's Git version control features.
- Files: The Files tab (shown open here) lists all pipelines in your current branch and allows you to create new pipelines.
- Components: When a pipeline is selected, the Components tab lists all components that can be used in the pipeline.
- Schemas: This tab opens a panel that displays the table and view objects within your database.
- Variables: Create two classes of variable; pipeline variables and project variables. To learn more about variables in Designer, read Variables.
- Copilot: This tab opens the Copilot AI-driven data productivity tool. Only available for transformation pipelines in Snowflake projects.
- Canvas: The canvas is the area where components are assembled into pipelines.
- Properties: When a component is selected on the canvas, the Properties panel displays the component's configurable properties. Properties may be displayed in collapsed groups, which you click to expand.
- Task history: When a pipeline runs, the Task history tab displays messages relating to the success or failure of the pipeline.
- Sample data: The Sample data tab displays data from validated transformation components.
- SQL: Click to open a panel that displays the SQL query generated by the component.
- Component name: Click the pencil icon to edit the component name that appears on the canvas.
- Help button: Click the question mark icon to open the documentation for the component in a separate browser tab.
Different panels in any given tab in the UI can be expanded or contracted as required, so you can see more or less detail in a particular panel. Click on the dividing line between any two panels and drag up/down or left/right, as required. Note that this change won't persist after you leave the Designer UI; it will always reopen with the default spacing.
Files panel
The Files panel, on the left side of the UI, lists every pipeline you have created on your current branch. Icons identify the type of pipeline: a blue O for orchestration, green T for transformation. Pipelines can be organized into folders within the panel.
In this tab you can add, delete, or edit pipelines. Read Pipelines for a full description of how to manage pipelines and use them in data workflows.
Our Pipelines tutorial is a step-by-step walkthrough that explains how to create an orchestration pipeline and a transformation pipeline using sample data provided by Matillion. This guide may be useful if you are new to Designer and want to practice creating pipelines and configuring components.
Components tab
When you have a pipeline open on the canvas, the components available to use in that pipeline are listed next to the Components tab, on the left side of the UI. Each component is listed with a unique icon, which is the icon that will represent the component when it is added to the canvas.
The Components tab only shows the components that can be used in the type of pipeline currently open. For example, Create Table is an orchestration component and can only be added to orchestration pipelines, never to transformation pipelines, and so is only visible in the Components tab when you have an orchestration pipeline open. To locate a specific component, type any part of its name into the Search bar.
Components are listed alphabetically and include a descriptor keyword, e.g. "Connectors" or "Flow". You can use these descriptors to filter out components that don't match that keyword. For example, typing "flow" in the search bar will return only the Flow components (And, End Failure, End Success, If, Or, Retry, Restart). The search also includes a number of synonym keywords to help filter components. For example, typing "logic" into the search bar will also return the Flow components.
To add a component to the pipeline, simply drag it from the Components tab to the canvas, and drop it in some convenient position. Alternatively, you can add a component by clicking + on the canvas, as described below.
You can also add iterators to components from the Components tab. Drag the iterator you want to use from the Components tab to the canvas, then use the connection ring beneath the iterator to connect it to the input of the component you want to iterate on. For more information, read Attaching, stacking and detaching iterators.
The pipeline canvas
The canvas is the main area of the UI which displays the details of a pipeline. A pipeline is a collection of connected components, displayed as labelled icons on the canvas. To add a component to a pipeline, you can use either of the following methods:
- Locate the component in the Components tab, and drag-and-drop it onto the canvas.
- Click + on the canvas, which appears to the right of any currently selected component. This opens the Add component dialog, displaying a pop-up list of components.
The following features apply, whether you add a component using the Components tab or by clicking + on the pipeline canvas:
| In the tab | In the dialog |
|---|---|
| Components are listed alphabetically | Components are listed alphabetically |
| Descriptor and synonym keywords help you refine your search | Descriptor and synonym keywords help you refine your search |
| When the component is dragged on to the pipeline canvas, this newly added component will become the selected component, meaning the + button will shift to the right. | Adds that component to the canvas in place of the + button, and closes the dialog. The new component will be connected to the previously selected component. |
| This newly added component will become the selected component, meaning the + button will shift to the right. |
For more details about constructing pipelines from components, read Components overview.
Any change you make to a pipeline on the canvas, such as adding, deleting or moving a component, is saved automatically. However, you need to commit and merge the changes before you can publish the pipeline for use in schedules.
You can open multiple pipelines at once. Each is shown as a separate tab at the top of the canvas, and you can click a tab to quickly switch to viewing a different pipeline. You can reorder your tabs by selecting and dragging a tab to your preferred order. Clicking the X on a tab will close that particular pipeline.
In the bottom-right, there are a number of tools in the control box that let you change how you view the canvas. These tools are useful if you have a large pipeline that doesn't fit in the default canvas view:
- Click the - button in the zoom toolbar to zoom in, or + to zoom out, in increments of 25%.
- Click the Fit all button to resize the canvas view, and show all components in the pipeline.
- Scroll the entire canvas in any direction. Click on an empty area of the canvas and drag up, down, left, or right.
- The Enable snap to grid feature is disabled by default. While enabled, components will always be positioned within the canvas grid. While disabled, components can be positioned freely.
- Drag the dividing lines between the canvas and the surrounding panels, to reduce the size of those panels and allow more room for the canvas.
Note
Click the "i" button in the bottom-right to open Designer overview on the documentation website for guidance and support.
None of these changes will persist after you leave the Designer UI; it will always reopen with the default view.
The control box in the top-right of the canvas gives additional controls which you can use to Validate or Run the pipeline.
Canvas context menu
As well as creating and configuring the pipeline, there are additional actions you can perform from the pipeline canvas. Right-click a blank place on the canvas to reveal the context menu. Refer to the table below for more information:
| Function | Description |
|---|---|
| Validate pipeline | Validate the pipeline. This performs a "dry run" of the pipeline, verifying that the pipeline will run without actually performing any data manipulation. |
| Run pipeline | Run the pipeline. |
| Add note | Adds a note box to the canvas. |
| Add note using AI | Invoke generative AI to annotate your data pipelines. Read AI Note for details. |
| Undo | Undo the last action performed. Actions that can be undone include deleting components, moving components, adding new components to the canvas, and connecting or disconnecting components. |
| Redo | Reverses the last Undo command, reverting whatever change was undone. |
| Copy | Copies any component, allowing you to then paste a copy of it. To perform this action, click the component to highlight it before opening the canvas context menu. |
| Paste | Paste a copy of a copied component at the location you have right-clicked. The copied component is identical to the original, including all configured properties. The Name will be slightly modified, and can be changed as normal. To perform this action, you must have first used the Copy action on a component. |
| Delete component | Remove a component from the pipeline canvas. To perform this action, click the component to highlight it before opening the canvas context menu. |
Component action bar
The component action bar allows you to perform additional actions on a single component in the pipeline. Click a component to show the component action bar. The table below lists all the additional actions available on the component action bar:
| Function | Description |
|---|---|
| Run only component | Orchestration pipelines only. Runs only the selected component, not the entire pipeline. For example, running a query component will perform the query but will not perform any downstream processing on the results. In some cases, running a single component in isolation will be meaningless. For example, running an If component without running the part of the pipeline that provides the inputs to the If. The context menu won't include this option in these cases. |
| Run from component | Orchestration pipelines only. Runs the current pipeline starting from the selected component. This may be useful to test the downstream effects of amending the pipeline at a specific point, for example. As the full pipeline is not running, this action may produce unexpected results and so should be used with care. |
| Skip on run | Temporarily disables the selected component, allowing you to run the pipeline as if this component was not present. When a component is skipped, this action changes to Unskip on run. |
| Copy | Copies the component, allowing you to then paste a copy of it using the Paste action from the canvas context menu. |
| Iterate | Opens the Add iterator dialog, where you can choose an iterator to add to the selected component. For more information, read Attaching, stacking, and detaching iterators. |
| Delete | Remove the component from the pipeline canvas. |
Adding notes
You can add any number of text notes to a pipeline canvas. These have no effect on the design or operation of the pipeline, but can be used to provide information for other users, such as the purpose of the pipeline.
To add a note, right-click on the canvas, and click Add note from the context menu.
A note is a text box with a solid outline. Click anywhere in the center of the note box to begin typing text. Text is aligned to the top-left of the box. You can format the text using common Markdown formatting. See this Markdown reference for details.
Click and drag the border of the box to resize it. Click inside the border and drag to move the box around the canvas. Click inside the box and then click the square button above the box to change the box's background color. Click inside the box and then click the trashcan button to delete the note.
The note box sits behind components on the canvas, so if you drag it over components it won't obscure them but will appear as a colored background and a solid border around those components. This allows you to position the box around a particular group of components, for example, and type a note relating to that group.
Properties panel
Every component has properties that can be configured to tailor the component's behavior. When a component is selected on the canvas, its properties are displayed in the Properties panel to the right of the canvas. A component's properties are described in the documentation for each component.
Note
For Public Preview only. Where Post-processing is available on a component, all properties will be displayed in the Configuration tab, next to the Post-processing tab.
The Properties panel shows the name of the selected component at the top, with a pencil icon next to it for editing purposes, the property names in bold, and the values of the properties underneath.
Toggle on Expand all underneath the name of the selected component to expand all headers and display all available properties. This toggle is off by default. Alternatively, click any header in the Properties panel to expand or collapse the corresponding properties.
To set or change the value of a property, click the button at the right of the value field. The type of icon tells you how you will set the value:
- Pencil: Type a value directly into the value field. Used for simple string and numeric properties.
- Down-arrow: Select a value from a drop-down list.
- Cog: Opens a dialog in which you can enter or select property values.
Note
Click the "?" button in the top-right of the properties panel to view the selected component's online documentation in a new tab.
There are two ways you can sample the data of a validated pipeline:
- Click the Sample data button next to the "?" button at the top of the Properties panel.
- Click the Sample data tab at the bottom of the Designer. For more information, read Sampling output.
Task history
Tasks are created whenever an orchestration or transformation operation is performed. This includes pipelines run from the canvas, tasks created by the scheduler, and the results of validated pipeline operations.
The Task history tab at the bottom of the UI shows a list of all tasks completed, running, queued, cancelled, or failed. These tasks can be expanded to show sub tasks.
Each row of the task history shows the following basic information about the task:
- Status icon:
- Red X icon: The task has failed to complete.
- Green tick: The task has completed successfully.
- Rotating disc: The task is currently in progress.
- Hourglass: The task has been queued, and will execute when a free thread is available.
- Task: The type of task, for example "Validate" or "Run".
- Environment: The environment that the task has run in.
- Pipeline: The name of the pipeline and the selected component.
- Queued: The time of day (hour:minute:second) that the task was queued for execution. This is not the same as the time the task execution started.
- Completed: The time of day (hour:minute:second) that the task execution completed, either successfully or unsuccessfully.
Note
To force a running pipeline to be cancelled, read Cancelling a running pipeline in Managing Pipelines for more information.
Detailed task history
To view further details about any task, click the corresponding row in the Task history tab at the bottom of the UI. This will open a tab on the pipeline canvas. In the top right, you can see when the task started, when it completed, and how long it took to finish.
While a task is running, this tab updates to show the progress of each step in the task. After the task is completed, this tab displays full details about each step in the task. You can expand some rows in the task to view more details about subtasks.
The task history shows:
- Pipeline: The name of the pipeline.
- Component: The name of each component in the task.
- Started: The time of day (hour:minute:second) that the task execution began. This time is shown for the entire task, not for each individual component.
- Duration & Completed: The time, in seconds, that the task took to execute, and the time of day (hour:minute:second) at which the task completed. This time is shown for the entire task, not for each individual component.
- Row Count: Where applicable, the number of rows of data processed by the task.
- Message: A message returned by each component when the task runs. This will indicate the success of the task, or give error information if the task failed.
Logging
To view detailed log information about a task, click the corresponding row in the Task history tab at the bottom of the UI. When the Task history tab opens on the pipeline canvas, click Logging.
The Logging tab shows the progress of the selected task step by step, and includes messages relevant to each step of the process. The log updates in real time as the task is running. The information shown here includes SQL code generated as part of the task, details of warehouses and agents involved in the task, as well as extremely detailed information that can be used to debug any pipeline issues.
Command palette
Designer offers a command palette to provide quick access to commands in Designer. Read on to discover the available commands.
File explorer
Use cmd + k or ctrl + k to activate a file explorer modal. Alternatively, click the search icon in the upper-right corner of the UI.
Commands
Use cmd + shift + k or ctrl + shift + k to open the command palette, where you can perform actions including Run pipeline, Validate pipeline, and Add component.