Designer UI basics
All work in Designer is accomplished by pipelines, and these pipelines are created, configured, and managed through the main Designer user interface (UI).
To open the Designer UI and create your first pipeline:
- Click Design data pipelines in the Services tab of the Hub.
- From Your projects, select your project.
- Select your branch. Designer will open.
- Get started by creating a pipeline. Click the Orchestration pipeline tile, which is the recommended first step, or the Transformation pipeline tile. For more information, read Pipelines.
The Designer UI has several different elements, illustrated below:
This image identifies the elements of the Designer UI, numbered as follows and described in detail in the following sections:
- Matillion logo: Click this to return to the Hub, from where you can navigate to any other Matillion service.
- User ID: Displays your Hub username and account you have joined. Click here to open a menu that allows you to view your personal details, switch to a different account, or log out of the Hub.
- Project bar: Displays an icon that indicates the data warehouse being used in the current project, followed by the name of your current branch and the name of the current environment. To the left of this information is a link labelled Your projects which will take you back to the project details screen.
- Add schedule: This button allows you to create a schedule.
- Git: This button opens a drop-down menu with Designer's Git version control features.
- Pipelines: The Pipelines tab lists all pipelines in your current branch and allows you to create new pipelines.
- Components: When a pipeline is selected, the Components tab lists all components that can be used in the pipeline.
- Properties: When a component is selected on the canvas, the Properties panel displays the component's configurable properties.
- Expand all: In the Properties panel, expand the headers to display the groups of properties associated with the selected component.
- Canvas: The canvas is the area where components are assembled into pipelines.
- Sample data: The Sample data tab displays data from validated transformation components.
- Task history: When a pipeline runs, the Task history tab displays messages relating the success or failure of the pipeline.
- Schemas: This tab opens a panel that displays the table and view objects within your database.
- Variables: Create two classes of variable; pipeline variables and project variables. To learn more about variables in Designer, read Variables.
Different panels in any given tab in the UI can be expanded or contracted as required, so you can see more or less detail in a particular panel. Click on the dividing line between any two panels and drag up/down or left/right, as required. Note that this change won't persist after you leave the Designer UI; it will always reopen with the default spacing.
Pipelines tab
The Pipelines tab, on the left side of the UI, lists in a panel, every pipeline you have created on your current branch. Icons identify the type of pipeline: a blue O for orchestration, green T for transformation. Pipelines can be organized into folders within the panel.
In this tab you can add, delete, or edit pipelines. Read Pipelines for a full description of how to manage pipelines and use them in data workflows.
Components tab
When you have a pipeline open on the canvas, the components available to use in that pipeline are listed next to the Components tab, on the left side of the UI. Each component is listed with a unique icon, which is the icon that will represent the component when it is added to the canvas.
The Components tab only shows the components that can be used in the type of pipeline currently open. For example, Create Table is an orchestration component and can only be added to orchestration pipelines, never to transformation pipelines, and so is only visible in the Components tab when you have an orchestration pipeline open. To locate a specific component, type any part of its name into the Search bar.
Components are listed alphabetically and include a descriptor keyword, e.g. "Connectors" or "Flow". You can use these descriptors to filter out components that don't match that keyword. For example, typing "flow" in the search bar will return only the Flow components (And, End Failure, End Success, If, Or, Retry, Restart). The search also includes a number of synonym keywords to help filter components. For example, typing "logic" into the search bar will also return the Flow components.
To add a component to the pipeline, simply drag it from the Components panel to the canvas, and drop it in some convenient position. Alternatively, you can add a component by clicking + on the canvas, as described below.
The pipeline canvas
The canvas is the main area of the UI which displays the details of a pipeline. A pipeline is a collection of connected components, displayed as labelled icons on the canvas. To add a component to a pipeline, you can use either of the following methods:
- Locate the component in the Components tab, and drag-and-drop it onto the canvas.
- Click + on the canvas, which appears to the right of any currently selected component. This opens the Add component dialog, displaying a pop-up list of components.
The following features apply, whether you add a component using the Components tab or by clicking + on the pipeline canvas:
| In the tab | In the dialog |
|---|---|
| Components are listed alphabetically | Components are listed alphabetically |
| Descriptor and synonym keywords help you refine your search | Descriptor and synonym keywords help you refine your search |
| When the component is dragged on to the pipeline canvas, this newly added component will become the selected component, meaning the + button will shift to the right. | Adds that component to the canvas in place of the + button, and closes the dialog. The new component will be connected to the previously selected component. |
| This newly added component will become the selected component, meaning the + button will shift to the right. |
For more details about constructing pipelines from components, read Components overview.
Any change you make to a pipeline on the canvas, such as adding, deleting or moving a component, is saved automatically. However, you need to commit and merge the changes before you can publish the pipeline for use in schedules.
You can open multiple pipelines at once. Each is shown as a separate tab at the top of the canvas, and you can click a tab to quickly switch to viewing a different pipeline. Clicking the X on a tab will close that particular pipeline.
In the bottom-right, there are a number of tools in the control box that let you change how you view the canvas. These tools are useful if you have a large pipeline that doesn't fit in the default canvas view:
- Click the - button in the zoom toolbar to zoom in, or + to zoom out, in increments of 25%.
- Click the Fit all button to resize the canvas view, and show all components in the pipeline.
- Scroll the entire canvas in any direction. Click on an empty area of the canvas and drag up, down, left, or right.
- The Enable snap to grid feature is disabled by default. While enabled, components will always be positioned within the canvas grid. While disabled, components can be positioned freely.
- Drag the dividing lines between the canvas and the surrounding panels, to reduce the size of those panels and allow more room for the canvas.
Note
Click the "i" button to the far-left to open Designer overview on the documentation website for guidance and support.
None of these changes will persist after you leave the Designer UI; it will always reopen with the default view.
The control box in the top-right of the canvas gives additional controls which you can use to Validate or Run the pipeline.
Canvas context menu
As well as creating and configuring the pipeline, there are additional actions you can perform from the pipeline canvas. Right-click a component to reveal the context menu. Refer to the table below for more information:
| Function | Description |
|---|---|
| Validate pipeline | Validate a pipeline. This performs a "dry run" of the pipeline, verifying that a pipeline will run without actually performing any data manipulation. To validate a pipeline, right-click on any connected component on the canvas, and select Validate pipeline. |
| Run pipeline | Run a pipeline from the canvas. Right-click on any connected component on the canvas, and select Run pipeline. |
| Add note | Adds a note box to the canvas. |
| Copy | Right-click on any connected component, and select Copy. |
| Paste | Right-click anywhere on the blank canvas and click Paste to create a copy of the component at that location. The copied component is identical to the original, including all configured properties. The Name will be slightly modified, and can be changed as normal. |
| Delete component | Right-click on any connected component, and select Delete to remove the component from the pipeline canvas. |
Adding notes
You can add any number of text notes to a pipeline canvas. These have no effect on the design or operation of the pipeline, but can be used to provide information for other users, such as the purpose of the pipeline.
To add a note, right-click on the canvas, and click Add note from the context menu.
A note is a text box with a solid outline. Click anywhere in the center of the note box to begin typing text. Text is aligned to the top-left of the box. You can format the text using common Markdown formatting. See this Markdown reference for details.
Click and drag the border of the box to resize it. Click inside the border and drag to move the box around the canvas. Click inside the box and then click the square button above the box to change the box's background color. Click inside the box and then click the trashcan button to delete the note.
The note box sits behind components on the canvas, so if you drag it over components it won't obscure them but will appear as a colored background and a solid border around those components. This allows you to position the box around a particular group of components, for example, and type a note relating to that group.
Properties panel
Every component has properties that can be configured to tailor the component's behavior. When a component is selected on the canvas, its properties are displayed in the Properties panel to the right of the canvas. A component's properties are described in the documentation for each component.
The Properties panel shows the name of the selected component at the top, with a pencil icon next to it for editing purposes, the property names in bold, and the values of the properties underneath.
Click the Expand all radio button underneath the name of the selected component to collapse all of the headers to display the group of properties available.
Note
- The first header will be collapsed by default.
- Alternatively, click any header in the Properties panel to expand the group of properties.
To set or change the value of a property, click the button at the right of the value field. The type of icon tells you how you will set the value:
- Pencil: Type a value directly into the value field. Used for simple string and numeric properties.
- Down-arrow: Select a value from a drop-down list.
- Cog: Opens a dialog in which you can enter or select property values.
Note
Click the "?" button in the far-right of the properties panel. A new tab will open in your browser, taking you to the selected component's online documentation.
There are two ways you can sample the data of a validated pipeline:
- Click the Sample data button next to the "?" button at the top of the Properties panel.
- Click the Sample data tab at the bottom of the Designer. For more information, read Sampling output.
Task history tab
Tasks are created whenever an orchestration or transformation operation is performed. This includes pipelines run from the canvas, tasks created by the scheduler, and the results of validated pipeline operations.
The Task history tab at the bottom of the UI shows a list of all tasks completed, running, queued, cancelled, or failed. These tasks can be expanded to show sub tasks.
Each row of the task history shows the following basic information about the task:
- Status icon:
- Red X icon: The task has failed to complete.
- Green tick: The task has completed successfully.
- Rotating disc: The task is currently in progress.
- Hourglass: The task has been queued, and will execute when a free thread is available.
- Task: The type of task, for example "Validate" or "Run".
- Environment: The environment that the task has run in.
- Pipeline: The name of the pipeline and the selected component.
- Queued: The time of day (hour:minute:second) that the task was queued for execution. This is not the same as the time the task execution started.
- Completed: The time of day (hour:minute:second) that the task execution completed, either successfully or unsuccessfully.
Note
To force a running pipeline to be cancelled, read Cancelling a running pipeline in Managing Pipelines for more information.
To see further details of any completed task, click the arrow button to the right of the task. This will open a tab on the pipeline canvas with full details of the task. You can expand the task to show each individual component in the pipeline. The task history shows:
- Pipeline: The name of the pipeline.
- Component: The name of each component in the task.
- Duration: The time, in seconds, that the task took to execute (either to completion or to failure). This time is shown for the entire task, not for each individual component.
- Queued: The time of day (hour:minute:second) that the task was queued for execution. This is not the same as the time the task execution started. This time is shown for the entire task, not for each individual component.
- Started: The time of day (hour:minute:second) that the task execution began. This time is shown for the entire task, not for each individual component.
- Completed: The time of day (hour:minute:second) that the task execution completed, either successfully or unsuccessfully. This time is shown for the entire task, not for each individual component.
- Row Count: Where applicable, the number of rows of data processed by the task.
- Message: A message returned by each component when the task runs. This will indicate the success of the task, or give error information if the task failed.