Designer UI basics
All work in Designer is accomplished by pipelines, and these pipelines are created, configured, and managed through the main Designer user interface (UI).
To open the Designer UI:
- Click Design data pipelines in the Services tab of the Hub.
- From Your projects, select your project.
- Select your branch. Matillion Designer will open.
The Designer UI has several different elements, illustrated below:
This image identifies the elements of the Designer UI, numbered as follows and described in detail in the following sections:
- Matillion logo: Click this to return to the Hub, from where you can navigate to any other Matillion service.
- User ID: Displays your Matillion user ID and the Matillion account you have joined. Click here to open a menu that allows you to view your personal details, switch to a different account, or log out of Matillion services.
- Project bar: Displays an icon that indicates the data warehouse being used in the current project, followed by the name of your current branch and the name of the current environment. To the left of this information is a link labelled Your projects which will take you back to the project details screen.
- Git version control: These buttons are used with Designer's Git version control features.
- Pipelines: The Pipelines panel lists all pipelines in your current branch and allows you to create new pipelines.
- Components: When a pipeline is selected, the Components panel lists all components which can be used in the pipeline.
- Canvas: The canvas is the area where components are assembled into pipelines.
- Properties: When a component is selected on the canvas, the Properties panel displays the component's configurable properties.
- Sample: The Sample tab displays data from validated transformation components.
- Task history: When a pipeline runs, the Task history panel displays messages relating the success or failure of the pipeline.
- Schemas: Opens a panel that displays the Table and View objects within your database.
Different panels in the UI can be expanded or contracted as required, so you can see more or less detail in a particular panel. Click on the dividing line between any two panels and drag up/down or left/right, as required. Note that this change won't persist after you leave the Designer UI; it will always re-open with the default spacing.
The Pipelines panel in the top-left of the UI lists every pipeline you have created on your current branch. Icons identify the type of pipeline: a blue
O for orchestration, green
T for transformation. Pipelines can be organized into folders within the panel.
In this panel you can add, delete, or edit pipelines. Read Pipelines for a full description of how to manage pipelines and use them in data workflows.
When you have a pipeline open on the canvas, the components available to use in that pipeline are shown in the Components panel at the bottom-left of the UI. Each component is listed with a unique icon which is the icon that will represent the component when it is added to the canvas.
Components are arranged in a tree view that groups similar components together. For example, expand the Scripting group to show the two scripting components, Bash Script and Python Script.
The Components panel only shows the components that can be used in the type of pipeline currently open. For example, Create Table is an orchestration component and can only be added to orchestration pipelines, never to transformation pipelines, and so is only visible in the Components panel when you have an orchestration pipeline open.
To add a component to the pipeline, simply drag it from the Components panel to the canvas, and drop it in some convenient position.
The pipeline canvas
The canvas is the main area of the UI which displays the details of a pipeline. A pipeline is a collection of connected components, displayed as labelled icons on the canvas. To add a component to a pipeline, you drag it from the Components panel and drop it onto the canvas. For more details of constructing pipelines from components, read Components overview.
Any change you make to a pipeline on the canvas, such as adding, deleting or moving a component, is saved automatically.
You can open multiple pipelines at once. Each is shown as a separate tab at the top of the canvas, and you can click a tab to quickly switch to viewing a different pipeline. Clicking the X on a tab will close that particular pipeline.
In the bottom-left, there are a number of tools in the control box that let you change how you view the canvas. These tools are useful if you have a large pipeline that doesn't fit in the default canvas view:
- Click the - button in the zoom toolbar to zoom in, or + to zoom out, in increments of 25%.
- Click the Fit all button to resize the canvas view, and show all components in the pipeline.
- Scroll the entire canvas in any direction. Click on an empty area of the canvas and drag up, down, left, or right.
- The Enable snap to grid feature is disabled by default. While enabled, components will always be positioned within the canvas grid. While disabled, components can be positioned freely.
- Drag the dividing lines between the canvas and the surrounding panels, to reduce the size of those panels and allow more room for the canvas.
None of these changes will persist after you leave the Designer UI; it will always re-open with the default view.
The control box in the top-right of the canvas notes the type of pipeline (orchestration or transformation) and gives additional controls which you can use to validate or run the pipeline. By clicking (x), you can manage any variables you add to your pipeline from this control box. For more information, refer to Variables.
Canvas context menu
As well as creating and configuring the pipeline, there are additional actions you can perform from the pipeline canvas. Right-click a component to reveal the context menu. Refer to the table below for more information:
|Validate pipeline||Validate a pipeline. This performs a "dry run" of the pipeline, verifying that a pipeline will run without actually performing any data manipulation. To validate a pipeline, right-click on any connected component on the canvas, and select Validate pipeline.|
|Run pipeline||Run a pipeline from the canvas. Right-click on any connected component on the canvas, and select Run pipeline.|
|Add note||Adds a note box to the canvas.|
|Copy||Right-click on any connected component, and select Copy.|
|Paste||Right-click anywhere on the blank canvas and click Paste to create a copy of the component at that location. The copied component is identical to the original, including all configured properties. The Name will be slightly modified, and can be changed as normal.|
|Delete component||Right-click on any connected component, and select Delete to remove the component from the pipeline canvas.|
You can add any number of text notes to a pipeline canvas. These have no effect on the design or operation of the pipeline, but can be used to provide information for other users, such as the purpose of the pipeline.
To add a note, right-click on the canvas and click Add note from the context menu.
A note is a text box with a solid outline. Click anywhere in the center of the note box to begin typing text. Text is aligned to the top-left of the box. You can format the text using common Markdown formatting. See this Markdown reference for details.
Click and drag the border of the box to resize it. Click inside the border and drag to move the box around the canvas. Click inside the box and then click the square icon above the box to change the box's background color. Click inside the box and then click the trashcan icon to delete the note.
The note box sits behind components on the canvas, so if you drag it over components it won't obscure them but will appear as a colored background and a solid border around those components. This allows you to position the box around a particular group of components, for example, and type a note relating to that group.
Every component has properties that can be configured to tailor the component's behavior. When a component is selected on the canvas, its properties are displayed in the Properties panel below the canvas. A component's properties are described in the documentation for each component.
The Property panel shows the Parameter (name) and Value of each property.
To set or change the value of a property, click the icon at the right of the Value field. The type of icon tells you how you will set the value:
- Pencil: Type a value directly into the Value field. Used for simple string and numeric properties.
- Down-arrow: Select a value from a drop-down list.
- Cog: Opens a dialog in which you can enter or select property values.
Task history panel
Tasks are created whenever an orchestration or transformation operation is performed. This includes pipelines run from the canvas, tasks created by the scheduler, and the results of validate pipeline operations.
The Task history panel at the lower-right of the UI shows a list of all tasks completed, running, queued, cancelled, or failed. These tasks can be expanded to show sub tasks. Each row of the task history shows the following basic information about the task:
- Status icon:
- Red X icon: The task has failed to complete.
- Green tick: The task has completed successfully.
- Rotating disc: The task is currently in progress.
- Hourglass: The task has been queued, and will execute when a free thread is available.
- Task: The type of task, for example "Validate" or "Run".
- Environment: The environment that the task has run in.
- Pipeline: The name of the pipeline.
- Queued: The time of day (hour:minute:second) that the task was queued for execution. This is not the same as the time the task execution started.
- Completed: The time of day (hour:minute:second) that the task execution completed, either successfully or unsuccessfully.
To see further details of any completed task, click the arrow icon to the right of the task. This will open a tab on the pipeline canvas with full details of the task. You can expand the task to show each individual component in the pipeline. The task history shows:
- Pipeline: The name of the pipeline.
- Component: The name of each component in the task.
- Duration: The time, in seconds, that the task took to execute (either to completion or to failure). This time is shown for the entire task, not for each individual component.
- Queued: The time of day (hour:minute:second) that the task was queued for execution. This is not the same as the time the task execution started. This time is shown for the entire task, not for each individual component.
- Started: The time of day (hour:minute:second) that the task execution began. This time is shown for the entire task, not for each individual component.
- Completed: The time of day (hour:minute:second) that the task execution completed, either successfully or unsuccessfully. This time is shown for the entire task, not for each individual component.
- Row Count: Where applicable, the number of rows of data processed by the task.
- Message: A message returned by each component when the task runs. This will indicate the success of the task, or give error information if the task failed.