Components overview
Components are the basic building blocks of pipelines. Each component represents a single task within the pipeline. For example, the Create Table component executes a task that creates a table when the pipeline runs, while the Salesforce Query component executes a task that returns data from Salesforce. Multiple components can be linked together to perform complex pipelines involving multiple tasks. For example, a pipeline may pull data from Salesforce, filter the data according to some criteria, and write the data to files in an S3 bucket, using the Salesforce Query, Filter, and S3 Unload components executed sequentially.
Each component is specifically applicable to one type of pipeline, orchestration or transformation, and it's impossible to add it to the other type of pipeline. However, see the Run Transformation component for a way to include a transformation pipeline (and its components) in an orchestration pipeline.
Note
Due to considerations such as security and data flow, some components can only be used in a Hybrid SaaS environment. This will be clearly stated in the component description. Read Matillion Full SaaS vs Hybrid SaaS for more details.
View components
To view the components that make up a pipeline, open the pipeline from the Pipelines tab by double-clicking its name, or by clicking the pipeline to select it then right-clicking and selecting Open pipeline. This will display the full pipeline on the canvas. Each component is represented by a separate icon on the canvas, and the connectors between the component icons shows the pipeline flow—the order in which components are executed.
When you have a pipeline open, the components available to use in that pipeline can be viewed by clicking the Components tab.
Add components to a pipeline
To add a component to a pipeline, open the pipeline in the canvas and use either of the following methods:
- Locate the component in the Components tab, drag it onto the canvas, and drop it in some convenient position. You can then connect it to any component already on the canvas.
- Click the + symbol on the canvas, which will appear to the right of any currently selected component. This opens a pop-up list of components. Select a component, and select the Output Connection you want it to connect to (On success, Unconditional, or On failure). The connection will be created automatically, giving you a rapid way of chaining together multiple components in a branching pipeline.
You can subsequently drag components around to new positions on the canvas if you wish. The connections between components will be maintained.
The Components tab and pop-up only show the components that can be used in the type of pipeline currently open. For example, Create Table is an orchestration component and can only be added to orchestration pipelines, never to transformation pipelines, and so is only visible in the Components tab when you have an orchestration pipeline open. The Components tab will be blank if no pipelines are open.
The Search box in the Components tab and pop-up allows you to search for a component by name, filtering the list to match your search string. If you don't know the exact name of a component, try searching for a logical type. For example, search for messaging to find a component such as "Send Email", which performs a messaging function. The search will also attempt to match based on synonyms, if you don't know the correct name.
Below the Search box is a set of filter buttons. Clicking one or more of these buttons will filter the list to show only those types of component. This can be used in conjunction with the Search box to further refine search results.
The component is given a default name when placed on the canvas, which can be changed in the component's Properties.
To delete a component from the canvas, select the component and click Delete on the context menu. You can undo the delete operation by pressing CTRL + Z (Windows) or CMD + Z (Mac). After undoing, you can redo the delete operation by pressing CTRL + Y (Windows) or CMD + Y (Mac).
Alternatively, click the delete button in the component action bar, which displays when a component is selected.
Copying components
You can copy a component to create a duplicate elsewhere in the pipeline. Use the component's context menu to copy the component, and then use the context menu to paste a copy of the component, including all configured properties, elsewhere on the canvas. Connect the copy into the pipeline as required.
Alternatively, click the copy button in the component action bar, which displays when a component is selected.
You can also copy a component between pipelines. Use the component's context menu to copy the component, switch to a different pipeline, and use the context menu to paste the component onto the new pipeline. The copied component is identical to the original, including all configured properties. The two pipelines must be of the same type; you can't paste from an orchestration pipeline to a transformation pipeline, or vice versa.
Select multiple components by holding down the SHIFT key while you click all the components you want to select, or hold down SHIFT while you drag the mouse pointer over the components you want to select. You can then copy all selected components as a single operation. To clear the selection, click on any empty part of the canvas.
Components context menu
| Function | Description | Windows shortcut | Mac shortcut |
|---|---|---|---|
| Validate pipeline | This performs a "dry run" of the pipeline, verifying that a pipeline will run without actually performing any data manipulation. | SHIFT + CTRL + ENTER | SHIFT + CMD + RETURN |
| Add note | Add a text note to the canvas to document your pipeline. | ||
| Run pipeline | Runs the pipeline. | CTRL + ENTER | CMD + RETURN |
| Copy | Make a copy of the component for pasting elsewhere. | CTRL + C | CMD + C |
| Paste | Right-click anywhere on the blank canvas and click Paste to paste a copied component to that location. | CTRL + V | CMD + V |
| Delete component | Remove the component from the pipeline canvas. You can undo the delete operation by pressing CTRL + Z (Windows) or CMD + Z (Mac). | DELETE | DELETE |
For more information about the Windows and Mac shortcuts, read Keyboard shortcuts in Windows and Mac keyboard shortcuts, respectively.
Connect components
The components will then need to be connected to one another in the order in which they will be executed in the pipeline. Each component is surrounded by connection rings: usually (but not always) one on the left and three on the right. To link two components, hover over a connection ring on the right of one component until the pointer changes to a + symbol. Then click and drag a line to the connection ring on the left of the second component. This will draw a permanent connection between the components.
To disconnect components, click the line that joins them and press Backspace or Delete.
The order that components are joined is crucial to the order that tasks are executed in the pipeline. The flow of pipeline execution proceeds from left to right, so the connection rings at the left of a component represent the input to the component, passed from the previous component, while the rings on the right represent the output of the component, which is passed to the next component.
To take an example of a Salesforce Query component taking data from Salesforce and writing it to an S3 bucket with S3 Unload, the pipeline starts with the Salesforce Query, so we place that on the left side of the canvas. One of the output rings on the right of that component should then be connected to the input ring at the left of the S3 Unload component, which naturally should be placed at the right side of the canvas for convenience. This tells the pipeline which order these components are to be executed in, and that the data generated by the first component should be made available for the second component to work with.
You can create complex, branching pipelines by linking two or more of a component's outputs to two or more other components.
The connection rings on the right of a component are color-coded, to denote the following specific functions:
- Green: Continue on to the next component if this component executed successfully.
- Red: Continue on to the next component if this component failed to execute.
- Grey: Continue on to the next component regardless of the success or failure of this component.
These connection options allow you to direct the pipeline flow according to whether the component task is successful or not, creating different branches for each outcome.
Once two components are connected a connecting line is visible, which will match the color-coding of the connection type. Most connection lines will also include an icon for added clarity of the connection type:
- Grey: Continue on to the next component regardless of the success or failure of this component.
- Green "thumb up": Continue on to the next component if this component executed successfully.
- Red "X": Continue on to the next component if this component failed to execute.
- Blue "thumb up": Continue on to the next component if this component's condition resolves to true.
- Orange "thumb down": Continue on to the next component if this component's condition resolves to false.
Skip components
Use the Skip on run option to temporarily disable a component in your pipeline. This allows you to run the pipeline as if it did not contain the skipped component, without actually needing to remove the component from the pipeline.
To skip a component, click the component and then click Skip on run in the component action bar. The component on the pipeline canvas will appear darker, and the Skip on run icon in the action bar will change to show that this component will be skipped.
Additionally, when you run a pipeline containing a skipped component and then open the detailed task view, the skipped component's success icon is greyed out and its Message notes that this component is skipped.
To un-skip a component, click the component and click Unskip on run. The appearance of the component and icon will return to normal.
Note
You can only skip components in orchestration pipelines, and you can't skip Flow components such as If.
Iterate on components
Use iterators to "loop" components. That is to say, repeat the component without adding multiple copies of the same component to your pipeline.
There are two ways to attach an iterator to a component:
- Click the component and then click Iterate in the component action bar. Next, select the iterator you want to use. The iterator will automatically be attached to the selected component on the canvas.
- Click and drag an iterator on to the canvas from the Components tab, and then use the connection ring beneath the iterator to connect it to the input of the component you want to iterate on.
The following iterators are available in Designer:
- Use the File Iterator to loop over matching files in a remote file system.
- Use the Fixed Iterator to loop over values of a simple sequence.
- Use the Grid Iterator to loop over the rows of a grid variable.
- Use the Loop Iterator to loop over values of a simple sequence using a for loop.
- Use the Table Iterator to loop over rows of data within a table.
For information about attaching and detaching multiple iterators, read Attaching, stacking, and detaching iterators.
Component properties
Every component has properties that can be configured to tailor the component's behavior. When a component is selected on the canvas, its properties are displayed in the Properties panel. A component's properties are described in the documentation for each component.
Note
For Public Preview only. Where Post-processing is available on a component, all properties will be displayed in the Configuration tab, next to the Post-processing tab.
Changing a component's properties won't affect the properties of any other component, even components of the same type.
For transformation components only, the Preview SQL toggle at the top of the properties panel displays the raw SQL of the selected component in a read-only editor. You must validate the component before the raw SQL will be displayed.
Many properties allow you to use variables or grid variables to set them dynamically. For example, any connectors that output to file storage can have the storage folder path set using variables, as described in Organizing file storage.
For many transformation components and for query components, once you have configured the properties, you can sample the data output by that component.
Reorder items
Many property dialogs allow you to reorder rows of information. For example, when you configure the Columns property of the Create Table component, you can reorder the columns. To do this in any applicable editor, use the ⋮⋮ icon to drag and drop a single row to change its position, or use the checkboxes to select multiple rows and then drag all selected rows into the required order.
It is important to note that reordering rows in this way can have a significant effect on your data. For example, it may change the order in which your data is displayed and processed, or the order in which filter conditions are applied.
Text mode
Text mode is a feature that lets you rapidly add information to a property dialog. Instead of typing or selecting values in the individual fields of the dialog, you can click the Text mode toggle to open a multi-line editor that lets you add all the items in a single block, one item to a line, with the "fields" separated by tabs.
For example, when you complete the Columns property of the Create Table component, you are required to specify Column Name, Data Type, Size, and so on, for every column in the table. In text mode you can type the following:
Name Text 64 0 Yes No
Descr Text 512 0 No No
Cost Number 5 0 Yes Yes
Once you have entered details in text mode, you must switch out of text mode by clicking the Text mode toggle once more, to verify the details are valid before you continue. For example, if you mistyped a data type as Numbor, the Data Type field in the dialog would reject the invalid type and remain blank. You must correct the type, either directly in the field or in text mode again, before you click Save in the dialog.
You can go into text mode to edit the fields of an existing dialog, regardless of whether it was originally completed in text mode or not.
An advantage of text mode is that it will allow you to paste in values from other text sources, as long as the source has values that match the columns in the dialog and are correctly separated by tabs and new lines. You can also rapidly copy the fields from a completed dialog to another identical dialog, allowing rapid duplication of properties between or within components:
- Open a completed dialog in text mode.
- Copy the text containing the column definitions.
- Open another dialog, either of the same type or having identical field layout.
- Paste the copied text into the text mode field of the new dialog.
Component SQL
Most components produce SQL statements that act upon your data. You can view (but not alter) these SQL statements to learn how the component is acting. This may be useful in debugging failed pipelines, for example.
To view the SQL statements for a component, select the component on the canvas and use the Preview SQL toggle above the component configuration fields to show or hide the SQL statements.