Skip to content

Components overview

Components are the basic building blocks of pipelines. Each component represents a single task within the pipeline. For example, the Create Table component executes a task that creates a table when the pipeline runs, while the Salesforce Query component executes a task that returns data from Salesforce. Multiple components can be linked together to perform complex pipelines involving multiple tasks. For example, a pipeline may pull data from Salesforce, filter the data according to some criteria, and write the data to files in an S3 bucket, using the Salesforce Query, Filter, and S3 Unload components executed sequentially.

Each component is specifically applicable to one type of pipeline, orchestration or transformation, and it's impossible to add it to the other type of pipeline. However, see the Run Transformation component for a way to include a transformation pipeline (and its components) in an orchestration pipeline.

View components

To view the components that make up a pipeline, open the pipeline from the Pipelines tab by double-clicking its name, or by clicking the pipeline to select it then right-clicking and selecting Open pipeline. This will display the full pipeline on the canvas. Each component is represented by a separate icon on the canvas, and the connectors between the component icons shows the pipeline flow—the order in which components are executed.

When you have a pipeline open, the components available to use in that pipeline can be viewed by clicking the Components tab.

Add components to a pipeline

To add a component to a pipeline, open the pipeline in the canvas and use either of the following methods:

  • Locate the component in the Components tab, drag it onto the canvas, and drop it in some convenient position. You can then connect it to any component already on the canvas.
  • Click the + symbol on the canvas, which will appear to the right of any currently selected component. This opens a pop-up list of components. Select a component, and select the Output Connection you want it to connect to (On success, Unconditional, or On failure). The connection will be created automatically, giving you a rapid way of chaining together multiple components in a branching pipeline.

You can subsequently drag components around to new positions on the canvas if you wish. The connections between components will be maintained.

The Components tab and pop-up only show the components that can be used in the type of pipeline currently open. For example, Create Table is an orchestration component and can only be added to orchestration pipelines, never to transformation pipelines, and so is only visible in the Components tab when you have an orchestration pipeline open. The Components tab will be blank if no pipelines are open.

The Search box in the Components tab and pop-up allows you to search for a component by name, filtering the list to match your search string. If you don't know the exact name of a component, try searching for a logical type. For example, search for messaging to find a component such as "Send Email", which performs a messaging function. The search will also attempt to match based on synonyms, if you don't know the correct name.

Below the Search box is a set of filter buttons. Clicking one or more of these buttons will filter the list to show only those types of component. This can be used in conjunction with the Search box to further refine search results.

The component is given a default name when placed on the canvas, which can be changed in the component's Properties.

To delete a component from the canvas, select the component and click Delete on the context menu. You can undo the delete operation by pressing CTRL + Z (Windows) or CMD + Z (Mac). After undoing, you can redo the delete operation by pressing CTRL + Y (Windows) or CMD + Y (Mac).

Copying components

You can copy a component to create a duplicate elsewhere in the pipeline. Use the component's context menu to copy the component, and then use the context menu to paste a copy of the component, including all configured properties, elsewhere on the canvas. Connect the copy into the pipeline as required.

You can also copy a component between pipelines. Use the component's context menu to copy the component, switch to a different pipeline, and use the context menu to paste the component onto the new pipeline. The copied component is identical to the original, including all configured properties. The two pipelines must be of the same type; you can't paste from an orchestration pipeline to a transformation pipeline, or vice versa.

Select multiple components by holding down the SHIFT key while you click all the components you want to select, or hold down SHIFT while you drag the mouse pointer over the components you want to select. You can then copy all selected components as a single operation. To clear the selection, click on any empty part of the canvas.

Components context menu

Function Description Windows shortcut Mac shortcut
Validate pipeline This performs a "dry run" of the pipeline, verifying that a pipeline will run without actually performing any data manipulation. SHIFT + CTRL + ENTER SHIFT + CMD + RETURN
Add note Add a text note to the canvas to document your pipeline.
Run pipeline Runs the pipeline. CTRL + ENTER CMD + RETURN
Copy Make a copy of the component for pasting elsewhere. CTRL + C CMD + C
Paste Right-click anywhere on the blank canvas and click Paste to paste a copied component to that location. CTRL + V CMD + V
Delete component Remove the component from the pipeline canvas. You can undo the delete operation by pressing CTRL + Z (Windows) or CMD + Z (Mac). DELETE FN + BACKSPACE or DELETE

Connect components

The components will then need to be connected to one another in the order in which they will be executed in the pipeline. Each component is surrounded by connection rings: usually (but not always) one on the left and three on the right. To link two components, hover over a connection ring on the right of one component until the pointer changes to a + symbol. Then click and drag a line to the connection ring on the left of the second component. This will draw a permanent connection between the components.

To disconnect components, click the line that joins them and press Backspace or Delete.

The order that components are joined is crucial to the order that tasks are executed in the pipeline. The flow of pipeline execution proceeds from left to right, so the connection rings at the left of a component represent the input to the component, passed from the previous component, while the rings on the right represent the output of the component, which is passed to the next component.

To take an example of a Salesforce Query component taking data from Salesforce and writing it to an S3 bucket with S3 Unload, the pipeline starts with the Salesforce Query, so we place that on the left side of the canvas. One of the output rings on the right of that component should then be connected to the input ring at the left of the S3 Unload component, which naturally should be placed at the right side of the canvas for convenience. This tells the pipeline which order these components are to be executed in, and that the data generated by the first component should be made available for the second component to work with.

You can create complex, branching pipelines by linking two or more of a component's outputs to two or more other components.

The connection rings on the right of a component are color-coded, to denote the following specific functions:

  • Green: Continue on to the next component if this component executed successfully.
  • Red: Continue on to the next component if this component failed to execute.
  • Grey: Continue on to the next component regardless of the success or failure of this component.

These connection options allow you to direct the pipeline flow according to whether the component task is successful or not, creating different branches for each outcome.

Component properties

Every component has properties that can be configured to tailor the component's behavior. When a component is selected on the canvas, its properties are displayed in the Properties panel. A component's properties are described in the documentation for each component.

Changing a component's properties won't affect the properties of any other component, even components of the same type.

Many properties allow you to use variables or grid variables to set them dynamically. For example, any connectors that output to file storage can have the storage folder path set using variables, as described in Organizing file storage.

For many transformation components and for query components, once you have configured the properties, you can sample the data output by that component.

Text mode

Text mode is a feature that lets you rapidly add information to a property dialog. Instead of typing or selecting values in the individual fields of the dialog, you can click the Text mode toggle to open a multi-line editor that lets you add all the items in a single block, one item to a line, with the "fields" separated by tabs.

For example, when you complete the Columns property of the Create Table component, you are required to specify Column Name, Data Type, Size, and so on, for every column in the table. In text mode you can type the following:

Name    Text    64  0   Yes No
Descr   Text    512 0   No  No
Cost    Number  5   0   Yes Yes

Once you have entered details in text mode, you must switch out of text mode by clicking the Text mode toggle once more, to verify the details are valid before you continue. For example, if you mistyped a data type as Numbor, the Data Type field in the dialog would reject the invalid type and remain blank. You must correct the type, either directly in the field or in text mode again, before you click Save in the dialog.

You can go into text mode to edit the fields of an existing dialog, regardless of whether it was originally completed in text mode or not.

An advantage of text mode is that it will allow you to paste in values from other text sources, as long as the source has values that match the columns in the dialog and are correctly separated by tabs and new lines. You can also rapidly copy the fields from a completed dialog to another identical dialog, allowing rapid duplication of properties between or within components:

  1. Open a completed dialog in text mode.
  2. Copy the text containing the column definitions.
  3. Open another dialog, either of the same type or having identical field layout.
  4. Paste the copied text into the text mode field of the new dialog.

Component SQL

Most components produce SQL statements that act upon your data. You can view (but not alter) these SQL statements to learn how the component is acting. This may be useful in debugging failed pipelines, for example.

To view the SQL statements for a component, select the component on the canvas and click the SQL tab in the right-hand panel.