Pipelines

Pipelines are the Data Productivity Cloud's way of designing, organizing, and executing workflows. You use Designer to build strings of configured components inside a pipeline and then run that pipeline to carry out a desired task such as loading or transforming data.

Pipelines are created, configured, and managed through the main Designer user interface.

There are two types of pipeline in Data Productivity Cloud: orchestration and transformation.

Orchestration pipelines deal with the loading of data from source system to target data warehouse. Typical orchestration components are connectors, flow logic components, and scripting components.
Transformation pipelines deal with transforming table data that exists in your target data warehouse, typically after loading that data with an orchestration pipeline. Transformation components are often analogs of SQL operations such as creating and deleting tables, joining data, or performing calculations.

Components are the basic building blocks of pipelines. Each component is specifically applicable to one type of pipeline, orchestration or transformation, and can't be added to the other type of pipeline. To learn more about components, read Components overview.

Adding a pipeline

To create a new pipeline:

In the Designer user interface, click the + icon in the top-right of the Pipelines panel and select Add new orchestration pipeline or Add new transformation pipeline.
Enter a name for the pipeline. The name may include alphanumerics, underscores, single spaces, parenthesis and hyphens. The name must be unique within each pipeline folder.
Click Add. The new pipeline immediately opens in a new tab on the canvas.

The newly created pipeline is blank at this stage. You now need to add components to the pipeline canvas to construct a workflow that will perform your data extraction and transformation tasks.

Managing pipelines

The Pipelines panel in the top-left of the Designer user interface lists every pipeline you have created on your current branch. Icons identify the type of pipeline: a blue O for orchestration, or green T for transformation. The pipelines can be organized into a hierarchy of folders, if required.

To view a pipeline's details on the canvas, do one of the following:

Double-click the pipeline in the list.
Right-click the pipeline and select Open pipeline.
Single-click the pipeline to select it, click the ... icon next to it, and click Open pipeline.
Single-click the pipeline to select it and press Enter.

To delete a pipeline, do one of the following:

Right-click the pipeline and select Delete pipeline.
Single-click the pipeline to select it, click the ... icon next to it, and click Delete pipeline.

Then click Yes, delete to confirm the deletion.

Deleted pipelines can't be recovered.

Note

Deleting a pipeline may break other pipelines that reference it via Run Orchestration or Run Transformation components. You will therefore need to update any pipelines that reference this deleted pipeline.

To rename a pipeline, do one of the following:

Right-click the pipeline and click Rename pipeline.
Single-click the pipeline to select it, click the ... icon next to it, and click Rename pipeline.

Note

Renaming a pipeline may break other pipelines that reference it. You will need to update any Run Orchestration or Run Transformation components that reference this pipeline with the new name.

Any existing schedules for a renamed pipeline will not be updated. You will need to create a new schedule using the new name.

Pipeline folders

A branch may contain a large number of pipelines, which by default are listed alphabetically in the Pipelines panel. To organize pipelines in a branch, you can arrange them into named folders. Folders can also be nested inside folders, up to 10 levels deep, to create a structure that makes sense to you and your team. Pipeline names must be unique within a folder but can be duplicated in different folders.

To create a folder, click the + icon in the top-right of the Pipelines panel and click Add new folder. Enter a name for the folder and click Create. Folder names can contain alphanumeric characters, dashes, and underscores.

You can collapse or expand a folder by clicking the > icon.

Each folder has its own context menu where you can add a pipeline to the folder, add another folder as a sub-folder, or delete the folder. Click ... next to the folder name to open the context menu.

When you delete a folder, all the pipelines and sub-folders it contains will also be deleted. However, active schedules for pipelines in a deleted folder will continue to use the latest published version of the pipelines. Deleting a folder can't be undone.

To add a pipeline or another folder in the root folder, click the + icon at the top-right of the Pipelines panel.

You can move a pipeline between folders, or between root and folder, by drag-and-drop. You can't move a pipeline to a folder where another pipeline with the same name already exists, and attempting to do so will result in a warning notice.

Warning

Be aware of the following when working with pipeline folders:

Moving a pipeline into a different folder will break other pipelines that reference it via Run Orchestration or Run Transformation components. You will therefore need to update any pipelines that reference the moved pipeline.
Any existing schedules that use the moved pipeline will not be updated. The pipelines will have to be republished and the schedule recreated.
Empty folders cannot be committed to Git. To commit a folder, you need to add a pipeline in that folder. Otherwise, the empty folder will disappear from your pipeline panel if you try to commit.
Moving a pipeline to a different folder is considered a change to your branch, and therefore needs to be committed to Git.