Skip to content

Post-processing

Public preview

You can update scalar variables in Designer using post-processing. Each component in an orchestration pipeline has a Post-processing tab where you can map your user-defined pipeline variables and project variables to system variables. System variables offer statistical and referential data about a component's execution after a pipeline has run. Examples of system variables include row count, execution duration, and project ID. For more information, read the full list of system variables. You may wish to make use of system variables for inline business logic, such as ensuring a table row count is greater than 0; or to perform custom status reporting, such as via a webhook or email; or for auditing purposes to capture relevant metrics via a shared pipeline.

Note the following behavior for usage:

  • Because system variables are limited to the Post-processing tab, you'll need to map any system variables you wish to use in your pipelines to user-defined pipeline or project variables.
  • System variable names cannot be changed.
  • System variable values are updated automatically as part of a component's execution.
  • The Start component does not include the Post-processing tab.
  • The main difference between post-processing and the Update Scalar component is the availability of additional system variables in post-processing that provide context about the component execution, such as row count.

Update scalar variables in post-processing

To update scalar variables, follow these steps:

  1. Select a component in your orchestration pipeline.
  2. Next to the Configuration tab, click the Post-proceesing tab.
  3. Click the cog (settings) button next to Update Scalar Variables.
  4. The Name column displays any existing pipeline and project variables. Select the variable you created earlier.
  5. The Value column is where you'll select a system variable to map your user-defined variable to.

Note

In the Value field, type ${sysvar to begin with, and all available system variables will be displayed.

You can use the Print Variables component to print the value of a variable at a given point in time to the task history tab. This is demonstrated in the example below.

Example: Using system variables in post-processing

The following example uses a simple orchestration pipeline to demonstrate the effectiveness of using system variables and post-processing. In this pipeline, we will connect to Jira, extract and load a table of issues, and then use an If component to read a user-defined variable's value and set the pipeline's end state to success or failure. The user-defined variable will be mapped to a system variable using the post-processing capabilities in Designer.

This example uses the following components:

  • Start: this is the starting point of any orchestration pipeline.
  • Jira Query: this is the connector we'll use to extract and load data, and which we'll base our system variable's value on.
  • If: this is how we'll conditionally define the success or failure of our pipeline based on the value of our system variable.
  • Print Variables: this will be used to print the value of our variable to the task history.
  • End Success: a successful end point for our pipeline.
  • End Failure: an unsuccessful end point for our pipeline.

Pipeline canvas

Create a user-defined variable

  1. In Designer, open the Variables menu.
  2. Click Add to begin creating a new variable.
  3. In the Add a variable dialog, configure the basic settings of your variable.
    1. Select whether the variable is a project variable or a pipeline variable. Read Variable scope for more information.
    2. Set the variable type. Read Variable type for more information.
    3. For the purposes of this example, the scope is set to Pipeline variable and the type is set to Number.

  4. Click Next.

    Add variable

  5. In the left-hand tab, provide a name for your new variable. You can provide a description too, but this is optional.

  6. For this example, the Visibility and Behavior settings will be left as-is, and the default value is left blank.
  7. Click Create.

Set up variable

Set up system variables

In this example, a Jira Query component is added to an orchestration pipeline to extract and load data from the Jira Issues data source.

The Post-processing tab is available in all orchestration pipeline components in the same location and works in the same manner, so using a different connector (such as Salesforce or Shopify) than Jira if you're following this example is fine.

To set up a system variable:

  1. Click on the orchestration component you wish to set up a system variable on. In this example, the Jira Query component is selected.
  2. Switch over to the Post-processing tab.

  3. Click into the Update Scalar Variables parameter.

    Set up system variables

  4. The Name column displays any existing pipeline and project variables. Select the variable you created earlier.

  5. The Value column is where you'll select a system variable to map your user-defined variable to.
    1. In the Value field, type ${sysvar to begin with, and all available system variables will be displayed.
    2. In this example, we will use ${sysvar.thisComponent.rowCount}, a system variable that returns the row count of the component at runtime, and we're going to map this to our user-defined variable named jira_row_count.
  6. Click Save to finish setting up your system variables.

Set up system variables

Set up Print Variables

This section is optional. The Print Variables component will display the value of any specified variables in the Task history tab.

In this case:

  • The Variables to print parameter is configured to print the value of the variable jira_row_count.
  • The Prefix text parameter is left blank.
  • The Include variable name parameter is set to Yes.

Set up success and failure conditions

  1. In this part of the example, an If component is used to define conditional logic.
  2. In the If component's Condition parameter, a condition is defined. In this example, the value of our user-defined variable jira_row_count must be greater than 0 for the condition to pass as true.
  3. When the condition is true, the If component will connect to an End Success component and finish the pipeline successfully. However, if the condition is false, the If component will connect to an End Failure component and the pipeline will fail.

Set up conditional logic

Run the pipeline

It's time to run our pipeline.

  1. Click Run.
  2. In the Task history tab, double-click the running task to view the task execution status of your pipeline.

In this example, the Jira Query component loaded 20 rows of data, which meets the conditional logic of jira_row_count being greater than 0. Therefore, the If component moves forward to an End Success component, rather than the End Failure component.

The Print Variables component displays its message jira_row_count = 20, confirming the value of ${sysvar.thisComponent.rowCount}, which has been mapped to jira_row_count.

Task history

Why was this useful?

A component can run successfully while loading 0 rows. For example, if the Jira Query connector remained set up correctly but was pulling data from an empty Jira project (i.e. no issues), then the connector could succeed with a row count of 0. By assigning our user-defined variable to a system variable that produces statistical and referential data and metrics about a component at runtime, we ensured that an If component could determine the success or failure of the pipeline based on the desired minimum row count value.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement