Skip to content

System variables

Public preview

System variables offer statistical and referential data about a component's execution after a pipeline has run. System variables are therefore useful for auditing and logging purposes. Examples of component data available through system variables include project ID, status, row count, and execution duration. Each component in an orchestration pipeline has a Post-processing tab where you can map your user-defined pipeline variables and project variables to system variables.

A complete list of available system variables can be found below.

Note the following behavior for usage:

  • Because system variables are limited to the Post-processing tab, you'll need to map any system variables you wish to use in your pipelines to user-defined pipeline or project variables.
  • System variable names cannot be changed.
  • System variable values are updated automatically as part of a component's execution.

Accessing system variables

System variables can be accessed in the Post-processing tab of any orchestration pipeline component, where you can map a system variable to a user-defined variable. For more information, read Post-processing.

System variables syntax

You can reference a system variable in a component property using the same ${} notation syntax as in user-defined variables. For more information, read Variables.

The ${} syntax provides access to various execution metrics, such as row count, status, and system constants, such as environment name and project ID. All system variables are prefixed with sysvar.. For example, a system variable can be used as follows:

${sysvar.thisComponent.rowCount}

List of system variables

This table lists the system variables currently available for use in post-processing.

System variable Description
.artifact.versionName Returns the version name of the artifact being executed. For schedules, API executions, and invocations in a shared pipeline, this corresponds to the version name provided during publishing. For runs in Designer, it is a dynamically generated UUID (since there is no version name in this scenario).
.environment.name Name of the environment the pipeline is being executed in.
.project.id ID of the project the pipeline is being executed in.
.rootPipeline.executionId The execution ID of the top level pipeline being triggered.
.thisComponent.rowCount Row count of the records loaded in the current component task execution.
.thisComponent.startedAt The starting timestamp of the current component task execution, in the ISO 8601 format UTC timezone, for example, 2024-11-11T11:26:06Z.
.thisComponent.mainTaskFinishedAt The main task finishing timestamp of the current component task execution, in the ISO 8601 format UTC timezone, for example, 2024-11-11T11:26:06Z. Note that the Task History finish time will also include any post-processing time, which, while minimal, may mean that that value may be slightly different to the value returned in this system variable.
.thisComponent.mainTaskDurationMs The duration of the main component task execution in milliseconds. Note that the Task History duration will also include any post-processing time, which, while minimal, may mean that that value may be slightly different to the value returned in this system variable.
.thisComponent.message The Task message of the current component task execution.
.thisComponent.status The status message of the main component task execution.
.thisComponent.name Component name of the current component task execution.
.thisComponent.taskId Execution ID of the current component task execution.
.thisPipeline.fullName Full name of the current pipeline, including the full folder path and file extension.
.thisPipeline.executionId Execution ID of the current pipeline execution. Where the current pipeline is the root pipeline, the ID will be the same as the .rootPipeline.executionId.

Example: Using system variables in post-processing

The following example uses a simple orchestration pipeline to demonstrate the effectiveness of using system variables and post-processing. In this pipeline, we will connect to Jira, extract and load a table of issues, and then use an If component to read a user-defined variable's value and set the pipeline's end state to success or failure. The user-defined variable will be mapped to a system variable using the post-processing capabilities in Designer.

This example uses the following components:

  • Start: this is the starting point of any orchestration pipeline.
  • Jira Query: this is the connector we'll use to extract and load data, and which we'll base our system variable's value on.
  • If: this is how we'll conditionally define the success or failure of our pipeline based on the value of our system variable.
  • Print Variables: this will be used to print the value of our variable to the task history.
  • End Success: a successful end point for our pipeline.
  • End Failure: an unsuccessful end point for our pipeline.

Pipeline canvas

Create a user-defined variable

  1. In Designer, open the Variables menu.
  2. Click Add to begin creating a new variable.
  3. In the Add a variable dialog, configure the basic settings of your variable.
    1. Select whether the variable is a project variable or a pipeline variable. Read Variable scope for more information.
    2. Set the variable type. Read Variable type for more information.
    3. For the purposes of this example, the scope is set to Pipeline variable and the type is set to Number.

  4. Click Next.

    Add variable

  5. In the left-hand tab, provide a name for your new variable. You can provide a description too, but this is optional.

  6. For this example, the Visibility and Behavior settings will be left as-is, and the default value is left blank.
  7. Click Create.

Set up variable

Set up system variables

In this example, a Jira Query component is added to an orchestration pipeline to extract and load data from the Jira Issues data source.

The Post-processing tab is available in all orchestration pipeline components in the same location and works in the same manner, so using a different connector (such as Salesforce or Shopify) than Jira if you're following this example is fine.

To set up a system variable:

  1. Click on the orchestration component you wish to set up a system variable on. In this example, the Jira Query component is selected.
  2. Switch over to the Post-processing tab.

  3. Click into the Update Scalar Variables parameter.

    Set up system variables

  4. The Name column displays any existing pipeline and project variables. Select the variable you created earlier.

  5. The Value column is where you'll select a system variable to map your user-defined variable to.
    1. In the Value field, type ${sysvar to begin with, and all available system variables will be displayed.
    2. In this example, we will use ${sysvar.thisComponent.rowCount}, a system variable that returns the row count of the component at runtime, and we're going to map this to our user-defined variable named jira_row_count.
  6. Click Save to finish setting up your system variables.

Set up system variables

Set up Print Variables

This section is optional. The Print Variables component will display the value of any specified variables in the Task history tab.

In this case:

  • The Variables to print parameter is configured to print the value of the variable jira_row_count.
  • The Prefix text parameter is left blank.
  • The Include variable name parameter is set to Yes.

Set up success and failure conditions

  1. In this part of the example, an If component is used to define conditional logic.
  2. In the If component's Condition parameter, a condition is defined. In this example, the value of our user-defined variable jira_row_count must be greater than 0 for the condition to pass as true.
  3. When the condition is true, the If component will connect to an End Success component and finish the pipeline successfully. However, if the condition is false, the If component will connect to an End Failure component and the pipeline will fail.

Set up conditional logic

Run the pipeline

It's time to run our pipeline.

  1. Click Run.
  2. In the Task history tab, double-click the running task to view the task execution status of your pipeline.

In this example, the Jira Query component loaded 20 rows of data, which meets the conditional logic of jira_row_count being greater than 0. Therefore, the If component moves forward to an End Success component, rather than the End Failure component.

The Print Variables component displays its message jira_row_count = 20, confirming the value of ${sysvar.thisComponent.rowCount}, which has been mapped to jira_row_count.

Task history

Why was this useful?

A component can run successfully while loading 0 rows. For example, if the Jira Query connector remained set up correctly but was pulling data from an empty Jira project (i.e. no issues), then the connector could succeed with a row count of 0. By assigning our user-defined variable to a system variable that produces statistical and referential data and metrics about a component at runtime, we ensured that an If component could determine the success or failure of the pipeline based on the desired minimum row count value.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement