Skip to content

Environment Variables

Environment variables are name:value pairs that are stored inside the Matillion ETL client and are fully configurable by its users. Unlike Job Variables, environment variables can be used across Matillion ETL, in configurations and in all jobs through many components.

Note

This article is part of a series on variables.

Creating environment variables

Environment variables must be created, or "declared", before being used. To create an environment variable:

  1. Click Project → Manage Environment Variables.
  2. Manage Environment Variables will open. The dialog lists every environment variable in a list. By default, they're sorted by Name, but you can sort the list on any column. To change the sort order, click a column title to sort in ascending order of that column, and click a second time to sort in descending order.
  3. To create a new environment variable, click + at the bottom of the overlay. This will create a blank row beneath the currently selected variable (or at the bottom of the list if none is selected), into which you can then type the details of the new variable.

  4. Provide the following details for the new variable:

    • Name: The name of the environment variable. Read Variables for variable naming guidelines. Note that you can't give a variable the same name as any of the automatic variables listed below.
    • Type: The Matillion ETL data type of the environment variable: Text, Numeric, or DateTime. Text is selected by default. Read Variables for more information on data types.
    • Behaviour: The behaviour of the environment variable will be Shared or Copied, depending on how the variable will be used by multiple job branches. Shared is selected by default.
    • Default Environment Value: The default value of the variable in each environment. The dialog includes a separate column for each environment, with the default values listed in the column. A variable's default value can be left blank, but failing to give a reasonable default value may cause problems in validating components that use that variable—for example, if a component uses a DateTime variable to which you have given a default value that is not in a valid DateTime format.
    • Description: An environment variable can optionally be given a description, which is a free-form text field that can contain any information about the variable that you wish to record. Descriptions are not displayed by default; to view descriptions, click + to the right of variable. To add or edit a description, click the edit button in the Description column, edit your text in the Description dialog, and then click OK.

To edit an existing environment variable, click in the column whose value you want to change, and then:

  • For Name or Default Environment Values, type the new text, then press the Enter key on your keyboard or click elsewhere in the dialog.
  • For Type or Behaviour, select the required value from the drop-down.

To delete an environment variable, click on the variable you want to delete (this will put you in editing mode, but ignore this) and then click the remove button at the bottom of the dialog.

Variables can also be added or edited in text mode by selecting the Text Mode checkbox.

Setting environment variable values

When a job begins, all variables are initialized with their default environment value, which should be set in the Manage Environment Variables overlay. The real power of a variable is that its value can be updated as a job runs, through the following methods:

Iteration components

Iteration components work by setting variables to a new value for each iteration. You must first define the variables you wish to iterate, using the Manage Environment Variables dialog, and then use those variables when configuring iteration components.

Iterations can be run in parallel, in which case the variable behaviour must be set to Copied to ensure each parallel component sees and iterates its own copy of the variable.

Python scripts

Python scripts can push new values into variables using their built-in context object, following this format:

context.updateVariable("variable", "new value")

Read the Python Script component documentation for more information.

SQS runs

Job executions triggered by SQS messages can also set variable values using the optional 'variables' item when sending the message. Read Manage SQS Configuration for more information, including the syntax for setting variables on the incoming message.

Variable exports

The Export tab of the Properties panel allows you to edit mappings between the runtime information that the component makes available and the variables you have already defined. All orchestration components and some transformation components support exporting runtime information into variables during job execution. A list of common exports is given in Component Exports.

Component export values are also used to populate the tasks view and the run history. To export a value, it must be mapped to an existing variable through the component's Export tab. It is important to map the value to a variable of the correct type according to the table given at Component Exports.

Note

When an environment variable is exported or imported, it's only the name of the variable that is exported/imported, not the value of the variable.

Automatic variables

The following environment variables are automatically available without first needing to be defined. The values of these variables can't be set in Manage Environment Variables but must be set as described in this table.

Variable Type Description
project_group_name Text Name of the current project group. Can be set via Rename Project Group from the Project menu.
project_group_id Numeric Internal ID of the current project group.
project_name Text Name of the current project. Can be set via Manage Project from the Project menu.
project_id Numeric Internal ID of the current project.
version_name Text Name of the current version. Versions can be renamed via Rename Version from the Project menu, unless locked. Read Version control for more information.
version_id Numeric The internal ID of the current version.
environment_name Text Name of the current environment. This can be set by right-clicking the environment in the Environments panel and selecting Edit Environment. Read Environments for more information.
environment_default_schema Text The name of the default schema for the current environment.
environment_database Text The name of the database for the current environment.
environment_endpoint Text URL of the Matillion ETL instance with current environment.
environment_id Numeric The internal ID of the current environment.
environment_port Numeric The port number of the current environment.
environment_username Text Username for the environment connection.
environment_catalog Text The default Databricks Unity Catalog set for the environment. Available on Databricks only.
job_name Text The name of the current job. This can be set by right-clicking the job in the Project panel and selecting Manage Job.
job_id Numeric The internal ID of the current job. All jobs have a unique ID that can be used to refer to it within a project. Note that this is not the ID of a particular run of a job.
component_name Text The name of the current component, as defined by the user. Components can be renamed by selecting them and editing the Name property.
component_id Numeric The internal ID of a given component in Matillion ETL.
component_message Text An error message returned by a component, which can be used in job error handling.
run_history_id Numeric The ID of a task in Matillion ETL. These can also be viewed via Task History.
detailed_error Text A detailed error description that contains the job_name, the component_name, and the component_message.
task_id Numeric The task ID associated with the current running job.

All of the above also have an id variable that is an internally generated ID and should be avoided in most cases.

Manipulating environment variables via the v1 API

Some examples of common ways to manipulate environment variables via the v1 API are given below.

When using these examples you should replace api-user and api-password with the username and password you use to access the instance. This user will require API access as well as any permissions required to alter environment variables.

These examples can be used through cURL by replacing [InstanceAddress], [GroupName], [ProjectName], [EnvironmentName], and [VariableName] with details appropriate for your resources.

To list all environment variables in an environment:

curl -X GET -o varlist.json -u api-user:api-password "http://<InstanceAddress>/rest/v1/group/name/<GroupName>/project/name/<ProjectName>/environment/name/<EnvironmentName>/variable"

Using that list of names, you can add /name/ to the cURL command to get the default value for variable :

curl -X GET -o varlist.json -u api-user:api-password "http://<InstanceAddress>/rest/v1/group/name/<GroupName>/project/name/<ProjectName>/environment/name/<EnvironmentName>/variable/name/<VariableName>"

This is not useful on its own, but can be used to manipulate that environment variable in the ways shown below.

Add /value to get the default value for variable :

curl -X GET -o varlist.json -u api-user:api-password "http://<InstanceAddress>/rest/v1/group/name/<GroupName>/project/name/<ProjectName>/environment/name/<EnvironmentName>/variable/name/<VariableName>/value"

Add /delete to delete the environment variable:

curl -X POST -u api-user:api-password "http://<InstanceAddress>/rest/v1/group/name/<GroupName>/project/name/<ProjectName>/environment/name/<EnvironmentName>/variable/name/<VariableName>/delete"

Add /set/value/ to update this environment variable to a new default value:

curl -X POST -u api-user:api-password "http://<InstanceAddress>/rest/v1/group/name/<GroupName>/project/name/<ProjectName>/environment/name/<EnvironmentName>/variable/name/<VariableName>/set/value/<NewValue>"

Environment variables video guide