Upgrade: Automatic variables
The Data Productivity Cloud supports most automatic variables and includes a similar concept to Matillion ETL called System variables. While both serve comparable purposes, they differ in syntax:
- Matillion ETL:
${my_variable_name}. - Data Productivity Cloud:
${sysvar.object.property}.
A clear pathway is established for migrating these variables and mapping them directly using the migration tool.
For more details of variable migration in general, read Upgrade: Variables.
Upgrade path
The table below lists all available automatic variables with their corresponding system variable mappings. To view the full list of system variables, read List of system variables. You will need to manually edit pipeline components to use the correct variables.
| Matillion ETL automatic variable | Data Productivity Cloud system variable |
|---|---|
| component_name | .thisComponent.name |
| component_message | .thisComponent.message |
| environment_name | .environment.name |
| job_name | .thisPipeline.fullName |
| project_id | .project.id |
| run_history_id | .rootPipeline.executionId |
| task_id | .thisComponent.taskId |
| version_name | .artifact.versionName |
Note
In the Data Productivity Cloud, the following variables are now UUIDs rather than integers:
.project.id.rootPipeline.executionID.thisComponent.taskID
This may mean you need to amend your pipelines and scripts to store these as TEXT rather than INTEGER datatypes.
Not yet supported
The following Matillion ETL automatic variables don't currently have equivalents in the Data Productivity cloud, but support will be added in a future release. See our Roadmap for details.
- Environment properties such as:
- environment_username
- environment_database
- environment_default_schema
- project_name
- queued_time
Not supported
Due to the differences in architecture between Matillion ETL and the Data Productivity Cloud, there are no equivalents to the following automatic variables. As a result, they do not have system variable mappings, as they cannot be migrated to the Data Productivity Cloud.
- detailed_error
- component_id
- job_id (see below)
- project_group_id
- project_group_name
- version_id
- environment_catalog
- environment_endpoint
- environment_id
- environment_port
Accessing through scripts
The Data Productivity cloud doesn't support directly accessing automatic variables through the Python Script or Bash Script components.
If you require this functionality, you can use an Update Scalar component to write the values to user-defined variables, which can then be passed to the script.
For more alternative options available when migrating Python and Bash scripts, read Upgrade: Bash and Upgrade: Python.
job_id
In Matillion ETL, the job_id variable is used to uniquely identify jobs, and therefore allow you to track them even if they're renamed.
In the Data Productivity Cloud, placing Git at the core of the architecture has meant that you can track jobs by knowing the artifact version and pipeline name, which are both available as system variables. The job_id variable is therefore not supported in the Data Productivity Cloud because it's not needed for tracking jobs.
You can retrieve the artifact version using ${sysvar.artifact.versionName} and the pipeline name using ${sysvar.thisPipeline.fullName}. This combination provides a unique identifier for each job execution, similar to how job_id is used in Matillion ETL.