Migration: Export variables
Matillion ETL export variables are called system variables in the Data Productivity Cloud, and operate in broadly the same way, with a few notable differences listed below.
For more details of variable migration in general, read Migration: Variables.
Migration path
Mappings of component exports to job or environment variables are automatically migrated to the Data Productivity Cloud. For more information, read Migration: Variables.
The following table lists the Data Productivity Cloud equivalents for some commonly used export variables. This list isn't exhaustive—consult the Data Productivity Cloud documentation for an up-to-date list of system variables.
| Matillion ETL component export | Data Productivity Cloud system variable |
|---|---|
| Completed At | .thisComponent.finishedAt |
| Component | .thisComponent.name |
| Duration | .thisComponent.duration |
| Message | .thisComponent.message |
| Row Count | .thisComponent.rowCount |
| Started At | .thisComponent.startedAt |
| Status | .thisComponent.status |
The following table outlines the component-specific export variables that can be migrated from Matillion ETL to the Data Productivity Cloud:
| Matillion ETL component | Matillion ETL export variable | Data Productivity Cloud system variable |
|---|---|---|
| SQL Query | Query ID | .thisComponent.queryId |
| Table update | Rows Deleted | .thisComponent.rowsDeleted |
| Table update | Rows Inserted | .thisComponent.rowsInserted |
| Table update | Rows Updated | .thisComponent.rowsUpdated |
Some Matillion ETL components have specific export variables that currently have no equivalents in the Data Productivity Cloud, and you will have to refactor to work around these. The following table has a (non-exhaustive) list of some common examples.
| Matillion ETL component | Variables |
|---|---|
| Append to Grid | Rows Added |
| Create Stream | Stream Recreated |
| Create Table | Table Recreated |
| Data Transfer | Bytes Written |
| Extract components (e.g. API Extract, Amplitude Extract, etc...) | Data Structure |
| Iterator components | Iterations Attempted, Iterations Generated, Iterations Successful |
| JDBC Table Metadata to Grid | Column Count |
| Query components (e.g. Database Query, DynamoDB Query, Email Query, etc...) | Filename, Time Taken to Load, Time Taken to Stage |
| Remove from Grid | Rows Removed |
| Retry | Iterations Attempted, Iterations Generated, Iterations Successful |
Functionality not supported
The following are not currently supported in the Data Productivity Cloud. If your Matillion ETL jobs incorporate any of these, you will need to refactor the migrated pipelines:
- Returning grid variables from a child pipeline.
- Component-specific properties, for example "Iterations generated".
- Use of system variables in an If component. This can only be done in the Data Productivity Cloud using "Advanced" mode.
- User defined variables.