Migration: Export variables
Matillion ETL export variables are called system variables in the Data Productivity Cloud, and operate in broadly the same way, with a few notable differences listed below.
For more details of variable migration in general, read Migration: Variables.
Migration path
Mappings of component exports to job or environment variables aren't automatically migrated to the Data Productivity Cloud. You must manually reconfigure the mapping in each migrated component, following the guidelines given here. Note that the job or environment variables you need to map to are migrated, as described in Migration: Variables.
The following table lists the Data Productivity Cloud equivalents for some commonly used export variables. This list isn't exhaustive—consult the Data Productivity Cloud documentation for an up-to-date list of system variables.
| Matillion ETL component export | Data Productivity Cloud system variable |
|---|---|
| Row Count | .thisComponent.rowCount |
| Started At | .thisComponent.startedAt |
| Completed At | .thisComponent.finishedAt |
| Duration | .thisComponent.duration |
| Message | .thisComponent.message |
| Status | .thisComponent.status |
| Component | .thisComponent.name |
Some Matillion ETL components have specific export variables that currently have no equivalents in the Data Productivity Cloud, and you will have to refactor to work around these. The following table has a (non-exhaustive) list of some common examples.
| Matillion ETL component | Variables |
|---|---|
| Append to Grid | Rows Added |
| Create Stream | Stream Recreated |
| Create Table | Table Recreated |
| Data Transfer | Bytes Written |
| Extract components (e.g. API Extract, Amplitude Extract, etc...) | Data Structure |
| Iterator components | Iterations Attempted, Iterations Generated, Iterations Successful |
| JDBC Table Metadata to Grid | Column Count |
| Query components (e.g. Database Query, DynamoDB Query, Email Query, etc...) | Filename, Time Taken to Load, Time Taken to Stage |
| Remove from Grid | Rows Removed |
| Retry | Iterations Attempted, Iterations Generated, Iterations Successful |
| SQL Script | Query ID |
| Table Metadata to Grid | Column Count |
| Table Update | Rows Deleted, Rows Inserted, Rows Updated |
Functionality not supported
The following are not currently supported in the Data Productivity Cloud. If your Matillion ETL jobs incorporate any of these, you will need to refactor the migrated pipelines:
- Returning grid variables from a child pipeline.
- Component-specific properties, for example "Iterations generated".
- Use of system variables in an If component. This can only be done in the Data Productivity Cloud using "Advanced" mode.
- User defined variables.