Upgrade: Iterators🔗

All Matillion ETL iterator components will be migrated to the equivalent iterator component in the Data Productivity Cloud.

Upgrade path🔗

In many cases, the migration will proceed with no manual refactoring of pipelines needed, but there are some specific differences to be aware of that may need to be addressed in your migrated pipelines. These are outlined below.

Stop On Condition🔗

Iterators with Stop On Condition set to No in Matillion ETL will migrate to the Data Productivity Cloud without any issues, and will perform as expected.

Iterators with Stop On Condition set to Yes in Matillion ETL will migrate to the Data Productivity Cloud, but pipelines may not perform as expected due to this property being missing. You should refactor your pipelines, if possible, to work without requiring this property.

Iterations Generated export variable🔗

Iterator components in Matillion ETL have an export variable called Iterations Generated, which returns the number of rows the iterator has looped through. This variable does not have an equivalent in the Data Productivity Cloud, but you can replicate its functionality in a number of ways, as described below. This will require some manual refactoring of your pipeline.

For a Grid Iterator, you can define a pipeline variable to replace Iterations Generated:

In your orchestration pipeline, create a numeric pipeline variable.
Use an Update Scalar component to set the variable as follows:
```
${grid_variable_name.column_name.length}
```
Replace grid_variable_name with the name of the grid variable that your iterator uses, and column_name with the name of any column in that grid. This will set the pipeline variable to the number of rows in the grid variable that the iterator is looping through. Any column can be chosen for this purpose, as all columns in the grid are the same length. The length property retrieves the number of items in that column, which gives you the number of iterations.

For a Fixed Iterator, the same process can be used if you are using a grid for the iteration values.

For a Loop Iterator, if you are using variables to define the iteration values, you can query the variables to check how many iterations there will be.

For a File Iterator or Table Iterator, there is no simple workaround to replace Iterations Generated, and you may need to refactor your pipeline to work without this functionality.