Iterator components🔗
Iterator components run an attached component multiple times, once for each item in a defined set. On each iteration, the iterator sets one or more variables to new values, which the attached component can then reference.
All iterator components:
- Support Snowflake, Databricks, and Amazon Redshift.
- Are limited to a maximum of 5000 iterations.
- Can be stacked to create nested loops.
The following iterator components are available:
| Iterator | Iterates over |
|---|---|
| File Iterator | Files in a remote file system |
| Fixed Iterator | Manually defined values |
| Grid Iterator | Rows of a grid variable |
| Loop Iterator | A numeric sequence |
| Table Iterator | Rows of a database table or view |
Attaching, stacking, and detaching iterators🔗
Use iterators to loop components—that is, repeat the component without adding multiple copies to your pipeline. You can stack multiple iterators of the same type or different types on a component.
Attaching iterators🔗
There are two ways to attach an iterator to a component:
- Click the component and then click Iterate in the component action bar. Next, select the iterator you want to use. It will automatically be attached to the selected component on the canvas.
- Open the Add component dialog, search for and click the iterator you want to use, and then use the connection ring beneath the iterator to connect it to the input of the component you want to iterate.
Stacking iterators🔗
To stack iterators on a component, use the connection ring beneath each iterator to connect it to the input of the component you want to iterate. As you repeat this process for each iterator, they will stack on top of the component.
When you stack iterators, they are nested from top to bottom. The top iterator runs one step of its loop, followed by the second iterator, and so on. Once the bottom iterator has finished its first full loop, the top iterator runs its second step.
For example, if you have the following stack of iterators on a Print Variables component:
- Sequential File Iterator (3 files)
- Concurrent Loop Iterator (values 1 to 10)
- Print Variables component
They are processed as follows:
- The File Iterator runs its first step (file 1), the Loop Iterator runs all steps (values 1 to 10), and the Print Variables component prints ten times (file 1 value 1, file 1 value 2, file 1 value 3, and so on).
- The File Iterator runs its second step (file 2), the Loop Iterator runs all steps (values 1 to 10), and the Print Variables component prints ten times (file 2 value 1, file 2 value 2, file 2 value 3, and so on).
- The File Iterator runs its third step (file 3), followed by the Loop Iterator and Print Variables component in the same way as for the first two steps.
Detaching iterators🔗
Detach all stacked iterators from a component to reorder them or remove individual iterators. All iterators that were stacked on the component will be detached and shown in a horizontal row above the component. To detach iterators from a component:
- Open the pipeline containing the component with stacked iterators.
- Click the component or iterators you want to detach.
- In the component action bar, click Detach.