> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# WhileIterator

> Recursively processes rows

<Accordion title="Dependencies & cluster requirements">
  **Dependencies:**

  * ProphecySparkAlteryxPython 0.0.1+

  **Cluster requirements:**

  * UC dedicated clusters 14.3+ supported
  * UC standard clusters 14.3+ supported
  * Livy clusters not supported
</Accordion>

The WhileIterator allows you to group a set of gems together, and then run them one after another repeatedly in a loop.

<img src="https://mintcdn.com/prophecy-62973bd0/YMU5yAViYYX3rOGW/data-engineering/gems/subgraph/img/while-iterator.png?fit=max&auto=format&n=YMU5yAViYYX3rOGW&q=85&s=28385e436f05f46dbfbc2138f90ddbac" alt="WhileIterator" width="2620" height="1508" data-path="data-engineering/gems/subgraph/img/while-iterator.png" />

## Input

The input of the WhileIterator can be any dataset. Importantly, the output schema of the WhileIterator **must** match or be a subset of the input schema. This is because the output feeds back into the input as the subgraph is iterating (looping).

## Configuration

<Info>
  Before setting the WhileIterator configuration, make sure you are familiar with [pipeline
  configurations](/data-engineering/development/pipelines/configuration).
</Info>

Once you have a WhileIterator subgraph on your canvas, set configurations by opening the subgraph and clicking on **Configuration**. The following table describes the parameters you need to set.

| Parameter                                                | Tab           | Description                                                                                                                                                                                                                          |
| -------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Max iterations                                           | Settings      | The maximum number of loops that will be performed.                                                                                                                                                                                  |
| Populate iteration number in config variable             | Settings      | A checkbox to enable if you want to keep track of iteration numbers in a variable.                                                                                                                                                   |
| Select config variable name to populate iteration number | Settings      | The name of the config variable that will store the iteration number.                                                                                                                                                                |
| Schema                                                   | Configuration | A table where you define configuration variables. You can **Copy Pipeline Configs** to inherit these from your pipeline. If you want to populate the iteration number in a config variable, make sure to include that variable here. |
| Config                                                   | Configuration | An area where you can define default values for variables in one or more configurations.                                                                                                                                             |

## Subgraph

Within the WhileIterator, you can design a flow with multiple gems to create your desired output.

Importantly, you also need to understand the two inputs of the WhileIterator\_Output. The order of your inputs is important:

* The **in0** dataset contains the DataFrame for the **following** iteration. If this dataset becomes empty, the loop is terminated.
* The **in1** dataset passes on an individual iteration. You'll see the entire list of iterations as the output of the WhileIterator gem.

## Break condition

There are two scenarios in which the WhileIterator will break:

* The **in0** dataset of the WhileIterator\_Output becomes empty.
* The maximum number of iterations is reached.

## Output

There are two outputs of the WhileIterator:

* The **out0** dataset that contains the output of the **last iteration** of the WhileIterator.
* The **out1** dataset that contains the whole list of iterations produced by the WhileIterator. This is produced by performing a union on all output iterations.

***

## Example

Imagine you are simulating an investment that earns a fixed percentage of interest per year. You want to keep compounding the interest until your investment reaches a desired amount. You can use the WhileIterator gem to produce a dataset demonstrating this yearly growth.

<img src="https://mintcdn.com/prophecy-62973bd0/YMU5yAViYYX3rOGW/data-engineering/gems/subgraph/img/while-iterator-output.png?fit=max&auto=format&n=YMU5yAViYYX3rOGW&q=85&s=c2dd7a452c441709d670312cf31fe03a" alt="WhileIterator" width="2620" height="1508" data-path="data-engineering/gems/subgraph/img/while-iterator-output.png" />

For this example, the input of the WhileIterator will be a simple table that describes your starting investment and the fixed interest rate.

<div class="table-example">
  | `year` | `principal` | `interest_rate` |
  | ------ | ----------- | --------------- |
  | 0      | 1000        | 0.05            |
</div>

### Add Reformat gem

Now, let's use a Reformat gem in the WhileIterator subgraph to update the **Year** and **Principal** each iteration.

1. Add the Reformat gem inside the WhileIterator.
2. Connect WhileIterator\_Input to the Reformat gem.
3. Connect the Reformat gem to the WhileIterator\_Output.
4. Configure the **Expressions** table as it is shown below.

| Target Column   | Expression                               |
| --------------- | ---------------------------------------- |
| `year`          | `year + 1 `                              |
| `principal`     | `principal + principal \* interest_rate` |
| `interest_rate` | `interest_rate`                          |

### Terminate loop

The while loop will stop if the **in0** dataset becomes empty. How can the **in0** dataset become empty? In this example:

1. Add a Filter gem with the condition `principal < 2000`.
2. Connect the Reformat gem output to the input of the Filter gem.
3. Connect the Filter gem output to the input of WhileIterator\_Output.

This way, the Filter gem will return data as-is until the principal reaches 2000. After that, the record is filtered out, and the loop will terminate.

<img src="https://mintcdn.com/prophecy-62973bd0/YMU5yAViYYX3rOGW/data-engineering/gems/subgraph/img/while-iterator.png?fit=max&auto=format&n=YMU5yAViYYX3rOGW&q=85&s=28385e436f05f46dbfbc2138f90ddbac" alt="WhileIterator" width="2620" height="1508" data-path="data-engineering/gems/subgraph/img/while-iterator.png" />

### View output data

The **out0** dataset should be empty because we used a Filter gem to return an empty set when the `principal > 2000`.

Below is an example **out1** dataset. This lets you project how your principal will grow over the years.

<div class="table-example">
  | `year` | `principal`    | `interest_rate` |
  | ------ | -------------- | --------------- |
  | 1      | 1050           | 0.05            |
  | 2      | 1102.5         | 0.05            |
  | 3      | 1157.625       | 0.05            |
  | 4      | 1215.50625     | 0.05            |
  | 5      | 1276.2815625   | 0.05            |
  | 6      | 1340.095640625 | 0.05            |
  | 7      | 1407.100422656 | 0.05            |
  | 8      | 1477.455443789 | 0.05            |
  | 9      | 1551.328215978 | 0.05            |
  | 10     | 1628.894626777 | 0.05            |
  | 11     | 1710.339358116 | 0.05            |
  | 12     | 1795.856326022 | 0.05            |
  | 13     | 1885.649142323 | 0.05            |
  | 14     | 1979.931599439 | 0.05            |
  | 15     | 2078.928179411 | 0.05            |
</div>
