> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# TableIterator

> Loop over each row of an input Dataframe

<Accordion title="Dependencies & cluster requirements">
  **Dependencies:**

  * ProphecySparkBasicPython 0.1.4+
  * ProphecyLibsPython 1.8.4+
  * ProphecyLibsScala 7.1.66+

  **Cluster requirements:**

  * UC dedicated clusters 14.3+ supported
  * UC standard clusters 14.3+ supported
  * Livy clusters 3.0.1+ supported
</Accordion>

TableIterator allows you to iterate over one or more gems for each row of the first input DataFrame.
Let's see how to create a Basic Loop which loops over a Metadata Table, and for each row of the table will run the gems inside the Subgraph.

## Creating a TableIterator gem

First add the Input gem on which you want to Iterate over. For this, simply use an existing dataset or create a new Source gem pointing to your Metadata table.
You can run this Source gem to see the data your loop would be running for.

Now, Drag and Drop the **(1) TableIterator** gem from the Subgraph menu, and connect it to the above created Source gem.

<img src="https://mintcdn.com/prophecy-62973bd0/YMU5yAViYYX3rOGW/data-engineering/gems/subgraph/img/Create_table_iterator.png?fit=max&auto=format&n=YMU5yAViYYX3rOGW&q=85&s=4ec739d2999ff0ddd2983489c25fe670" alt="Create table iterator" width="2880" height="1084" data-path="data-engineering/gems/subgraph/img/Create_table_iterator.png" />

## Configure the TableIterator

Open the TableIterator gem, and click on **(1) Configure** to open the Settings dialog.
Here, on the left side panel you can edit the \*\*(2) Name \*\* of your gem, check the **(3) Input Schema** for your DataFrame on which the loop will iterate.

On the right side, you can define your Iterator Settings, and any other Subgraph Configs you want to use in the Subgraph.
In the **(4) Max Iterations** field, you can control the maximum number of Iterations this loop can have. This is to safeguard that nobody runs the loop on a very large DataFrame by mistake. The default value is set to 1000.

You can also **(5) Enable Parallel Execution**.

<img src="https://mintcdn.com/prophecy-62973bd0/YMU5yAViYYX3rOGW/data-engineering/gems/subgraph/img/Configure_table_iterator.png?fit=max&auto=format&n=YMU5yAViYYX3rOGW&q=85&s=a4701de1121478849e9500d678bc59a9" alt="Configure loop" width="2880" height="1084" data-path="data-engineering/gems/subgraph/img/Configure_table_iterator.png" />

When you check Enable Parallel Execution, you can also control **(1) Number of threads** to run in parallel as shown below.

<img src="https://mintcdn.com/prophecy-62973bd0/YMU5yAViYYX3rOGW/data-engineering/gems/subgraph/img/loop_settings.png?fit=max&auto=format&n=YMU5yAViYYX3rOGW&q=85&s=6b5b02ec7a8d5fff57546713fb9932c5" alt="Loop settings" width="2880" height="1726" data-path="data-engineering/gems/subgraph/img/loop_settings.png" />

In the table below that, map the columns from the incoming DataFrame, to the configs of the Subgraph.
When you pick a column name in **(2) Source column**, a config is automatically created with the same name in **(3) Config name**. You can see the configs created by switching to the **(4) Configuration** tab.

This will show all the configurations created for this Subgraph as shown below.

<img src="https://mintcdn.com/prophecy-62973bd0/YMU5yAViYYX3rOGW/data-engineering/gems/subgraph/img/loop_configs.png?fit=max&auto=format&n=YMU5yAViYYX3rOGW&q=85&s=36fb46617128cc3f9bdb6ff71add0f33" alt="Loop configs" width="2880" height="1084" data-path="data-engineering/gems/subgraph/img/loop_configs.png" />

You can add default values for any of these by switching to the **(1) Config Tab**, as shown above. You can also add more configurations in case you want to use in your subgraph.

<Info>
  Note, the configurations associated with a source column will have different values for each
  iteration based on the input data's current row value for that column. The configs which are not
  mapped to a source column, will have a fixed value for each Iteration provided in the Config tab.
</Info>

Once done, click on **(2) Save**, to save the Iterator configurations.

Now you can add the gems to your Subgraph on which you want to loop on. To do this simply Drag and Drop any gem onto the Subgraph Canvas.
You can add any Source, Target or any other transformation gem and use the configs inside these gems to change the values for each Iteration.

## Running the Loop

Once you have added the gems to your Subgraph, Click on the **(1) Run button** to start execution.
As the execution starts, you will see a new **(2) Iteration** button. This will show for which Iteration you are seeing the current interims.
You can click on the **(3) Interim** to check values for that Iteration.

<img src="https://mintcdn.com/prophecy-62973bd0/YMU5yAViYYX3rOGW/data-engineering/gems/subgraph/img/Run_loop.png?fit=max&auto=format&n=YMU5yAViYYX3rOGW&q=85&s=b35f8923e91bfa3b1cca6a25f500d3d9" alt="Run loop" width="2880" height="1726" data-path="data-engineering/gems/subgraph/img/Run_loop.png" />

Click on the **(2) Iteration** button, and it will open up the Iterations table as shown below. Here you can see all iterations and config values for each of them.

<img src="https://mintcdn.com/prophecy-62973bd0/YMU5yAViYYX3rOGW/data-engineering/gems/subgraph/img/Iterations.png?fit=max&auto=format&n=YMU5yAViYYX3rOGW&q=85&s=07b63ff8ead53bebee7e11bb851d3024" alt="Iterations" width="2880" height="1726" data-path="data-engineering/gems/subgraph/img/Iterations.png" />

## Adding Inputs and Outputs to TableIterator

For a TableIterator gem, the first input port is for your DataFrame on which you want to Iterate Over.
You can **(1)Add** more Inputs or Switch to **(2) Output** tab to add more Outputs as needed. These extra inputs would not change for every iteration.
Also, the output will be a Union of output of all Iterations. You can **(3) Delete** any port by hovering over it.

<img src="https://mintcdn.com/prophecy-62973bd0/YMU5yAViYYX3rOGW/data-engineering/gems/subgraph/img/loop_additional_ports.png?fit=max&auto=format&n=YMU5yAViYYX3rOGW&q=85&s=adeb78f3d5bc87f6f020cec0a4c3ca03" alt="Table iterator ports" width="2880" height="1084" data-path="data-engineering/gems/subgraph/img/loop_additional_ports.png" />
