> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Data sampling

> Choose when to sample data during interactive execution

<Callout icon="/images/icon.png" color="#FFC107">
  Available for [Enterprise Edition](/data-engineering/administration/platform/editions) only.
</Callout>

Prophecy gives you control over when and where data samples are generated during interactive pipeline execution. This helps you optimize for speed, visibility, or compatibility with your fabric setup.

You can customize data sampling at three levels:

* **Gem level**: Turn sampling on or off for individual or multiple gems.
* **Pipeline level**: Set the sampling mode used during interactive runs.
* **Fabric level**: Enable or disable data sampling for pipelines running on particular fabrics.

This page describes how to set up and use data samples for your use cases.

## Interactive run configuration

You can adjust data sampling settings for each pipeline through the **Interactive Run Configuration** panel. You can also access the same settings in [Pipeline Settings](/data-engineering/development/pipelines/pipeline-settings#run-settings).

1. Hover the large **play** button in the canvas.
2. Click on the **ellipses** that appears on hover.
3. Toggle data sampling on or off.
4. When data sampling is on, select your [preferred mode](#data-sampling-modes) from the **Data Sampling** dropdown.

<img src="https://mintcdn.com/prophecy-62973bd0/RfFLi6i-TgWOlf5t/data-engineering/development/runs/img/interactive-run-config.png?fit=max&auto=format&n=RfFLi6i-TgWOlf5t&q=85&s=2937f67da81e284a613319855dca8493" alt="Interactive run configuration" width="2620" height="1508" data-path="data-engineering/development/runs/img/interactive-run-config.png" />

## Data sampling modes

Prophecy provides the following data sampling modes.

| Mode              | Samples generated                                                         | Use case                           |
| ----------------- | ------------------------------------------------------------------------- | ---------------------------------- |
| **All** (default) | After every gem, excluding Target gems.                                   | Full visibility                    |
| **Selective**     | When **Data Preview** enabled per gem. [Learn more](#selective-sampling). | Full control per gem               |
| **Sources**       | Only after Source gems.                                                   | Focus on inputs                    |
| **Targets**       | Only before Target gems.                                                  | Focus on outputs                   |
| **IO**            | Only after Sources and before Targets (not between intermediate gems).    | High-level input/output inspection |

### Selective sampling

Selective data sampling gives you granular control by letting you enable or disable data samples for individual gems. To control data sampling for gems:

* **Single gem**: Select the Data Preview checkbox in the gem's [action menu](/data-engineering/gems/gems).
* **Multiple gems**: Select multiple gems by dragging, then click the Data Preview button in the bottom menu.

When Data Preview is disabled for a gem, its output appears pale after pipeline execution, indicating no sample was generated. Click the pale output to load the data sample on demand. The output icon will then display in normal bold colors.

<img src="https://mintcdn.com/prophecy-62973bd0/RfFLi6i-TgWOlf5t/data-engineering/development/runs/img/selective-interims.png?fit=max&auto=format&n=RfFLi6i-TgWOlf5t&q=85&s=70f6814deff56f949d24f4b2c4329357" alt="Selective" width="2620" height="1508" data-path="data-engineering/development/runs/img/selective-interims.png" />

<Tip>
  Prophecy recommends using selective sampling mode for all users, regardless of your Spark
  provider.
</Tip>

<AccordionGroup>
  <Accordion title="Modify the behavior of selective sampling">
    Selectively-generated samples load up to 10,000 rows (or 2 MB payload) by default. Set the following environment variables for your Spark cluster to modify this behavior:

    * `EXECUTION_DATA_SAMPLE_LOADER_MAX_ROWS`: Max number of rows (default is 10,000 rows).
    * `EXECUTION_DATA_SAMPLE_LOADER_PAYLOAD_SIZE_LIMIT`: Max payload size (default 2 MB).
    * `EXECUTION_DATA_SAMPLE_LOADER_CHAR_LIMIT`: Per column character limit (default 200 KB). Values exceeding the limit are truncated.
  </Accordion>
</AccordionGroup>

## Cached interims

When you change data sampling settings and re-run a pipeline, some data samples may appear grayed out. These cached samples are from previous runs and may not reflect your current data or pipeline changes.

<img src="https://mintcdn.com/prophecy-62973bd0/d9a4F1BI2-0KzuSv/data-engineering/development/runs/img/cached-interims.png?fit=max&auto=format&n=d9a4F1BI2-0KzuSv&q=85&s=96304300e8f53305571115cc4df0b74c" alt="Cached interims" width="2620" height="1508" data-path="data-engineering/development/runs/img/cached-interims.png" />

## Record counts

In addition to data samples, Prophecy can also display the total record count of datasets between gems. This works for **selective data sampling** mode only.

To display the record count for a certain gem output:

1. Click the **...** (ellipsis) on the gem.
2. Select the **Record Count** checkbox.

<img src="https://mintcdn.com/prophecy-62973bd0/2XdHm-fm7TNk_sYY/data-engineering/development/runs/img/gem-menu.png?fit=max&auto=format&n=2XdHm-fm7TNk_sYY&q=85&s=fbcb9949288f0ae2a24384dc59d76087" alt="Gem menu with Record Count checkbox" width="2872" height="1610" data-path="data-engineering/development/runs/img/gem-menu.png" />

You can enable the record count and leave the data preview option disabled. They are independent of each other.

The following image shows a pipeline with the record count enabled on the DataCleansing and Reformat gems. Notice that the Reformat gem is the only gem that has data preview enabled.

<img src="https://mintcdn.com/prophecy-62973bd0/2XdHm-fm7TNk_sYY/data-engineering/development/runs/img/record-count.png?fit=max&auto=format&n=2XdHm-fm7TNk_sYY&q=85&s=3f5df9eb5064671442b33425348d2d3d" alt="Pipeline with record count enabled" width="2872" height="1610" data-path="data-engineering/development/runs/img/record-count.png" />

## Fabric settings

In a fabric, you can enable or disable data sampling and override pipeline-level settings when a pipeline runs on that fabric. You can access this option in the **Advanced** tab of a fabric. A common use case is preventing sample data generation in **production** pipelines.

<img src="https://mintcdn.com/prophecy-62973bd0/RfFLi6i-TgWOlf5t/data-engineering/development/runs/img/limit-data-preview-interims.png?fit=max&auto=format&n=RfFLi6i-TgWOlf5t&q=85&s=85b2caba52c1d157de27143b90a49da7" alt="Create a new model test" width="2620" height="1508" data-path="data-engineering/development/runs/img/limit-data-preview-interims.png" />

By default, only team admins can access the Advanced tab in a fabric. However, there are two flags you can set in your deployment to change this behavior:

* `ALLOW_FABRIC_ACCESS_CLUSTER_ADMIN`: Grants cluster admins full access to fabrics, even if they are not team admins.
* `DISALLOW_FABRIC_CODEDEPS_UPDATE_TEAM_ADMIN`: Prevents team admins from modifying the data sampling settings within a fabric.