Skip to main content
/images/icon.png
Available for Enterprise Edition only.
Prophecy gives you control over when and where data samples are generated during interactive pipeline execution. This helps you optimize for speed, visibility, or compatibility with your fabric setup. You can customize data sampling at three levels:
  • Gem level: Turn sampling on or off for individual or multiple gems.
  • Pipeline level: Set the sampling mode used during interactive runs.
  • Fabric level: Enable or disable data sampling for pipelines running on particular fabrics.
This page describes how to set up and use data samples for your use cases.

​
Interactive run configuration

You can adjust data sampling settings for each pipeline through the Interactive Run Configuration panel. You can also access the same settings in Pipeline Settings.
  1. Hover the large play button in the canvas.
  2. Click on the ellipses that appears on hover.
  3. Toggle data sampling on or off.
  4. When data sampling is on, select your preferred mode from the Data Sampling dropdown.
Interactive run configuration

​
Data sampling modes

Prophecy provides the following data sampling modes.
ModeSamples generatedUse case
All (default)After every gem, excluding Target gems.Full visibility
SelectiveWhen Data Preview enabled per gem. Learn more.Full control per gem
SourcesOnly after Source gems.Focus on inputs
TargetsOnly before Target gems.Focus on outputs
IOOnly after Sources and before Targets (not between intermediate gems).High-level input/output inspection

​
Selective sampling

Selective data sampling gives you granular control by letting you enable or disable data samples for individual gems. To control data sampling for gems:
  • Single gem: Select the Data Preview checkbox in the gem’s action menu.
  • Multiple gems: Select multiple gems by dragging, then click the Data Preview button in the bottom menu.
When Data Preview is disabled for a gem, its output appears pale after pipeline execution, indicating no sample was generated. Click the pale output to load the data sample on demand. The output icon will then display in normal bold colors. Selective
Prophecy recommends using selective sampling mode for all users, regardless of your Spark provider.
Selectively-generated samples load up to 10,000 rows (or 2 MB payload) by default. Set the following environment variables for your Spark cluster to modify this behavior:
  • EXECUTION_DATA_SAMPLE_LOADER_MAX_ROWS: Max number of rows (default is 10,000 rows).
  • EXECUTION_DATA_SAMPLE_LOADER_PAYLOAD_SIZE_LIMIT: Max payload size (default 2 MB).
  • EXECUTION_DATA_SAMPLE_LOADER_CHAR_LIMIT: Per column character limit (default 200 KB). Values exceeding the limit are truncated.

​
Cached interims

When you change data sampling settings and re-run a pipeline, some data samples may appear grayed out. These cached samples are from previous runs and may not reflect your current data or pipeline changes. Cached interims

​
Record counts

In addition to data samples, Prophecy can also display the total record count of datasets between gems. This works for selective data sampling mode only. To display the record count for a certain gem output:
  1. Click the … (ellipsis) on the gem.
  2. Select the Record Count checkbox.
Gem menu with Record Count checkbox You can enable the record count and leave the data preview option disabled. They are independent of each other. The following image shows a pipeline with the record count enabled on the DataCleansing and Reformat gems. Notice that the Reformat gem is the only gem that has data preview enabled. Pipeline with record count enabled

​
Fabric settings

In a fabric, you can enable or disable data sampling and override pipeline-level settings when a pipeline runs on that fabric. You can access this option in the Advanced tab of a fabric. A common use case is preventing sample data generation in production pipelines. Create a new model test By default, only team admins can access the Advanced tab in a fabric. However, there are two flags you can set in your deployment to change this behavior:
  • ALLOW_FABRIC_ACCESS_CLUSTER_ADMIN: Grants cluster admins full access to fabrics, even if they are not team admins.
  • DISALLOW_FABRIC_CODEDEPS_UPDATE_TEAM_ADMIN: Prevents team admins from modifying the data sampling settings within a fabric.