Skip to main content
Gems are functional units in a pipeline that perform tasks such as reading, transforming, writing, or handling other data operations. Combine gems on the canvas to build a pipeline, connecting each gem’s output to the next gem’s input.
Use the Prophecy Agent to help you add gems throughout your pipeline.

​
Categories

You can select gems from the following categories. Each category opens a dropdown menu when you click it.

​
Bookmark favorite gems

You can bookmark frequently used gems so they’re easiliy available from the gem drawer. To bookmark a gem, click the star icon next to the gem in the category’s dropdown menu. Starred gems appear in the Favorites section of the toolbar, where they’re displayed as icon-only shortcuts. From Favorites, you can:
  • Click a gem to add it directly to the canvas.
  • Drag a gem onto the canvas.
  • Click the star again to remove it from your favorites.

​
Interactive gem examples

To test a gem hands-on, you can try the interactive example of the gem. If you search for a gem in the project sidebar, you can open the associated example and run the preconfigured pipeline! Gem example

​
Expressions

Many gems include expression components where you can implement custom or complex logic. You can either build expressions visually, or use SQL syntax to write expression in code. Prophecy will automatically convert visual expressions to code expressions, and vice versa.
The SQL dialect for expressions depends on the SQL warehouse connection of your fabric. Project code must match the requirements of the execution environment. As a result, expressions can differ slightly between projects that run on Databricks, Snowflake, or Google BigQuery.

​
Gem instance

When you click on a gem from the gem drawer, Prophecy adds the gem to your pipeline. The callouts in the image below correspond to the numbered UI elements described in the table. Gem instance
If you select one or more gems, you can copy and paste them within the same pipeline or across pipelines. However, you cannot paste across projects that use different languages (for example, from SQL to Scala).

​
Gem configuration

Once a gem is on the canvas, you can configure its behavior. Available fields available depend on the gem type. See the documentation for each gem to learn about its specific parameters.

​
Gem ports

Each gem handles ports differently. Some require specific input and output ports. Others allow only one port or none at all. Refer to the documentation for each gem to understand its port requirements. To add a port:
  1. Open the gem to access its configuration.
  2. On the left, choose the Input or Output tab under Ports.
  3. Click the + button next to Ports. If the + button doesn’t appear, the gem does not support adding more ports.
To remove ports from a gem:
  1. Open the gem to access its configuration.
  2. On the left, choose the Input or Output tab under Ports.
  3. Click the pencil icon to enter Edit mode.
  4. Hover over the port you want to remove and click the trash icon.
  5. Click Done to save your changes and exit Edit mode.
To hide ports from the gem configuration, you can collapse the ports panel using the collapse icon next to Ports.

​
Visual and code view

Most gems can be configured in the visual view or the code view. Use the visual expression builder to populate fields in the visual view. Prophecy will automatically convert visual expressions into SQL expressions. You can edit these SQL statements or write your own in the code view.

​
Add a gem from a connection

You can quickly connect a new gem by hovering over the connector between two gems to reveal a + button. Click it to open the gem palette and insert a new gem directly at that point in the flow.
  • Hover over a gem’s input port to add a gem before it. If a connection already feeds that port, the new gem is spliced into it automatically
  • Hover over a gem’s output port to add a gem after it. If the port has no downstream connections, the new gem attaches directly. If the port already feeds one or more gems, you choose how to insert it:
    • Insert into the flow — the new gem is placed between the existing gems, so data now passes through it (A → New Gem → B).
    • Add as a separate path — the new gem reads the same output as a new branch, and existing connections stay unchanged.
options for adding gem to output The gem palette only shows gems compatible with the selected insertion point, grouped by category. Use the search field at the top of the palette to filter by name instead of scrolling through categories. Prophecy adjusts the canvas layout automatically to fit the new gem, and the entire change — the new gem, any rewired connections, and any repositioned gems — undoes as a single step.

​
Action menu

Selecting one or more gems reveals a toolbar at the bottom of the canvas with the following buttons: The Actions dropdown includes:

​
Applying actions to multiple gems

You can select multiple gems and apply Change Phase, Attach Annotation, Data Preview, Record Count, and Group to all of them at once from the Actions dropdown.
Bulk Data Preview and Record Count toggles skip gems that don’t support them, such as canvas comments, target-process gems, and containers. If your selection has a mix of current values (for example, some selected gems have Data Preview on and others off), the checkbox shows an indeterminate state until you click it.
To apply actions to multiple gems:
  1. Drag to select multiple gems on the canvas.
  2. Click Actions at the bottom of the canvas.
  3. Select desired action.

​
Gem phase

In a data pipeline, the phase of a gem determines the sequence in which it runs. Here’s how it works:
  • Gems are assigned a numerical phase (e.g., 0, 1, -1), where lower values run first. For example, a gem with phase 0 will execute before a gem with phase 1.
  • When a gem runs, all its upstream gems must also run. This means that if a downstream gem has phase 0 and an upstream gem has phase 1, the upstream gem will be grouped into phase 0 to ensure proper execution.
  • Because of this dependency, the phase assigned to the last gem in a branch determines the phase of the entire branch. This means that when configuring gem phases, you only need to focus on the leaf nodes—the final gems in each branch of the pipeline.
Normally, pipeline branches run in parallel. Using gem phases, you can develop your pipeline to run in different stages. This can be useful when one part of the pipeline depends on the results of another, allowing you to control the execution order and ensure that data flows correctly from one stage to the next.