When you click on a gem from the gem drawer, an instance of that gem gets added to your pipeline canvas. Use the image and the table below to understand the UI of a gem.
Callout
UI element
Description
1
Gem label
The name of this particular gem instance. It must be unique within a given pipeline.
2
Gem type name
The type of gem.
3
Input ports
One or more ports that accept connections from upstream gems.
4
Output ports
One or more ports that connect to downstream gems.
5
Gem phase
The phase for this gem instance, which defines the order in which gem instances are executed.
6
Open
The button that lets you open the gem configuration.
7
Run button
A button that runs the pipeline up to and including the gem.
8
Action menu
A menu that includes options to change the phase of the gem, add run conditions, delete the gem, and more.
If you select one or more gems, you can copy and paste them within the same pipeline or across
pipelines. However, you cannot paste across projects that use different languages (for example,
from SQL to Scala).
The Input tab defines the incoming connections accepted by the gem. Most gem types only accept one connection, but some (such as Join) allow for multiple inputs.
UI element
Description
1
Search
A field that will filter your input columns.
2
Add Input
If the gem supports multiple inputs, you can click this button to add more input ports to this instance.
3
Input
The name of the input port and the name of the input gem instance.
4
Port schema
The schema of the port (columns and column types). The schema will only appear when an input port is present.
The Output tab defines the outgoing schemas that will be available to downstream gems. In some cases, the Prophecy compiler can’t infer the output schema automatically, so you have the option to infer the schema using your connected fabric or specify it manually.
UI element
Description
1
Output schema
Output schema for this gem instance. This will be the schema of the data that downstream gem instances will use
2
Custom schema
Toggle this to enable custom output schema editing
3
Infer from cluster
Run the gem code on the connected cluster and infer the schema from the result
Cache the entire gem output dataset. This is useful if you have multiple pipeline branches from one gem. If you cache the dataset from that gem, Spark will not have to compute upstream gems multiple times for each branch.
Data Preview
Selective data sampling only: Use this checkbox to enable/disable the generation of the data sample for the gem.
Record Count
Selective data sampling only: Use this checkbox to enable/disable display of the total record count for a gem’s output. The record count appears above the interim after the pipeline runs.
Data Diff
When Data Diff is configured for a Target gem, use this checkbox to enable or disable the data diff output.
In a data pipeline, the phase of a gem determines the sequence in which it runs. This sequencing is managed by adjusting the order of the generated pipeline code.
def apply(spark: SparkSession): Unit = { val df_my_orders = my_orders(spark).cache() val df_Repartition_1 = Repartition_1(spark, df_my_orders) Write_CSV(spark, df_Repartition_1) val df_SchemaTransform_1 = SchemaTransform_1(spark, df_my_orders)}
Here’s how it works:
Gems are assigned a numerical phase (e.g., 0, 1, -1), where lower values run first. For example, a gem with phase 0 will execute before a gem with phase 1.
When a gem runs, all its upstream gems must also run. This means that if a downstream gem has phase 0 and an upstream gem has phase 1, the upstream gem will be grouped into phase 0 to ensure proper execution.
Because of this dependency, the phase assigned to the last gem in a branch determines the phase of the entire branch. This means that when configuring gem phases, you only need to focus on the leaf nodes—the final gems in each branch of the pipeline.
