Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt

Use this file to discover all available pages before exploring further.

The Pipeline gem allows you to run another pipeline from within your current pipeline. It supports conditional triggering, parameter passing, and returning metadata for each pipeline run. This gem is useful for building orchestrated workflows directly in the visual canvas. You can find it under the Custom category in the gem drawer. A single Pipeline gem execution can trigger the target pipeline multiple times. Triggered pipeline runs execute sequentially, and the Pipeline gem completes only after all triggered runs finish.
You can create pipelines solely dedicated to pipeline orchestration using multiple instances of this gem. We recommend labeling your orchestration pipelines to differentiate them from standard data pipelines.

​
Limitations

The Pipeline gem has the following limitations:
  • It can only trigger pipelines for the current project.
  • The Monitoring page doesn’t show the parent pipeline that triggered a run using the Pipeline gem.

​
Input and Output

The following table describes what the Pipeline gem expects as input and what it will produce as output.
PortDescription
in0Optional input dataset that may include a status column and additional columns.

Valid statuses include success, failure, or skipped.

Each row of the input triggers a separate pipeline run per Pipeline gem execution.
outOne output dataset that contains the metadata for each triggered pipeline run.

The Pipeline gem returns one row per triggered run.
When no input is connected:
  • The Pipeline gem triggers the child pipeline once.
  • The output contains a single row of metadata for that run.
When an input dataset is connected:
  • If a status column is present, it is used to evaluate trigger conditions. If the trigger condition is met, the Pipeline gem triggers the child pipeline once for each input row.
  • If a status column is not present, the Pipeline gem always triggers the child pipeline once for each input row.
  • Additional columns can be used to pass parameter values into each run.
  • The output contains one row of metadata per input row.
The input dataset can be any dataset (including the output of an upstream Pipeline gem).

​
Output schema

Each row in the output table represents a pipeline run triggered from the Pipeline gem. The output schema provides you with the following information for each pipeline run.
ColumnTypeDescription
statusStringFinal status of the triggered pipeline: success, failure, or skipped.
pipelineRunIDStringUnique ID of the triggered pipeline run.
startTimeTimestampTime when the triggered pipeline started.
endTimeTimestampTime when the triggered pipeline finished.
errorStringError message, if any.
logsString ArrayLog messages from the triggered pipeline.
Pipelines are skipped when the input does not meet the trigger condition set in the gem.

​
Parameters

The Pipeline gem accepts the following parameters.
ParameterDescription
Pipeline to runSelect a pipeline to trigger in the list of pipelines from the same project.
Trigger only ifChoose a condition to control when the trigger fires based on the status column of the input dataset (if present). If the condition is not met, the Pipeline gem runs, but no pipelines are triggered. The output will show that all pipeline runs were skipped.
Maximum number of pipeline triggersSet a maximum number of times the child pipeline runs per Pipeline gem execution. Maximum is 10,000.
Set pipeline parametersSet values for the pipeline parameters defined in the child pipeline. You can specify constants, expressions, or column values for each parameter. If you don’t specify a value, the child pipeline uses its default parameter value.
Use multiple input rows to launch runs of the same pipeline with different sets of parameter values.

​
Trigger conditions

A Pipeline gem requires at least one non-empty input row to execute. If input is empty, the pipeline will not run, even when trigger condition is set to Always run.
Trigger conditionDescription
Always run (default)Executes the pipeline whenever at least one input row is present, regardless of input status.
All pipelines succeededExecutes only when every input row has a success status.
All pipelines failedExecutes only when every input row has a failure status.
All pipelines finishedExecutes after all input pipeline runs have completed, regardless of their success or failure.
Any pipeline succeededExecutes if at least one input row has a success status.
Any pipeline failedExecutes if at least one input row has a failure status.

​
Execution behavior

Prophecy supports the following execution behaviors for Pipeline gems.
Execution behaviorDescription
Sequential executionIf a Pipeline gem triggers the same target pipeline multiple times, the triggered pipeline runs execute sequentially. The Pipeline gem completes only after all triggered runs finish.
Sequential triggeringPipeline gems execute in sequence when they are connected sequentially or assigned sequential gem phases. Downstream Pipeline gems wait for upstream Pipeline gems to finish before triggering.
Parallel triggeringPipeline gems can execute in parallel when separate pipeline branches use the same gem phase.
Recursive triggersPipeline gems support recursive execution. If a triggered pipeline contains another Pipeline gem, that Pipeline gem executes normally.