Pipeline gem

The Pipeline gem allows you to run another pipeline from within your current pipeline. It supports conditional triggering, parameter passing, and returning metadata for each pipeline run. This gem is useful for building orchestrated workflows directly in the visual canvas. You can find it under the Custom category in the gem drawer. A single Pipeline gem execution can trigger the target pipeline multiple times. Triggered pipeline runs execute sequentially, and the Pipeline gem completes only after all triggered runs finish.

You can create pipelines solely dedicated to pipeline orchestration using multiple instances of this gem. We recommend labeling your orchestration pipelines to differentiate them from standard data pipelines.

Limitations

The Pipeline gem has the following limitations:

It can only trigger pipelines for the current project.
The Monitoring page doesn’t show the parent pipeline that triggered a run using the Pipeline gem.

Input and Output

The following table describes what the Pipeline gem expects as input and what it will produce as output.

Port	Description
in0	Optional input dataset that may include a `status` column and additional columns. Valid statuses include `success`, `failure`, or `skipped`. Each row of the input triggers a separate pipeline run per Pipeline gem execution.
out	One output dataset that contains the metadata for each triggered pipeline run. The Pipeline gem returns one row per triggered run.

When no input is connected:

The Pipeline gem triggers the child pipeline once.
The output contains a single row of metadata for that run.

When an input dataset is connected:

If a status column is present, it is used to evaluate trigger conditions. If the trigger condition is met, the Pipeline gem triggers the child pipeline once for each input row.
If a status column is not present, the Pipeline gem always triggers the child pipeline once for each input row.
Additional columns can be used to pass parameter values into each run.
The output contains one row of metadata per input row.

The input dataset can be any dataset (including the output of an upstream Pipeline gem).

Output schema

Each row in the output table represents a pipeline run triggered from the Pipeline gem. The output schema provides you with the following information for each pipeline run.

Column	Type	Description
`status`	String	Final status of the triggered pipeline: `success`, `failure`, or `skipped`.
`pipelineRunID`	String	Unique ID of the triggered pipeline run.
`startTime`	Timestamp	Time when the triggered pipeline started.
`endTime`	Timestamp	Time when the triggered pipeline finished.
`error`	String	Error message, if any.
`logs`	String Array	Log messages from the triggered pipeline.

Pipelines are skipped when the input does not meet the trigger condition set in the gem.

Parameters

The Pipeline gem accepts the following parameters.

Parameter	Description
Pipeline to run	Select a pipeline to trigger in the list of pipelines from the same project.
Trigger only if	Choose a condition to control when the trigger fires based on the `status` column of the input dataset (if present). If the condition is not met, the Pipeline gem runs, but no pipelines are triggered. The output will show that all pipeline runs were `skipped`.
Maximum number of pipeline triggers	Set a maximum number of times the child pipeline runs per Pipeline gem execution. Maximum is `10,000`.
Set pipeline parameters	Set values for the pipeline parameters defined in the child pipeline. You can specify constants, expressions, or column values for each parameter. If you don’t specify a value, the child pipeline uses its default parameter value.

Use multiple input rows to launch runs of the same pipeline with different sets of parameter values.

Trigger conditions

A Pipeline gem requires at least one non-empty input row to execute. If input is empty, the pipeline will not run, even when trigger condition is set to Always run.

Trigger condition	Description
Always run (default)	Executes the pipeline whenever at least one input row is present, regardless of input status.
All pipelines succeeded	Executes only when every input row has a `success` status.
All pipelines failed	Executes only when every input row has a `failure` status.
All pipelines finished	Executes after all input pipeline runs have completed, regardless of their success or failure.
Any pipeline succeeded	Executes if at least one input row has a `success` status.
Any pipeline failed	Executes if at least one input row has a `failure` status.

Execution behavior

Prophecy supports the following execution behaviors for Pipeline gems.

Execution behavior	Description
Sequential execution	If a Pipeline gem triggers the same target pipeline multiple times, the triggered pipeline runs execute sequentially. The Pipeline gem completes only after all triggered runs finish.
Sequential triggering	Pipeline gems execute in sequence when they are connected sequentially or assigned sequential gem phases. Downstream Pipeline gems wait for upstream Pipeline gems to finish before triggering.
Parallel triggering	Pipeline gems can execute in parallel when separate pipeline branches use the same gem phase.
Recursive triggers	Pipeline gems support recursive execution. If a triggered pipeline contains another Pipeline gem, that Pipeline gem executes normally.

Getting started

Build with Agent

Build and inspect pipelines

Explore and visualize

Platform setup

Run and share

Collaboration

Limitations

Input and Output

Output schema

Parameters

Trigger conditions

Execution behavior

​Limitations

​Input and Output

​Output schema

​Parameters

​Trigger conditions

​Execution behavior

Limitations

Input and Output

Output schema

Parameters

Trigger conditions

Execution behavior