Condition

The Condition gem routes rows from an input model to one of two or more outputs based on conditions that you define for each output port. It evaluates those conditions in order and sends each row to the first output whose condition is satisfied. You can use this gem when you need branching logic in a pipeline. This includes splitting datasets, handling exceptions, adding alerts or guardrails, and implementing control flow based on dataset-level checks such as row counts or sums.

​

How it works

Each output port has a condition that evaluates to true or false. Conditions are evaluated from top to bottom. Each row is routed to the first output whose condition evaluates to true. Any rows that do not match earlier conditions are routed to the final output port. After a row is routed to an output, it is not evaluated against later outputs. The Conditional gem supports both row-level routing and dataset-level routing:

• Row-level routing splits data based on column values.
• Dataset-level routing supports control-flow decisions using aggregate conditions such as count or sum.

Dataset-level conditions (such as using count) evaluate the same way for all rows: If the condition is true, all rows are routed to that output. If the condition is false, no rows are routed to that output. The final output port acts as a default (else) route. Any rows that do not match earlier conditions are routed to the last output.

​

Use the Condition gem

1. Add a Condition gem to your pipeline from the Custom category.
2. Connect an input to the gem.
3. Define a condition for each output port by using the condition builder. The condition builder supports columns, functions, parameters, and expressions.
4. Click Add Routing Rule or the + to add more output ports when you need additional branches.
5. Reorder the output ports as needed to change evaluation priority.
6. Connect downstream gems to each output port.

The gem starts with two output ports by default.

​

Condition types

Condition gem conditions can reference pipeline parameters and project parameters. You can define row-level conditions such as OrderAmount > 1000 or dataset-level conditions such as count < threshold.

​

Output behavior

If no rows satisfy a condition, that output receives an empty dataframe. Downstream transformations still execute even when the input dataframe is empty. Aggregations applied downstream of an empty branch may still produce rows. For example, a count aggregation can return a single-row result.

​

Example

Assume you have the following orders table. Suppose you configure the Condition gem with two outputs. Output out0 uses the condition OrderAmount > 1000. Output out1 is the default (else) route. Rows where OrderAmount > 1000 are routed to out0. All remaining rows are routed to out1.

​

Example with dataset-level condition

You can route based on properties of the entire dataset. Output out0 uses the condition Count < row_limit. If the dataset has fewer rows than row_limit, all rows are routed to out0. Otherwise, they continue to the next output. This approach is useful for alerting when data volume is too low, routing to fallback logic, and enforcing data quality thresholds.

​

Limitations

• Output order affects behavior because earlier conditions take priority over later ones.
• A condition that always evaluates to true captures all rows and prevents later outputs from receiving data.
• Dataset-level conditions such as using Count do not split rows. They route all rows or no rows.
• You must account for empty dataframes in downstream logic.
• If an earlier condition always evaluates to true, all rows will be routed to that output and later outputs will not receive any data.

​

Example use cases

You can use the Condition gem for these scenarios:

• Data splitting, such as routing high and low values to different branches
• Exception handling
• Alerts or guardrails, such as reacting to a low row count
• Branching pipeline logic

​

Filter vs Condition gem

Both the Filter gem and the Condition gem evaluate conditions on a dataset, but they serve different purposes.

​

Key differences

​

When to use the Filter gem

Use the Filter gem when you want to keep only the rows that match a condition and discard the rest. For example, with OrderAmount > 1000, only rows with OrderAmount > 1000 remain, and all other rows are removed from the pipeline.

​

When to use Condition gem

Use the Condition gem when you want to route rows to different paths based on conditions. For example, when output out0 uses OrderAmount > 1000 and output out1 has no condition, rows with OrderAmount > 1000 go to out0, and all remaining rows go to out1.

​

Dataset-level vs row-level behavior

• The Filter gem always operates at the row level.
• The Condition gem can operate at the row level and the dataset level.

​

Summary

The Filter gem keeps what matches. The Condition gem decides where data goes. Use the Filter gem for simple filtering. Use the Condition gem when you need branching or control flow in your pipeline.