Skip to main content
This gem runs in .

​
Overview

Use the Filter gem to keep rows that match a condition and remove all other rows from the pipeline. Common use cases include:
  • filtering active customers
  • removing null values
  • keeping recent records
  • filtering by date ranges
  • selecting high-value transactions
The Filter gem evaluates each row individually and keeps only rows that match the filter condition.
The gem has a corresponding interactive gem example. See Interactive gem examples to learn how to run sample pipelines for this and other gems.

​
Parameters

ParameterDescriptionRequired
ModelInput dataset to filter.True
Filter ConditionBoolean expression used to determine which rows are kept.True

​
Common filter examples

GoalFilter condition
Keep active customersStatus = 'ACTIVE'
Keep high-value ordersOrderAmount > 1000
Filter recent ordersOrderDate >= '2025-01-01'
Remove null valuesCustomerId IS NOT NULL
Keep specific countriesCountry IN ('US', 'CA')

​
Example

Assume you have the following weather prediction table.
DatePredictionTemperatureCelsiusHumidityPercentWindSpeedCondition
2025-03-01156510Sunny
2025-03-02177012Cloudy
2025-03-03166811Rainy
2025-03-0414729Sunny
Using the following filter condition: 
DatePrediction > '2025-03-02'
returns:
DatePredictionTemperatureCelsiusHumidityPercentWindSpeedCondition
2025-03-03166811Rainy
2025-03-0414729Sunny

​
Using pipeline parameters in filter conditions

You can reference pipeline parameters in filter conditions to make filtering dynamic at runtime. In Visual mode, select Configuration Variables from the expression builder to insert a parameter directly. In Code mode, use Jinja syntax: 
{{ var('parameter_name') }}
Parameter typeExample filter condition
Stringsensor_id = {{ var('sensor') }}
Datefrom_utc_timestamp(timestamp_col, 'UTC') > {{ var('start_date') }}
Numeric (Int, Long, Float, Double)total_usage_mb > {{ var('usage_cap_mb') }}
ArrayUse array_contains in the visual expression builder. See Use parameters.
Booleanarchived = {{ var('include_archived') }}
For a full walkthrough of each parameter type including dashboard integration, see Use parameters.

​
Common issues

​
No rows returned

Verify that:
  • the filter condition matches the column data type
  • the values exist in the dataset
  • string comparisons use the expected capitalization

​
Null values not matching

Comparisons with NULL may return unexpected results. Use:
  • IS NULL
  • IS NOT NULL
instead of:
  • = NULL
  • != NULL

​
Date comparisons not working

Ensure date values use the correct format and data type.

​
Similar tools and concepts

The Filter gem works similarly to a SQL WHERE clause. For example, this filter condition: 
OrderAmount > 1000
is equivalent to: 
SELECT *
FROM Orders
WHERE OrderAmount > 1000
If you’ve used SQL before, you can apply similar comparison operators and expressions in the Filter gem. You may also recognize similar behavior from:
  • the Alteryx Filter tool
  • df.filter() in PySpark
  • boolean indexing in Pandas

​
Filter gem vs Conditional gem

The Filter gem and the Conditional gem both evaluate conditions on a dataset, but they serve different purposes.

​
Key differences

Filter gemConditional gem
PurposeReduce dataRoute data
OutputsOne outputTwo or more outputs
BehaviorKeeps rows that match the condition and removes the restSends rows to different outputs based on conditions
RoutingNo routingYes
Order mattersNoYes (first matching output wins)

​
Row-level vs dataset-level behavior

  • The Filter gem always operates at the row level.
  • The Conditional gem can operate at:
    • row level (for example, OrderAmount > 1000)
    • dataset level (for example, Count < threshold)

​
When to use the Filter gem

Use the Filter gem when you want to:
  • keep only matching rows
  • remove unwanted records
  • reduce the size of a dataset
  • apply filtering logic similar to a SQL WHERE clause

​
When to use the Conditional gem

Use the Conditional gem when you want to:
  • route rows to different outputs
  • create branching logic
  • split data into multiple paths
For example:
  • rows matching OrderAmount > 1000 can be routed to out0
  • all remaining rows can be routed to out1

​
Summary

  • Filter gem → keeps matching rows
  • Conditional gem → routes rows to different outputs
Use the Filter gem for simple row filtering. Use the Conditional gem when you need branching or control flow in your pipeline.