> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Filter gem for Data Engineering

> Filter your data based on a custom filter condition

<Panel>
  <Info>
    Dependencies:

    * ProphecySparkBasicsPython 0.0.1+
    * ProphecySparkBasicsScala 0.0.1+
  </Info>

  <Info>
    Cluster requirements:

    * UC dedicated clusters 14.3+ supported
    * UC standard clusters 14.3+ supported
    * Livy clusters 3.0.1+ supported
  </Info>
</Panel>

Filters a DataFrame based on the provided filter condition.

## Parameters

| Parameter        | Description                                                                           | Required |
| :--------------- | :------------------------------------------------------------------------------------ | :------- |
| DataFrame        | Input DataFrame on which the filter condition will be applied.                        | True     |
| Filter Condition | BooleanType column or boolean expression. Supports SQL, Python and Scala expressions. | True     |

<Note>
  Use the [visual language syntax](/data-engineering/development/pipelines/configuration#syntax-for-different-languages) to call configuration variables in the Filter gem.
</Note>

## Example

In this example, the Filter gem is used to return only marketing orders that are either finished or approved, while excluding any orders that have been discounted.

<img src="https://mintcdn.com/prophecy-62973bd0/YMU5yAViYYX3rOGW/data-engineering/gems/transform/img/filter_eg_1.png?fit=max&auto=format&n=YMU5yAViYYX3rOGW&q=85&s=f7181d046d373624606dac7c719a8b35" alt="Example usage of Filter" width="2034" height="802" data-path="data-engineering/gems/transform/img/filter_eg_1.png" />

<Info>
  The Filter gem configuration translates into the Spark code shown below, which applies the same filtering logic.
</Info>

## Spark code

<CodeGroup>
  ```python example.py theme={null}
  def Filter_Orders(spark: SparkSession, in0: DataFrame) -> DataFrame:
   return in0.filter(
   (
   ((col("order_category") == lit("Marketing"))
   & ((col("order_status") == lit("Finished")) | (col("order_status") == lit("Approved"))))
   & ~ col("is_discounted")
   )
   )
  ```

  ```scala example.scala theme={null}
  object Filter_Orders {

   def apply(spark: SparkSession, in: DataFrame): DataFrame =
   in.filter(
   (
   col("order_category") === lit("Marketing"))
   .and(
   (col("order_status") === lit("Finished"))
   .or(col("order_status") === lit("Approved"))
   )
   .and(!col("is_discounted"))
   )
  }
  ```
</CodeGroup>