> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# DynamicSelect gem for Data Engineering

> Dynamically filter columns of your dataset based on a set of conditions

<Panel>
  <Info>
    Dependencies:

    * ProphecySparkBasicsPython 0.2.27+
    * ProphecyLibsPython 1.9.42+
    * ProphecyLibsScala 8.2.1+
  </Info>

  <Info>
    Cluster requirements:

    * UC dedicated clusters 14.3+ supported
    * UC standard clusters 14.3+ supported
    * Livy clusters 3.0.1+ supported
  </Info>
</Panel>

Use the DynamicSelect gem to dynamically filter columns of your dataset based on a set of conditions.

## Configuration

There are two ways to configure the DynamicSelect.

| Configuration         | Description                                                                                   |
| --------------------- | --------------------------------------------------------------------------------------------- |
| Select field types    | Choose one or more types of columns to keep in the dataset, such as string, decimal, or date. |
| Select via expression | Create an expression that limits the type of columns to keep in the dataset.                  |

## Examples

You'll use DynamicSelect when you want to avoid hard-coding your choice of columns. In other words, rather than define each column to keep in your pipeline, you let the system automatically choose the columns based on certain conditions or rules.

### Remove date columns using field type

Assume you would like to remove irrelevant date and timestamp columns from your dataset. You can do so with the **Select field types** method by selecting all field types to maintain, except for date and timestamp.

<img src="https://mintcdn.com/prophecy-62973bd0/5tvw_2e98LqB5rz7/data-engineering/gems/transform/img/remove-date-timestamp.png?fit=max&auto=format&n=5tvw_2e98LqB5rz7&q=85&s=970a1945b0e41cc0acbe731afd472f79" alt="Keep all columns except Date and Timestamp column using the visual interface" width="2620" height="1508" data-path="data-engineering/gems/transform/img/remove-date-timestamp.png" />

### Remove date columns with an expression

Using the same example, you can accomplish the same task with the **Select via expression** method by inputting the expression `column_type NOT IN ('date', 'timestamp')`.
