DynamicSelect gem for Data Engineering

Dependencies & cluster requirements

Dependencies:

• ProphecySparkBasicsPython 0.2.27+
• ProphecyLibsPython 1.9.42+
• ProphecyLibsScala 8.2.1+

Cluster requirements:

• UC dedicated clusters 14.3+ supported
• UC standard clusters 14.3+ supported
• Livy clusters 3.0.1+ supported

Use the DynamicSelect gem to dynamically filter columns of your dataset based on a set of conditions.

​

Configuration

There are two ways to configure the DynamicSelect.

​

Examples

You’ll use DynamicSelect when you want to avoid hard-coding your choice of columns. In other words, rather than define each column to keep in your pipeline, you let the system automatically choose the columns based on certain conditions or rules.

​

Remove date columns using field type

Assume you would like to remove irrelevant date and timestamp columns from your dataset. You can do so with the Select field types method by selecting all field types to maintain, except for date and timestamp.

​

Remove date columns with an expression

Using the same example, you can accomplish the same task with the Select via expression method by inputting the expression column_type NOT IN ('date', 'timestamp').