OrderBy gem for Data Engineering

Dependencies & cluster requirements

Dependencies:

Cluster requirements:

Sorts a DataFrame on one or more columns in ascending or descending order.

Parameters

Parameter	Description	Required
DataFrame	Input DataFrame to be sorted	True
Order columns	Columns to sort DataFrame by	True
Sort	Order of sorting - ascending or descending	True

def Sort(spark: SparkSession, in0: DataFrame) -> DataFrame:
 return in0.orderBy(col("name").asc(), col("updated_at").desc())

object Sort {
 def apply(spark: SparkSession, in: DataFrame): DataFrame =
 in.orderBy(col("updated_at").desc, col("name").asc)
}