Skip to main content
Dependencies:
  • ProphecySparkBasicsPython 0.0.1+
  • ProphecySparkBasicsScala 0.0.1+
Cluster requirements:
  • UC dedicated clusters 14.3+ supported
  • UC standard clusters 14.3+ supported
  • Livy clusters 3.0.1+ supported
Use the RowDistributor gem to create multiple DataFrames based on provided filter conditions from an input DataFrame. This is useful for cases where rows from the input DataFrame needs to be distributed into multiple DataFrames in different ways for downstream gems.

​
Parameters

ParameterDescriptionRequired
DataFrameInput DataFrame for which rows needs to be distributed into multiple DataFramesTrue
Filter ConditionsBoolean Type column or boolean expression for each output tab. Supports SQL, Python and Scala expressionsTrue

​
Example

Row distributor 1
Number of outputs can be changed as needed by clicking the + button.

​
Generated Code

def RowDistributor(spark: SparkSession, in0: DataFrame) -> (DataFrame, DataFrame, DataFrame):
 df1 = in0.filter((col("order_status") == lit("Started")))
 df2 = in0.filter((col("order_status") == lit("Approved")))
 df3 = in0.filter((col("order_status") == lit("Finished")))

 return df1, df2, df3

object RowDistributor {

 def apply(
 spark: SparkSession,
 in: DataFrame
 ): (DataFrame, DataFrame, DataFrame) =
 (in.filter(col("order_status") === lit("Started")),
 in.filter(col("order_status") === lit("Approved")),
 in.filter(col("order_status") === lit("Finished"))
 )

}