Skip to main content
Dependencies:
  • ProphecySparkBasicsPython 0.0.1+
  • ProphecySparkBasicsScala 0.0.1+
Cluster requirements:
  • UC dedicated clusters 14.3+ supported
  • UC standard clusters 14.3+ supported
  • Livy clusters 3.0.1+ supported
This will repartition or coalesce the input DataFrame based on the specified configuration. There are four different repartitioning options:

Hash Repartitoning

Repartitions the data evenly across various partitions based on the hash value of the specified key.

Parameters

Compiled code

Random Repartitioning

Repartitions without data distribution defined.

Parameters

Compiled code

Range Repartitoning

Repartitions the data with tuples having keys within the same range on the same worker.

Parameters

Compiled code

Coalesce

Reduces the number of partitions without shuffling the dataset.

Parameters

Compiled code

Video demo