Skip to main content
Dependencies:
  • ProphecySparkBasicsPython 0.0.1+
  • ProphecySparkBasicsScala 0.0.1+
Cluster requirements:
  • UC dedicated clusters 14.3+ supported
  • UC standard clusters 14.3+ supported
  • Livy clusters 3.0.1+ supported
Limits the number of rows in the output.

​
Parameters

ParameterDescription
DataFrameInput DataFrame
LimitNumber of rows required in output. Allowed range: [0, 231 -1]

​
Limit to 10 rows

If you want to limit your output to 10 rows, you can input 10 in the Limit gem. Example usage of Limit
Data samples generated before the Limit gem might also be limited. This is because Spark tries to push the limit down to earlier stages of execution to minimize data processing. This means Spark may reduce the number of rows fetched from the source or processed in earlier transformations.

​
Code

def limit(spark: SparkSession, in0: DataFrame) -> DataFrame:
 return in0.limit(10)