Documentation Index
Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
Use this file to discover all available pages before exploring further.
Use the SetOperation gem to perform addition or subtraction of rows from DataFrames with identical schemas and different data.
Parameters
| Parameter | Description |
|---|
| DataFrame 1 | First input DataFrame |
| DataFrame 2 | Second input DataFrame |
| DataFrame N | Nth input DataFrame |
| Operation type | Choose an operation type: - Union: Returns a DataFrame containing rows in any one of the input DataFrames, while preserving duplicates.
- Intersect All: Returns a DataFrame containing rows in all of the input DataFrames, while preserving duplicates.
- Except All: Returns a DataFrames containing rows in the first DataFrame, but not in the other DataFrames, while preserving duplicates.
|
+ icon on the left sidebar.
Examples
Operation Type: Union
def union(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
return in0.unionAll(in1)
Operation Type: Intersect All
def intersectAll(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
return in0.intersectAll(in1)
Operation Type: Except All
def exceptAll(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
return in0.exceptAll(in1)
