> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# SetOperation
> Union, Intersect and Difference
Dependencies:
* ProphecySparkBasicsPython 0.0.1+
* ProphecySparkBasicsScala 0.0.1+
Cluster requirements:
* UC dedicated clusters 14.3+ supported
* UC standard clusters 14.3+ supported
* Livy clusters 3.0.1+ supported
Use the SetOperation gem to perform addition or subtraction of rows from DataFrames with identical schemas and different data.
## Parameters
| Parameter | Description |
| :------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| DataFrame 1 | First input DataFrame |
| DataFrame 2 | Second input DataFrame |
| DataFrame N | Nth input DataFrame |
| Operation type | Choose an operation type: - Union: Returns a DataFrame containing rows in any one of the input DataFrames, while preserving duplicates.
- Intersect All: Returns a DataFrame containing rows in all of the input DataFrames, while preserving duplicates.
- Except All: Returns a DataFrames containing rows in the first DataFrame, but not in the other DataFrames, while preserving duplicates.
|
To add more input DataFrames, you can click the `+` icon on the left sidebar.
## Examples
### Operation Type: `Union`
```python example.py theme={null}
def union(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
return in0.unionAll(in1)
```
```scala example.scala theme={null}
object union {
def apply(spark: SparkSession, in0: DataFrame, in1: DataFrame): DataFrame =
in0.unionAll(in1)
}
```
### Operation Type: `Intersect All`
```python example.py theme={null}
def intersectAll(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
return in0.intersectAll(in1)
```
```scala example.scala theme={null}
object intersectAll {
def apply(spark: SparkSession, in0: DataFrame, in1: DataFrame): DataFrame =
in0.intersectAll(in1)
}
```
### Operation Type: `Except All`
```python example.py theme={null}
def exceptAll(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
return in0.exceptAll(in1)
```
```scala example.scala theme={null}
object exceptAll {
def apply(spark: SparkSession, in0: DataFrame, in1: DataFrame): DataFrame =
in0.exceptAll(in1)
}
```