SetOperation

Dependencies:

Cluster requirements:

Parameters

Parameter	Description
DataFrame 1	First input DataFrame
DataFrame 2	Second input DataFrame
DataFrame N	Nth input DataFrame
Operation type	Choose an operation type: Union: Returns a DataFrame containing rows in any one of the input DataFrames, while preserving duplicates. Intersect All: Returns a DataFrame containing rows in all of the input DataFrames, while preserving duplicates. Except All: Returns a DataFrames containing rows in the first DataFrame, but not in the other DataFrames, while preserving duplicates.

To add more input DataFrames, you can click the + icon on the left sidebar.

Examples

Operation Type: Union

def union(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
 return in0.unionAll(in1)

object union {
 def apply(spark: SparkSession, in0: DataFrame, in1: DataFrame): DataFrame =
 in0.unionAll(in1)
}

Operation Type: Intersect All

def intersectAll(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
 return in0.intersectAll(in1)

object intersectAll {
 def apply(spark: SparkSession, in0: DataFrame, in1: DataFrame): DataFrame =
 in0.intersectAll(in1)
}

Operation Type: Except All

def exceptAll(spark: SparkSession, in0: DataFrame, in1: DataFrame, ) -> DataFrame:
 return in0.exceptAll(in1)

object exceptAll {
 def apply(spark: SparkSession, in0: DataFrame, in1: DataFrame): DataFrame =
 in0.exceptAll(in1)
}

Parameters