Skip to main content
Dependencies:
  • ProphecySparkAlteryxPython 0.0.7+
Cluster requirements:
  • UC dedicated clusters 14.3+ supported
  • UC standard clusters not supported
  • Livy clusters not supported
Use the FuzzyMatch gem to identify non-identical duplicates in your data.

​
Input and Output

DataFrameDescription
in0Includes the DataFrame on which duplicates will be checked. Note: FuzzyMatch only allows one input.
outGenerates one record per fuzzy match.

​
Parameters

​
Configuration

ParameterDescription
Merge/Purge ModeRecords are either compared from a single source (Purge) or across multiple sources (Merge). Merge mode assumes that multiple sources exist in the same DataFrame in0.
Source ID FieldUnique identifier for each source when using Merge mode. This is necessary because the different sources exist in the same DataFrame in0.
Record ID FieldUnique identifier for each record.
Match threshold percentageIf the match score is less than the threshold, the record does not qualify as a match.
Include similarity scoreCheckbox to enable for an additional output column that includes the similarity score.

​
Match Fields

ParameterDescription
Field nameName of the column that you want to check for duplicates.
Match functionThe method that generates the similarity score.

​
Example

One common use case for the FuzzyMatch gem is to match similarly spelled names. This can be useful for identifying accidentally misspelled names.
  1. Create a FuzzyMatch gem and use the customer_id as the Record ID. Then, add a match field for the first_name column. FuzzyMatch names
  2. Run the gem and see that the output includes the Record IDs of the records with fuzzy matches. FuzzyMatch output
  3. Join the output with the original dataset to view the matched names. FuzzyMatch joined

​
Example code

To see the compiled code of your project, switch to the Code view in the project header.
def fuzzy_match_customers(spark: SparkSession, in0: DataFrame) -> DataFrame:
 from prophecy.utils.transpiler.dataframe_fcns import fuzzyPurgeMode

 return fuzzyPurgeMode(
 in0,
 spark,
 recordId = "customer_id",
 matchFields = {"first_name" : "name"},
 threshold = 0.8,
 includeSimilarityScore = True
 )