UC cluster compatibility

Available for Enterprise Edition only.

When you configure a Databricks cluster, you must specify the cluster access mode. This page provides an overview of how different Prophecy features perform across various Unity Catalog cluster types.

High-level features

Review the table below to determine which Prophecy features are available based on the cluster access mode you choose.

Prophecy Feature	Dedicated (formerly single user)	Standard (formerly shared)	No isolation-shared (legacy)	Unity Catalog Dedicated (formerly single user)	Unity Catalog Standard (formerly shared)
Library installations	Supported	Supported but interim nodes can’t be attached	Supported	Supported	Version-dependent
Data samples	Default & Selective	Vanilla	Default & Selective	Default & Selective	Vanilla & Selective
Execution metrics	Supported	Not Supported	Supported	Supported	Not Supported
Run history	Supported	Not Supported	Supported	Supported	Not Supported
Interactive run progress	Supported	Not Supported	Supported	Supported	Not Supported
Interactive runs on pre-existing clusters	Supported	Not Supported	Supported	Supported	Not Supported
User-defined functions	Supported	Not supported	Supported	Supported	Supported*

*UDF support Graviton instance support for UDFs on Unity Catalog-enabled clusters is available in Databricks Runtime 15.2 and above. Clusters must have standard access mode for Python UDFs.

Gem support

A subset of Prophecy gems are not supported on UC standard clusters according to the UC standard cluster version (12.2, 14.3, or 15.4). Each row indicates the minimum Prophecy Package version required for that gem to be supported on the relevant UC standard cluster version listed.

The following tables apply to Python gems only. Support for Scala gems may differ.

Legacy Shared clusters 12.2 and below are shown for reference only. Later versions are strongly recommended.

Sources / Targets

Minimum Prophecy Package version required to support Databricks Unity Catalog Standard Clusters:

Gem	Package	12.2 - Shared	14.3 - UC standard	15.4 - UC standard
Avro	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
CSV (Only File Path)	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
CSV (SFTP - Source)	ProphecySparkBasicsPython	Not Supported	0.2.44	0.2.44
CSV (SharePoint - Source)	ProphecySparkBasicsPython	Not Supported	0.2.44	0.2.44
Delta	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
JSON	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
ORC	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Parquet	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Text	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Fixed Format	ProphecySparkBasicsPython	Not Supported	Not Supported	Not Supported
XLSX	ProphecySparkBasicsPython	Not Supported	Not Supported	0.2.47
XML	ProphecySparkBasicsPython	Not Supported	0.2.39	0.2.39
Seed	ProphecySparkBasicsPython	Not Supported	0.2.39	0.2.39
Createdata	ProphecySparkAbinitioPython	Not Supported	0.0.28	0.0.28
Mtime	ProphecySparkAbinitioPython	Not Supported	0.0.28	0.0.28

Warehouses

Minimum Prophecy Package version required to support Databricks Unity Catalog Standard Clusters:

Gem	Package	12.2 - Shared	14.3 - UC standard	15.4 - UC standard
JDBC	ProphecySparkBasicsPython	Not Supported	0.2.39	0.2.39
synapse	ProphecyWarehousePython	0.0.8	0.0.9	0.0.9
Data Generator	ProphecySparkBasicsPython	Not Supported	Not Supported	0.2.39
Kafka (Source)	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Kafka (Target)	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Bigquery (Source)	ProphecyWarehousePython	Not Supported	0.0.9	0.0.9
Bigquery (Target)	ProphecyWarehousePython	Not Supported	0.0.9	0.0.9
Mongodb (Driver Mongodb+srv)	ProphecyWarehousePython	Not Supported	Not Supported	Not Supported
Mongodb (Driver Mongodb)	ProphecyWarehousePython	Not Supported	Not Supported	Not Supported
Redshift	ProphecyWarehousePython	Not Supported	0.0.9	0.0.9

Catalog Table

Minimum Prophecy Package version required to support Databricks Unity Catalog Standard Clusters:

Gem	Package	12.2 - Shared	14.3 - UC standard	15.4 - UC standard
Catalog Table	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Iceberg	ProphecySparkBasicsPython	Pending	Pending	Pending

Lookup

The following table defines Lookup support Databricks Unity Catalog Standard Clusters. If Lookup is not supported, use a Left join instead.

Gem	Package	12.2 - Shared	14.3 - UC standard	15.4 - UC standard
Lookup	N/A	Not Supported	Not Supported	Supported*

*Lookup support Lookups are implemented as user-defined functions under the hood in Prophecy. Graviton instance support for UDFs on Unity Catalog-enabled clusters is available in Databricks Runtime 15.2 and above.

Transform

Minimum Prophecy Package version required to support Databricks Unity Catalog Standard Clusters:

Gem	Package	12.2 - Shared	14.3 - UC standard	15.4 - UC standard
Aggregate	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
BulkColumnExpressions	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
SampleRows	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Unpivot	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
ColumnParser	ProphecySparkBasicsPython	Not Supported	0.2.39	0.2.39
DynamicSelect	ProphecySparkBasicsPython	Not Supported	0.2.45	0.2.45
Deduplicate	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Filter	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
FlattenSchema	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Limit	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
OrderBy	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Reformat	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
DataCleansing	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
SchemaTransform	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
SetOperation	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
WindowFunction	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
BulkColumnRename	ProphecySparkBasicsPython	Not Supported	0.2.45	0.2.45
Normalize	ProphecySparkAbinitioPython	Not Supported	0.2.24	0.2.24
MetaPivot	ProphecySparkAbinitioPython	Not Supported	0.2.24	0.2.24
ReadRaw	ProphecySparkAbinitioPython	Not Supported	Pending	Pending
ReadSeparated Values	ProphecySparkAbinitioPython	Not Supported	0.2.24	0.2.24
WriteSeparated Values	ProphecySparkAbinitioPython	Not Supported	0.2.24	0.2.24
SyncDataFrameColumnsWithSchema	ProphecySparkAbinitioPython	Not Supported	0.2.24	0.2.24
Sequence	ProphecySparkAbinitioPython	Not Supported	0.2.24	0.2.24
AssignKeys	ProphecySparkAbinitioPython	Not Supported	0.2.24	0.2.24
RoundRobinPartition	ProphecySparkAbinitioPython	Not Supported	0.2.24	0.2.24
CompareRecords	ProphecySparkAbinitioPython	Not Supported	0.2.24	0.2.24
DynamicReplace	ProphecySparkAlteryxPython	Not Supported	Pending	Pending
FuzzyMatch	ProphecySparkAlteryxPython	Not Supported	Pending	Pending

Join/Split

Minimum Prophecy Package version required to support Databricks Unity Catalog Standard Clusters:

Gem	Package	12.2 - Shared	14.3 - UC standard	15.4 - UC standard
Join	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Repartition	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
RowDistributor	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
CompareColumns	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39

Custom

Minimum Prophecy Package version required to support Databricks Unity Catalog Standard Clusters:

Gem	Package	12.2 - Shared	14.3 - UC standard	15.4 - UC standard
Script	N/A	Supported	Supported	Supported
DeltaTableOperations	ProphecySparkBasicsPython	Supported	Supported	Supported
FileOperation (DBFS Copy)	ProphecySparkBasicsPython	Supported	Supported	Supported
FileOperation	ProphecySparkBasicsPython	Pending	Pending	Pending
Directory	ProphecySparkBasicsPython	Not Supported	0.2.45	0.2.45
RestAPlEnrich	ProphecySparkBasicsPython	Not Supported	Not Supported	0.0.24
Email	ProphecyWebAppPython	Not Supported	0.1.2	0.1.2
EmailData	ProphecyWebAppPython	Not Supported	0.1.2	0.1.2
Tableau	ProphecyWebAppPython	Not Supported	0.1.2	0.1.2
ReadMultipleFiles	ProphecySparkAbinitioPython	Not Supported	Pending	Pending
WriteMultipleFiles	ProphecySparkAbinitioPython	Not Supported	Pending	Pending
Display	ProphecySparkAbinitioPython	Not Supported	Pending	Pending
Trash	ProphecySparkAbinitioPython	Not Supported	Pending	Pending
Todo	ProphecySparkAbinitioPython	Not Supported	Pending	Pending
Assertions	ProphecySparkAbinitioPython	Not Supported	0.0.24	0.0.24
Log	ProphecySparkAbinitioPython	Not Supported	0.0.24	0.0.24

Subgraph

Minimum Prophecy Package version required to support Databricks Unity Catalog Standard Clusters:

Gem	Package	12.2 - Shared	14.3 - UC standard	15.4 - UC standard
Basic	N/A	Supported	Supported	Supported
Published subgraph	N/A	Supported	Supported	Supported
WhileIterator	ProphecySparkAlteryxPython	Not Supported	0.0.4	0.0.4
Table Iterator	ProphecySparkBasicsPython	0.2.39	0.2.39	0.2.39
Catalog	ProphecyStreamingPython	Pending	Pending	Pending

Streaming

Streaming gems and capabilities are not supported on UC standard clusters as of Prophecy 3.4.x.

Getting started

Development

Production

Execution

Extensibility

High-level features

Gem support

Sources / Targets

Warehouses

Catalog Table

Lookup

Transform

Join/Split

Custom

Subgraph

Streaming

​High-level features

​Gem support

​Sources / Targets

​Warehouses

​Catalog Table

​Lookup

​Transform

​Join/Split

​Custom

​Subgraph

​Streaming

High-level features

Gem support

Sources / Targets

Warehouses

Catalog Table

Lookup

Transform

Join/Split

Custom

Subgraph

Streaming