> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks serverless compute for PySpark

> Use Databricks serverless compute to execute PySpark pipelines

<Callout icon="/images/icon.png" color="#FFC107">
  Available for [Enterprise Edition](/data-engineering/administration/platform/editions) only.
</Callout>

[Databricks serverless compute](https://docs.databricks.com/aws/en/compute/serverless/) allows you to run workloads without manually provisioning a Spark cluster. With serverless compute, Databricks takes care of the infrastructure in the background, so your jobs start up quickly and scale as needed. Prophecy supports serverless compute for interactively running pipelines in PySpark projects on Databricks.

This page explains how to use serverless compute with Prophecy, including supported data sources, data sampling modes, and current limitations.

<Info>
  Databricks serverless compute differs from [serverless SQL
  warehouses](https://docs.databricks.com/aws/en/compute/sql-warehouse/#what-is-serverless-sql).
  Prophecy uses serverless compute to run Spark pipelines on Spark fabrics. In contrast, serverless
  SQL warehouses are connected to Prophecy via JDBC and are used to run SQL queries generated from
  pipelines in SQL projects.
</Info>

## Prerequisites

To use serverless compute in Prophecy, you need:

* [Access to serverless compute](https://docs.databricks.com/aws/en/compute/serverless/#enable-serverless-compute) in Databricks
* PySpark projects in Prophecy (Scala not supported)

## Supported data sources

You can run the following sources on Databricks serverless compute:

* [Avro](/data-engineering/gems/source-target/file/avro)
* [CSV](/data-engineering/gems/source-target/file/csv)
* [Data Generator](/data-engineering/gems/source-target/file/synthetic-data-generator)
* [Delta files](/data-engineering/gems/source-target/file/delta)
* [JSON](/data-engineering/gems/source-target/file/json)
* [Kafka](/data-engineering/gems/source-target/file/kafka-stream)
* [ORC](/data-engineering/gems/source-target/file/orc)
* [Parquet](/data-engineering/gems/source-target/file/parquet)
* [Seed files](/data-engineering/gems/source-target/file/seed)
* [Text files](/data-engineering/gems/source-target/file/text)
* [Unity Catalog tables](https://docs.databricks.com/aws/en/tables/)
* [XLSX](/data-engineering/gems/source-target/file/xlsx)
* [XML](/data-engineering/gems/source-target/file/xml)

## Supported data sampling modes

You can use the following data sampling modes when using Databricks serverless compute:

* [Selective](/data-engineering/development/runs/data-sampling#selective-sampling) mode
* Vanilla mode (deprecated)

## Limitations

Below are the current limitations of Databricks Serverless and how they impact Prophecy project development.

| **Feature**                                                 | **Limitation**                                                                                                                                                                                                                                           |
| ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Scala support                                               | Databricks serverless only supports Python and SQL. <br />Scala projects cannot run on Databricks Serverless.                                                                                                                                            |
| Dependencies                                                | Only Python dependencies are supported. <br />[Dependencies](/data-engineering/extensibility/dependencies/spark-dependencies) must be added through the Prophecy UI. <br />You cannot install dependencies to serverless compute directly in Databricks. |
| Jobs                                                        | Scheduled pipeline runs cannot run on Databricks serverless. <br />Prophecy only supports running pipelines on-demand on serverless.                                                                                                                     |
| Authentication                                              | To use serverless compute, you must authenticate with Databricks using a [Databricks personal access token](https://docs.databricks.com/aws/en/dev-tools/auth/pat). Prophecy does not support OAuth for serverless compute.                              |
| Row size                                                    | Maximum row size is 128MB.                                                                                                                                                                                                                               |
| Driver size                                                 | Databricks serverless driver size is unknown and cannot be changed.                                                                                                                                                                                      |
| Supported data formats                                      | XLSX, fixed format, and custom formats are not supported.                                                                                                                                                                                                |
| UDF network access                                          | [UDFs](/data-engineering/development/functions/user-defined-functions) cannot access the internet.                                                                                                                                                       |
| Spark configuration                                         | Databricks Serverless only supports a limited number of [Spark configuration properties](https://docs.databricks.com/aws/en/spark/conf#configure-spark-properties-for-serverless-notebooks-and-jobs).                                                    |
| APIs in [Script gems](/data-engineering/gems/custom/script) | Spark Connect APIs are supported. <br />Spark RDD APIs are not supported. <br />DataFrame and SQL cache APIs are not supported.                                                                                                                          |

<Note>
  For the complete list of limitations, visit [Serverless compute
  limitations](https://docs.databricks.com/aws/en/compute/serverless/limitations) in the Databricks
  documentation.
</Note>