Skip to main content
This page explains how to use and configure a Databricks connection in Prophecy. A Databricks connection allows Prophecy to access files, tables, and compute resources in your Databricks workspace. You can use the same connection to access any of these resources, as long as the authenticated account in the connection has the appropriate permissions.

Prerequisites

Prophecy connects to Databricks using the credentials you provide. These credentials are used to authenticate your session and authorize all data operations during pipeline execution, including reading from and writing to tables. To use a Databricks connection effectively, your user or service principal must have the following:
  • Basic table permissions defined in the Databricks documentation.
  • Additional Unity Catalog privileges:
    • CREATE VOLUME for permission to create the PROPHECY_ORCHESTRATOR_VOLUME
    • READ VOLUME for the path /Volumes/<catalog>/<schema>/PROPHECY_ORCHESTRATOR_VOLUME
    • WRITE VOLUME for permission to delete intermediate files from the volume

Connection type

Prophecy supports Databricks as both a SQL Warehouse connection and an Ingress/Egress connection. To learn more about these different connection types, visit Prophecy fabrics.

Feature support

The table below outlines whether the connection supports certain Prophecy features.

Connection parameters

To create a connection with Databricks, enter the following parameters.
When you use Databricks as your primary SQL warehouse, Prophecy also uses the catalog and schema you define in the connection to store temporary tables during pipeline execution. Therefore, you must have write access to the schema in Databricks. To avoid conflicts, define distinct catalog and schema locations for each fabric.

Authentication methods

You can configure your Databricks connection using the following authentication methods.

OAuth

When you select OAuth as your authentication method for the connection, Prophecy can authenticate using either user-based (U2M) or service principal-based (M2M) OAuth. A single fabric cannot use both methods for the same connection. To use both U2M and M2M, create separate fabrics. You can schedule pipelines with user-based OAuth, but you’ll need to re-authenticate periodically. Prophecy estimates token expiry based on Databricks’ response and prompts you when re-authentication is needed. To avoid interruptions, switch to service principal-based OAuth for production workloads. Use different fabrics for development and production to align authentication with your environment’s needs.

Personal Access Token (PAT)

When you choose Personal Access Token (PAT) for the authentication method, you’ll authenticate using a Databricks personal access token. When you set up the connection, you will use a secret to enter your PAT. Using the PAT authentication method:
  • All team members who have access to the fabric can use the connection in their projects.
  • No additional authentication is required. Team members automatically inherit the access and permissions of the stored connection credentials.

OIDC

When you choose OIDC for the authentication method, Prophecy authenticates as a service principal whose credentials are issued by your identity provider. Prophecy exchanges these credentials for a Databricks bearer token through an OIDC federator, caches the token, and mints a new one before it expires. No interactive sign-in is required, so OIDC connections work for unattended runs. OIDC authentication applies to SQL warehouse queries, dbt, and Databricks workspace access. Enter the following parameters: The certificate and private key are required because the federator signs a client assertion with the private key. A client secret alone cannot authenticate the connection.
OIDC authentication requires an OIDC federator URL in the Prophecy deployment configuration. If the federator URL isn’t configured, connections that use OIDC fail with the error OIDC federator URL is not configured. Contact your Prophecy administrator to configure it.

Data type mapping

When Prophecy processes data from Databricks using an external SQL warehouse, it converts Databricks data types to compatible types.
Learn more in Supported data types.