Skip to main content
/images/icon.png
Available for Enterprise Edition only.
If Maven or PyPI is blocked in your Databricks environment, you can add libraries directly to Databricks volumes.

​
Prerequisites

Before you set up Prophecy libraries in a Databricks volume, you need to know which library version your project uses. That version must be present in the volume for the setup to work. To check your project’s Prophecy library version:
  1. Open your project metadata.
  2. Click on the Dependencies tab.
  3. Find the relevant Prophecy library (Scala or Python) and note the version.
If using ProphecyLibsScala, you can look up versions on Maven Repository that are compatible with your Spark version.

​
Prepare a Databricks volume

To get started, you’ll need to define a Databricks volume where you will store the Prophecy libraries.
  1. Create a volume in your workspace where you’ll maintain Prophecy libraries: 
    databricks volumes create --name prophecy-libs --storage-location <your-volume-location>

​
Download library dependencies

Depending on your Databricks Runtime version, you will need to download certain additional libraries (Python only).
Review the following required and optional libraries:
NameVersionPurposeWheel file
pyhocon0.3.59 or laterGeneral purpose dependencypyhocon-0.3.61-py3-none-any.whl
hvac2.3.0To access Hashicorp secretshvac-2.3.0-py3-none-any.whl
zstandard0.23.0 or later(Optional) For selective data sampling performance improvementszstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
msgspec0.18.6 or later(Optional) For selective data sampling performance improvementsmsgspec-0.20.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
Review the following required and optional libraries:
NameVersionPurposeWheel file
pyhocon0.3.59 or laterGeneral purpose dependencypyhocon-0.3.61-py3-none-any.whl
hvac2.3.0To access Hashicorp secretshvac-2.3.0-py3-none-any.whl
zstandard0.23.0 or later(Optional) For selective data sampling performance improvementszstandard-0.25.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
msgspec0.18.6 or later(Optional) For selective data sampling performance improvementsmsgspec-0.20.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
Review the following required and optional libraries:
NameVersionPurposeWheel file
pyhocon0.3.59 or laterGeneral purpose dependencypyhocon-0.3.61-py3-none-any.whl
hvac2.3.0To access Hashicorp secretshvac-2.3.0-py3-none-any.whl
zstandard0.23.0 or later(Optional) For selective data sampling performance improvementszstandard-0.25.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
msgspec0.18.6 or later(Optional) For selective data sampling performance improvementsmsgspec-0.20.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
Review the following required and optional libraries:
NameVersionPurposeWheel file
pyhocon0.3.59 or laterGeneral purpose dependencypyhocon-0.3.61-py3-none-any.whl
hvac2.3.0To access Hashicorp secretshvac-2.3.0-py3-none-any.whl
zstandard0.23.0 or later(Optional) For selective data sampling performance improvementszstandard-0.25.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
msgspec0.18.6 or later(Optional) For selective data sampling performance improvementsmsgspec-0.20.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
To install dependencies, follow the steps below:
  1. Download the appropriate wheel files for each dependency you need to install. Refer to the tables above for the specific wheel file names based on your Databricks Runtime version.
  2. Upload each dependency to the volumes path. Repeat this step for every dependency: 
    databricks fs cp pyhocon-0.3.61-py3-none-any.whl /Volumes/<your-volume-location>/prophecy-libs/
databricks fs cp hvac-2.3.0-py3-none-any.whl /Volumes/<your-volume-location>/prophecy-libs/
# Add additional dependencies as needed
  3. Create a local Init Script that installs all dependencies. For example, create install_dependencies.sh with the following content, including all dependencies you uploaded: 
    #!/bin/bash
pip install /Volumes/<your-volume-location>/prophecy-libs/pyhocon-0.3.61-py3-none-any.whl
pip install /Volumes/<your-volume-location>/prophecy-libs/hvac-2.3.0-py3-none-any.whl
# Add pip install commands for each additional dependency
  4. Upload the script to your volumes path. 
    databricks fs cp install_dependencies.sh /Volumes/<your-volume-location>/prophecy-libs/

​
Upload Prophecy libraries

Download the appropriate Prophecy library versions from the public buckets and upload them to the same volume path. Replace <version> in the example URLs based on what’s used in your Prophecy project.
  • ProphecyPythonLibs 
    wget https://prophecy-public-bucket.s3.us-east-2.amazonaws.com/python-prophecy-libs/prophecy_libs-<version>-py3-none-any.whl

databricks fs cp prophecy_libs-<version>-py3-none-any.whl /Volumes/<your-volume-location>/prophecy-libs/
  • ProphecyScalaLibs 
    wget https://prophecy-public-bucket.s3.us-east-2.amazonaws.com/prophecy-libs/prophecy-libs-assembly-<version>.jar

databricks fs cp prophecy-libs-assembly-<version>.jar /Volumes/<your-volume-location>/prophecy-libs/
You can also download Prophecy libraries from Azure and GCP. For a full list of paths, visit Prophecy libraries.

​
Update fabric configuration

For all relevant Databricks fabrics, you’ll need to update the location of the Prophecy libraries.
  1. Open the fabric settings.
  2. Toward the end of the page, find the Prophecy Library settings.
  3. For Scala and/or Python, change the Resolution mode to File System.
  4. Under Path, add the volume path that you created earlier. Example: /Volumes/<your-volume-location>/prophecy-libs/
  5. Click Update to save this change.

​
Whitelist for UC standard clusters

To whitelist the Init Script and the JAR path, follow the instructions to add Prophecy libraries to your Databricks allowlist.