> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Set up libraries in Databricks volumes
> Add libraries directly to volumes when Maven/PyPI is blocked on Databricks
Available for [Enterprise Edition](/data-engineering/administration/platform/editions) only.
If Maven or PyPI is blocked in your Databricks environment, you can add libraries directly to Databricks volumes.
## Prerequisites
Before you set up Prophecy libraries in a Databricks volume, you need to know which library version your project uses. That version must be present in the volume for the setup to work.
To check your project's Prophecy library version:
1. Open your project metadata.
2. Click on the **Dependencies** tab.
3. Find the relevant Prophecy library (Scala or Python) and note the version.
If using ProphecyLibsScala, you can look up versions on [Maven
Repository](https://mvnrepository.com/artifact/io.prophecy/prophecy-libs) that are compatible with
your Spark version.
## Prepare a Databricks volume
To get started, you'll need to define a Databricks volume where you will store the Prophecy libraries.
1. Create a volume in your workspace where you'll maintain Prophecy libraries:
```bash theme={null}
databricks volumes create --name prophecy-libs --storage-location
```
## Download library dependencies
Depending on your Databricks Runtime version, you will need to download certain additional libraries (Python only).
Review the following required and optional libraries:
| Name | Version | Purpose | Wheel file |
| --------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| pyhocon | 0.3.59 or later | General purpose dependency | `pyhocon-0.3.61-py3-none-any.whl` |
| hvac | 2.3.0 | To access Hashicorp secrets | `hvac-2.3.0-py3-none-any.whl` |
| zstandard | 0.23.0 or later | (Optional) For [selective data sampling](/data-engineering/development/runs/data-sampling#selective-sampling) performance improvements | `zstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl` |
| msgspec | 0.18.6 or later | (Optional) For [selective data sampling](/data-engineering/development/runs/data-sampling#selective-sampling) performance improvements | `msgspec-0.20.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl` |
Review the following required and optional libraries:
| Name | Version | Purpose | Wheel file |
| --------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| pyhocon | 0.3.59 or later | General purpose dependency | `pyhocon-0.3.61-py3-none-any.whl` |
| hvac | 2.3.0 | To access Hashicorp secrets | `hvac-2.3.0-py3-none-any.whl` |
| zstandard | 0.23.0 or later | (Optional) For [selective data sampling](/data-engineering/development/runs/data-sampling#selective-sampling) performance improvements | `zstandard-0.25.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl` |
| msgspec | 0.18.6 or later | (Optional) For [selective data sampling](/data-engineering/development/runs/data-sampling#selective-sampling) performance improvements | `msgspec-0.20.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl` |
Review the following required and optional libraries:
| Name | Version | Purpose | Wheel file |
| --------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| pyhocon | 0.3.59 or later | General purpose dependency | `pyhocon-0.3.61-py3-none-any.whl` |
| hvac | 2.3.0 | To access Hashicorp secrets | `hvac-2.3.0-py3-none-any.whl` |
| zstandard | 0.23.0 or later | (Optional) For [selective data sampling](/data-engineering/development/runs/data-sampling#selective-sampling) performance improvements | `zstandard-0.25.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl` |
| msgspec | 0.18.6 or later | (Optional) For [selective data sampling](/data-engineering/development/runs/data-sampling#selective-sampling) performance improvements | `msgspec-0.20.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl` |
Review the following required and optional libraries:
| Name | Version | Purpose | Wheel file |
| --------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| pyhocon | 0.3.59 or later | General purpose dependency | `pyhocon-0.3.61-py3-none-any.whl` |
| hvac | 2.3.0 | To access Hashicorp secrets | `hvac-2.3.0-py3-none-any.whl` |
| zstandard | 0.23.0 or later | (Optional) For [selective data sampling](/data-engineering/development/runs/data-sampling#selective-sampling) performance improvements | `zstandard-0.25.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl` |
| msgspec | 0.18.6 or later | (Optional) For [selective data sampling](/data-engineering/development/runs/data-sampling#selective-sampling) performance improvements | `msgspec-0.20.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl` |
To install dependencies, follow the steps below:
1. Download the appropriate wheel files for each dependency you need to install. Refer to the tables above for the specific wheel file names based on your Databricks Runtime version.
2. Upload each dependency to the volumes path. Repeat this step for every dependency:
```
databricks fs cp pyhocon-0.3.61-py3-none-any.whl /Volumes//prophecy-libs/
databricks fs cp hvac-2.3.0-py3-none-any.whl /Volumes//prophecy-libs/
# Add additional dependencies as needed
```
3. Create a local Init Script that installs all dependencies. For example, create `install_dependencies.sh` with the following content, including all dependencies you uploaded:
```
#!/bin/bash
pip install /Volumes//prophecy-libs/pyhocon-0.3.61-py3-none-any.whl
pip install /Volumes//prophecy-libs/hvac-2.3.0-py3-none-any.whl
# Add pip install commands for each additional dependency
```
4. Upload the script to your volumes path.
```
databricks fs cp install_dependencies.sh /Volumes//prophecy-libs/
```
## Upload Prophecy libraries
Download the appropriate Prophecy library versions from the public buckets and upload them to the same volume path. Replace `` in the example URLs based on what's used in your Prophecy project.
* ProphecyPythonLibs
```
wget https://prophecy-public-bucket.s3.us-east-2.amazonaws.com/python-prophecy-libs/prophecy_libs--py3-none-any.whl
databricks fs cp prophecy_libs--py3-none-any.whl /Volumes//prophecy-libs/
```
* ProphecyScalaLibs
```
wget https://prophecy-public-bucket.s3.us-east-2.amazonaws.com/prophecy-libs/prophecy-libs-assembly-.jar
databricks fs cp prophecy-libs-assembly-.jar /Volumes//prophecy-libs/
```
You can also download Prophecy libraries from Azure and GCP. For a full list of paths, visit [Prophecy libraries](/data-engineering/extensibility/dependencies/prophecy-libs#download-prophecy-libraries).
## Update fabric configuration
For all relevant Databricks fabrics, you'll need to update the location of the Prophecy libraries.
1. Open the fabric settings.
2. Toward the end of the page, find the **Prophecy Library** settings.
3. For Scala and/or Python, change the **Resolution mode** to **File System**.
4. Under **Path**, add the volume path that you created earlier.
Example: `/Volumes//prophecy-libs/`
5. Click **Update** to save this change.
## Whitelist for UC standard clusters
To whitelist the Init Script and the JAR path, follow the instructions to [add Prophecy libraries to your Databricks allowlist](/data-engineering/fabrics/spark-provider/databricks/whitelist-plibs).