Skip to main content
/images/icon.png
Available for Enterprise Edition only.
The Prophecy Build Tool (PBT) is a command-line utility for building, testing, validating, and deploying Prophecy-generated projects. The Prophecy Build Tool lets you integrate Prophecy pipelines into existing CI/CD systems (such as GitHub Actions or Jenkins) and orchestration platforms (such as Databricks Workflows). You can use the PBT to run the same set of tasks described in Project release and deployment from the command line or in a CI/CD script.
The Prophecy Build Tool only supports building Python and Scala projects.

​
Features

Using the Prophecy Build tool, you can:
  • Build pipelines (all or a subset) in Prophecy projects.
  • Unit test pipelines in Prophecy projects.
  • Deploy jobs with built pipelines on Databricks.
  • Deploy jobs filtered by fabric IDs on Databricks.
  • Integrate with CI/CD tools like GitHub Actions.
  • Verify the project structure of Prophecy projects.
  • Deploy pipeline configurations.
  • Add Git tags to a deployment.
  • Set versions for PySpark projects.

​
Requirements

To install and run the Prophecy Build Tool, you need:
  • pip
  • python >=3.7 (Recommended 3.9.13)
  • pyspark (Recommended 3.3.0)

​
Installation

To install the Prophecy Build Tool, run: 
pip3 install prophecy-build-tool
See the PyPI package for the latest version.

​
Usage

Usage: pbt [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show help.

Commands:
  build     	Build Prophecy pipelines
  build-v2      Same as build but with additional options
  deploy    	Deploy pipelines and jobs
  deploy-v2     Same as deploy but with additional options
  test      	Run unit tests
  validate  	Validate pipelines for diagnostics
  versioning	Add versions to PySpark pipelines
  tag           Create a Git tag for the version in `pbt_project.yml`

​
Configuration

Before using PBT with Databricks, set the following environment variables: 
export DATABRICKS_HOST="https://example_databricks_host.cloud.databricks.com"
export DATABRICKS_TOKEN="exampledatabrickstoken"
These variables define the Databricks workspace and credentials used for building and deploying.

​
Build pipelines

The build command compiles all or specific pipelines within a Prophecy project. 
pbt build --path /path/to/your/prophecy_project/
You can also use a v2 version of the command, which adds an --add-pom-python option. 
pbt build-v2 --path /path/to/your/prophecy_project/

​
Build options

OptionDescription
--path TEXTRequired. Path to the directory containing the pbt_project.yml file.
--pipelines TEXTComma-separated list of pipelines to build.
--ignore-build-errorsContinue even if build errors occur.
Refer to logs for details.
--ignore-parse-errorsContinue even if pipeline parsing errors occur.
Returns success (EXIT_CODE = 0).
Refer to logs for details.
--add-pom-pythonAvailable with --build-v2. Adds pom.xml and MAVEN_COORDINATES files to PySpark builds.
--helpShow help for this command.
To build only specific pipelines: 
pbt build --pipelines customers_orders,join_agg_sort --path /path/to/your/prophecy_project/
To continue despite build or parsing errors: 
pbt build --path /path/to/your/prophecy_project/ --ignore-build-errors --ignore-parse-errors
If any pipeline fails to build, the Build tool exits with code 1 unless error-skipping flags are used.

​
Deploy pipelines and jobs

The deploy command builds and deploys Prophecy pipelines and jobs to your Databricks workspace. 
pbt deploy --path /path/to/your/prophecy_project/ --release-version 1.0 --project-id 10
PBT supports the --release-version and --project-id parameters, used to replace placeholders in your job definition file (databricks-job.json). These values determine the DBFS path where artifacts are uploaded.
Use the project’s ID (from its URL) and a unique release version for each deployment.

​
Sample deploy output

Prophecy Build Tool v1.0.4.1

Found 1 job: daily
Found 1 pipeline: customers_orders (python)

Building 1 pipeline 🚰
  Building pipeline pipelines/customers_orders [1/1]
✅ Build complete!

Deploying 1 job ⏱
  Deploying job jobs/daily [1/1]
  Uploading customers_orders-1.0-py3-none-any.whl to dbfs:/FileStore/prophecy/artifacts/...
  Updating existing job: daily
✅ Deployment completed successfully!

​
Deploy dependent projects

Use --dependent-projects-path to include dependent Prophecy projects located in subdirectories. 
pbt deploy --path /path/to/your/prophecy_project/ --release-version 1.0 --project-id 10 --dependent-projects-path /path/to/dependent/prophecy/projects

​
Deploy by fabric ID

Use --fabric-ids to deploy jobs associated with specific Fabric IDs (helpful for multi-workspace environments). 
pbt deploy --fabric-ids 647,1527 --path /path/to/your/prophecy_project/
You can find Fabric IDs in the Prophecy UI by visiting the Metadata page of a Fabric and checking its URL.

​
Skip builds

To deploy previously built pipelines without rebuilding: 
pbt deploy --skip-builds --path /path/to/your/prophecy_project/

​
Deploy specific jobs

By default, all jobs are deployed. To deploy selected jobs, use --job-ids. 
pbt deploy --path /path/to/your/prophecy_project/ --job-ids "TestJob1,TestJob2"
The Prophecy Build Tool automatically identifies and builds only the pipelines required by those jobs. You can also use a v2 version of the command, which adds several options described in the table below. 
pbt deploy-v2 --path /path/to/your/prophecy_project/ --job-ids "TestJob1,TestJob2"

​
Deploy options summary

OptionDescription
--path TEXTRequired. Path containing the pbt_project.yml file.
--dependent-projects-path TEXTPath containing dependent Prophecy projects.
--release-version TEXTRelease version tag for this deployment.
--project-id TEXTIn deploy, Prophecy project ID (used to replace placeholders). In deploy-v2, path to the directory containing the pbt_project.yml file.
--prophecy-url TEXTProphecy base URL for deployment. Removed in build-v2.
--fabric-ids TEXTComma-separated Fabric IDs to filter jobs.
--skip-buildsSkip building pipelines.
--job-ids TEXTComma-separated list of Job IDs to deploy.
--conf-dir TEXTAvailable with --deploy-v2. Path to configuration file folders.
--release-tag TEXTAvailable with --deploy-v2. Specify a release. tag.
--skip-pipeline-deployAvailable with --deploy-v2. Skip pipeline deployment and deploy only job definitions.
--migrateAvailable with --deploy-v2. Migrates a v1 project to v2.format.
--artifactory TEXTAvailable with --deploy-v2. Allows use of PyPI/Maven packages instead of DBFS files for deployment.
--skip-artifactory-uploadAvailable with --deploy-v2. Skips uploading to private artifactory (must be used with --artifactory).
--helpShow help for this command.

​
Test pipelines

The Prophecy Build Tool supports unit testing of pipelines within a Prophecy project. Tests run with the default configuration under configs/resources/config. 
pbt test --path /path/to/your/prophecy_project/

​
Test options

OptionDescription
--path TEXTRequired. Path containing the pbt_project.yml file.
--driver-library-path TEXTPath to JARs for prophecy-python-libs or other dependencies.
--pipelines TEXTComma-separated list of pipelines to test.
--helpShow help for this command.
If --driver-library-path is omitted, dependencies are fetched automatically from Maven Central.

​
Sample test output

Prophecy Build Tool v1.0.1

Found 1 job: daily
Found 1 pipeline: customers_orders (python)

Unit Testing pipeline pipelines/customers_orders [1/1]

============================= test session starts ==============================
platform darwin -- Python 3.8.9, pytest-7.1.2
collected 1 item

test/TestSuite.py::CleanupTest::test_unit_test_0 PASSED [100%]
============================== 1 passed in 17.4s ===============================

✅ Unit test for pipeline: customers_orders succeeded.

​
Validate pipelines

Validation checks all pipelines in a project for warnings and errors, similar to Prophecy’s in-IDE diagnostics. This helps ensure pipelines are production-ready before deployment. 
pbt validate --path /path/to/your/prophecy_project/

​
Validate options

OptionDescription
--path TEXTRequired. Path containing the pbt_project.yml file.
--treat-warnings-as-errorsTreat warnings as errors during validation.

​
Applying versions to PySpark projects

PySpark projects often rely on specific versions of Spark, Python libraries, and data connectors. Versioning the project (via pbt_project.yml or setup.py) ensures compatibility between your code and these dependencies, helping avoid runtime errors when pipelines are deployed to different environments (local, Databricks). The Prophecy Build Tool lets you set various options for versioning as follows.

​
Versioning options

OptionDescription
--path <PATH>Required. Path to the directory containing the pbt_project.yml file.
--repo-path <PATH>Path to the repository root. If left blank, the tool uses --path.
--bump [major|minor|patch|build|prerelease]Bumps one of the semantic version numbers for the project and all pipelines based on the current value. Only works if existing versions follow Semantic Versioning.
--set TEXTExplicitly set the exact version.
--force, --spikeBypass errors if the version set is lower than the base branch.
--syncEnsure all files are set to the same version defined in pbt_project.yml. (Implies --force.)
--set-suffix TEXTSet a suffix string (e.g., -SNAPSHOT or -rc.4). If this is not a valid semVer string, an error will be thrown.
--check-syncCheck to see if versions are synced. Exit code 0 = success, 1 = failure.
--compare-to-target, --compare <TARGET_BRANCH>Checks if the current branch has a greater version number than the <TARGET_BRANCH> provided. Returns 0 (true) or 1 (false). Also performs a --sync check.
Note: If --bump is also provided, it compares versions and applies the bump strategy if the current version is lower.
--make-uniqueMakes a version unique for feature branches by adding build-metadata and prerelease identifiers.
Format: MAJOR.MINOR.PATCH-PRERELEASE+BUILDMETADATA
Examples:
Python → 3.3.0 → 3.3.0-dev0+sha.j0239ruf0ew
Scala → 3.3.0 → 3.3.0-SNAPSHOT+sha.j0239ruf0ew
--pbt-onlyApply version operation to pbt_project.yml file only. Applicable with --compare, --make-unique, --bump, --set, or --set-suffix.
--helpShow help for this command.

​
Tagging builds

The pbt tag command creates a Git tag for the version listed in pbt_project.yml. This tag marks a specific point in your project’s history so you can track or redeploy that version later. By default, the tag name includes the branch (for example, main/1.4.0) and is pushed to the remote automatically. You can change or remove the branch name with --branch, create a custom tag with --custom, or skip pushing with --no-push.

​
Tag options

OptionDescription
--path TEXTPath to the directory containing the pbt_project.yml file [required]
--repo-path TEXTPath to the repository root. If left blank, it will use --path.
--no-pushBy default, the tag will be pushed to the origin after it is created. Use this flag to skip pushing the tag.
--branch TEXTNormally, the tag is prefixed with the branch name: <branch_name>/<version>. This option overrides <branch_name>. Provide "" to omit the branch name.
--custom TEXTExplicitly set the exact tag using a string. Ignores other options.
--helpShow help for this command.

​
Sample output

Prophecy Build Tool v1.0.3.4

Project name: HelloWorld
Found 1 job: default_schedule
Found 4 pipelines: customers_orders, report_top_customers, join_agg_sort, farmers-markets-irs

Validating 4 pipelines
  Validating pipeline pipelines/customers_orders [1/4]
  Pipeline validated: customers_orders
  ...
✅ All pipelines validated successfully.

​
Quick reference

CommandDescriptionCommon Options
pbt buildBuild all or selected pipelines within a Prophecy project.--path (required)
--pipelines (comma-separated)
--ignore-build-errors
--ignore-parse-errors
pbt testRun unit tests for pipelines using the default configuration.--path (required)
--pipelines (comma-separated)
--driver-library-path (optional)
pbt validateValidate pipelines for warnings or errors before deployment.--path (required)
--treat-warnings-as-errors
pbt deployBuild and deploy pipelines and jobs to Databricks.--path (required)
--release-version
--project-id
--fabric-ids
--job-ids
--skip-builds
pbt versionSet versions for PySpark projects.
Environment variablesRequired for Databricks connections.DATABRICKS_HOST
DATABRICKS_TOKEN

​
Example workflow

# 1. Build project pipelines
pbt build --path /path/to/project

# 2. Run tests
pbt test --path /path/to/project

# 3. Validate before deployment
pbt validate --path /path/to/project

# 4. Deploy
pbt deploy --path /path/to/project --release-version 1.0 --project-id 123

​
What’s next

To learn how to integrate the Prophecy Build Tool with GitHub Actions and Jenkins, see the following examples: