Skip to main content
Available for Enterprise Edition only.
This example shows how to use Jenkins to:
  • validate and test Prophecy pipelines on pull requests
  • deploy pipelines to Databricks environments after merge
Each environment (develop, qa, prod) maps to a separate Databricks workspace. Typical promotion flow:

Prerequisites

You should have:
  • A Git repository containing a Prophecy project
  • A Jenkins server with permission to create pipelines
  • Databricks workspaces for each environment

Required Jenkins plugins

  • GitHub Pull Request Builder (for test pipeline)
  • GitHub plugin (for deploy pipeline)
Check plugin compatibility with your Jenkins version before installing.

Configuration

Secrets

Configure the following credentials in Jenkins:
  • DEMO_DATABRICKS_HOST
  • DEMO_DATABRICKS_TOKEN
  • PROD_DATABRICKS_HOST
  • PROD_DATABRICKS_TOKEN

Fabric ID

Find your Fabric ID from:
Metadata → Fabrics → <your fabric>

Testing pipeline (PR validation)

This pipeline:
  • runs on pull requests to develop, qa, and prod
  • validates pipelines
  • runs unit tests

Trigger

Use GitHub Pull Request Builder to trigger on:
  • new PRs
  • updates to PRs

Jenkinsfile (test)

What this pipeline does

  1. Checks out the PR branch.
  2. Installs PBT and dependencies.
  3. Validates pipeline syntax.
  4. Runs unit tests.

Deploy pipeline (post-merge)

This pipeline:
  • runs on commits to develop, qa, prod.
  • deploys pipelines to the corresponding Databricks environment.

Trigger

Use a GitHub webhook to trigger on push events.

Jenkinsfile (deploy)

What this pipeline does

  1. Selects the target environment based on branch.
  2. Installs PBT.
  3. Builds pipelines into .jar / .whl artifacts.
  4. Uploads artifacts to Databricks.
  5. Creates or updates jobs.

Notes

  • Each sh step runs in a separate shell, so the virtual environment must be reactivated.
  • For Scala pipelines, ensure JDK 11 is installed on Jenkins nodes.
  • Jenkins files are stored in the repository; Jenkins stores only triggers and credentials.