Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt

Use this file to discover all available pages before exploring further.

/images/icon.png
Available for Enterprise Edition only.

​
Using PBT with GitHub Actions

Prophecy Build Tool (PBT) can be integrated with GitHub Actions to:
  • validate pipelines
  • build artifacts (.jar / .whl)
  • run unit tests
  • deploy pipelines to Databricks
View an Example GitHub repository.

​
Prerequisites

  • A Prophecy project hosted in a GitHub repository
  • A Databricks workspace for deployment

​
Configuration

​
Environment variables

PBT requires the following:
  • DATABRICKS_HOST
  • DATABRICKS_TOKEN
Store the token as a GitHub Actions secret:
Settings → Secrets → Actions → New repository secret
Then reference it in your workflow: 
env:
  DATABRICKS_HOST: "https://<your-databricks-instance>"
  DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}

​
Example workflow (deploy on push to prod)

This workflow:
  • runs on every push to prod
  • validates, builds, and tests pipelines
  • deploys artifacts to Databricks
Create the workflow file: 
.github/workflows/exampleWorkflow.yml

​
Workflow definition

name: Example CI/CD with GitHub actions

on:
  push:
    branches:
      - "prod"

env:
  DATABRICKS_HOST: "https://sample_databricks_url.cloud.databricks.com"
  DATABRICKS_TOKEN: ${{ secrets.PROD_DATABRICKS_TOKEN }}
  FABRIC_ID: "4004"  # replace with your fabric id

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Set up JDK 11
        uses: actions/setup-java@v3
        with:
          java-version: "11"
          distribution: "adopt"

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.9.13"

      - name: Install dependencies
        run: |
          python3 -m pip install --upgrade pip
          pip install build pytest wheel pytest-html pyspark==3.3.0 prophecy-build-tool

      - name: Validate pipelines
        run: pbt validate --path .

      - name: Build pipelines
        run: pbt build --path .

      - name: Run tests
        run: pbt test --path .

      - name: Deploy pipelines
        run: pbt deploy --path . --release-version 1.0 --project-id example_project_id

​
What this workflow does

  1. Triggers on pushes to the prod branch
  2. Sets required environment variables for Databricks access
  3. Installs Java, Python, and PBT dependencies
  4. Validates pipeline syntax (pbt validate)
  5. Builds pipelines into .jar / .whl artifacts (pbt build)
  6. Runs unit tests (pbt test)
  7. Deploys artifacts and jobs to Databricks (pbt deploy)
    • Uploads artifacts referenced in databricks-job.json
    • Creates or updates Databricks jobs
    • Deploys pipeline configurations to DBFS if defined
If any step fails, the workflow stops and the run is marked as failed.