On-demand pipeline compilation
This feature lets you make multiple pipeline changes without triggering pipeline compilation after each edit. When ready, you can compile on-demand using the Compile button. This reduces latency when developing large pipelines and projects.To use this feature, have your administrator or your Prophecy contact setEDITOR_ON_DEMAND_COMPILATION: 'true' in your Prophecy deployment configuration.Data tests compatible with pipelines
Prophecy now supports running data tests on Prophecy fabrics. (Previously, data tests were only supported on SQL fabrics.)Learn more in Table tests and Project tests.Schema validation for JSON and XML files
Prophecy lets you enable schema validation for JSON and XML files in a Source gem. Use schema validation to ensure files conform to a predefined structure before ingesting the data. Schema validation works for all file storage connections.Knowledge graph scope
The knowledge graph indexer now scans all connections in a fabric, not just the SQL warehouse connection. This way, the Agent can access all data from the attached fabric.Charts improvements
Now, when you configure a chart in a Prophecy App or in the Data Explorer, you can change the colors used in the chart.
Selective sampling mode
There have been several improvements to selective sampling:- Prophecy can now display the total record count of selected gems.
- You can now use selective data sampling mode when running pipelines on Livy clusters.
Prophecy Library versions
- ProphecyLibsPython 2.1.7
- ProphecyLibsScala 8.15.1
Fabric APIs
We’ve exposed many new endpoints for creating and managing fabrics. You can now create, update, and delete fabrics, as well as manage fabric connections and secrets, via REST API.Agent documentation templates
Create custom templates for AI pipeline documentation generation using Copilot Template Language (CTL). CTL extends Markdown with special markers that generate interactive components like question forms, visual pipeline diagrams, and dynamic content that automatically updates based on your pipeline structure.Azure Data Lake Storage (ADLS) Source and Target gems
Use Azure Data Lake Storage (ADLS) as a source and target in your pipelines with the ADLS gem. The gem supports reading and writing CSV, JSON, Parquet, XLSX, and XML files. Source gems can read from specific file paths or from a file arrival/change trigger. Target gems can write to specific file paths.Knowledge graph indexer
You can now schedule automated indexing for the knowledge graph indexer to run hourly, daily, or weekly. The schedule supports custom timezone configuration, with the default timezone matching your Prophecy access location.You can also configure separate authentication credentials for the knowledge graph indexer, independent from pipeline execution credentials. Use Service Principal OAuth for production environments where credentials don’t expire, or User OAuth for development scenarios.LLM configuration
We’ve added the ability to configure LLM provider credentials and model specifications directly from the Prophecy UI in Copilot Settings. Add credentials for multiple LLM providers, define smart and fast models for different task types, and configure speech-to-text and text-to-speech models. Sensitive credential values are automatically masked after saving.Prophecy Library versions
- ProphecyLibsPython 2.1.5
- ProphecyLibsScala 8.14.0
Free and Professional Editions
We have released two new editions of Prophecy for analysts: Free and Professional Edition. Free and Professional Editions are designed for data analysts who want to collaborate without managing infrastructure. Both editions provide Prophecy-managed resources that are metered by credits.Review the Prophecy Editions page for a full
comparison between Free, Professional, and Enterprise Editions of Prophecy.
Agent-generated pipeline documentation
You can now use the Prophecy Agent to generate pipeline documentation. Instead of writing documentation manually, this lets you create detailed specifications automatically.New gems
- Regex: Enables pattern matching and text extraction using regular expressions.
- GenerateRows: Create new rows of data using iterative expressions.
- GCS Source and Target: Read from or write to Google Cloud Storage (GCS) buckets.
- Salesforce Target: Supports writing to Salesforce Objects (CRM Analytics datasets not supported for targets).
Gem improvements
- The Email gem now supports a Use column option for several parameters, including To, Cc, Bc, Subject, and Body. When enabled, these fields can be dynamically populated from columns in your input dataset. Each row in your dataset can specify a different recipient and/or custom message while sending the same attachment to all recipients.
-
The RestAPI gem has been updated in the following ways:
- The gem now supports parsing JSON responses into multiple output columns.
- There are two new authentication methods — Basic Auth and Bearer Token — that automatically send credentials to APIs using new Prophecy secret types.
-
XLSX (Excel) Source and Target gems have been enhanced:
- Read from multiple sheets in a single Excel file and union all rows into one output table.
- Write to multiple sheets by providing a column whose values are target sheet names. Sheets are created if they don’t exist.
- We have added an option to the Directory gem that enables you to list all the sheet names in XLSX files detected in a specified directory.
- You can now use the DynamicInput gem to union data from files defined in a column of file paths. Schemas are matched based on column names.
Automatic package upgrades
When configuring Prophecy project dependencies, you can now enable automatic package upgrades. When enabled, the dependency will automatically upgrade to the latest version available.EMR fabric improvements
- You can use Selective data sampling mode when running pipelines on EMR and EMR Serverless clusters.
- When using EMR SAML authentication with Okta, you will see updated job size configurations in EMR fabrics.
Gem improvements
You can now run the following gems in Databricks Serverless:| Gem | Requirements |
|---|---|
| XLSX Source and Target | ProphecySparkBasicsPython >= 0.2.5 ProphecyLibsPython >= 2.1.5 Pandas Library Type only (Crealytics not supported) |
| Text Source and Target | ProphecySparkBasicsPython >= 0.2.5 |
Databricks jobs
For Databricks Jobs, we have expanded configuration options and improved integration with Prophecy orchestration.- Email notifications: Now includes a duration threshold, providing email alerts when jobs exceed expected runtimes.
- If Else gem: Implement conditional branching within a job to control execution flow between two paths.
- RunJob gem: Allow one Prophecy job to trigger another job programmatically.
- Notebook gem: Execute a Databricks notebook from a Prophecy job.
- Delta Live Tables Pipeline gem: Run a Databricks Delta Live Tables pipeline from a Prophecy job.
- Run Conditions for job gems: Control when a job gem executes based on the outcome of upstream job gems.
- For Each execution for job gems: Configure most job gems to run multiple iterations dynamically using the For Each option in the gem’s Conditional tab.
Prophecy Library versions
- ProphecyLibsPython 2.1.5
- ProphecyLibsScala 8.13.0
Automatic team creation
You can now enable automatic team creation when using Microsoft Entra ID (Azure Active Directory) SSO.Team-level Copilot
We’ve added the ability to enable or disable Copilot at the team level.Knowledge graph improvements
- After the first full crawl of a fabric, subsequent crawls are now incremental, based on differences between cached Knowledge Graph data and the latest warehouse data. This greatly improves crawl performance.
- Progress is now displayed at the schema level instead of the database level, providing more granular tracking.
- Crawls can now be cancelled while in progress.
Pipeline gem
The Pipeline gem lets you trigger another pipeline from within the current pipeline. You can use trigger conditions, pass parameters, and receive metadata for each run. Each gem execution can trigger the target pipeline multiple times, running sequentially until all runs finish.Data processing gems
- CountRecords: Count the number of rows in a dataset using different methods.
- Directory: List files and folders of a specified directory from a data ingress/egress connection.
- DataMasking: Obfuscate sensitive string data in one or more columns.
- DataEncoderDecoder: Encode or decode data in selected columns using a variety of techniques.
- FindDuplicates: Filters rows in a dataset based on how frequently they appear.
- RecordID: Generate row-level IDs using UUIDs for randomly generated values or incremental IDs for sequential values.
- DynamicInput: Run SQL queries that update automatically based on your incoming data.
Connections and source/target gems
The following are new connections and source/target gems:- Fixed-width Source: Read fixed-width files from various file storage accessed from connections.
- Azure Synapse Source: Using the new Azure Synapse connection, you can read data from Microsoft SQL Server hosted on Azure Synapse dedicated SQL pools.
- Hana Source and Target: Using the new HANA connection, you can read and write to an SAP HANA database (Added in 4.1.3.3).
- We’ve added a File Encoding property to CSV Source and Target gems. This lets you decode CSV files when reading and encode CSV files when writing.
- SFTP and S3 Source gems have the following enhancements that allow you to (Added in 4.1.3.3):
- Use wildcard patterns in file paths.
- Move or delete files after they are successfully read.
- Dynamically read files provided by file arrival/change triggers through the new Configuration mode.
Visual containers
We’ve added a feature that lets you divide your pipeline into logical sections using visual containers for canvas organization.Version control improvements
You can now move projects hosted in a Prophecy-managed Git repository into an external repository without having to clone the project.Enhancements
- You can now enable or disable auto-run for Prophecy Apps in the app settings.
- When you open a gem, you can now collapse the Ports panel of the gem configuration dialog.
- Write strategies for Table gems have been renamed. However, they are still mapped to the same dbt configs under the hood (Changed in 4.1.3.9).
- We’ve enhanced the lineage extractor Python tool to be able to extract lineage from pipelines in SQL projects in Excel and OpenLineage format (Added in 4.1.3.10).
Project-level configurations
Previously, pipeline configurations were always scoped to the individual pipeline. You can now define project-level configurations that are shared across all pipelines within a project.Data sampling improvements
- The Vanilla data sampling mode has been removed from the UI. Pipelines that currently use Vanilla will continue to run as expected; however, once the sampling mode is changed, you won’t be able to switch back to Vanilla. We recommend updating all pipelines from Vanilla to Selective mode.
- Prophecy can now capture execution metrics for pipelines that run using Selective data sampling mode.
Fabric improvements
- The Amazon EMR Spark fabric now accepts SAML as an authentication method, using Okta as the identity provider (Added in 4.1.3.3).
- When configuring a Databricks job size in Prophecy, you can now set the cluster access mode to Auto (Added in 4.1.3.3). This aligns with recent Databricks changes around cluster access modes. Note that pipeline behavior may vary depending on the cluster Databricks allocates.
Prophecy Library versions
- ProphecyLibsPython 2.1.3
- ProphecyLibsScala 8.12.0
Supported LLMs
Prophecy uses external LLMs for Copilot and Agents. We’ve added support for additional LLMs if you want to switch from the default OpenAI endpoint:- Google Gemini 2.5
- Claud Sonnet 4
OpenAI GPT-4o and GPT-4o models mini are fully supported. Gemini and Claud models are under
integration testing at this time.
BigQuery warehouse
Prophecy now supports running SQL transformations in your BigQuery warehouse. Learn how to configure a BigQuery connection in fabrics to leverage this capability.Spatial gems
- Buffer: Take any polygon or line and expand or contract its boundaries.
- FindNearest: Find the closest spatial point(s) between two datasets based on geographic distance.
- Simplify: Reduce the number of vertices in polygons and polylines while preserving overall shape.
- HeatMap: Transform latitude and longitude point data into a spatial heatmap using hexagonal tiling.
- PolyBuild: Build spatial shapes from coordinate data by grouping and ordering points into either polygons or polylines.
- SpatialMatch: Compare geometries and return only the pairs that have the spatial relationship you specify.
Transformation gems
- Pivot: Reshape your data by transforming unique row values from one column into new columns.
- Unpivot: Convert multiple columns into row-level key-value pairs to simplify columnar data.
Connections and source/target gems
- Connect to Salesforce and read data into your pipelines.
- Prophecy now supports uploading Parquet files to Databricks connections.
File arrival trigger
Pipeline schedules can now be triggered by file arrivals or changes in a specified SFTP or S3 location.Stored procedures
You can now define and call stored procedures within SQL projects running on BigQuery. This is particularly useful when migrating workflows from other systems.Enhancements
- Prophecy uses temporary tables to process data from external systems in a SQL warehouse. Previously, these tables were written and then removed, which meant you could briefly see them in your warehouse. Now, they use ephemeral materialization, so they only exist during execution and no longer appear in your warehouse.
- You can now overwrite a Table gem using the File Upload component in a Prophecy App. Previously, you could only overwrite Source gems. (Added in 4.1.2.2).
Databricks serverless
PySpark project pipelines can now run on Databricks serverless compute. See supported features and limitations in Databricks serverless compute for PySpark.Version control improvements
- We have improved the release flow for protected branches. When a branch like
mainis protected, direct commits to it are not allowed. Our previous release process required a direct commit tomain, which caused issues in these scenarios. We now provide an option on the Open Pull Request screen to bump the version of the source branch before merging into main. To learn more, visit Pull requests. - You can now merge from the default branch (e.g.,
main) into another branch. Previously, the merge UI only allowed merging into the default branch. This enhancement is available in both the Git workflow dialog within the project editor and the Commits tab on the project metadata page.
Catalog table improvements
The Catalog Table Target gem includes the following enhancements. Update theProphecySparkBasicsPython package to version 0.2.47 to access these updates.- Soft Delete for SCD Type 2 (SCD2) Merge: You can now enable soft delete behavior during SCD2 merges. When the source system deletes a record, the target table retains its history by updating the record’s end-time. This treats the deletion as a Type 2 change instead of removing the row.
- Schema Evolution for Merge: When writing to a Delta table on Databricks using merge mode, the table can now automatically evolve its schema to accommodate changes in the incoming data (such as added or modified columns). This feature helps prevent merge failures due to schema mismatches. To enable it, turn on the “Schema evolution for merge” property in the Catalog Table Target gem. (Requires Databricks Runtime 15.4 or later.)
XLSX support on UC clusters
The XLSX gem now runs on Databricks UC clusters configured with standard (formerly shared) access mode. To leverage this capability, choose to read file with Pandas. Theopenpyxl library is needed for Pandas to work. Files can be read from and written to UC volumes only.| Gem | Package | Minimum version |
|---|---|---|
| XLSX | ProphecySparkBasicsPython | 0.2.47 |
Prophecy Library versions
- ProphecyLibsPython 2.0.11
- ProphecyLibsScala 8.11.0
Audit file downloads
You can now enable auditing for file downloads from Prophecy. When this feature is turned on via an environment flag, audit logs will capture the file name, the user who downloaded it, and the page in Prophecy where the download occurred. Contact your administrator or support team to enable this feature.Service principal OAuth
Databricks connections in Prophecy fabrics now accept service principal credentials for OAuth. Now you can deploy scheduled pipelines using the service principal credentials. To learn more, visit the Databricks connection documentation.Run history improvement
The Run History tab in the Observability interface now displays the Run Type and Run ID for each historical pipeline run.Knowledge graph crawling
We’ve added the ability to reindex your SQL warehouse for the knowledge graph. This ensures that the Prophecy Agent has the latest information about your datasets (Added in 4.1.1.2).Reporting connection and gem
The new Power BI connection and PowerBIWrite gem allow you to you publish pipeline results directly to Power BI tables (Added in 4.1.1.5).Enhancements
- You can now fetch and reference the hidden
_metadatacolumn from input files in Databricks. Spark projects only. - For target models running on BigQuery SQL fabrics, you can now configure overwrite modes to partition by a specific column and appropriate partitions (Added in 4.1.1.1).
- Spark project deployment is no longer blocked when you are missing service principal on an unrelated fabric using Databricks OAuth (Fixed in 4.1.1.1).
Prophecy Library versions
- ProphecyLibsPython 1.9.49
- ProphecyLibsScala 8.10.1
Agents V1
We’ve integrated an Agent that you can chat with to help build out pipelines. Open a pipeline and ask the Agent to help you search for data, transform it, and save it to a table.Data visualizations
We’ve added data visualization capabilities to the Data Explorer, Prophecy Apps, and Agent.Models under the hood
To view the dbt models created in the backend for pipelines in SQL projects, select the Show Models checkbox in the left sidebar.Fabric connections
- New Oracle, Redshift, and OneDrive connections
- Parquet file support enabled for Amazon S3 and SFTP connections
ProphecyDatabricksSqlSpatial 0.0.1
This package contains the following new spatial gems:- CreatePoint
- Distance
Fixes
- Fixed the remove letters and remove numbers operations in the DataCleansing gem. Available in ProphecyDatabricksSqlBasics 0.0.5+ and ProphecySnowflakeSqlBasics 0.0.4+.
New pipeline canvas for data preparation and analysis
Prophecy 4.0.0 introduces a new way to build analytics projects. Not only do you have access to SQL transformation power, but you also can use Prophecy Automate to extend these capabilities.While your core business logic is still written in SQL and run using dbt, data ingress, data egress, and pipeline orchestration are now handled by Prophecy. The new canvas not only incorporates these capabilities, but also simplifies the development process by adding visual Git abstractions and other collaboration features.Gem enhancements
-
The CSV Source gem in Python/PySpark projects now includes a property that lets you skip
nnumber of first or last lines when reading in a CSV file. -
More gems in Python/PySpark projects are now compatible with Databricks UC clusters configured with standard (formerly shared) access mode. The table below shows the minimum package version required to enable compatibility.
Gem Package Minimum version CSV Source ProphecySparkBasicsPython0.2.44BigQuery Source and Target ProphecyWarehousePython0.0.9EmailData ProphecyWebAppPython0.1.2Seed Source ProphecySparkBasicsPython0.2.39
Data diff
Data diff is a new feature that lets you see if your target data at the end of your pipeline matches your expectations.Enhancements
- You can now set a specific Spark Config in a Livy fabric for a job size configuration.
- Selective data sampling mode now works with Databricks UC standard clusters in Scala projects (Python already supported).
Prophecy Library versions
- ProphecyLibsPython: 1.9.45
- ProphecyLibsScala: 8.9.0