> ## Documentation Index > Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt > Use this file to discover all available pages before exploring further. # 2025 releases > Features and enhancements from new releases ## On-demand pipeline compilation This feature lets you make multiple pipeline changes without triggering pipeline compilation after each edit. When ready, you can [compile on-demand](/data-analysis/development/studio/development-settings#enable-manual-compilation) using the **Compile** button. This reduces latency when developing large pipelines and projects. To use this feature, have your administrator or your Prophecy contact set `EDITOR_ON_DEMAND_COMPILATION: 'true'` in your Prophecy deployment configuration. ## Data test API Run data tests programmatically using the new [API](/api-reference/pipelines/run-data-tests). Execute column tests, table tests, and project tests together from a single API call. Trigger model tests from a separate API call. ## Data tests compatible with pipelines Prophecy now supports running data tests on Prophecy fabrics. (Previously, data tests were only supported on SQL fabrics.) Learn more in [Table tests](/data-analysis/development/tests/table-tests) and [Project tests](/data-analysis/development/tests/project-tests). ## Schema validation for JSON and XML files Prophecy lets you enable schema validation for [JSON](/data-analysis/gems/source-target/file/file-types/json) and [XML](/data-analysis/gems/source-target/file/file-types/xml) files in a Source gem. Use schema validation to ensure files conform to a predefined structure before ingesting the data. Schema validation works for all file storage connections. ## Knowledge graph scope The [knowledge graph](/data-analysis/ai/knowledge-graph/knowledge-graph) indexer now scans all connections in a fabric, not just the SQL warehouse connection. This way, the Agent can access all data from the attached fabric. ## Charts improvements Now, when you configure a chart in a [Prophecy App](/data-analysis/analysis/overview) or in the [Data Explorer](/data-analysis/development/runs/data-explorer/data-explorer), you can change the colors used in the chart. Charts color picker ## Selective sampling mode There have been several improvements to [selective sampling](/data-engineering/development/runs/data-sampling#selective-sampling): * Prophecy can now display the total [record count](/data-engineering/development/runs/data-sampling#record-counts) of selected gems. * You can now use selective data sampling mode when running pipelines on [Livy](/data-engineering/fabrics/spark-provider/livy) clusters. ## Prophecy Library versions * ProphecyLibsPython 2.1.7 * ProphecyLibsScala 8.15.1 ## Fabric APIs We've exposed many [new endpoints](/api-reference/introduction) for creating and managing [fabrics](/data-analysis/environment/fabrics/prophecy-fabrics). You can now create, update, and delete fabrics, as well as manage fabric connections and secrets, via REST API. ## Agent documentation templates Create custom templates for AI pipeline documentation generation using [Copilot Template Language (CTL)](/data-analysis/ai/agent/documentation/ctl-reference). CTL extends Markdown with special markers that generate interactive components like question forms, visual pipeline diagrams, and dynamic content that automatically updates based on your pipeline structure. ## Azure Data Lake Storage (ADLS) Source and Target gems Use Azure Data Lake Storage (ADLS) as a source and target in your pipelines with the [ADLS gem](/data-analysis/gems/source-target/file/adls). The gem supports reading and writing CSV, JSON, Parquet, XLSX, and XML files. Source gems can read from specific file paths or from a file arrival/change trigger. Target gems can write to specific file paths. ## Knowledge graph indexer You can now schedule automated indexing for the [knowledge graph indexer](/data-analysis/ai/knowledge-graph/indexer) to run hourly, daily, or weekly. The schedule supports custom timezone configuration, with the default timezone matching your Prophecy access location. You can also configure separate authentication credentials for the [knowledge graph indexer](/data-analysis/ai/knowledge-graph/indexer#add-separate-authentication-for-the-indexer), independent from pipeline execution credentials. Use Service Principal OAuth for production environments where credentials don't expire, or User OAuth for development scenarios. ## LLM configuration We've added the ability to configure LLM provider credentials and model specifications directly from the Prophecy UI in [Copilot Settings](/data-engineering/administration/management/cluster-admin-settings/copilot-settings). Add credentials for multiple LLM providers, define smart and fast models for different task types, and configure speech-to-text and text-to-speech models. Sensitive credential values are automatically masked after saving. ## Prophecy Library versions * ProphecyLibsPython 2.1.5 * ProphecyLibsScala 8.14.0 ## Free and Professional Editions We have released two new editions of Prophecy for analysts: Free and Professional Edition. Free and Professional Editions are designed for data analysts who want to collaborate without managing infrastructure. Both editions provide Prophecy-managed resources that are metered by credits. Review the [Prophecy Editions](/data-engineering/administration/platform/editions) page for a full comparison between Free, Professional, and Enterprise Editions of Prophecy. ## Agent-generated pipeline documentation You can now use the Prophecy Agent to [generate pipeline documentation](/data-analysis/ai/agent/documentation/documentation). Instead of writing documentation manually, this lets you create detailed specifications automatically. ## New gems * [Regex](/data-analysis/gems/parse/regex): Enables pattern matching and text extraction using regular expressions. * [GenerateRows](/data-analysis/gems/prepare/generate-rows): Create new rows of data using iterative expressions. * [GCS Source and Target](/data-analysis/gems/source-target/file/gcs): Read from or write to Google Cloud Storage (GCS) buckets. * [Salesforce Target](/data-analysis/gems/source-target/external-table/salesforce): Supports writing to Salesforce Objects (CRM Analytics datasets not supported for targets). ## Gem improvements * The [Email gem](/data-analysis/gems/report/email) now supports a **Use column** option for several parameters, including **To**, **Cc**, **Bc**, **Subject**, and **Body**. When enabled, these fields can be dynamically populated from columns in your input dataset. Each row in your dataset can specify a different recipient and/or custom message while sending the same attachment to all recipients. * The [RestAPI gem](/data-analysis/gems/custom/rest-api) has been updated in the following ways: * The gem now supports parsing JSON responses into multiple output columns. * There are two new authentication methods — **Basic Auth** and **Bearer Token** — that automatically send credentials to APIs using new [Prophecy secret types](/data-analysis/environment/secrets/secrets). * XLSX (Excel) Source and Target gems have been enhanced: * Read from multiple sheets in a single Excel file and union all rows into one output table. * Write to multiple sheets by providing a column whose values are target sheet names. Sheets are created if they don't exist. * We have added an option to the [Directory gem](/data-analysis/gems/custom/directory) that enables you to list all the sheet names in XLSX files detected in a specified directory. * You can now use the [DynamicInput gem](/data-analysis/gems/custom/dynamic-input) to union data from files defined in a column of file paths. Schemas are matched based on column names. ## Automatic package upgrades When configuring [Prophecy project dependencies](/data-analysis/development/extensibility/dependencies), you can now enable automatic package upgrades. When enabled, the dependency will automatically upgrade to the latest version available. ## EMR fabric improvements * You can use [Selective](/data-engineering/development/runs/data-sampling#selective-sampling) data sampling mode when running pipelines on EMR and EMR Serverless clusters. * When using EMR [SAML authentication](/data-engineering/fabrics/spark-provider/emr/emr-saml) with Okta, you will see updated job size [configurations](/data-engineering/fabrics/spark-provider/emr/emr-job-sizes) in EMR fabrics. ## Gem improvements You can now run the following gems in Databricks Serverless: | Gem | Requirements | | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | | [XLSX Source and Target](/data-engineering/gems/source-target/file/xlsx) | ProphecySparkBasicsPython >= 0.2.5
ProphecyLibsPython >= 2.1.5
Pandas Library Type only (Crealytics not supported) | | [Text Source and Target](/data-engineering/gems/source-target/file/text) | ProphecySparkBasicsPython >= 0.2.5 | ## Databricks jobs For [Databricks Jobs](/data-engineering/orchestration/databricks-jobs), we have expanded configuration options and improved integration with Prophecy orchestration. * **Email notifications**: Now includes a duration threshold, providing email alerts when jobs exceed expected runtimes. * **If Else gem**: Implement conditional branching within a job to control execution flow between two paths. * **RunJob gem**: Allow one Prophecy job to trigger another job programmatically. * **Notebook gem**: Execute a Databricks notebook from a Prophecy job. * **Delta Live Tables Pipeline gem**: Run a Databricks Delta Live Tables pipeline from a Prophecy job. * **Run Conditions for job gems**: Control when a job gem executes based on the outcome of upstream job gems. * **For Each execution for job gems**: Configure most job gems to run multiple iterations dynamically using the For Each option in the gem's Conditional tab. ## Prophecy Library versions * ProphecyLibsPython 2.1.5 * ProphecyLibsScala 8.13.0 ## Automatic team creation You can now enable automatic team creation when using Microsoft Entra ID (Azure Active Directory) [SSO](/data-analysis/administration/management/authentication/azure-ad#step-2-optional-enable-automatic-team-creation). ## Team-level Copilot We've added the ability to [enable or disable Copilot](/data-analysis/administration/management/teams/teams) at the team level. ## Knowledge graph improvements * After the first full crawl of a fabric, subsequent crawls are now incremental, based on differences between cached Knowledge Graph data and the latest warehouse data. This greatly improves crawl performance. * Progress is now displayed at the schema level instead of the database level, providing more granular tracking. * Crawls can now be cancelled while in progress. ## Pipeline gem The [Pipeline gem](/data-analysis/production/scheduling/pipeline-trigger-gem) lets you trigger another pipeline from within the current pipeline. You can use trigger conditions, pass parameters, and receive metadata for each run. Each gem execution can trigger the target pipeline multiple times, running sequentially until all runs finish. ## Data processing gems * [CountRecords](/data-analysis/gems/transform/count-records): Count the number of rows in a dataset using different methods. * [Directory](/data-analysis/gems/custom/directory): List files and folders of a specified directory from a data ingress/egress connection. * [DataMasking](/data-analysis/gems/prepare/masking): Obfuscate sensitive string data in one or more columns. * [DataEncoderDecoder](/data-analysis/gems/transform/encoder-decoder): Encode or decode data in selected columns using a variety of techniques. * [FindDuplicates](/data-analysis/gems/prepare/find-duplicates): Filters rows in a dataset based on how frequently they appear. * [RecordID](/data-analysis/gems/prepare/record-id): Generate row-level IDs using UUIDs for randomly generated values or incremental IDs for sequential values. * [DynamicInput](/data-analysis/gems/custom/dynamic-input): Run SQL queries that update automatically based on your incoming data. ## Connections and source/target gems The following are new connections and source/target gems: * [Fixed-width Source](/data-analysis/gems/source-target/file/file-types/fixed-width): Read fixed-width files from various file storage accessed from connections. * [Azure Synapse Source](/data-analysis/gems/source-target/external-table/synapse): Using the new Azure Synapse [connection](/data-analysis/environment/connections/synapse), you can read data from Microsoft SQL Server hosted on Azure Synapse dedicated SQL pools. * [Hana Source and Target](/data-analysis/gems/source-target/external-table/hana/hana): Using the new HANA connection, you can read and write to an SAP HANA database (Added in 4.1.3.3). Additionally, the following enhancements have been made to existing gems: * We've added a **File Encoding** property to [CSV Source and Target](/data-analysis/gems/source-target/file/file-types/csv) gems. This lets you decode CSV files when reading and encode CSV files when writing. * [SFTP](/data-analysis/gems/source-target/file/sftp) and [S3](/data-analysis/gems/source-target/file/s3) Source gems have the following enhancements that allow you to (Added in 4.1.3.3): * Use wildcard patterns in file paths. * Move or delete files after they are successfully read. * Dynamically read files provided by file arrival/change triggers through the new Configuration mode. ## Visual containers We've added a feature that lets you divide your pipeline into logical sections using [visual containers](/data-analysis/development/studio/containers) for canvas organization. ## Version control improvements You can now [move projects](/data-analysis/development/versioning/migrate-managed) hosted in a Prophecy-managed Git repository into an external repository without having to clone the project. ## Enhancements * You can now enable or disable [auto-run](/data-analysis/analysis/analysis-settings) for Prophecy Apps in the app settings. * When you open a gem, you can now collapse the **Ports** panel of the gem configuration dialog. * [Write strategies](/data-analysis/gems/source-target/table/write/write-options) for Table gems have been renamed. However, they are still mapped to the same dbt configs under the hood (Changed in 4.1.3.9). * We've enhanced the [lineage extractor](/data-engineering/lineage/lineage-extractor) Python tool to be able to extract lineage from pipelines in SQL projects in Excel and OpenLineage format (Added in 4.1.3.10). ## Project-level configurations Previously, pipeline configurations were always scoped to the individual pipeline. You can now define [project-level configurations](/data-engineering/development/pipelines/configuration) that are shared across all pipelines within a project. ## Data sampling improvements * The **Vanilla** [data sampling](/data-engineering/development/runs/data-sampling) mode has been removed from the UI. Pipelines that currently use Vanilla will continue to run as expected; however, once the sampling mode is changed, you won't be able to switch back to Vanilla. We recommend updating all pipelines from Vanilla to Selective mode. * Prophecy can now capture [execution metrics](/data-engineering/fabrics/execution-metrics) for pipelines that run using Selective [data sampling](/data-engineering/development/runs/data-sampling) mode. ## Fabric improvements * The [Amazon EMR Spark fabric](/data-engineering/fabrics/spark-provider/emr/emr) now [accepts SAML](/data-engineering/fabrics/spark-provider/emr/emr-saml) as an authentication method, using Okta as the identity provider (Added in 4.1.3.3). * When configuring a Databricks job size in Prophecy, you can now set the cluster access mode to **Auto** (Added in 4.1.3.3). This aligns with recent Databricks changes around cluster access modes. Note that pipeline behavior may vary depending on the cluster Databricks allocates. ## Prophecy Library versions * ProphecyLibsPython 2.1.3 * ProphecyLibsScala 8.12.0 ## Supported LLMs Prophecy uses external LLMs for Copilot and Agents. We've added support for additional LLMs if you want to switch from the default OpenAI endpoint: * Google Gemini 2.5 * Claud Sonnet 4 OpenAI GPT-4o and GPT-4o models mini are fully supported. Gemini and Claud models are under integration testing at this time. ## BigQuery warehouse Prophecy now supports running SQL transformations in your BigQuery warehouse. Learn how to configure a [BigQuery connection](/data-analysis/environment/connections/bigquery) in fabrics to leverage this capability. ## Spatial gems * [Buffer](/data-analysis/gems/spatial/buffer): Take any polygon or line and expand or contract its boundaries. * [FindNearest](/data-analysis/gems/spatial/nearest-point): Find the closest spatial point(s) between two datasets based on geographic distance. * [Simplify](/data-analysis/gems/spatial/simplify): Reduce the number of vertices in polygons and polylines while preserving overall shape. * [HeatMap](/data-analysis/gems/spatial/heatmap): Transform latitude and longitude point data into a spatial heatmap using hexagonal tiling. * [PolyBuild](/data-analysis/gems/spatial/polybuild): Build spatial shapes from coordinate data by grouping and ordering points into either polygons or polylines. * [SpatialMatch](/data-analysis/gems/spatial/spatial-match): Compare geometries and return only the pairs that have the spatial relationship you specify. ## Transformation gems * [Pivot](/data-analysis/gems/transform/pivot): Reshape your data by transforming unique row values from one column into new columns. * [Unpivot](/data-analysis/gems/transform/unpivot): Convert multiple columns into row-level key-value pairs to simplify columnar data. ## Connections and source/target gems * Connect to [Salesforce](/data-analysis/gems/source-target/external-table/salesforce) and read data into your pipelines. * Prophecy now supports uploading [Parquet](/data-analysis/gems/source-target/file/file-types/parquet) files to Databricks connections. ## File arrival trigger Pipeline schedules can now be triggered by [file arrivals or changes](/data-analysis/production/scheduling/triggers#file-arrival-or-change-trigger) in a specified SFTP or S3 location. ## Stored procedures You can now define and call [stored procedures](/data-analysis/gems/custom/stored-procedure) within SQL projects running on BigQuery. This is particularly useful when migrating workflows from other systems. ## Enhancements * Prophecy uses temporary tables to process data from external systems in a SQL warehouse. Previously, these tables were written and then removed, which meant you could briefly see them in your warehouse. Now, they use **ephemeral materialization**, so they only exist during execution and no longer appear in your warehouse. * You can now overwrite a Table gem using the [File Upload component](/data-analysis/analysis/analysis-components#file-upload) in a Prophecy App. Previously, you could only overwrite Source gems. (Added in 4.1.2.2). ## Databricks serverless PySpark project pipelines can now run on Databricks serverless compute. See supported features and limitations in [Databricks serverless compute for PySpark](/data-engineering/fabrics/spark-provider/databricks/databricks-serverless). ## Version control improvements * We have improved the release flow for protected branches. When a branch like `main` is protected, direct commits to it are not allowed. Our previous release process required a direct commit to `main`, which caused issues in these scenarios. We now provide an option on the Open Pull Request screen to bump the version of the source branch before merging into main. To learn more, visit [Pull requests](/data-engineering/ci-cd/git/git). * You can now merge from the default branch (e.g., `main`) into another branch. Previously, the merge UI only allowed merging into the default branch. This enhancement is available in both the [Git workflow dialog](/data-engineering/ci-cd/git/git-workflow) within the project editor and the Commits tab on the project metadata page. ## Catalog table improvements The [Catalog Table](/data-engineering/gems/source-target/catalog-table/delta) Target gem includes the following enhancements. Update the `ProphecySparkBasicsPython` package to version 0.2.47 to access these updates. * Soft Delete for SCD Type 2 (SCD2) Merge: You can now enable soft delete behavior during SCD2 merges. When the source system deletes a record, the target table retains its history by updating the record's end-time. This treats the deletion as a Type 2 change instead of removing the row. * Schema Evolution for Merge: When writing to a Delta table on Databricks using merge mode, the table can now automatically evolve its schema to accommodate changes in the incoming data (such as added or modified columns). This feature helps prevent merge failures due to schema mismatches. To enable it, turn on the "Schema evolution for merge" property in the Catalog Table Target gem. (Requires Databricks Runtime 15.4 or later.) ## XLSX support on UC clusters The XLSX gem now runs on [Databricks UC clusters](/data-engineering/fabrics/spark-provider/databricks/UCShared) configured with standard (formerly shared) access mode. To leverage this capability, choose to read file with Pandas. The `openpyxl` library is needed for Pandas to work. Files can be read from and written to UC volumes only. | Gem | Package | Minimum version | | ------------------------------------------------------ | --------------------------- | --------------- | | [XLSX](/data-engineering/gems/source-target/file/xlsx) | `ProphecySparkBasicsPython` | `0.2.47` | ## Prophecy Library versions * ProphecyLibsPython 2.0.11 * ProphecyLibsScala 8.11.0 ## Audit file downloads You can now enable auditing for file downloads from Prophecy. When this feature is turned on via an environment flag, audit logs will capture the file name, the user who downloaded it, and the page in Prophecy where the download occurred. Contact your administrator or support team to enable this feature. ## Service principal OAuth Databricks connections in Prophecy fabrics now accept [service principal](https://docs.databricks.com/aws/en/admin/users-groups/service-principals) credentials for OAuth. Now you can deploy scheduled pipelines using the service principal credentials. To learn more, visit the [Databricks connection](/data-analysis/environment/connections/databricks) documentation. ## Run history improvement The Run History tab in the Observability interface now displays the **Run Type** and **Run ID** for each historical pipeline run. ## Knowledge graph crawling We've added the ability to reindex your SQL warehouse for the knowledge graph. This ensures that the Prophecy Agent has the latest information about your datasets (Added in 4.1.1.2). ## Reporting connection and gem The new [Power BI connection](/data-analysis/environment/connections/power-bi) and [PowerBIWrite gem](/data-analysis/gems/report/power-bi) allow you to you publish pipeline results directly to Power BI tables (Added in 4.1.1.5). ## Enhancements * You can now fetch and reference the hidden `_metadata` column from input files in Databricks. Spark projects only. * For target models running on BigQuery SQL fabrics, you can now configure overwrite modes to partition by a specific column and appropriate partitions (Added in 4.1.1.1). * Spark project deployment is no longer blocked when you are missing service principal on an unrelated fabric using Databricks OAuth (Fixed in 4.1.1.1). ## Prophecy Library versions * ProphecyLibsPython 1.9.49 * ProphecyLibsScala 8.10.1 ## Agents V1 We've integrated an Agent that you can chat with to help build out pipelines. Open a pipeline and ask the Agent to help you search for data, transform it, and save it to a table. ## Data visualizations We've added data visualization capabilities to the Data Explorer, Prophecy Apps, and Agent. ## Models under the hood To view the dbt models created in the backend for pipelines in SQL projects, select the **Show Models** checkbox in the left sidebar. ## Update all dependencies You can now update all Spark dependencies simultaneously using the new **Update all dependencies** option in the project editor or metadata page. ## Prophecy Library versions * ProphecyLibsPython 1.9.49 * ProphecyLibsScala 8.10.0 ## Fabric connections * New Oracle, Redshift, and OneDrive connections * Parquet file support enabled for Amazon S3 and SFTP connections ## ProphecyDatabricksSqlSpatial v0.0.1 This package contains the following new spatial gems: * CreatePoint * Distance ## Fixes * Fixed the remove letters and remove numbers operations in the DataCleansing gem. Available in ProphecyDatabricksSqlBasics 0.0.5+ and ProphecySnowflakeSqlBasics 0.0.4+. ## New data types for Spark pipeline configs You can now set `date` and `timestamp` values in Spark pipeline configs. ## Prophecy Library versions * ProphecyLibsPython 1.9.46 * ProphecyLibsScala 8.9.1 ## New pipeline canvas for data preparation and analysis Prophecy 4.0.0 introduces a new way to build analytics projects. Not only do you have access to SQL transformation power, but you also can use Prophecy Automate to extend these capabilities. While your core business logic is still written in SQL and run using dbt, data ingress, data egress, and pipeline orchestration are now handled by Prophecy. The new canvas not only incorporates these capabilities, but also simplifies the development process by adding visual Git abstractions and other collaboration features. ## Gem enhancements * The CSV Source gem in Python/PySpark projects now includes a property that lets you skip `n` number of first or last lines when reading in a CSV file. * More gems in Python/PySpark projects are now compatible with [Databricks UC clusters](/data-engineering/fabrics/spark-provider/databricks/UCShared) configured with standard (formerly shared) access mode. The table below shows the minimum package version required to enable compatibility. | Gem | Package | Minimum version | | ------------------------------------------------------------------------------------- | --------------------------- | --------------- | | [CSV Source](/data-engineering/gems/source-target/file/csv) | `ProphecySparkBasicsPython` | `0.2.44` | | [BigQuery Source and Target](/data-engineering/gems/source-target/warehouse/bigquery) | `ProphecyWarehousePython` | `0.0.9` | | [EmailData](/data-engineering/gems/custom/email-data) | `ProphecyWebAppPython` | `0.1.2` | | [Seed Source](/data-engineering/gems/source-target/file/seed) | `ProphecySparkBasicsPython` | `0.2.39` | ## Data diff Data diff is a new feature that lets you see if your target data at the end of your pipeline matches your expectations. ## Enhancements * You can now set a specific Spark Config in a Livy fabric for a job size configuration. * Selective data sampling mode now works with Databricks UC standard clusters in Scala projects (Python already supported). ## Prophecy Library versions * ProphecyLibsPython: 1.9.45 * ProphecyLibsScala: 8.9.0