Huge improvements to the Agent
Private PreviewProphecy Agents have been completely redesigned to be more powerful, accurate, and delightful to work with. Try using the upgraded Agent to transform, document, and harmonize your data.The Agent will now describe how it’s thinking as it works through your request, making the process transparent and easier to debug. This additional thinking also helps the Agent make better decisions and produce more accurate results. Because of this, you can prompt the Agent with more complex requests and decrease the time to results during development.Analysis dashboards
You can now quickly gain insight about your data by asking the Agent to create data analyses for you. These analyses are available and customizable using dashboards, which you can share with your team or deploy on a schedule. See the analysis overview for more information.Analysis dashboards were built on Prophecy Apps, which are now deprecated.
Visual container enhancements
Visual containers let you organize related gems in your pipeline for easier navigation and understanding. We now support nesting containers within other containers.Parameter sets
You can now create parameter sets to assign values to variables for different environments or deployment scenarios. Parameters can be global (project-level) or local (pipeline-level). You’ll be able to select the appropriate parameter set during project deployment or app runs.
Source and target file enhancements
We’ve made several enhancements to the source and target gems for file storage systems:- Compression: Source gems can now read compressed files and target gems can now write compressed files in the following formats:
gzip,zstd,lz4,zlib,snappy, andlzop. - Examine: Gems now have an Examine button in the Location tab. When you click the Examine button, Prophecy reads a small sample of the file to automatically determine the file format, compression type (if any), and schema.
- Encryption: Target gems can now write encrypted files using the following encryption methods:
AES-192,AES-296, andBlowFish. Source gems cannot read encrypted files.
PySpark for data analysis projects
Private PreviewProphecy now offers Simplified PySpark for data analysis. This project type abstracts PySpark into a simple, data analysis interface. You can switch between SQL and PySpark project types in the development settings of a project.We’ve also added support for unit tests for PySpark projects. In future releases, we will support unit tests for SQL projects as well.SAP HANA target gem
The SAP HANA target gem now supports Delete and Insert and SCD2 write modes.ProphecySparkBasicsPython v0.2.54
This new version of the ProphecySparkBasicsPython package includes the following enhancements:- Added support for bulk updating target columns during SCD1 Delta merge. This way, you don’t have to write individual update expressions for each column.
- Added support for lateral column aliases in Reformat and Join gems. Lateral column aliasing in PySpark allows you to reuse a column alias defined earlier in the same clause.
Prophecy Library versions
- ProphecyLibsPython 2.1.9
- ProphecyLibsScala 8.16.0