Prophecy operates as a distributed system built on microservices architecture, orchestrated by Kubernetes across multiple cloud platforms. The platform consists of several core components that work together to provide data transformation, orchestration, and management capabilities.
Architecture by Edition
Free and Professional Edition
The Free and Professional Editions provide a complete data platform with managed components.
| Component | Description |
|---|
| Prophecy Studio | The control plane that provides the user interface for developing visual data pipelines and managing projects. |
| Prophecy Automate | The native runtime designed for data ingestion, egress, and built-in scheduling capabilities. |
| Prophecy In Memory | The Prophecy-managed SQL warehouse that processes data transformations. |
| Data storage | Data outside of the execution environment that will flow in and out of the pipeline. |
| AI endpoint | Prophecy-managed LLM subscription and endpoint. |
| Version control | Git integration supporting both Prophecy-managed and external Git repositories. |
| Deployment model | SaaS only. Learn more in Deployment models. |
Express Edition
The Express Edition provides enterprise-grade features scoped to leverage your existing SQL warehouse infrastructure.
| Component | Description |
|---|
| Prophecy Studio | The control plane that provides the user interface for developing visual data pipelines and managing projects. |
| Prophecy Automate | The native runtime designed for data ingestion, egress, and built-in scheduling capabilities. |
| External SQL Warehouse | Your own Databricks SQL engine that executes data transformations. |
| Data storage | Data outside of the execution environment that will flow in and out of the pipeline. |
| AI endpoint | Customer-managed LLM subscription and endpoint. |
| Version control | Git integration supporting both Prophecy-managed and external Git repositories. |
| Deployment model | Dedicated SaaS only. Learn more in Deployment models. |
This diagram shows the architecture for the Express Edition. Users on the Enterprise Edition can
also leverage this architecture. However, Enterprise users can also connect to additional SQL
warehouses, like BigQuery and Snowflake.
Enterprise Edition
The Enterprise edition offers maximum flexibility with multiple execution engine options and deployment models.
| Component | Description |
|---|
| Prophecy Studio | The control plane that provides the user interface for developing visual data pipelines and managing projects across various data platforms. |
| Execution engine | Flexible compute options including Spark clusters or external SQL warehouses combined with Prophecy Automate. Prophecy executes data transformations on your chosen execution environment. Fabrics enable users to execute pipelines on these platforms. Prophecy does not persist your data. |
| Data storage | Data outside of the execution environment that will flow in and out of the pipeline. |
| AI | Customer-managed LLM subscription and endpoint. |
| Version control | Git integration supporting both Prophecy-managed and external Git repositories. |
| Deployment model | Dedicated SaaS preferred and SaaS available. Learn more in Deployment models. |
The Enterprise Edition supports both SQL-based and Spark-based architectures. The diagram above shows the architecture for a deployment using Spark.
Prophecy can accommodate a wide variety of architectures beyond this diagram. For example:
- The diagram demonstrates Databricks as the execution engine. You can connect to other platforms like Amazon EMR and Google Cloud Dataproc, or use another Spark engine through Apache Livy.
- The diagram displays a connection to an external Git repository. You can connect to a variety of providers such as GitHub, Bitbucket, GitLab, and more.
What is Prophecy Automate?
Prophecy Automate is the native runtime available across all Prophecy editions. It extends pipeline execution by providing orchestration, external data handling, and observability for both SQL and PySpark pipelines.
-
Ingress / Egress: Automate enables pipelines to interact with external systems such as databases, file storage, and APIs.
-
Orchestration: Automate manages pipeline execution across environments through time-based and trigger-based scheduling, as well as orchestration APIs for programmatic control. It also supports dynamic execution patterns through features like the DynamicInput gem, which adapts queries based on incoming data, and the Directory gem, which enumerates files and folders from storage systems.
-
Observe: Automate provides visibility into pipeline activity, including run history and status tracking, project deployment tracking, and monitoring of active pipeline schedules.
Supported compute engines
| Provider | SQL Pipelines | Spark Pipelines | SQL Models |
|---|
| Prophecy In Memory | ✔ | | ✔ |
| Databricks | ✔ | ✔ | ✔ |
| BigQuery | ✔ | | ✔ |
| Snowflake | ✔ | | ✔ |
| EMR | | ✔ | |
| Dataproc | | ✔ | |
| Synapse | | ✔ | |
| Livy | | ✔ | |
