Available for Enterprise Edition only.
| Benefit | Description |
|---|---|
| Error Tracking | Quickly trace issues back to their source |
| Quality Assurance | Verify data integrity throughout its lifecycle |
| Change Management | Plan system migrations with complete visibility |
| Discovery | Find and understand available data assets |
| Governance | Maintain compliance with data policies |
Prophecy supports lineage for Python/PySpark and Scala projects.
Accessing lineage
There are two ways to access the lineage view:- Option 1: Hover over your pipeline in the Pipeline tab of the Metadata page.
- Option 2: For SQL projects where the default transformation is Model, open Project Lineage from the project settings dropdown.
- Entity type
- Project
- Author
- Last modified time period
Lineage Summary
The Summary tab visually illustrates the flow of data through pipelines, models, and datasets depending on the entities in your project. Use the Browse Datasets panel to search for and select the column or entity that you want to inspect.
Clicking the project icon or project name opens the project landing page in the canvas. To open
lineage, use the lineage navigation described here.
Column-level lineage
Column-level lineage tracks the flow and transformation of individual data columns through the pipeline, demonstrating how they are processed at each stage. When you select a column in the lineage view, Prophecy provides the column’s upstream and downstream transformations.
Lineage Zoom In
The Zoom In tab allows you to dive deeper into specific components of your pipeline or dataset to examine detailed information.Pipeline lineage
For pipelines, you can see code-level information about each component present in the pipeline. Select a component to view its:- Transformation code
- Input columns
- Output columns
Dataset lineage
For datasets, you can find information around all the upstream and downstream transformations (if any) for all the columns of the selected dataset.