Lineage

Data lineage visualizes the complete journey of your data from source to destination, including all transformations it undergoes along the way. This visibility provides critical context for data operations and governance. Understanding data lineage delivers multiple benefits for your engineering team:

Benefit	Description
Error Tracking	Quickly trace issues back to their source
Quality Assurance	Verify data integrity throughout its lifecycle
Change Management	Plan system migrations with complete visibility
Discovery	Find and understand available data assets
Governance	Maintain compliance with data policies

Lineage is computed on-demand directly from Git code. This means you can experiment in branches and immediately see how changes affect overall lineage before committing.

Prophecy supports lineage for Python/PySpark and Scala projects.

Accessing lineage

There are two ways to access the lineage view:

Option 1: Hover over your pipeline in the Pipeline tab of the Metadata page.
Option 2: Open the Lineage page from the left navigation bar.

When you open Lineage from the left navigation bar, you are brought to the lineage search page. Once you enter a search query, you can see a list of matching entities and a list of columns. You can filter these lists by:

Entity type
Project
Author
Last modified time period

Once you click an entity or column, you’ll see the lineage visualization.

Lineage Summary

The Summary tab visually illustrates the flow of data through pipelines, models, and datasets depending on the entities in your project. Use the Browse Datasets panel to search for and select the column or entity that you want to inspect.

When you open a project in the project editor, you will see the project’s dataset and pipeline lineage. However, you cannot drill-down into lineage details unless you navigate to the lineage view described here.

Column-level lineage

Column-level lineage tracks the flow and transformation of individual data columns through the pipeline, demonstrating how they are processed at each stage. When you select a column in the lineage view, Prophecy provides the column’s upstream and downstream transformations.

Lineage Zoom In

The Zoom In tab allows you to dive deeper into specific components of your pipeline or dataset to examine detailed information.

Pipeline lineage

For pipelines, you can see code-level information about each component present in the pipeline. Select a component to view its:

Transformation code
Input columns
Output columns

Dataset lineage

For datasets, you can find information around all the upstream and downstream transformations (if any) for all the columns of the selected dataset.

Getting started

Development

Production

Execution

Extensibility

Accessing lineage