Skip to main content
Applicable to the Enterprise Edition only.
Pipelines are groups of data transformations that you can build from a visual or code interface. When using the visual interface, each component of a pipeline is automatically compiled into code that you can reuse and customize. Under the hood, pipelines are based on Spark-native code. Pipelines are ideal for Spark environments like Databricks or EMR, particularly for tasks such as complex data ingestion (e.g., loading data from Salesforce or JDBC), handling advanced data transformations (e.g., working with complex data types), and supporting machine learning workflows.

​
Creation

If you want to create a new pipeline, you can do so from the Create Entity page in the left sidebar. You can also create pipelines directly within the project editor. The following table describes the parameters for pipeline creation.

​
Project editor

When building your pipelines, it helps to be familiar with the project editor interface. The following table describes different areas of the project editor. See these components marked in the image below. Project Editor

​
Canvas

Let’s take a closer look at the pipeline canvas. The canvas includes:
  • Canvas: space to add and connect gems.
  • Gem drawer: toolbox that contains all available gems.
  • Run button: click to execute the pipeline interactively.
  • Copilot: AI assistant to help build your pipeline.
Pipeline canvas

​
Metadata

To view a list of pipelines in Prophecy, navigate to the Metadata page from the left sidebar. For more granular metadata, click into a pipeline. Pipeline metadata can also be accessed from the header of the project editor. Pipeline metadata The table below describes the different tabs inside an individual pipeline’s metadata.