Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt

Use this file to discover all available pages before exploring further.

Analyzing and manipulating data with Spark functions often requires building complex expressions that combine multiple functions. Prophecy makes this easier with the Expression Builder, which streamlines the creation of complex expressions, saving time and effort. It also helps you better understand the relationships between functions and their arguments.

​
Spark Expression Builder

To navigate to the Spark Expression Builder:
  1. Navigate to the column you want to edit, and open the expanded editor.
  2. Click on the Expression Builder button.
Now, you can search for, and insert functions, columns or configurations in your canvas. To search for a function in Spark Expression Builder:
  1. Click the fx button.
  2. In the search bar at the top of the screen, type the function name.
A list of matching functions appears. When you click on a function, you see information about its syntax, arguments, and an example on how to use it.

​
Insert

You can insert a function, column, or configuration into your expression. To insert a function:
  1. Navigate to, or search for the function you want to use.
  2. Click on the Insert Function button.
To specify an argument for the function, click on the function and fill in the required fields. To insert a column:
  1. Click the Columns tab.
  2. Click on the column you want to insert, and click Insert Column.
To insert a configuration:
  1. Click the Configurations tab.
  2. Click on the column you want to insert, and click Insert Configuration.

​
Run and Verify the output

You can attach to a cluster and run your pipeline for the current gem on the same screen. To run your code:
  1. Click the Run button.
  2. Click the Data button to see your results.
Verify the results to ensure that your data analysis tasks are accurate and reliable.
This data is same as what you see in interims view.