This gem runs in .
Overview
Use a Source or Target gem to read from or write to Databricks Volumes in Prophecy pipelines. This page covers supported file formats, how to create the gem, and how to configure connection details and paths for both Source and Target gems.Supported file formats
Create a Databricks Volumes gem
To create a Databricks Volumes Source or Target gem in your pipeline:1
Add a Source or Target gem to the pipeline
- Open your pipeline in the Studio.
- Click on Source/Target in the canvas.
- Select Source or Target from the dropdown.
- Click on the gem to open the configuration.
2
Select Databricks format
In the Type tab, select Databricks under File. Do not select Databricks under
Table. Then, click Next.
3
Set location details
In the Location tab, set your file format and connection details. To learn more, jump to Location.
4
Set file properties
In the Properties tab, set the file properties. These vary based on the file type that you are working with.
See the list of properties per file type, such as CSV.
5
Preview data (Source only)
In the Preview tab, load a sample of the data and verify that it looks correct.
Location
| Parameter | Description |
|---|---|
| Format type | Type of file to read or write, such as csv or json. |
| Select or create connection | Select an existing Databricks Volumes connection or create a new one. |
| Filepath | Path to the file in Databricks Volumes. Example: /Volumes/catalog/schema/volume/file.csv |