Skip to main content
This gem runs in .

​
Overview

This page describes how to use Databricks external Source and Target gems to read from or write to tables. Only use an external Source and Target gem when Databricks is not the configured SQL warehouse connection. Otherwise, use the Table gem to read from and write to Databricks.
If you’re working with file types like CSV or Parquet from Databricks file storage, see File types for guidance. This page focuses only on catalog tables.

​
Create a Databricks gem

To create a Databricks Source or Target gem in your pipeline:
1

Add a Source or Target gem to the pipeline

  1. Open your pipeline in the Studio.
  2. Click on Source/Target in the canvas.
  3. Select Source or Target from the dropdown.
  4. Click on the gem to open the configuration.
2

Select Databricks format

In the Type tab, select Databricks under Table. Do not select Databricks under File. Then, click Next.
3

Set location details

In the Location tab, set your connection details and table location. To learn more, jump to Source location and Target location.
4

Set table properties

In the Properties tab, set the table properties. To learn more, jump to Source properties and Target properties.
5

Preview data (Source only)

In the Preview tab, load a sample of the data and verify that it looks correct.

​
Source configuration

Use these settings to configure a Databricks Source gem for reading data.

​
Source location

​
Target configuration

Use these settings to configure a Databricks Target gem for writing data.

​
Target location

​
Target properties

​
Cross-workspace access

If your fabric uses Databricks as the SQL warehouse, you can’t select Databricks in an external Source or Target gem. Instead, you must use Table gems, which are limited to the Databricks warehouse defined in the SQL warehouse connection. To work with tables from a different Databricks workspace, use Delta Sharing. Delta Sharing lets you access data across workspaces without creating additional Databricks connections.
Prophecy implements this guardrail to avoid using external connections when the data can be made available in your warehouse. External connections introduce an extra data transfer step, which slows down pipeline execution and adds unnecessary complexity. For best performance, Prophecy always prefers reading and writing directly within the warehouse.