> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks external table gem

> Read and write catalog tables in Databricks

export const execution_engine_0 = "Prophecy Automate"

<Info>This gem runs in {execution_engine_0}.</Info>

## Overview

This page describes how to use Databricks external Source and Target gems to read from or write to tables. Only use an external Source and Target gem when Databricks is not the configured [SQL warehouse connection](/data-analysis/environment/fabrics/prophecy-fabrics). Otherwise, use the [Table gem](/data-analysis/gems/source-target/table/bigquery).

<Info>
  If you're working with file types like CSV or Parquet from Databricks file storage, see [File
  types](/data-analysis/gems/source-target/file) for guidance. This page focuses only on catalog
  tables.
</Info>

## Create a Databricks gem

To create a Databricks Source or Target gem in your pipeline:

<Steps>
  <Step title="Add a Source or Target gem to the pipeline">
    1. Open your pipeline in the [Studio](/data-analysis/development/studio/studio).
    2. Click on **Source/Target** in the canvas.
    3. Select **Source** or **Target** from the dropdown.
    4. Click on the gem to open the configuration.
  </Step>

  <Step title="Select Databricks format">
    In the **Type** tab, select **Databricks** under **Table**. Do not select Databricks under
    **File**. Then, click **Next**.
  </Step>

  <Step title="Set location details">
    In the **Location** tab, set your connection details and table location. To learn more, jump to [Source location](#source-location) and [Target location](#target-location).
  </Step>

  <Step title="Set table properties">
    In the **Properties** tab, set the table properties. To learn more, jump to [Source
    properties](#source-properties) and [Target properties](#target-properties).
  </Step>

  <Step title="Preview data (Source only)">
    In the **Preview** tab, load a sample of the data and verify that it looks correct.
  </Step>
</Steps>

## Source configuration

Use these settings to configure a Databricks Source gem for reading data.

### Source location

| Parameter                   | Description                                                                                                                            |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Format type                 | Table format for the source. For Databricks tables, set to `databricks`.                                                               |
| Select or create connection | Select or create a new [Databricks connection](/data-analysis/environment/connections/databricks) in the Prophecy fabric you will use. |
| Database                    | Database including the schema where the table is located.                                                                              |
| Schema                      | Schema containing the table you want to read from.                                                                                     |
| Name                        | Exact name of the Databricks table to read data from.                                                                                  |

## Target configuration

Use these settings to configure a Databricks Target gem for writing data.

### Target location

| Parameter                   | Description                                                                                                                            |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Format type                 | Table format for the target. For Databricks tables, set to `databricks`.                                                               |
| Select or create connection | Select or create a new [Databricks connection](/data-analysis/environment/connections/databricks) in the Prophecy fabric you will use. |
| Database                    | Database including the schema where the table is/will be located.                                                                      |
| Schema                      | Schema where the target table will be created or updated.                                                                              |
| Name                        | Name of the Databricks table to write data to. If the table doesn't exist, it will be created automatically.                           |

### Target properties

| Property    | Description                                                                                                     | Default |
| ----------- | --------------------------------------------------------------------------------------------------------------- | ------- |
| Description | Description of the table.                                                                                       | None    |
| Write Mode  | Whether to overwrite the table completely, append new data to the table, or throw an error if the table exists. | None    |

## Cross-workspace access

If your fabric uses Databricks as the SQL warehouse, you can't select Databricks in an external Source or Target gem. Instead, you must use Table gems, which are limited to the Databricks warehouse defined in the SQL warehouse connection.

To work with tables from a different Databricks workspace, use [Delta Sharing](https://docs.databricks.com/aws/en/delta-sharing/). Delta Sharing lets you access data across workspaces without creating additional Databricks connections.

<Info>
  Prophecy implements this guardrail to avoid using external connections when the data can be made
  available in your warehouse. External connections introduce an extra data transfer step, which
  slows down pipeline execution and adds unnecessary complexity. For best performance, Prophecy
  always prefers reading and writing directly within the warehouse.
</Info>