Skip to main content
This gem runs in .

​
Overview

In Prophecy, datasets stored in the SQL Warehouse Connection defined in your fabric are accessed using Table gems. Unlike Source and Target gems, Table gems run directly within the data warehouse, eliminating extra orchestration steps and improving performance. This page explains how to read from a Databricks table, view, or seed using the Table gem. To write to a Databricks table or view instead, see Write to Databricks tables.

​
Table types

The following table types are supported for Databricks connections.
For more information, visit the Databricks documentation on Tables and Views.

​
Configure table

Once you create a Table gem, you can reuse the table throughout your project. All created tables appear in the Project tab in the left sidebar. To read from a table in your pipeline:
1

Add a table gem to the pipeline

  1. Open your pipeline in the Studio.
  2. Click on Source/Target in the canvas.
  3. Select Table from the dropdown.
  4. Click on the gem to open the configuration.
2

Select a table

Choose the table, view, or seed you want to read from the list.
Seeds configure differently from Tables and Views — see Configure seeds below.
3

Configure location

The Location tab defines where a table lives and how it is identified within your project.A parameter set selector appears in the upper-right corner of the Location tab:
  • If the table gem is used inside a pipeline, the selector shows that pipeline’s active parameter set automatically.
  • If you’re working on the table outside of a pipeline (for example, from the Project browser), the selector shows Select Pipeline and Parameter set until you choose one. You only need to do this if one or more Location fields are set to Advanced mode (see below) and you need their values resolved outside pipeline context.
At the bottom of the Location tab, Prophecy shows a live preview of the resolved database.schema.table location: the hardcoded values if all fields are in Simple mode, or the resolved parameter values if any field is in Advanced mode. If a value can’t be resolved yet — for example, because no parameter set is selected — Prophecy displays the raw value instead.

​
Make a location field dynamic

In the default Simple mode, each Location field (database, schema, table) takes a fixed value that you type directly.Switch a field to Advanced mode to bind it to a project or pipeline parameter instead of a fixed value.
Only parameters of type sql_expression can be used in Advanced mode. Using a parameter of a different type will cause the table location to fail to resolve.
Switching a field from Advanced back to Simple mode clears its current value.
Once a field is in Advanced mode, its value depends on which parameter set is active (see the parameter set selector above). This makes it possible to define a table once and reuse it across multiple pipelines, each supplying different values for the parameterized fields via their own parameter sets. To reuse a table you’ve already created, select it from Table > [alias] in the Project browser.
Prophecy does not validate that tables resolved from different parameter sets share the same schema. If your parameter sets point to tables with different schemas, downstream steps in your pipeline may fail or behave unexpectedly.
For a View, enter the database, schema, and view name here instead of a table name.
4

Configure properties

Click Infer Schema to infer the table’s schema. We recommend keeping values as generated, because these match column names and types in your SQL warehouse.When you click Infer Schema, Prophecy also generates a description for the table. You can edit the description by clicking the field.
For a View, this defines or infers the schema from the view’s underlying query.
5

Preview data

Click Preview to view a sample of the table’s data.
For a View, this loads data based on the view’s underlying query rather than stored data.
6

(Optional) Create and run data tests for the table

Here, you can create and run table tests for the table. Table tests are reusable, parameterized SQL queries that validate your data quality. See Table tests vs. project tests if you’re deciding which approach fits your validation needs.When you are satisfied with the table’s configuration, click Save.

​
Configure seeds

Seeds are lightweight CSV datasets defined in your project. Seeds are source-only.
Seeds are implemented as dbt seeds under the hood. The CSV data you define is stored in your Prophecy project files and materialized as a table in your data warehouse. This table is created in the default target schema specified in your Databricks connection.Tables in pipelines do not support dbt properties, which are only applicable to model sources and targets. The properties referenced above are the seed’s own properties (the dbt seed config), not pipeline table properties.

​
Cross-workspace access

If your fabric uses Databricks as the SQL warehouse, you can’t select Databricks in an external Source or Target gem. Instead, you must use Table gems, which are limited to the Databricks warehouse defined in the SQL warehouse connection. To work with tables from a different Databricks workspace, use Delta Sharing. Delta Sharing lets you access data across workspaces without creating additional Databricks connections.
Prophecy implements this guardrail to avoid using external connections when the data can be made available in your warehouse. External connections introduce an extra data transfer step, which slows down pipeline execution and adds unnecessary complexity. For best performance, Prophecy always prefers reading and writing directly within the warehouse.

​
Reusing and sharing tables

After you create a table in Prophecy, you can reuse its configuration across your entire project. All created tables appear in the Project tab in the left sidebar. To make tables available to other teams, you can share your project as a package in the Package Hub. Other users will be able to use the shared table configuration, provided they have the necessary permissions in Databricks to access the underlying data.