Knowledge graph indexer

The knowledge graph indexer scans your SQL warehouse and external data storage to build and maintain the metadata index that powers AI features. Prophecy automatically indexes your data environment when you create a fabric using your default credentials. After the initial run, you can configure indexing behavior to control when and how the indexer runs. This page covers:

• Scheduling automatic indexing
• Triggering manual runs
• Configuring separate authentication credentials for the indexer

​

How indexing works

The knowledge graph indexer crawls each connection in your fabric separately. For each run, the indexer:

1. Uses the credentials stored in the connection to authenticate as an identity in the external system.
2. Scans databases and file storage systems that the identity has access to, limited by the permissions granted to that identity.
3. Indexes table names, schemas, column names, data types, and other metadata.
4. Updates the knowledge graph with this information.

​

Configure automatic indexing

Configure scheduled crawling to keep your index up-to-date without manual intervention.

1. In Prophecy, navigate to Metadata > Fabrics.
2. Select the fabric where you will enable indexing.
3. Open the Connections tab.
4. Click to open the connection that you wish to schedule indexing for.
5. In the connection dialog, scroll to the Knowledge Graph Indexer tile and toggle on Enable Knowledge Graph Periodic Indexing.
6. Configure the schedule to run hourly, daily, or weekly.

The schedule must have a defined frequency and timezone. The default timezone is the timezone from where you access Prophecy.

​

Scheduling parameters

Hourly Daily Weekly

​

Manually trigger indexing

You may need to manually trigger indexing if you know that certain tables are missing from the knowledge graph. To do so:

1. In Prophecy, navigate to Metadata > Fabrics.
2. Select the fabric where you will enable indexing.
3. Open the Connections tab.
4. Click to open the connection that you wish to reindex.
5. Scroll to the Knowledge Graph Indexing Status tile in the connection dialog.
6. Click Start to reindex the tables and track its progress. You’ll be able to view the progress of processed schemas and directories.

If more convenient, you can also start this process from the Environment tab in your project:

1. Open a project in the project editor.
2. Attach to the fabric that you wish to reindex.
3. In the left sidebar, open the Environment tab.
4. Below your connections, you’ll see a Missing Tables? callout.
5. Click Refresh.

​

Add separate authentication for the indexer

In certain scenarios, you may want more granular control over which tables get indexed. For Databricks connections only, Prophecy lets you do so with dedicated authentication credentials for the knowledge graph indexer. There are two types of credentials stored in a connection:

• Pipeline Development and Scheduled Execution credentials control how pipelines authenticate when they run.
• Knowledge Graph Indexer credentials control how the crawler authenticates when it indexes your warehouse on an automated schedule.

If you don’t add separate authentication for the indexer, it will use the pipeline development credentials when running.

​

Prerequisites

Before configuring dedicated credentials for the knowledge graph indexer, you must:

• Upgrade to Prophecy 4.2.2 or later.
• Configure your SQL warehouse connection with a Databricks connection. Other SQL warehouses are not supported.
• Be a Prophecy administrator. Though there are no role-based restrictions for configuring the knowledge graph indexer, you need to understand how authentication works in Prophecy.
• Be a Databricks administrator. This lets you assign appropriate permissions to the identity that will be used to run the indexer. The identity must have MANAGE access on the assets that you wish to index in the knowledge graph.

​

Procedure

To configure the knowledge graph indexer for a fabric:

1. In Prophecy, navigate to Metadata > Fabrics.
2. Select the fabric where you will enable indexing.
3. Open the Connections tab.
4. Click the pencil icon to edit the SQL Warehouse Connection.
5. In the dialog, scroll to the Knowledge Graph Indexer tile.
6. Configure authentication based on your pipeline development authentication method: If you use User OAuth for pipeline development:
   • Choose either OAuth (User) or OAuth (Service Principal) for the knowledge graph indexer.
   If you use Service Principal OAuth for pipeline development:
   • You can only use Service Principal OAuth for the knowledge graph indexer.

​

Service Principal OAuth (recommended)

Recommended for production and scheduled indexing. Credentials don’t expire.

• Configuration: Reuse pipeline development credentials or provide a different Service Principal Client ID and Client Secret.
• What gets indexed: All tables that the service principal can access.

​

User OAuth

User OAuth should only be used for development.

• Configuration: Uses the same app registration as pipeline development.
• What gets indexed: All tables that the individual user can access.
• Limitations: Requires frequent user logins. Scheduled crawling can fail when user credentials expire.