Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt

Use this file to discover all available pages before exploring further.

The Knowledge Graph indexer retrieves metadata from your SQL warehouse and external data storage systems to build and maintain the metadata index that powers AI features in Prophecy. Prophecy automatically indexes your data environment when you create a fabric using your default credentials. After the initial run, you can configure indexing behavior to control when and how the indexer runs. This page covers:

​
How indexing works

The Knowledge Graph indexer processes each connection in your fabric separately. For each indexing run, the indexer:
  1. Uses the credentials stored in the connection to authenticate with the external system.
  2. Retrieves metadata only for databases, schemas, tables, and storage locations that the configured identity is authorized to access.
  3. Indexes metadata such as table names, schemas, column names, data types, and available object descriptions.
  4. Updates the Knowledge Graph with the latest metadata.
Prophecy indexes metadata using both structured search and embedding-based retrieval techniques to support AI-powered dataset discovery and contextual assistance.
The Knowledge Graph stores metadata only. It does not store actual warehouse data values, sampled records, or summaries of table contents.

​
Knowledge graph dependency checklist

Before configuring or troubleshooting the indexer, verify that the following requirements are satisfied.
#ItemWhy it mattersRequired?
1KG indexer has run successfullyAI features rely on indexed metadata for dataset discovery and schema-aware assistanceRequired
2Knowledge Graph services are running and reachableAI features require access to Knowledge Graph retrieval servicesRequired
3KNOWLEDGE_GRAPH_BASE_URL configured in sql-sandbox-config-mapEnables AI services to access Knowledge Graph retrieval APIsRequired
4CLAUDE_MODEL configured in sql-sandbox-config-mapSpecifies the Anthropic model used for Knowledge Graph-powered AI assistanceRequired
5AI_DATA_ACCESS_CLUSTER_ENABLED flag state is knownDetermines whether the AI agent can perform live data operations or operate in metadata-only modeRequired
6Sufficient worker resources are availableIndexing operations require adequate compute and memory resourcesRequired for indexing
7KG indexing completed with SUCCESS statusSchema and dataset lookups depend on up-to-date metadataRequired
8Incremental indexing schedule confirmedKeeps metadata current as warehouse schemas evolveRecommended
Items 3 and 4 require updating the sql-sandbox-config-map. See Knowledge graph configuration for setup instructions.

​
Configure automatic indexing

You can configure scheduled indexing to keep your Knowledge Graph up to date without manual intervention.
  1. In Prophecy, open Metadata > Fabrics.
  2. Select the fabric where you wish to enable indexing.
  3. Open the Connections tab.
  4. Open the connection to be indexed.
  5. In the connection dialog, scroll to the Knowledge Graph Indexer tile and enable Knowledge Graph Periodic Indexing.
  6. Configure the schedule to run hourly, daily, or weekly.
The schedule must have a defined frequency and timezone. By default, Prophecy uses the timezone from where you access the application.

​
Scheduling parameters

Schedule typeParameterDescriptionDefault
HourlyRepeat every … fromThe interval in hours between indexing runs, starting at a specific time.
Example: Repeat every 2 hours from 12:00 AM.		Every 1 hour
starting at 2:00 AM
DailyRepeat atThe time of day when indexing runs.

Example: Repeat at 9:00 AM.		2:00 AM
WeeklyRepeat onThe day(s) of the week when indexing runs.

Example: Repeat on Monday, Wednesday, Friday.		Sunday
WeeklyRepeat atThe time of day when indexing runs.

Example: Repeat at 9:00 AM.		2:00 AM

​
Manually trigger indexing

You may need to manually trigger indexing if newly created tables or schemas are not yet available to AI features. To manually trigger indexing:
  1. In Prophecy, open Metadata > Fabrics.
  2. Select the fabric you want to index.
  3. Open the Connections tab.
  4. Open the connection to be indexed.
  5. Scroll to the Knowledge Graph Indexing Status tile in the connection dialog.
  6. Click Start to begin indexing and monitor progress.
You can also trigger indexing from the Environment tab in your project:
  1. Open a project in the project editor.
  2. Attach the fabric you want to index.
  3. Open the Environment tab in the left sidebar.
  4. Locate the Missing Tables? callout below your connections.
  5. Click Refresh.
Prophecy may prompt you to manually trigger indexing if the AI agent cannot locate a table or schema during a conversation.

​
Add separate authentication for the indexer

In some environments, administrators may want more granular control over which metadata is indexed into the Knowledge Graph. For Databricks connections, Prophecy supports configuring separate authentication credentials specifically for the Knowledge Graph indexer. There are two types of credentials stored in a connection:
  • Pipeline Development and Scheduled Execution credentials control how pipelines authenticate when they run.
  • Knowledge Graph Indexer credentials control how the indexer authenticates when retrieving metadata on a schedule.
If separate credentials are not configured, the indexer uses the pipeline development credentials.
The Knowledge Graph indexer always uses the same identity as the pipeline development identity if the pipeline development authentication strategy is Personal Access Token (rather than OAuth). This section does not apply when using PAT authentication.

​
Prerequisites

Before configuring dedicated credentials for the Knowledge Graph indexer, you must:
  • Upgrade to Prophecy 4.2.2 or later.
  • Configure your SQL warehouse connection with a Databricks connection. Other SQL warehouses are not currently supported for separate indexer authentication.
  • Be a Prophecy administrator.
  • Be a Databricks administrator with permission to assign appropriate access to the indexing identity.
The configured identity must have sufficient permissions to retrieve metadata for the warehouse objects you want indexed into the Knowledge Graph.
Knowledge Graph indexing permissions should generally match or exceed the permissions used for pipeline execution. This helps ensure that metadata for the datasets used in pipelines is also available to AI features. Prophecy does not enforce this automatically.

​
Procedure

To configure separate authentication for the Knowledge Graph indexer:
  1. In Prophecy, navigate to Metadata > Fabrics.
  2. Select the target fabric.
  3. Open the Connections tab.
  4. Edit the SQL Warehouse Connection.
  5. Scroll to the Knowledge Graph Indexer section.
  6. Configure authentication based on your pipeline development authentication method:
    • If you use User OAuth for **pipeline development, choose either OAuth (User) or OAuth (Service Principal) for the Knowledge Graph indexer.
    • If you use Service Principal OAuth for **pipeline development, you can only use Service Principal OAuth for the Knowledge Graph indexer.
Recommended for production and scheduled indexing because credentials do not expire.
  • Configuration: Reuse pipeline development credentials or provide a separate Service Principal Client ID and Client Secret.
  • Indexed metadata: Metadata for all warehouse objects the service principal is authorized to access.
If pipeline development uses User OAuth, Prophecy continues to enforce user-level permissions even when the Knowledge Graph indexer uses service principal credentials.

​
User OAuth

Recommended primarily for development environments.
  • Configuration: Uses the same app registration as pipeline development.
  • Indexed metadata: Metadata for warehouse objects the authenticated user is authorized to access.
  • Limitations: Scheduled indexing can fail when user credentials expire or require reauthentication.