Runtime logs

Documentation Index

Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt

Use this file to discover all available pages before exploring further.

Runtime logs provide visibility into pipeline execution in Prophecy. They help you understand how a pipeline ran, which steps succeeded or failed, how long operations took, and where errors occurred. Prophecy supports both:

• Live runtime logs for active or recent runs in the project editor.
• Historical runtime logs for completed runs, which are snapshots of the pipeline at execution time accessed through the Observability interface.

​

Open runtime logs

You can open runtime logs from the information footer at the bottom of the project editor. Runtime logs update in real time while your pipeline runs. Each log entry provides structured details that help you trace execution and diagnose issues. Logs include:

• Pipeline steps: Execution details for every processing step, mapped directly to the gems in your visual pipeline.
• Execution sequence: The exact order in which steps were carried out.
• Timestamp: The start time of each step.
• Duration: How long each step took.
• Status information: Success, warning, and failure details for each operation.

When an error occurs, the message usually comes directly from the execution engine. For example, if your pipeline is running on a Databricks SQL warehouse, errors originate from Databricks. If you’re using Prophecy In Memory, errors come from DuckDB (the underlying warehouse). For deeper troubleshooting, consult the corresponding warehouse documentation to interpret execution-specific error details.

​

Use runtime logs

In the Runtime Logs tab, you can use several tools to help navigate and analyze execution details.

1. Filter logs by keywords: Find specific information by entering search terms
2. Sort logs: Organize logs chronologically
3. Filter logs by log level: Focus on info, warning, or error messages
4. Download logs: Save logs to your machine as a text file
5. Expand log screen: Display a larger view of your logs
6. Clear all logs: Remove log information for previous runs

Closing Runtime Logs does not immediately clear log information for recent runs. Recent runtime logs may be restored automatically while they remain available in persistent storage.

​

Access historical runtime logs

You can reopen runtime logs for completed pipeline runs through the Observability interface. Historical runtime logs help you investigate failures, compare past executions, validate pipeline behavior, and understand how a pipeline operated at a specific point in time. To access historical runtime logs:

1. Click Observability in the left sidebar and open the Run History tab.
2. Locate the pipeline run you want to inspect and click See logs in the far-right column.

Prophecy opens the selected pipeline run in Historical Mode, showing:

• Runtime logs for that execution.
• The pipeline snapshot associated with the run.
• Historical component execution states and statuses.

​

Historical Mode

When you open historical runtime logs, the pipeline opens in Historical Mode. Historical Mode reconstructs the pipeline exactly as it existed during the selected execution. This includes the pipeline graph, runtime logs, and component execution statuses. In Historical Mode:

• The pipeline canvas becomes read-only.
• Runtime logs are restored for the selected run.
• Components display their historical execution states.
• Execution progress and failure states are preserved.
• You can switch between historical runs to compare executions.
• You can rerun historical pipeline snapshots.

A banner at the top of the canvas indicates that you are viewing a historical pipeline snapshot. From this banner, you can:

• Return to overall run history.
• Exit Historical Mode and return to the latest pipeline version.

​

Pipeline snapshots and retention

Historical runtime logs rely on persisted pipeline snapshots and execution metadata. When available, Prophecy stores:

• Runtime log output.
• Pipeline execution state.
• Component execution progress and statuses.
• Pipeline snapshots associated with completed runs.

These artifacts are retained for a limited period of time and may eventually expire depending on system configuration and storage limits. Not every historical run may have an available pipeline snapshot. If a snapshot cannot be restored, Prophecy displays a message indicating that the historical pipeline snapshot could not be found and provides an option to return to the latest pipeline version. When loading a historical run, Prophecy displays a loading state while runtime logs and pipeline snapshots are restored.

​

SQL warehouse logs

You can find additional logs specific to pipeline queries in your SQL warehouse. If you are connected to a Databricks SQL warehouse, you can find these logs directly in Databricks. This extra visibility into query execution in Databricks helps you troubleshoot pipeline behavior and improve performance. If you have access to your Databricks workspace:

1. Navigate to SQL > Query History in the Databricks sidebar.
2. Locate the SQL queries run during your pipeline’s execution window.
3. Click on a query to see:
   • Full SQL query
   • Status (Success or Error)
   • Start and end times
   • Error messages
   • Execution plan or profile

​

What’s next

Now that you know about runtime and publish logs, you might want to learn about audit logs and system logs, which provide visibility into platform-level and administrative activity.

• Audit logs track events across entities like projects, pipelines, jobs, and fabrics. These logs help administrators understand who did what and when. For example, these logs can capture who creates a pipeline, releases a project, modifies a dataset, etc.
• System logs capture backend infrastructure details to support platform monitoring and troubleshooting. These include Kubernetes cluster configuration (such as resource quotas and node settings), cluster custom resources, config maps and files, and resource consumption metrics.

Unlike runtime logs, which focus on the execution of a specific pipeline run, audit and system logs provide a broader view of how users and the platform are operating over time.