Prompting best practices

The Prophecy Agent enables you to build and modify pipelines, analyses, and documentation using natural language. Like any agent, it performs best when you provide the Agent with clear intent, structured context, and opportunities for review. This guide outlines practical techniques for working effectively with the Transform, Harmonization, and Documentation Agents.

​

Why prompting best practices matter

The Agent responds to prompts by implementing concrete project changes such as generating pipelines, adding gems, or updating analyses. Clear, well-structured prompts improve:

• Accuracy of generated transformations.
• Consistency of pipeline logic.
• Efficiency of iteration.
• Alignment with business requirements.

​

Be specific and explicit

Describe exactly what you want the Agent to do. Instead of:

“Clean the data”

Use:

Remove duplicate records based on customer_id, filter to orders from 2024, and replace null values in email with 'unknown@example.com'

When prompting the Transform Agent, specify:

• Source tables or gems.
• Join types and keys.
• Filter conditions.
• Column derivations.
• Output expectations.

Specific instructions reduce ambiguity and improve reliability.

​

Common prompting patterns

Prophecy’s agents supports common data workflow actions, such as:

• Finding and previewing datasets.
• Filtering and cleaning records.
• Joining and transforming tables.
• Parsing semi-structured data.
• Aggregating results.
• Generating visualizations.
• Saving outputs as tables.

Thinking in these discrete actions can help you structure clear, effective prompts.

​

Build pipelines incrementally

For multi-stage workflows, we recommend generating transformations step by step instead of all at once. Incremental development produces more reliable results and makes each stage easier to validate, refine, and debug. For example, a structured approach might look like:

1. Add source tables to the canvas.
2. Join datasets.
3. Apply filters.
4. Derive new columns.
5. Aggregate results.
6. Save the output as a table.

After each step:

• Use Inspect to review modified gems (these gems are highlighted in yellow).
• Validate both input and output data.
• Confirm the transformation matches your intent.

​

Use @ mentions to reference project assets

Use the @ symbol to reference specific tables or gems in your project. See @ mentions for more details. Examples:

Aggregate the @orders_cleaned table by month.

Add a filter after @last_join to include only records where region = 'West'.

Show a sample of @daily_sales.

Referencing project assets explicitly helps the Agent identify the correct objects and reduces unintended modifications.

​

Define the expected output

If you need structured output, state that explicitly. Examples:

Return only the SQL query.

Save the final result as a table.

Provide a summary of the pipeline in bullet points.

Generate documentation for the current pipeline.

Clear output expectations reduce follow-up clarification and rework. The Agent can also generate visualizations when prompted clearly, as in

Create a bar chart of monthly sales from @orders_cleaned.

​

Use descriptive column and gem names

Descriptive names improve the Agent’s ability to generate accurate logic. If source data contains unclear labels, consider renaming columns early in the pipeline to improve downstream suggestions. Use names such as:

• customer_email
• order_date
• total_amount

Avoid ambiguous names such as:

• col1
• field_a
• x

​

Inspect, validate, and restore

The Agent assists development, but you should always validate results. After each generated change:

• Review modified gems using Inspect.
• Examine input and output datasets.
• Confirm logic aligns with business requirements.
• Use Restore to revert to a previous state if needed.

All changes remain versioned and visible in project history.

​

Tailor prompts to the active agent

Prompting best practices apply across all Prophecy Agents, with slight variations.

• Transform Agent — Focus on explicit transformation logic and step-by-step pipeline construction.
• Harmonization Agent — Clearly define the target Common Data Model (CDM) and review generated source-to-target mappings and data quality tests.
• Documentation Agent — Specify the scope (pipeline, analysis, or project) and desired level of detail for generated documentation.

Understanding the active agent mode helps you structure prompts effectively.

​

Summary

Effective prompting combines clarity, structure, and inspection. By providing explicit instructions, building incrementally, referencing project assets, and validating results, you can maximize the reliability and efficiency of the Prophecy Agent.