> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Spark Copilot

> See how Copilot can help you build your pipeline

export const WistiaVideo = ({mediaId, autoplay = true, muted = true, loop = true}) => {
  const containerRef = React.useRef(null);
  const playerRef = React.useRef(null);
  const wistiaPlayerRef = React.useRef(null);
  const initWistia = () => {
    if (!containerRef.current || playerRef.current) return;
    const container = containerRef.current;
    const playerId = `wistia_${mediaId}_${Math.random().toString(36).substr(2, 9)}`;
    container.innerHTML = '';
    const embedDiv = document.createElement('div');
    embedDiv.className = `wistia_embed wistia_async_${mediaId}`;
    embedDiv.id = playerId;
    container.appendChild(embedDiv);
    if (window._wq) {
      window._wq.push({
        id: playerId,
        onReady: video => {
          wistiaPlayerRef.current = video;
          if (loop) {
            video.bind('end', () => {
              video.play();
            });
          }
        },
        options: {
          autoPlay: autoplay,
          muted: muted,
          videoFoam: true
        }
      });
    }
    playerRef.current = playerId;
  };
  React.useEffect(() => {
    if (window._wq) {
      initWistia();
      return;
    }
    const script = document.createElement('script');
    script.src = 'https://fast.wistia.com/assets/external/E-v1.js';
    script.async = true;
    script.onload = () => {
      window._wq = window._wq || [];
      initWistia();
      window.dispatchEvent(new Event('wistia-loaded'));
    };
    document.head.appendChild(script);
    const checkWistia = () => {
      if (window._wq) {
        initWistia();
        window.removeEventListener('wistia-loaded', checkWistia);
      }
    };
    window.addEventListener('wistia-loaded', checkWistia);
    return () => {
      window.removeEventListener('wistia-loaded', checkWistia);
      if (playerRef.current && window.Wistia && window.Wistia.api) {
        const player = window.Wistia.api(playerRef.current);
        if (player) {
          player.remove();
        }
      }
    };
  }, [mediaId, autoplay, muted, loop]);
  return <div ref={containerRef} className="aspect-video w-full rounded-xl" style={{
    position: 'relative'
  }} />;
};

<Callout icon="/images/icon.png" color="#FFC107">
  Applicable to the [Enterprise Edition](/data-engineering/administration/platform/editions) only.
</Callout>

Prophecy's Spark Copilot provides suggestions from an AI model as you develop your data pipelines and models. You can view and incorporate suggestions directly within the Prophecy visual editor and code editor.

<Info>
  The Spark Copilot fetches context from environment metadata only. The Spark Copilot **does not**
  use knowledge graphs, which additionally store metadata relationships.
</Info>

## Text to pipelines

Get started on a new pipeline quickly by typing your prompt into the text box and Data Copilot will generate a new pipeline or modify an existing one.

### Start a new pipeline

You can use Data Copilot to start a new pipeline by typing a simple English text prompt.

<img src="https://mintcdn.com/prophecy-62973bd0/dljM4e5bzMFf5w3s/data-engineering/development/copilot/img/copilot_text_to_Pipeline.png?fit=max&auto=format&n=dljM4e5bzMFf5w3s&q=85&s=8b1d946da0887ee17d1517c5098a3246" alt="Start a pipeline" width="2880" height="1726" data-path="data-engineering/development/copilot/img/copilot_text_to_Pipeline.png" />

The following example uses Data Copilot to help start a pipeline:

1. Type a prompt with English text, such as `Which customers shipped the largest orders this year?`
2. If you'd like, review the suggested changes before you decide to keep or reject the suggested pipeline. Then interactively execute it to see the results.
3. View Data Copilot's suggested changes in the visual editor.

### Modify an existing pipeline

You can also call Data Copilot to modify an existing model. Type a new text prompt, and Data Copilot will suggest a new sequence of data transformations. You don't necessarily have to select where you want to make your modification for Data Copilot to make its suggestion.

<Frame>
  <WistiaVideo mediaId="a2x6kkzy8y" />
</Frame>

Added/updated gems are highlighted in yellow.

## Next-transformation suggestions

Data Copilot can suggest the next transformation in a series or the next expression within a gem.

### Suggest gems

Data Copilot can suggest the next transformation for Leaf Nodes in a graph.

<img src="https://mintcdn.com/prophecy-62973bd0/dljM4e5bzMFf5w3s/data-engineering/development/copilot/img/copilot_next_suggestion.png?fit=max&auto=format&n=dljM4e5bzMFf5w3s&q=85&s=db3926d73dc4f93f466b582dbf1e2f33" alt="Suggest gems" width="2880" height="1726" data-path="data-engineering/development/copilot/img/copilot_next_suggestion.png" />

See the following Join suggestion example:

1. Select and drop a dataset of interest on the canvas.
2. Data Copilot suggests datasets which are frequently used with the selected dataset.
3. Data Copilot then suggests a next transformation, in this case, a Join gem.

### Suggest Expressions

As we continue development within gems, Data Copilot can suggest expressions within gems.

<img src="https://mintcdn.com/prophecy-62973bd0/dljM4e5bzMFf5w3s/data-engineering/development/copilot/img/copilot_next_suggestion_expression.png?fit=max&auto=format&n=dljM4e5bzMFf5w3s&q=85&s=30e7a49a127b47af40cb665064cdfe1b" alt="Suggest expressions" width="2870" height="1722" data-path="data-engineering/development/copilot/img/copilot_next_suggestion_expression.png" />

Within our [advanced Expression Builder](/data-engineering/gems/expression-builder) you can:

1. Type an English text prompt.
2. Data Copilot generates a code expression for a particular column.
3. Review the code expression, and if you'd like, try again with a different prompt.
4. Run the pipeline up to and including this gem, and observe the resulting data sample.

## Generate with AI

Data Copilot can generate script gems, user-defined functions in Spark, or macro functions in SQL.

<Frame>
  <WistiaVideo mediaId="vhf63tjnun" />
</Frame>

## Map with AI

You don't have to worry about mapping the schema across your model. Data Copilot will map the target schema with the existing gems and datasets.

<Frame>
  <WistiaVideo mediaId="p5ab7sgfdw" />
</Frame>

## Code with AI

In addition to the visual editor above, you'll also see code completion suggestions in the code editor.

Data Copilot helps you build your model in the code interface by making predictions as you type your code. And when you go back to the visual interface, you'll see your code represented as a model.

<Frame>
  <WistiaVideo mediaId="7lff0jpqnm" />
</Frame>

## Fix with AI

If there are any errors in your gems, perhaps introduced upstream without your knowledge, Data Copilot will automatically suggest one-click fixes.

The Fix with AI option appears on the diagnostic screen where you see the error messages or directly with the expression itself.

<Frame>
  <WistiaVideo mediaId="17q9stsz6n" />
</Frame>

## Auto Documentation

Understanding data assets is much easier with Data Copilot's auto-documentation. Data Copilot delivers summary documentation suggestions for all aatasets, pipelines, models, and orchestrations.

### Explain gems

Here Data Copilot provides a high-level summary of a pipeline and more detailed description of each gem.

<Frame>
  <WistiaVideo mediaId="9r86rl0lbz" />
</Frame>

### Describe Datasets and Metadata

How did a dataset change? Data Copilot recommends a description of the change for every edit you make.
How was a column computed? Data Copilot suggests a plain English description that explains data sources and how every column is generated and what it represents.

<Frame>
  <WistiaVideo mediaId="rec6bcdwet" />
</Frame>

This is a big time saver! You can edit the documentation suggestions and commit them to your repository.

### Write Commit Messages and Release Notes

Data Copilot auto-documents anywhere you need it - from the granular data sources and columns to gem labels, all the way to project descriptions. Copilot even helps you write commit messages and release notes.

<Frame>
  <WistiaVideo mediaId="z3x5tfu1cf" />
</Frame>

## Data Tests and Quality Checks

Unit tests and data quality checks are crucial for pipeline and job productionalization, yet many teams leave little time to develop these tests or worse, don't build them at all. With Data Copilot, you'll have one or more suggested [unit tests](/data-engineering/ci-cd/tests) that can be seamlessly integrated into your CI/CD process.

<Frame>
  <WistiaVideo mediaId="i1bjyf2zae" />
</Frame>

Data Copilot also suggests data quality checks based on the data profile and expectations.