> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# RestAPI

> Call REST APIs from your pipeline

export const execution_engine_0 = "Prophecy Automate"

<Info>This gem runs in {execution_engine_0}.</Info>

## Overview

Use the RestAPI gem to make HTTP requests to external REST APIs from your pipeline. You can retrieve data, send data to downstream systems, or trigger external services as part of your workflow.

## Use cases

Common uses for the RestAPI gem include:

* Enriching pipeline data with information from an external service.
* Sending pipeline results to another application.
* Triggering external workflows or notifications.
* Retrieving reference or lookup data during pipeline execution.

## Input and output

The RestAPI gem uses the following input and output ports:

| Port | Description |
| - | - |
| **in0** | (Optional) Input table. If connected, Prophecy makes one API request for each input row. |
| **out** | Output table containing the API response, along with the input columns when applicable. |

By default, the RestAPI gem does not include an input port. You can add one by clicking the **+** button next to **Ports**, or by connecting another gem directly to the RestAPI gem.

<Info>
  You can change the number of input and output ports for most gems. To learn more, see [Gem ports](/data-analysis/gems/gems#gem-ports).
</Info>

## Configure the RestAPI gem

When you first open the RestAPI gem, Prophecy guides you through a multi-step configuration flow.

<Steps>
  <Step title="Request">
    The **Request** page defines the HTTP request that will be sent.

    Configure the following settings:

    | Setting | Description |
    | - | - |
    | HTTP Method | Select the HTTP method to use. Supported methods include `GET`, `POST`, `PUT`, `DELETE`, and `PATCH`. |
    | Target URL | Enter the URL of the REST API endpoint to call. |
    | Request Body | For methods that send a request body, choose the body format and provide the request payload. Supported formats include JSON, XML, and plain text. |

    You can reference input columns anywhere the gem accepts text by enclosing the column name in double curly braces:

    ```text theme={null}
    https://api.company.com/customers/{{CustomerID}}
    ```

    You can also reference columns inside a request body:

    ```json theme={null}
    {
      "customerId": "{{CustomerID}}",
      "region": "{{Region}}"
    }
    ```

    Once you have configured the HTTP request, click **Continue** to proceed.
  </Step>

  <Step title="Authentication and headers">
    The **Auth & Headers** page lets you configure authentication and additional request metadata.

    Choose one of the supported authentication methods:

    * **None** — Send the request without automatic authentication. You can add an `Authorization` header manually in the **Request headers** section.
    * **Basic Auth** — Authenticate using a username and password stored in a [Username & Password secret](/data-analysis/environment/secrets/secrets).
    * **Bearer Token** — Authenticate using a bearer token. Select an existing secret to supply the token value, or enter a new token as a secret.
    * **API Key** — Send an API key in either a request header or query parameter.

    You can also configure:

    * **Request parameters** to append query parameters to the URL. For example, `limit:250` appends `?limit=250` to the request URL.
    * **Request headers** to send additional HTTP headers. Header values support hard-coded strings, pipeline parameters, and secrets.

    <Tip>
      Store credentials as [Prophecy secrets](/data-analysis/environment/secrets/secrets) rather than hard-coding them in headers or parameters. Secrets are never visible after they are saved.
    </Tip>

    Click **Continue** to proceed.
  </Step>

  <Step title="Response">
    The **Response** page controls how Prophecy parses the API response.

    Choose one of the following parsing options.

    ### None

    Return the API response as a single text column. You can specify a name for the output column.

    ### JSON

    Parse the response as JSON.

    You can optionally:

    1. **Flatten the parsed response into columns.** If you leave this option disabled, Prophecy returns the parsed response as a single struct column.
    2. **Wait for async results.** Enable this option for APIs that initially return an "in progress" response. Enter a JSONPath condition that identifies an in-progress response. Prophecy continues polling while the condition evaluates to true and stops when it evaluates to false or no longer matches.

    ### XML

    Parse the response as XML.

    You can optionally:

    1. **Flatten the parsed response into columns.** If you leave this option disabled, Prophecy returns the parsed response as a single struct column.
    2. **Wait for async results.** Enable this option for APIs that initially return an "in progress" response. Enter an XPath condition that identifies an in-progress response. Prophecy continues polling while the condition evaluates to true and stops when it evaluates to false or no longer matches.

    ### Parsed response output

    When you choose JSON or XML, the output depends on whether you flatten the parsed response.

    <Note>
      Parsing the response does not automatically define the output schema. After configuring the gem, open the output port and click **Infer from Cluster** to retrieve the response schema.
    </Note>

    | Flatten parsed response into columns | Output |
    | - | - |
    | Disabled | The parsed response is returned as a single struct column. |
    | Enabled | Fields from the parsed response are returned as individual output columns. |

    Click **Continue** to proceed.
  </Step>

  <Step title="Resilience">
    The **Resilience** page lets you configure retry behavior for failed requests.

    Enable **Retry if the API fails** to retry unsuccessful requests automatically.

    When retries are enabled, configure:

    * **Retry attempts** — Maximum number of retry attempts.
    * **Retry delay** — Number of seconds to wait between attempts.

    Prophecy retries requests that fail because of:

    * Network errors or timeouts.
    * HTTP `408 Request Timeout` responses.
    * HTTP `429 Too Many Requests` responses.
    * HTTP `5xx` server errors.

    Other HTTP `4xx` responses are not retried.

    Click **Save Configuration** to finish configuring the gem.
  </Step>
</Steps>

## Example

Suppose you have a dataset of customer records with the following columns:

| Column name | Description |
| - | - |
| `customer_id` | Unique identifier for each customer. |
| `region` | Geographic region associated with the customer. |

You can use the RestAPI gem to enrich each row by calling an external CRM API that returns account details for a given customer.

### Example configuration

| Setting | Value |
| - | - |
| HTTP Method | `GET` |
| Target URL | `https://api.crm.com/accounts/{{customer_id}}` |
| Authentication | Bearer Token |
| Response parsing | JSON, flattened into columns |

### Result

This configuration:

* Sends one `GET` request per input row, using the `customer_id` column to construct the URL dynamically.
* Authenticates each request using a bearer token stored as a Prophecy secret.
* Parses the JSON response and flattens the returned fields into individual output columns alongside the original input columns.

## Common issues

### Authentication errors

If requests return `401 Unauthorized` or `403 Forbidden`:

* Verify that the token or credentials are correct and have not expired.
* Check that the secret is attached to the correct fabric.
* If using **None** authentication, confirm that the `Authorization` header is formatted correctly. Bearer tokens should use the format `Bearer <token>`.

### Unexpected response shape

If the output schema looks wrong after inferring:

* Check the raw response by temporarily disabling JSON or XML parsing and inspecting the `api_data` column.
* Some APIs wrap results in a top-level key such as `data` or `results`. You may need a subsequent gem to extract the nested field.

### Pagination

Most APIs return a limited number of results per request, controlled by parameters such as `limit` and `offset` or `page`. The RestAPI gem sends one request per input row and does not handle pagination automatically. To retrieve all pages, build a loop using pipeline parameters or pre-compute the required page offsets in an upstream gem.

### API not reachable from Prophecy

If the API call fails with a network or connection error, the API may not be accessible from Prophecy's execution environment. In this case, consider calling the API externally and writing the results to a table your fabric can read (such as a Databricks notebook writing to a Delta table, a Snowflake task, or a BigQuery scheduled query) then reading from that table as a source in your Prophecy pipeline.

## Similar tools and concepts

| Tool or platform | Similar concept |
| - | - |
| SQL | `JOIN` against an external data source |
| Alteryx | Download tool |
| Pandas | `requests` library with `DataFrame.apply()` |
| PySpark | UDF wrapping `requests` library |
| Postman | Manual API testing and request configuration |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.