This gem runs in .
Overview
Use the RestAPI gem to make HTTP requests to external REST APIs from your pipeline. You can retrieve data, send data to downstream systems, or trigger external services as part of your workflow.Use cases
Common uses for the RestAPI gem include:- Enriching pipeline data with information from an external service.
- Sending pipeline results to another application.
- Triggering external workflows or notifications.
- Retrieving reference or lookup data during pipeline execution.
Input and output
The RestAPI gem uses the following input and output ports:
By default, the RestAPI gem does not include an input port. You can add one by clicking the + button next to Ports, or by connecting another gem directly to the RestAPI gem.
You can change the number of input and output ports for most gems. To learn more, see Gem ports.
Configure the RestAPI gem
When you first open the RestAPI gem, Prophecy guides you through a multi-step configuration flow.1
Request
The Request page defines the HTTP request that will be sent.Configure the following settings:You can also reference columns inside a request body:Once you have configured the HTTP request, click Continue to proceed.
You can reference input columns anywhere the gem accepts text by enclosing the column name in double curly braces:
2
Authentication and headers
The Auth & Headers page lets you configure authentication and additional request metadata.Choose one of the supported authentication methods:
- None — Send the request without automatic authentication. You can add an
Authorizationheader manually in the Request headers section. - Basic Auth — Authenticate using a username and password stored in a Username & Password secret.
- Bearer Token — Authenticate using a bearer token. Select an existing secret to supply the token value, or enter a new token as a secret.
- API Key — Send an API key in either a request header or query parameter.
- Request parameters to append query parameters to the URL. For example,
limit:250appends?limit=250to the request URL. - Request headers to send additional HTTP headers. Header values support hard-coded strings, pipeline parameters, and secrets.
3
Response
The Response page controls how Prophecy parses the API response.Choose one of the following parsing options.
None
Return the API response as a single text column. You can specify a name for the output column.JSON
Parse the response as JSON.You can optionally:- Flatten the parsed response into columns. If you leave this option disabled, Prophecy returns the parsed response as a single struct column.
- Wait for async results. Enable this option for APIs that initially return an “in progress” response. Enter a JSONPath condition that identifies an in-progress response. Prophecy continues polling while the condition evaluates to true and stops when it evaluates to false or no longer matches.
XML
Parse the response as XML.You can optionally:- Flatten the parsed response into columns. If you leave this option disabled, Prophecy returns the parsed response as a single struct column.
- Wait for async results. Enable this option for APIs that initially return an “in progress” response. Enter an XPath condition that identifies an in-progress response. Prophecy continues polling while the condition evaluates to true and stops when it evaluates to false or no longer matches.
Parsed response output
When you choose JSON or XML, the output depends on whether you flatten the parsed response.Parsing the response does not automatically define the output schema. After configuring the gem, open the output port and click Infer from Cluster to retrieve the response schema.
Click Continue to proceed.
4
Resilience
The Resilience page lets you configure retry behavior for failed requests.Enable Retry if the API fails to retry unsuccessful requests automatically.When retries are enabled, configure:
- Retry attempts — Maximum number of retry attempts.
- Retry delay — Number of seconds to wait between attempts.
- Network errors or timeouts.
- HTTP
408 Request Timeoutresponses. - HTTP
429 Too Many Requestsresponses. - HTTP
5xxserver errors.
4xx responses are not retried.Click Save Configuration to finish configuring the gem.Example
Suppose you have a dataset of customer records with the following columns:
You can use the RestAPI gem to enrich each row by calling an external CRM API that returns account details for a given customer.
Example configuration
Result
This configuration:- Sends one
GETrequest per input row, using thecustomer_idcolumn to construct the URL dynamically. - Authenticates each request using a bearer token stored as a Prophecy secret.
- Parses the JSON response and flattens the returned fields into individual output columns alongside the original input columns.
Common issues
Authentication errors
If requests return401 Unauthorized or 403 Forbidden:
- Verify that the token or credentials are correct and have not expired.
- Check that the secret is attached to the correct fabric.
- If using None authentication, confirm that the
Authorizationheader is formatted correctly. Bearer tokens should use the formatBearer <token>.
Unexpected response shape
If the output schema looks wrong after inferring:- Check the raw response by temporarily disabling JSON or XML parsing and inspecting the
api_datacolumn. - Some APIs wrap results in a top-level key such as
dataorresults. You may need a subsequent gem to extract the nested field.
Pagination
Most APIs return a limited number of results per request, controlled by parameters such aslimit and offset or page. The RestAPI gem sends one request per input row and does not handle pagination automatically. To retrieve all pages, build a loop using pipeline parameters or pre-compute the required page offsets in an upstream gem.