Skip to main content
This gem runs in .

​
Overview

Use a Source and Target gem to read from or write to Google Cloud Storage (GCS) locations in Prophecy pipelines. This page covers supported file formats, how to create the gem, and how to configure connection details and paths for both Source and Target gems.

​
Supported file formats

​
Create a GCS gem

To create a GCS Source or Target gem in your pipeline:
1

Add a Source or Target gem to the pipeline

  1. Open your pipeline in the Studio.
  2. Click on Source/Target in the canvas.
  3. Select Source or Target from the dropdown.
  4. Click on the gem to open the configuration.
2

Select GCS format

In the Type tab, select GCS. Then, click Next.
3

Set location details

In the Location tab, set your file format and connection details. To learn more, jump to Source location and Target location.
4

Set file properties

In the Properties tab, set the file properties. These vary based on the file type that you are working with.
See the list of properties per file type, such as CSV.
5

Preview data (Source only)

In the Preview tab, load a sample of the data and verify that it looks correct.

​
Source location

When setting up a GCS Source gem, you need to decide how Prophecy should locate files at runtime. There are two modes:
  • Filepath: Always read from a specific file path. You can use wildcards in your path definition. If multiple files match, they are unioned into a single output table.
  • Configuration: Dynamically read files provided by a file arrival/change trigger in the pipeline’s schedule. When new or updated files are detected in the monitored directory, the trigger starts the pipeline and passes those files to the GCS Source gem, which unions them into a single output table.
Use the table below to understand how to configure each option.

​
Configuration

If you select Configuration, the gem will only run successfully during a triggered pipeline run. This is because the gem expects files from the trigger. In other cases, such as an interactive run or an API-triggered run, there will be no files to read. In these situations, you will encounter the following error:
For the same reason, you’ll see an error if you try to infer the schema in the Properties tab or load a preview in the Preview tab of the Source gem.

​
Target location

When setting up a GCS Target gem, you need to set the location and file type to correctly write the file.
Configure encryption in a Target gem to encrypt an entire file. Use the DataEncoderDecoder gem to encrypt individual columns.