> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prophecy.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Azure Data Lake Storage (ADLS)

> Use Azure Data Lake Storage as a file source or target in a gem

export const execution_engine_0 = "Prophecy Automate"

<Info>This gem runs in {execution_engine_0}.</Info>

Use a Source and Target gem to read from or write to [Azure Data Lake Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) (ADLS) locations in Prophecy pipelines. This page covers supported file formats, how to create the gem, and how to configure connection details and paths for both Source and Target gems.

## Supported file formats

| Format                                                                       | Read | Write |
| ---------------------------------------------------------------------------- | ---- | ----- |
| [CSV](/data-analysis/gems/source-target/file/file-types/csv)                 | ✔    | ✔     |
| [Fixed width](/data-analysis/gems/source-target/file/file-types/fixed-width) | ✔    |       |
| [JSON](/data-analysis/gems/source-target/file/file-types/json)               | ✔    | ✔     |
| [Parquet](/data-analysis/gems/source-target/file/file-types/parquet)         | ✔    | ✔     |
| [XLSX](/data-analysis/gems/source-target/file/file-types/excel)              | ✔    | ✔     |
| [XML](/data-analysis/gems/source-target/file/file-types/xml)                 | ✔    | ✔     |

## Create an ADLS gem

To create an ADLS Source or Target gem in your pipeline:

<Steps>
  <Step title="Add a Source or Target gem to the pipeline">
    1. Open your pipeline in the [Studio](/data-analysis/development/studio/studio).
    2. Click on **Source/Target** in the canvas.
    3. Select **Source** or **Target** from the dropdown.
    4. Click on the gem to open the configuration.
  </Step>

  <Step title="Select ADLS format">In the **Type** tab, select **ADLS**. Then, click **Next**.</Step>

  <Step title="Set location details">
    In the **Location** tab, set your file format and connection details. To learn more, jump to [Source location](#source-location) and [Target location](#target-location).
  </Step>

  <Step title="Set file properties">
    In the **Properties** tab, set the file properties. These vary based on the file type that you are working with.

    <Info>See the list of properties per file type, such as [CSV](/data-analysis/gems/source-target/file/file-types/csv).</Info>
  </Step>

  <Step title="Preview data (Source only)">
    In the **Preview** tab, load a sample of the data and verify that it looks correct.
  </Step>
</Steps>

## Source location

When setting up an ADLS Source gem, you need to decide how Prophecy should locate files at runtime. There are two modes:

* **Filepath**: Always read from a specific file path. You can use wildcards in your path definition. If multiple files match, they are unioned into a single output table.
* **Configuration**: Dynamically read files provided by a [file arrival/change trigger](/data-analysis/production/scheduling/triggers) in the pipeline's schedule. When new or updated files are detected in the monitored directory, the trigger starts the pipeline and passes those files to the ADLS Source gem, which unions them into a single output table.

Use the table below to understand how to configure each option.

| Parameter                                             | Description                                                                                                                                                                                                                                                  |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Select or create connection                           | Select an existing ADLS connection or [create a new one](/data-analysis/environment/connections/adls).                                                                                                                                                       |
| Choose Path or Configuration                          | Choose between the following options.<ul><li>**Filepath**: Read one file from a specified file path.</li><li>**Configuration**: Dynamically read files provided by a [file arrival/change trigger](/data-analysis/production/scheduling/triggers).</li></ul> |
| File Path<br />*Filepath option only*                 | Path to the file in the ADLS container. Supports wildcards. <br />Example: `abfss://container-name@account-name.dfs.core.windows.net/temp/dir/*.csv`                                                                                                         |
| Examine                                               | Automatically determine the file format, compression type (if any), properties, and schema of the file to read.                                                                                                                                              |
| Select Configuration<br />*Configuration option only* | File arrival/change trigger [configuration](/data-analysis/production/scheduling/triggers#trigger-configuration) that provides the added or modified files for that run.                                                                                     |
| Include filename Column                               | Appends a column containing the source filename for each row in the output table.                                                                                                                                                                            |
| Delete files after successfully processed             | Deletes objects after they are successfully read.                                                                                                                                                                                                            |
| Move files after successfully processed               | Moves objects to a specified directory after they are successfully read.                                                                                                                                                                                     |
| Format type                                           | Type of file to read, such as csv or json.                                                                                                                                                                                                                   |
| Compression                                           | The compression type of the file to read. <br /><br />Supported types: `uncompressed`,`gzip`, `zstd`, `lz4`, `zlib`, `snappy`, `lzop`                                                                                                                        |

### Configuration

If you select **Configuration**, the gem will only run successfully during a **triggered pipeline run**. This is because the gem expects files from the trigger. In other cases, such as an interactive run or an API-triggered run, there will be no files to read. In these situations, you will encounter the following error:

```
Failed due to: Unable to detect modified files for provided File Trigger
```

For the same reason, you'll see an error if you try to infer the schema in the **Properties** tab or load a preview in the **Preview** tab of the Source gem.

## Target location

When setting up an ADLS Target gem, you need to set the location and file type to correctly write the file.

| Parameter                   | Description                                                                                                                                   |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| Select or create connection | Select an existing ADLS connection or [create a new one](/data-analysis/environment/connections/adls).                                        |
| File Path                   | ADLS path where the output file will be written. <br />Example: `abfss://container-name@account-name.dfs.core.windows.net/data/orders.csv`    |
| Examine                     | If an existing file exists at the target path, click **Examine** to automatically determine the file format and compression type (if any).    |
| Encryption Algorithm        | The encryption algorithm to use when writing the file.<br /><br />Supported algorithms: `AES-192`, `AES-296`, `BlowFish`                      |
| Format type                 | Type of file to write, such as csv or json.                                                                                                   |
| Compression                 | The compression type to use when writing the file.<br /><br />Supported types: `uncompressed`,`gzip`, `zstd`, `lz4`, `zlib`, `snappy`, `lzop` |

<Note>
  Configure encryption in a Target gem to encrypt an entire file. Use the
  [DataEncoderDecoder](/data-analysis/gems/transform/encoder-decoder) gem to encrypt individual
  columns.
</Note>