DataEncoderDecoder

​

Overview

The DataEncoderDecoder gem allows you to encode or decode data in selected columns using a variety of standard techniques, including Base64, Hex, and AES encryption. You can transform values in-place or create new output columns with a prefix or suffix.

​

Prerequisites

• Add prophecy_basics package version 1.0.0 or higher to your project.

​

Input and Output

The DataEncoderDecoder gem accepts the following input and output.

​

Parameters

Configure the DataEncoderDecoder gem using the following parameters.

​

Encode/Decode methods

Choose from the following methods:

​

base64

Encodes the selected column(s) using Base64.

​

unbase64

Decodes Base64-encoded column values.

​

hex

Encodes the selected column(s) into hexadecimal format.

​

unhex

Decodes hexadecimal-encoded values.

​

encode

Encodes the string column(s) using a specified character set.

• Charset: Character set to use, such as UTF-8.

​

decode

Decodes the string column(s) using a specified character set.

• Charset: Character set to use, such as UTF-8.

​

aes_encrypt

Encrypts the selected column(s) using AES encryption.

• Secret scope: The name of the Databricks secret scope.
• Secret key: The key name within the scope that stores the encryption key.
• Mode: AES encryption mode to use.
  • GCM
  • CBC
  • EBC
• (Optional) AAD scope and key: For GCM mode, you can specify Databricks scope and key for the AAD.
• (Optional) Initialization vector scope and key: For CBC mode, specify a Databricks secret scope and key for the IV.

​

Transformed column options

• Substitute the new columns in place: Replaces the original column(s) with the transformed values.
• Add new columns with a prefix/suffix attached: Adds a new column for each transformed input column, appending a prefix or suffix to the name.

​

Example

Assume you have the following dataset: Using the base64 method, adding new columns with the suffix _encoded, the output would be: