TextProcessing

The TextProcessing gem enables text data preparation for machine learning in two different ways:

Load web URLs and extract text.
Split text data into equal chunks.

Follow along to see how to use the TextProcessing gem. For an example set of pipelines that uses this gem to create a Generative AI Chatbot, see this guide.

1. Load web URLs and Extract Text

Given a column with web URLs, the Load web URLs operation will scrape the content from each URL, and output the content as a binary format or as a human readable text format, depending on the operation type selected. The figure below shows the Load web URL and Extract Text operation.

1a. Configure web scrape

Configure the (1) Operation Type to Load url (web scrape), and optionally extract the text. Specify which input (2) Column name contains the web urls. If the extract text operation is selected, the text will be converted from binary to human readable format. When would you want to use the binary format? Binary web scraping is useful for downloading content including images or archived documents.

1b. Input

Parameter	Description	Required
Column name (string with urls)	string - the input column which contains the strings of web URLs	True

1c. Output

Parameter	Description
Result content `Load url (web scrape)`	binary - the contents of each web page
Result content `Load url (web scrape) and extract text`	string - the contents of each web page, converted from binary to human readable text

1d. Generated Code

def scrape_pages(spark: SparkSession, in0: DataFrame) -> DataFrame:
 from pyspark.sql.functions import expr, array, struct
 from spark_ai.webapps import WebUtils
 WebUtils().register_udfs(spark)

 return in0.withColumn("result_content", expr(f"web_scrape(loc)"))

[Not yet supported]

2. Split text data into equal chunks

Sometimes you’d like to send text data to a foundational model or store in a vector database, but the text is too long. For this case, just split the text into “chunks” of characters.

2a. Configure text splitting

Given a text input, the Split data operation will separate the input column entries into chunks of specified size.

Select the (1) Operation type to split text into equal chunks. Specify which input (2) Column name contains the relevant content. Specify an integer chunk (3) Size relevant for your generative AI use case.

2b. Input

Parameter	Description	Required
Column name	string - the text content which should be split into equal chunks	True
Size	integer - the size of each chunk, number of characters. Example: `1000`	True

2c. Output

Parameter	Description
result_chunks	array(string) - an array of text strings, each string representing one chunk of the larger text content

2d. Generated code

def Chunkify(spark: SparkSession, web_bronze_content: DataFrame) -> DataFrame:
 from pyspark.sql.functions import expr, array, struct
 from spark_ai.files.text import FileTextUtils
 FileTextUtils().register_udfs(spark)

 return web_bronze_content.withColumn("result_chunks", expr(f"text_split_into_chunks(content, 1000)"))

[Not yet supported]

Overview

Sources and targets

Data processing

Advanced

1. Load web URLs and Extract Text

1a. Configure web scrape

1b. Input

1c. Output

1d. Generated Code

2. Split text data into equal chunks

2a. Configure text splitting

2b. Input

2c. Output

2d. Generated code

​1. Load web URLs and Extract Text

​1a. Configure web scrape

​1b. Input

​1c. Output

​1d. Generated Code

​2. Split text data into equal chunks

​2a. Configure text splitting

​2b. Input

​2c. Output

​2d. Generated code

1. Load web URLs and Extract Text

1a. Configure web scrape

1b. Input

1c. Output

1d. Generated Code

2. Split text data into equal chunks

2a. Configure text splitting

2b. Input

2c. Output

2d. Generated code