Skip to main content
Dependencies:
  • ProphecySparkBasicsPython 0.0.1+
  • ProphecySparkBasicsScala 0.0.1+
Cluster requirements:
  • UC dedicated clusters 14.3+ supported
  • UC standard clusters not supported
  • Livy clusters not supported
A Fixed Format (Fixed-Length Format) file type:
  • Is a text file where each field or column occupies a predetermined, constant number of characters in each record.
  • Can parse and process quickly because the software knows exactly where to find each field.
  • Is often used in legacy systems, data exchange, and performance-critical applications.

​
Parameters

ParameterTabDescription
LocationLocationFile path to read from or write to the Fixed Format file.
SchemaPropertiesSchema to apply on the loaded data. In the Source gem, you can define or edit the schema visually or in JSON code. In the Target gem, you can view the schema visually or as JSON code.

​
Source

The Source gem reads data from Fixed Format files and allows you to optionally specify the following additional properties.

​
Source Properties

Property nameDescriptionDefault
DescriptionDescription of your dataset.None
Skip header linesNumber of lines to skip at the beginning of the file.None
Skip footer linesNumber of lines to skip at the end of the file.None
Fixed Format SchemaSchema string for the fixed format file. Supports either EBCDIC or ASCII formats.None

​
Example

Fixed format source example

​
Compiled code

To see the compiled code of your project, switch to the Code view in the project header.
def read_ebcdic(spark: SparkSession) -> DataFrame:
 from prophecy.utils.transpiler import parse

 return spark.read\
 .option("schema", parse("ebcdic record\nstring(18) c_name;\ndecimal(10, 0) c_custkey ;\nend"))\
 .format("io.prophecy.libs.FixedFileFormat")\
 .load("/FileStore/tables/fixed_format/test/read_ebcdic")

​
Target

The Target gem writes data to Fixed Format files and allows you to optionally specify the following additional properties.

​
Target Properties

Property nameDescriptionDefault
Write ModeHow to handle existing data. For a list of the possible values, see Supported write modes.error
DescriptionDescription of your dataset.None
Fixed Format SchemaSchema string for the fixed format file. Supports either EBCDIC or ASCII formats.None

​
Supported write modes

Write modeDescription
errorIf the data already exists, throw an exception.
overwriteIf the data already exists, overwrite the data with the contents of the DataFrame.
appendIf the data already exists, append the contents of the DataFrame.
ignoreIf the data already exists, do nothing with the contents of the DataFrame. This is similar to the CREATE TABLE IF NOT EXISTS clause in SQL.

​
Example

Fixed format target Example

​
Compiled code

To see the compiled code of your project, switch to the Code view in the project header.
def write_ebcdic(spark: SparkSession, in0: DataFrame):
 from prophecy.utils.transpiler import parse
 in0.write\
 .mode("overwrite")\
 .option("schema", parse("ebcdic record\nstring(18) c_name ;\ndecimal(10, 0) c_custkey ;\nend"))\
 .format("io.prophecy.libs.FixedFileFormat")\
 .save("/FileStore/tables/fixed_format/test/write_ebcdic_alt")