You can also use the upload file feature
to use CSV files. These will be stored in the SQL warehouse configured in your fabric.
Properties
Properties for reading CSVs
The following properties are available when CSV is used as a source format. You can remove some properties by clicking the trash can icon to its right. You can also add additional properties using the + at the top of the page.Add additional properties
You can add the following properties by clicking the + at the top of the Properties page.Schema options
By default, Infer Schema rebuilds the schema table from the files each time you click it. Use the following properties to control how the gem determines the schema:- Enable Schema Merging: When the gem reads multiple files, Prophecy combines their columns by name. Rows from a file that doesn’t contain a column get null values in that column. If the schema table already has columns, Prophecy maps the merged columns onto the schema table by position and drops any extra columns.
- Master Schema File Path: Infer Schema uses the first row of this file as the column names and infers data types from its data rows. When the gem reads data, Prophecy fills an empty schema table from this file. If the schema table already has columns, Prophecy uses its names and types instead and maps data columns by position. Leave this property empty to infer the schema from the data files.
- Enforce Schema: Infer Schema returns the saved schema table instead of rebuilding it from the files, and Prophecy reads the data with those column names and data types. For example, if an
idcolumn is typed asString, values such as010keep their leading zero. This property has no effect while the schema table is empty.
The source file name column must be the last column in the schema table, and its name can’t match an existing column in the data. If the names collide, the read fails with the error
column name "<name>" already exists in the data schema. To fix this, change Source File Column Name.Properties for writing CSVs
The following properties are available when CSV is used as a target format.Write to multiple files
When File Write Method is set to Dynamic data write to multiple files as per file column name, the gem writes one CSV file for each unique value in the File Column Name column. Each file is named<prefix><value>.csv, where <prefix> is the File Prefix for multi file write value.
Where the files are written depends on the file path configured in the gem:
- If the path ends with
/or has no file extension, it is treated as a folder, and the files are written inside it. For example, with the prefixFile_, the pathexports/orders/producesexports/orders/File_US.csv,exports/orders/File_EU.csv, and so on. - If the path includes a file name, such as
exports/orders.csv, the files are written to that file’s parent folder. The file name itself is ignored.
<prefix>unknown.csv. Header, separator, encoding, and other CSV properties apply to every file.
Supported character encodings
Supported character encodings
- UTF-8 - UTF-16 - ISO-8859-1 - ISO-8859-2 - ISO-8859-3 - ISO-8859-4 - ISO-8859-5 - ISO-8859-6 - ISO-8859-7 - ISO-8859-8 - ISO-8859-9 - ISO-8859-10 - ISO-8859-13 - ISO-8859-14 - ISO-8859-15 - ISO-8859-16 - Windows-1250 - Windows-1251 - Windows-1252 - Windows-1253 - Windows-1254 - Windows-1255 - Windows-1256 - Windows-1257 - Windows-1258 - Windows-874 - CodePage437 - CodePage850 - CodePage852 - CodePage855 - CodePage858 - CodePage860 - CodePage862 - CodePage863
- CodePage865 - CodePage866 - Macintosh - MacintoshCyrillic - KOI8R - KOI8U - XUserDefined - ASCII

