Skip to main content
Dependencies:
  • ProphecySparkBasicsPython 0.2.27+
  • ProphecyLibsPython 1.9.42+
  • ProphecyLibsScala 8.2.1+
Cluster requirements:
  • UC dedicated clusters 14.3+ supported
  • UC standard clusters 14.3+ supported
  • Livy clusters not supported
The Directory gem returns a listing of all the files in a specified directory including file metadata such as file creation time, type, size, path, and more.

​
Parameters

ParametersDescription
LocationThe location of the directory that you want to return
Include SubdirectoriesA checkbox you can enable to return all of the subdirectories in the defined location
File SpecificationA regex filter to help you find your files in the file browser

​
Sample output

Below is an example output of a Directory gem. It includes seven columns:
  • name
  • path
  • size
  • creation_time
  • modification_time
  • parent_directory
  • file_type
Directory output

​
Example code

To see the compiled code of your project, switch to the Code view in the project header.
def directory_listing(spark: SparkSession) -> DataFrame:
 from prophecy.libs.utils import directory_listing

 return directory_listing(spark, "dbfs:/source_file.txt", False, "*.*")