fs command group

The fs command group within the Databricks CLI allows you to automate volumes in Unity Catalog and to automate Databricks File System (DBFS) objects.

You run fs commands by appending them to databricks fs. To display help for the fs command, run databricks fs -h.

fs commands require volume paths to begin with dbfs:/Volumes and require directory and file paths in DBFS to begin with dbfs:/.

List the contents of a directory

To list the contents of a directory in a volume or in DBFS, use the ls command. Specify the path to the directory. The following examples list the names of the objects found in the specified volume's root or in the DBFS root:

databricks fs ls dbfs:/Volumes/main/default/my-volume
databricks fs ls dbfs:/

To display full information including object types, sizes, modification times since Epoch in milliseconds, as well as the objects' names, use the --long or -l option. The following examples list the full information of the objects found in the specified volume's root or in a tmp directory within the DBFS root:

databricks fs ls dbfs:/Volumes/main/default/my-volume -l
databricks fs ls dbfs:/tmp -l

To display full object paths, use the --absolute option. The following examples list the full information of the objects, and the objects' full paths, found in the specified volume's root or in a tmp directory within the DBFS root:

databricks fs ls dbfs:/Volumes/main/default/my-volume -l --absolute
databricks fs ls dbfs:/tmp -l --absolute

Output the contents of a file

To output the contents of a file, use the cat command. Specify the path to the file in DBFS. The following examples output the contents of the file named babynames.csv found in the specified volume's root or in a tmp directory within the DBFS root:

databricks fs cat dbfs:/Volumes/main/default/my-volume/babynames.csv
databricks fs cat dbfs:/tmp/babynames.csv

The following examples return errors, as they try to output the contents of a directory instead of a file:

databricks fs cat dbfs:/Volumes/main/default/my-volume
databricks fs cat dbfs:/tmp

Create a directory

To create a directory, use the mkdir command. Specify the path to the directory to be created in a volume or in DBFS. If the directory already exists, nothing happens. The following examples create a directory named squirrel-data within the specified volume's root or in a directory named tmp within the DBFS root:

databricks fs mkdir dbfs:/Volumes/main/default/my-volume/squirrel-data
databricks fs mkdir dbfs:/tmp/squirrel-data

Copy a directory or a file

To copy a directory or a file, use the cp command. Specify the paths to the source directory or file to copy and its destination. You can copy directories and files between the local filesystem and DBFS, and you can copy directories and files between DBFS paths. To overwrite existing files, use the --overwrite option. To recursively copy files within a directory, use the --recursive or -r option.

The cp command assumes file:/, if file:/ is omitted.

The following examples copy a directory named squirrel-data and its contents, from a local filesystem path to a squirrels directory within the specified volume's root or the DBFS root.

databricks fs cp /Users/<username>/squirrel-data dbfs:/Volumes/main/default/my-volume/squirrels -r
databricks fs cp /Users/<username>/squirrel-data dbfs:/squirrels -r

The following example copies a file named squirrels.csv from a local filesystem path to a directory named squirrel-data within the specified volume's root or the DBFS root. If the file already exists in the destination, it is overwritten.

databricks fs cp /Users/<username>/squirrels.csv dbfs:/Volumes/main/default/my-volume/squirrel-data --overwrite
databricks fs cp /Users/<username>/squirrels.csv dbfs:/squirrel-data --overwrite

Remove a directory

To remove a directory, use the rm command. Specify the path to the existing directory in DBFS. The following examples remove a directory named squirrel-data from the specified volume's root or from a tmp directory in the DBFS root:

databricks fs rm dbfs:/Volumes/main/default/my-volume/squirrel-data
databricks fs rm dbfs:/tmp/squirrel-data

If the directory does not exist, nothing happens, and no error is returned.

If the directory exists but is not empty, an error is returned. To remove a non-empty directory and all of its contents, use the --recursive or -r option. The following examples remove a non-empty directory named squirrel-data from the specified volume's root or from a tmp directory in the DBFS root:

databricks fs rm dbfs:/Volumes/main/default/my-volume/squirrel-data -r
databricks fs rm dbfs:/tmp/squirrel-data -r