Use PowerShell to manage directories and files in Azure Data Lake Storage

This article shows you how to use PowerShell to create and manage directories and files in storage accounts that have a hierarchical namespace.

To learn about how to get, set, and update the access control lists (ACL) of directories and files, see Use PowerShell to manage ACLs in Azure Data Lake Storage.

Reference | Give feedback |

Prerequisites

• An Azure subscription. For more information, see Get Azure trial.

• A storage account that has hierarchical namespace enabled. Follow these instructions to create one.

• .NET Framework is 4.7.2 or greater installed. For more information, see Download .NET Framework.

• PowerShell version 5.1 or higher.

Install the PowerShell module

1. Verify that the version of PowerShell that 's installed is 5.1 or higher by using the following command.

   echo $PSVersionTable.PSVersion.ToString()
   

   To upgrade your version of PowerShell, see Upgrading existing Windows PowerShell

2. Install Az.Storage module.

   Install-Module Az.Storage -Repository PSGallery -Force  
   

   For more information about how to install PowerShell modules, see Install the Azure PowerShell module

Connect to the account

Choose how you want your commands to obtain authorization to the storage account.

Option 1: Obtain authorization by using Microsoft Entra ID

With this approach, the system ensures that your user account has the appropriate Azure role-based access control (Azure RBAC) assignments and ACL permissions.

1. Open a Windows PowerShell command window, and then sign in to your Azure subscription with the Connect-AzAccount command and follow the on-screen directions.

   Connect-AzAccount -Environment AzureChinaCloud
   

2. If your identity is associated with more than one subscription, then set your active subscription to subscription of the storage account that you want create and manage directories in. In this example, replace the <subscription-id> placeholder value with the ID of your subscription.

   Select-AzSubscription -SubscriptionId <subscription-id>
   

3. Get the storage account context.

   $ctx = New-AzStorageContext -StorageAccountName '<storage-account-name>' -UseConnectedAccount
   

Option 2: Obtain authorization by using the storage account key

With this approach, the system doesn't check Azure RBAC or ACL permissions. Get the storage account context by using an account key.

$ctx = New-AzStorageContext -StorageAccountName '<storage-account-name>' -StorageAccountKey '<storage-account-key>'

Create a container

A container acts as a file system for your files. You can create one by using the New-AzStorageContainer cmdlet.

This example creates a container named my-file-system.

$filesystemName = "my-file-system"
New-AzStorageContainer -Context $ctx -Name $filesystemName

Create a directory

Create a directory reference by using the New-AzDataLakeGen2Item cmdlet.

This example adds a directory named my-directory to a container.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname -Directory

This example adds the same directory, but also sets the permissions, umask, property values, and metadata values.

$dir = New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname -Directory -Permission rwxrwxrwx -Umask ---rwx---  -Property @{"ContentEncoding" = "UDF8"; "CacheControl" = "READ"} -Metadata  @{"tag1" = "value1"; "tag2" = "value2" }

Show directory properties

This example gets a directory by using the Get-AzDataLakeGen2Item cmdlet, and then prints property values to the console.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
$dir =  Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname
$dir.ACL
$dir.Permissions
$dir.Group
$dir.Owner
$dir.Properties
$dir.Properties.Metadata

Rename or move a directory

Rename or move a directory by using the Move-AzDataLakeGen2Item cmdlet.

This example renames a directory from the name my-directory to the name my-new-directory.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
$dirname2 = "my-new-directory/"
Move-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname -DestFileSystem $filesystemName -DestPath $dirname2

This example moves a directory named my-directory to a subdirectory of my-directory-2 named my-subdirectory.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
$dirname2 = "my-directory-2/my-subdirectory/"
Move-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname -DestFileSystem $filesystemName -DestPath $dirname2

Delete a directory

Delete a directory by using the Remove-AzDataLakeGen2Item cmdlet.

This example deletes a directory named my-directory.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
Remove-AzDataLakeGen2Item  -Context $ctx -FileSystem $filesystemName -Path $dirname

You can use the -Force parameter to remove the file without a prompt.

Download from a directory

Download a file from a directory by using the Get-AzDataLakeGen2ItemContent cmdlet.

This example downloads a file named upload.txt from a directory named my-directory.

$filesystemName = "my-file-system"
$filePath = "my-directory/upload.txt"
$downloadFilePath = "download.txt"
Get-AzDataLakeGen2ItemContent -Context $ctx -FileSystem $filesystemName -Path $filePath -Destination $downloadFilePath

List directory contents

List the contents of a directory by using the Get-AzDataLakeGen2ChildItem cmdlet. You can use the optional parameter -OutputUserPrincipalName to get the name (instead of the object ID) of users.

This example lists the contents of a directory named my-directory.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $filesystemName -Path $dirname -OutputUserPrincipalName

The following example lists the ACL, Permissions, Group, and Owner properties of each item in the directory. The -FetchProperty parameter is required to get values for the ACL property.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
$properties = Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $filesystemName -Path $dirname -Recurse -FetchProperty
$properties.ACL
$properties.Permissions
$properties.Group
$properties.Owner

Upload a file to a directory

Upload a file to a directory by using the New-AzDataLakeGen2Item cmdlet.

This example uploads a file named upload.txt to a directory named my-directory.

$localSrcFile =  "upload.txt"
$filesystemName = "my-file-system"
$dirname = "my-directory/"
$destPath = $dirname + (Get-Item $localSrcFile).Name
New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $destPath -Source $localSrcFile -Force

This example uploads the same file, but then sets the permissions, umask, property values, and metadata values of the destination file. This example also prints these values to the console.

$file = New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $destPath -Source $localSrcFile -Permission rwxrwxrwx -Umask ---rwx--- -Property @{"ContentEncoding" = "UDF8"; "CacheControl" = "READ"} -Metadata  @{"tag1" = "value1"; "tag2" = "value2" }
$file1
$file1.Properties
$file1.Properties.Metadata

Show file properties

This example gets a file by using the Get-AzDataLakeGen2Item cmdlet, and then prints property values to the console.

$filepath =  "my-directory/upload.txt"
$filesystemName = "my-file-system"
$file = Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $filepath
$file
$file.ACL
$file.Permissions
$file.Group
$file.Owner
$file.Properties
$file.Properties.Metadata

Delete a file

Delete a file by using the Remove-AzDataLakeGen2Item cmdlet.

This example deletes a file named upload.txt.

$filesystemName = "my-file-system"
$filepath = "upload.txt"
Remove-AzDataLakeGen2Item  -Context $ctx -FileSystem $filesystemName -Path $filepath

You can use the -Force parameter to remove the file without a prompt.

See also

• Known issues
• Storage PowerShell cmdlets