AdlsClient Class

Definition

Client of Azure data lake store. It contains the public APIs to perform operations of REST API which are easier to call and more usable than Core APIs. Core APIs provide more freedom but ADLSClient provide more commonly used forms. It encapsulates the Authorization token and token refresh. Contains factory methods that takes a ServiceClientCredential or a string auth token and returns instance of this class. For every operation it provides a async and sync version. Every sync method is a wait on async method with exception of Create and Concurrent append. All APIs are thread safe with some exceptions in CreateFile and GetAppendStream. CreateFile and GetAppendStream cannot be called for the same path from different threads because writing is done with a lease so there will lease conflicts If an application wants to perform multi-threaded operations using this SDK it is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2. Adls use NLog for logging. adls.dotnet.* is the name of the logger to obtain all logs.

public class AdlsClient
type AdlsClient = class
Public Class AdlsClient
Inheritance
System.Object
AdlsClient
Derived

Constructors

AdlsClient()

Protected constructor for moq tests

Fields

ClientLogger

Logger to log information (debug/error/trace) regarding client

Properties

AccountFQDN

Azure data lake store account name including full domain name

ClientId

Client object ID

DefaultNumThreads

Default number of threads used by tools like uploader/downloader, recursive acl change and other multi-threaded tools using the SDK. Can be used to set ServicePointManager.DefaultConnectionLimit if you want the SDK to decide number of threads it uses for multi-threaded tools.

Methods

AddUserAgentSuffix(String)

Adds the user agent suffix

BulkDownload(String, String, Int32, IfExists, Boolean, IProgress<TransferStatus>, Boolean, Boolean, CancellationToken)

Download directory or file from remote server to local. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. By default logs the transfer progress in system's temp path, so that user can recover using resume if upload has crashed. This progress logging can be disabled using disableTransferLogging. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.

BulkDownload(String, String, Int32, IfExists, IProgress<TransferStatus>, Boolean, Boolean, CancellationToken)

Download directory or file from remote server to local. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.

BulkUpload(String, String, Int32, IfExists, Boolean, IProgress<TransferStatus>, Boolean, Boolean, Boolean, CancellationToken)

Upload directory or file from local to remote. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. By default logs the transfer progress in system's temp path, so that user can recover using resume if upload has crashed. This progress logging can be disabled using disableTransferLogging. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2. By default files are uploaded at new line boundaries. However if files does not have newline within 4MB chunks, the transfer will fail. In that case it is required to pass true to isBinary to avoid uploads at newline boundaries.

BulkUpload(String, String, Int32, IfExists, IProgress<TransferStatus>, Boolean, Boolean, Boolean, CancellationToken)

Upload directory or file from local to remote. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2. By default files are uploaded at new line boundaries. However if files does not have newline within 4MB chunks, the transfer will fail. In that case it is required to pass true to isBinary to avoid uploads at newline boundaries.

ChangeAcl(String, List<AclEntry>, RequestedAclType, Int32)

Change Acl (Modify, set and remove) recursively on a directory tree It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.

ChangeAcl(String, List<AclEntry>, RequestedAclType, Int32, IProgress<AclProcessorStats>, CancellationToken)

Change Acl (Modify, set and remove) recursively on a directory tree It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.

CheckAccess(String, String, CancellationToken)

Checks if the user/group has specified access of the given path

CheckAccessAsync(String, String, CancellationToken)

Asynchronously checks if the user/group has specified access of the given path

CheckExists(String)

Checks whether file or directory exists

ConcatenateFiles(String, List<String>, Boolean, CancellationToken)

Synchronous API to concatenate source files to a destination file

ConcatenateFilesAsync(String, List<String>, Boolean, CancellationToken)

Asynchronous API to concatenate source files to a destination file

ConcurrentAppend(String, Boolean, Byte[], Int32, Int32)

Synchronous API to perform concurrent append at server. The offset at which append will occur is determined by server. It is highly recomended to call this api with data size less than equals 4MB. Backend gurantees 4MB atomic appends.

ConcurrentAppendAsync(String, Boolean, Byte[], Int32, Int32, CancellationToken)

Asynchronous API to perform concurrent append at server. The offset at which append will occur is determined by server. Asynchronous operation. It is highly recomended to call this api with data size less than equals 4MB. Backend gurantees 4MB atomic appends.

CreateClient(String, ServiceClientCredentials)

Factory method that creates an instance of AdlsClient using ServiceClientCredential. If an application wants to perform multi-threaded operations using this SDK it is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.

CreateClient(String, String)

Factory method that creates an instance AdlsClient using the token key. If an application wants to perform multi-threaded operations using this SDK it is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.

CreateDirectory(String, String, CancellationToken)

Synchronous API to create a directory

CreateDirectoryAsync(String, String, CancellationToken)

Asynchronous API to create a directory.

CreateFile(String, IfExists, String, Boolean)

Synchronous API that creates a file and returns the stream to write data to that file in ADLS. The file is opened with exclusive access - any attempt to open the same file for append will fail while this stream is open.

Threading: The returned stream is not thread-safe.

CreateFileAsync(String, IfExists, String, Boolean, CancellationToken)

Asynchronous API that creates a file and returns the stream to write data to that file in ADLS. The file is opened with exclusive access - any attempt to open the same file for append will fail while this stream is open.

Threading: The returned stream is not thread-safe.

Delete(String, CancellationToken)

Synchronous api to delete a file or directory. For directory it will only delete if it is empty.

DeleteAsync(String, CancellationToken)

Asynchronous api to delete a file or directory. For directory it will only delete if it is empty.

DeleteRecursive(String, CancellationToken)

Synchronous api to delete a file or directory recursively. If it is a non-empty directory then it deletes the sub-directories or files.

DeleteRecursiveAsync(String, CancellationToken)

Asynchronous api to delete a file or directory recursively

EnumerateDeletedItems(String, String, Int32, IProgress<EnumerateDeletedItemsProgress>, CancellationToken)

Search trash under a account with hint and a starting point. This is a long running operation, and user is updated with progress periodically. Caution: Undeleting files is a best effort operation. There are no guarantees that a file can be restored once it is deleted. The use of this API is enabled via whitelisting. If your ADL account is not whitelisted, then using this api will throw Not immplemented exception. For further information and assistance please contact Microsoft support.

EnumerateDeletedItemsAsync(String, String, Int32, IProgress<EnumerateDeletedItemsProgress>, CancellationToken)

Asynchronously gets the trash entries Caution: Undeleting files is a best effort operation. There are no guarantees that a file can be restored once it is deleted. The use of this API is enabled via whitelisting. If your ADL account is not whitelisted, then using this api will throw Not immplemented exception. For further information and assistance please contact Microsoft support.

EnumerateDirectory(String, UserGroupRepresentation, CancellationToken)

Returns a enumerable that enumerates the sub-directories or files contained in a directory. By default listAfter and listBefore is empty and we enumerate all the directory entries.

GetAclStatus(String, UserGroupRepresentation, CancellationToken)

Gets the ACL entry list, owner ID, group ID, octal permission and sticky bit (only for a directory) of the file/directory

GetAclStatusAsync(String, UserGroupRepresentation, CancellationToken)

Asynchronously gets the ACL entry list, owner ID, group ID, octal permission and sticky bit (only for a directory) of the file/directory

GetAppendStream(String, CancellationToken)

Synchronous API that returns the stream to write data to a file in ADLS. The file is opened with exclusive access - any attempt to open the same file for append will fail while this stream is open.

Threading: The returned stream is not thread-safe.

GetAppendStreamAsync(String, CancellationToken)

Asynchronous API that returns the stream to write data to a file in ADLS. The file is opened with exclusive access - any attempt to open the same file for append will fail while this stream is open.

Threading: The returned stream is not thread-safe.

GetContentSummary(String, Int32, CancellationToken)

Gets content summary of a file or directory. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.

GetDirectoryEntry(String, UserGroupRepresentation, CancellationToken)

Synchronously gets meta data like full path, type (file or directory), group, user, permission, length,last Access Time,last Modified Time, expiry time, acl Bit, replication Factor

GetDirectoryEntryAsync(String, UserGroupRepresentation, CancellationToken)

Asynchronously gets meta data like full path, type (file or directory), group, user, permission, length,last Access Time,last Modified Time, expiry time, acl Bit, replication Factor

GetExceptionFromResponse(OperationResponse, String)

Returns a ADLS Exception based on response from the server

GetFileProperties(String, Boolean, String, Boolean, Boolean, Int32, Boolean, Boolean, Int64, CancellationToken)

Recursively dumps file property of alldirectories or/and files under the given path to a local or adl file. File property can be disk usage or Acl or both. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.

GetReadStream(String, CancellationToken)

Synchronous API that returns the stream to read data from file in ADLS

GetReadStream(String, Int32, CancellationToken)

Synchronous API that returns the stream to read data from file in ADLS

GetReadStreamAsync(String, CancellationToken)

Asynchronous API that returns the stream to read data from file in ADLS

GetReadStreamAsync(String, Int32, CancellationToken)

Asynchronous API that returns the stream to read data from file in ADLS

ModifyAclEntries(String, List<AclEntry>, CancellationToken)

Modifies acl entries of a file or directory with given ACL list. It merges the exisitng ACL list with given list.

ModifyAclEntriesAsync(String, List<AclEntry>, CancellationToken)

Asynchronously modifies acl entries of a file or directory with given ACL list. It merges the exisitng ACL list with given list.

RemoveAclEntries(String, List<AclEntry>, CancellationToken)

Removes specified Acl Entries for a file or directory.

RemoveAclEntriesAsync(String, List<AclEntry>, CancellationToken)

Asynchronously removes specified Acl Entries for a file or directory.

RemoveAllAcls(String, CancellationToken)

Removes all Acl Entries for a file or directory.

RemoveAllAclsAsync(String, CancellationToken)

Asynchronously removes all Acl Entries for a file or directory.

RemoveDefaultAcls(String, CancellationToken)

Removes all Acl Entries of AclScope default for a file or directory.

RemoveDefaultAclsAsync(String, CancellationToken)

Asynchronously removes all Acl Entries of AclScope default for a file or directory.

Rename(String, String, Boolean, CancellationToken)

Synchronous API to rename a file or directory. For renaming directory: If the destination exists then it puts the source directory one level under the destination.

RenameAsync(String, String, Boolean, CancellationToken)

Asynchronous API to rename a file or directory. For renaming directory: If the destination exists then it puts the source directory one level under the destination.

RestoreDeletedItems(String, String, String, String, CancellationToken)

Synchronously Restores trash entry Caution: Undeleting files is a best effort operation. There are no guarantees that a file can be restored once it is deleted. The use of this API is enabled via whitelisting. If your ADL account is not whitelisted, then using this api will throw Not immplemented exception. For further information and assistance please contact Microsoft support.

RestoreDeletedItemsAsync(String, String, String, String, CancellationToken)

Asynchronously Restores trash entry Caution: Undeleting files is a best effort operation. There are no guarantees that a file can be restored once it is deleted. The use of this API is enabled via whitelisting. If your ADL account is not whitelisted, then using this api will throw Not immplemented exception. For further information and assistance please contact Microsoft support.

SetAcl(String, List<AclEntry>, CancellationToken)

Sets Acl Entries for a file or directory. It wipes out the existing Acl entries for the path.

SetAclAsync(String, List<AclEntry>, CancellationToken)

Asynchronously sets Acl Entries for a file or directory. It wipes out the existing Acl entries for the path.

SetConditionalCreateWithOverwrite(Boolean)

Sets whether to perform conditional create with overwrite. Helps resolving atomic issues with retries

SetExpiryTime(String, ExpiryOption, Int64, CancellationToken)

Synchronously sets the expiry time

SetExpiryTimeAsync(String, ExpiryOption, Int64, CancellationToken)

Asynchronously sets the expiry time

SetOwner(String, String, String, CancellationToken)

Sets the owner or/and group of the path

SetOwnerAsync(String, String, String, CancellationToken)

Asynchronously sets the owner or/and group of the path

SetPermission(String, String, CancellationToken)

Sets the permission of the specified path

SetPermissionAsync(String, String, CancellationToken)

Asynchronously sets the permission of the specified path

SetPerRequestTimeout(TimeSpan)

Sets the per request timeout. Highly recommended to set it after creating client. Not recommended to set it while requests are underway from a different thread.

SetToken(String)

Sets the auth token.

Applies to