Microsoft.Azure.DataLake.Store Namespace

Classes

AdlsArrayPool<T>

Abstract class built on top of ArrayPool that exposes async methods of Rent Return

AdlsClient

Client of Azure data lake store. It contains the public APIs to perform operations of REST API which are easier to call and more usable than Core APIs. Core APIs provide more freedom but ADLSClient provide more commonly used forms. It encapsulates the Authorization token and token refresh. Contains factory methods that takes a ServiceClientCredential or a string auth token and returns instance of this class. For every operation it provides a async and sync version. Every sync method is a wait on async method with exception of Create and Concurrent append. All APIs are thread safe with some exceptions in CreateFile and GetAppendStream. CreateFile and GetAppendStream cannot be called for the same path from different threads because writing is done with a lease so there will lease conflicts If an application wants to perform multi-threaded operations using this SDK it is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2. Adls use NLog for logging. adls.dotnet.* is the name of the logger to obtain all logs.

AdlsException

Exception returned by client. It encapsulates any remote excepotion returned from server, Unhandled and handled exception.

AdlsInputStream

ADLS Input stream that reads data from a file on Data lake. It reads data in bulk from server to a buffer and then provides buffered output to the client as per request. Data can be read asynchronously/synchronously. Data can be read serially or from arbitrary points in file. Read is fully synchronous till the transport layer. ReadAsync is fully asynchronous till the transport layer. AdlsInputStream is not threadsafe since it uses buffer (maintains state so not stateless).

AdlsOutputStream

ADLS Output stream that writes data to a file on Data lake. It writes data to a buffer and when the buffer gets filled, writes data in bulk to server Data can be written asynchronously/synchronously. Write is fully synchronous till the transport layer. WriteAsync is fully asynchronous till the transport layer. AdlsOutputStream is not threadsafe since it uses buffer (maintains state so not stateless).

ContentSummary

Encapsulates total directory count, file count, space consumed

Core

Core is a stateless class. It contains thread safe methods for REST APIs. For each rest api command it sends a HTTP request to server. Every API is threadsafe with some exceptions in Create and Append (Listed in the documentation of the respective apis). We have both async and sync versions of CREATE, APPEND, OPEN, CONCURRENTAPPEND. The reason we have that is if the application is doing these operations heavily using explicit threads, then using async-await internally creates unecessary threads in threadpool and performance degrades due to context switching. Application can create explicit threads in cases of uploader and downloader. All these operation also call sync versions of MakeCall in WebTransport layer.

DirectoryEntry

Class that encapsulates the metadata of the directory entry

EnumerateDeletedItemsProgress

Class

LatencyTracker

Single entry, comma separated: 1. Client Request ID 2. latency in milliseconds 3. error code(if request failed) 4. Operation 5. Request+response body Size(if available, zero otherwise) 6. Instance of ADLStoreClient(a unique number per instance in this VM)

Multiple entries can be on a single request.Entries will be separated by semicolons
Limit max entries on a single request to three, to limit increase in HTTP request size.

All apis are threadsafe.

OperationResponse

Class encapsulates the response of one operation. Available operations are in Operation.cs For one operation, Http request can sent more than one time (More than one retry).

RequestOptions

This class contains the request parameters like timeout, Retry Policy and unique requestId

TrashEntry

Class that encapsulates the metadata of the trash entry

TrashStatus

Class

Enums

DirectoryEntryType

Enum containing two types of directory entry

ExpiryOption

Different types of expiry options

IfExists

Enum specifying actions to take while creating new file if the file exists

SyncFlag

Flags that are passed as parameters as a part of Http request for operations Write and Append

TrashEntryType

Enum containing two types of trash entry

UserGroupRepresentation

Enumerater defining how the user or group objects will be represented in calls that return group or user