使用 C# 为 Azure 数据资源管理器创建事件网格数据连接Create an Event Grid data connection for Azure Data Explorer by using C#

• 05/11/2021
• • o
  • a

Azure 数据资源管理器是一项快速且高度可缩放的数据探索服务，适用于日志和遥测数据。Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. Azure 数据资源管理器提供了从事件中心、IoT 中心和写入 blob 容器的 blob 引入数据（数据加载）的功能。Azure Data Explorer offers ingestion (data loading) from Event Hubs, IoT Hubs, and blobs written to blob containers.

在本文中，你将使用 C# 为 Azure 数据资源管理器创建事件网格数据连接。In this article, you create an Event Grid data connection for Azure Data Explorer by using C#.

必备条件Prerequisites

• 如果尚未安装 Visual Studio 2019，可以下载并使用 免费的 Visual Studio 2019 Community Edition。If you don't have Visual Studio 2019 installed, you can download and use the free Visual Studio 2019 Community Edition. 在安装 Visual Studio 的过程中，请确保启用“Azure 开发”。Make sure that you enable Azure development during the Visual Studio setup.
• 如果还没有 Azure 订阅，可以在开始前创建一个 Azure 帐户。If you don't have an Azure subscription, create a Azure account before you begin.
• 创建群集和数据库Create a cluster and database
• 创建表和列映射Create table and column mapping
• 设置数据库和表策略（可选）Set database and table policies (optional)
• 创建包含事件网格订阅的存储帐户。Create a storage account with an Event Grid subscription.

安装 C# NuGetInstall C# NuGet

• 安装 Microsoft.Azure.Management.Kusto NuGet 包。Install the Microsoft.Azure.Management.Kusto NuGet package.

AuthenticationAuthentication

若要运行以下示例，需要可以访问资源的 Azure Active Directory (Azure AD) 应用程序和服务主体。To run the following example, you need an Azure Active Directory (Azure AD) application and service principal that can access resources. 若要创建免费的 Azure AD 应用程序并在订阅级别添加角色分配，请参阅创建 Azure AD 应用程序。To create a free Azure AD application and add role assignment at the subscription level, see Create an Azure AD application. 还需要目录（租户）ID、应用程序 ID 和客户端密码。You also need the directory (tenant) ID, application ID, and client secret.

添加事件网格数据连接Add an Event Grid data connection

以下示例演示如何以编程方式添加事件网格数据连接。The following example shows you how to add an Event Grid data connection programmatically. 请参阅在 Azure 数据资源管理器中创建事件网格数据连接，以使用 Azure 门户添加事件网格数据连接。See create an Event Grid data connection in Azure Data Explorer for adding an Event Grid data connection using the Azure portal.

var tenantId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Directory (tenant) ID
var clientId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Application ID
var clientSecret = "xxxxxxxxxxxxxx";//Client Secret
var subscriptionId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";
var authenticationContext = new AuthenticationContext($"https://login.chinacloudapi.cn/{tenantId}");
var credential = new ClientCredential(clientId, clientSecret);
var result = await authenticationContext.AcquireTokenAsync(resource: "https://management.core.chinacloudapi.cn/", clientCredential: credential);

var credentials = new TokenCredentials(result.AccessToken, result.AccessTokenType);

var kustoManagementClient = new KustoManagementClient(credentials)
{
    SubscriptionId = subscriptionId
};

var resourceGroupName = "testrg";
//The cluster and database that are created as part of the Prerequisites
var clusterName = "mykustocluster";
var databaseName = "mykustodatabase";
var dataConnectionName = "myeventhubconnect";
//The event hub and storage account that are created as part of the Prerequisites
var eventHubResourceId = "/subscriptions/xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.EventHub/namespaces/xxxxxx/eventhubs/xxxxxx";
var storageAccountResourceId = "/subscriptions/xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Storage/storageAccounts/xxxxxx";
var consumerGroup = "$Default";
var location = "China East 2";
//The table and column mapping are created as part of the Prerequisites
var tableName = "StormEvents";
var mappingRuleName = "StormEvents_CSV_Mapping";
var dataFormat = DataFormat.CSV;
var blobStorageEventType = "Microsoft.Storage.BlobCreated";

await kustoManagementClient.DataConnections.CreateOrUpdateAsync(resourceGroupName, clusterName, databaseName, dataConnectionName,
    new EventGridDataConnection(storageAccountResourceId, eventHubResourceId, consumerGroup, tableName: tableName, location: location, mappingRuleName: mappingRuleName, dataFormat: dataFormat, blobStorageEventType: blobStorageEventType));

生成示例数据Generate sample data

连接 Azure 数据资源管理器和存储帐户后，可以创建示例数据并将其上传到存储。Now that Azure Data Explorer and the storage account are connected, you can create a sample data and upload it to the storage.

使用 Azure Blob 存储 SDK 上传文件Upload file using Azure Blob Storage SDK

下面的代码片段在存储帐户中创建新容器，将现有文件（作为 blob）上传到该容器，然后列出容器中的 blob。The following code snippet creates a new container in your storage account, uploads an existing file (as a blob) to that container, and then lists the blobs in the container.

var azureStorageAccountConnectionString=<storage_account_connection_string>;

var containerName = <container_name>;
var blobName = <blob_name>;
var localFileName = <file_to_upload>;
var uncompressedSizeInBytes = <uncompressed_size_in_bytes>;
var mapping = <mappingReference>;

// Creating the container
var azureStorageAccount = CloudStorageAccount.Parse(azureStorageAccountConnectionString);
var blobClient = azureStorageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference(containerName);
container.CreateIfNotExists();

// Set metadata and upload file to blob
var blob = container.GetBlockBlobReference(blobName);
blob.Metadata.Add("rawSizeBytes", uncompressedSizeInBytes);
blob.Metadata.Add("kustoIngestionMappingReference", mapping);
blob.UploadFromFile(localFileName);

// List blobs
var blobs = container.ListBlobs();

使用 Azure Data Lake SDK 上传文件Upload file using Azure Data Lake SDK

使用 Data Lake Storage Gen2 时，可以使用 Azure Data Lake SDK 将文件上传到存储。When working with Data Lake Storage Gen2, you can use Azure Data Lake SDK to upload files to storage. 以下代码片段使用 Azure.Storage.Files.DataLake v12.5.0 在 Azure Data Lake Storage 中创建新的文件系统，并将具有元数据的本地文件上传到该文件系统。The following code snippet uses Azure.Storage.Files.DataLake v12.5.0 to create a new filesystem in Azure Data Lake storage and to upload a local file with metadata to that filesystem.

var accountName = <storage_account_name>;
var accountKey = <storage_account_key>;
var fileSystemName = <file_system_name>;
var fileName = <file_name>;
var localFileName = <file_to_upload>;
var uncompressedSizeInBytes = <uncompressed_size_in_bytes>;
var mapping = <mapping_reference>;

var sharedKeyCredential = new StorageSharedKeyCredential(accountName, accountKey);
var dfsUri = "https://" + accountName + ".dfs.core.chinacloudapi.cn";
var dataLakeServiceClient = new DataLakeServiceClient(new Uri(dfsUri), sharedKeyCredential);

// Create the filesystem
var dataLakeFileSystemClient = dataLakeServiceClient.CreateFileSystem(fileSystemName).Value;

// Define metadata
IDictionary<String, String> metadata = new Dictionary<string, string>();
metadata.Add("rawSizeBytes", uncompressedSizeInBytes);
metadata.Add("kustoIngestionMappingReference", mapping);

// Set uploading options
var uploadOptions = new DataLakeFileUploadOptions
{
    Metadata = metadata,
    Close = true // Note: The close option triggers the event being processed by the data connection
};

// Write to the file
var dataLakeFileClient = dataLakeFileSystemClient.GetFileClient(fileName);
dataLakeFileClient.Upload(localFileName, uploadOptions);

使用 Azure Data Lake SDK 重命名文件Rename file using Azure Data Lake SDK

以下代码片段使用 Azure Data Lake SDK 重命名 ADLSv2 存储帐户中的 blob。The following code snippet uses Azure Data Lake SDK to rename a blob in an ADLSv2 storage account.

var accountName = <storage_account_name>;
var accountKey = <storage_account_key>;
var fileSystemName = <file_system_name>;
var sourceFilePath = <source_file_path>;
var destinationFilePath = <destination_file_path>;

var sharedKeyCredential = new StorageSharedKeyCredential(accountName, accountKey);
var dfsUri = "https://" + accountName + ".dfs.core.chinacloudapi.cn";
var dataLakeServiceClient = new DataLakeServiceClient(new Uri(dfsUri), sharedKeyCredential);

// Get a client to the the filesystem
var dataLakeFileSystemClient = dataLakeServiceClient.GetFileSystemClient(fileSystemName);

// Rename a file in the file system
var dataLakeFileClient = dataLakeFileSystemClient.GetFileClient(sourceFilePath);
dataLakeFileClient.Rename(destinationFilePath);

清理资源Clean up resources

若要删除数据连接，请使用以下命令：To delete the data connection, use the following command:

kustoManagementClient.DataConnections.Delete(resourceGroupName, clusterName, databaseName, dataConnectionName);