Manually create resources for Event Grid ingestion

Azure Data Explorer offers continuous ingestion from Azure Storage (Azure Blob storage and Azure Data Lake Storage Gen2) using an Event Grid Ingestion pipeline. In the Event Grid ingestion pipeline, an Azure Event Grid service routes blob created or blob renamed events from a storage account to Azure Data Explorer via an Azure Event Hubs.

In this article, you learn how to manually create the resources needed for Event Grid Ingestion: an Event Grid subscription, an Event Hubs namespace, and an event hub. Event Hubs namespace and event hub creation are described in the Prerequisites. To use automatic creation of these resources while defining the Event Grid ingestion, see Create an Event Grid data connection in Azure Data Explorer.

Prerequisites

• An Azure subscription. Create a Azure account.
• An Azure Data Explorer cluster and database. Create a cluster and database.
• A storage account. An Event Grid notification subscription can be set on Azure Storage accounts for BlobStorage, StorageV2, or Data Lake Storage Gen2.
• An Event Hubs namespace and event hub.

Create an Event Grid subscription

1. In the Azure portal, go to your storage account.

2. In the left menu, select Events > Event Subscription.

   

3. In the Create Event Subscription window within the Basic tab, provide the following values:

   

   Setting	Suggested value	Field description
   Name	test-grid-connection	The name of the Event Grid subscription that you want to create.
   Event Schema	Event Grid Schema	The schema that should be used for the Event Grid.
   Topic Type	Storage account	The type of Event Grid topic. Automatically populated.
   Source Resource	gridteststorage1	The name of your storage account. Automatically populated.
   System Topic Name	gridteststorage1...	The system topic where Azure Storage publishes events. This system topic then forwards the event to a subscriber that receives and processes events. Automatically populated.
   Filter to Event Types	Blob Created	Which specific events to get notified for. When creating the subscription, select one of the supported types: Microsoft.Storage.BlobCreated or Microsoft.Storage.BlobRenamed. Blob renaming is supported only for ADLSv2 storage.

4. In ENDPOINT DETAILS, select Event Hubs.

   

5. Click Select an endpoint and fill in the event hub you created, for example test-hub.

6. Select the Filters tab if you want to filter events.

   Use Subject Filters to track specific subjects events. Set the filters for the notifications as follows:

   

   1. Select Enable subject filtering
   2. Subject Begins With field is the literal prefix of the subject. Since the pattern applied is startswith, it can span multiple containers, folders, or blobs. No wildcards are allowed.
      • To define a filter on the blob container, set the field as follows: /blobServices/default/containers/[container prefix].
      • To define a filter on a blob prefix (or a folder in Azure Data Lake Gen2), set the field as follows: /blobServices/default/containers/[container name]/blobs/[folder/blob prefix].
   3. Subject Ends With field is the literal suffix of the blob. No wildcards are allowed.
   4. Case-sensitive subject matching field indicates whether the prefix and suffix filters are case-sensitive.

   For more information about filtering events, see blob storage events.

   Use Advanced Filters to add custom filters that meet your requirements.

   For example, When using the Azure Data Lake SDK to upload a file, file creation triggers an Event Grid event with size 0. This event is discarded by Azure Data Explorer. File flushing triggers another event if the Close parameter is set to true. This event indicates that this is the final update and the file stream has been closed. To avoid unnecessary processing and empty file ingestion errors, filter out CreateFile events using the following filter:

   

   Setting	Suggested value	Field description
   Key	data.api	The field in the event schema used for filtering.
   Operator	String is not in	An operator to be evaluates on the selected key.
   Value	CreateFile	The value used to evaluate the key.

7. Select Create

Next step