Configure Event Hubs with Microsoft Purview to send and receive Atlas Kafka topics messages

This article will show you how to configure Microsoft Purview to be able to send and receive Atlas Kafka topics events with Azure Event Hubs.

If you have already configured your environment, you can follow our guide to get started with the Azure.Messaging.EventHubs .NET library to send and receive messages.

Prerequisites

To configure your environment, you need certain prerequisites in place:

• A Azure subscription. To use Azure services, including Event Hubs, you need an Azure subscription. If you don't have an Azure account, you can sign up for a trial or use your MSDN subscriber benefits when you create an account.
• An active Microsoft Purview account.

Configure Event Hubs

To send or receive Atlas Kafka topics messages, you'll need to configure at least one Event Hubs namespace. You can either:

• Create your own Azure Event Hubs namespace and configure it to work with Microsoft Purview
• Use the REST API to create a managed namespace

Bring your own Event Hubs namespace

You can configure an existing Azure Event Hubs namespace with an Event Hubs or create a new one to connect with Microsoft Purview.

Event Hubs permissions

To authenticate with your Event Hubs, you can either use:

• Microsoft Purview managed identity(Recommended)
• User assigned managed identity - only available when configuring namespaces after account creation and if using Event Hubs with an open network.

These identities will need at least contributor permissions on your Event Hubs to be able to configure them to use with Microsoft Purview.

Event Hubs networking

1. If your Microsoft Purview account is using private endpoints, set up an ingestion private endpoint to connect to Event Hubs.

2. Your Event Hubs workspace network needs to be configured in one of two ways:

   • Public network access set to Selected networks or Disabled, Allow trusted Microsoft services to bypass this firewall set to Yes, and using the Microsoft Purview managed identity to authenticate.
   • Public network access set to All networks and using the Microsoft Purview managed identity or a user assigned managed identity to authenticate.

   Microsoft Purview's status as a trusted service means that, as long as Allow trusted Microsoft services to bypass this firewall is set to Yes on your Event Hubs resource, your hooks can still receive messages from Event Hubs.

For more information about networking for Event Hubs, see the Azure Event Hubs documentation.

Configure Event Hubs to publish messages to Microsoft Purview

1. Navigate to Kafka configuration under settings on your Microsoft Purview account page in the Azure portal.

   

2. Select Add configuration and Hook configuration.

   Note

   You can add as many hook configurations as you need.

   

3. Give a name to your hook configuration, select your subscription, an existing Event Hubs namespace, an existing Event Hubs to send the notifications to, the consumer group you want to use, and the kind of authentication you would like to use.

   Tip

   You can use the same Event Hubs namespace more than once, but each configuration will need its own Event Hubs.

   

4. Select Save. It will take a couple minutes for your configuration to complete.

5. Once configuration is complete, you can begin the steps to publish messages to Microsoft Purview.

Configure Event Hubs to receive messages from Microsoft Purview

1. Navigate to Kafka configuration under settings on your Microsoft Purview account page in the Azure portal.

   

2. If there's a configuration already listed as type Notification, Event Hubs is already configured, and you can begin the steps to receive Microsoft Purview messages.

   Note

   Only one Notification Event Hubs can be configured at a time.

   

3. If there isn't a Notification configuration already listed, select Add configuration and Notification configuration.

   

4. Give a name to your notification configuration, select your subscription, an existing Event Hubs namespace, an existing Event Hubs to send the notifications to, the partitions you want to use, and the kind of authentication you would like to use.

   Tip

   You can use the same Event Hubs namespace more than once, but each configuration will need its own Event Hubs.

   

5. Select Save. It will take a couple minutes for your configuration to complete.

6. Once configuration is complete, you can begin the steps to receive Microsoft Purview messages.

Remove configured Event Hubs

To remove configured Event Hubs namespaces, you can follow these steps:

1. Search for and open your Microsoft Purview account in the Azure portal.

2. Select Kafka configuration under settings on your Microsoft Purview account page in the Azure portal.

3. Select the Event Hubs you want to disable. (Hook hubs send messages to Microsoft Purview. Notification hubs receive notifications.)

4. Select Remove to save the choice and begin the disablement process. This can take several minutes to complete.

   

Create a managed namespace

If you prefer to have a managed Event Hubs namespace, you'll need to use the REST API to configure your account.

You can use below script (set apiVersion = 2021-12-01) to update an account and this way will only affect the event hub configuration:

$body = @{"properties" = @{ "managedEventHubState" = "Enabled" }; "location" = $($location); "identity" = @{ "type"= $($type) } } | ConvertTo-Json
$Token=Get-AzAccessToken
$Headers = @{ "Authorization" = "Bearer $($Token.Token)" }
$Uri = "https://management.chinacloudapi.cn/subscriptions/$($subscription)/resourceGroups/$($rg)/providers/Microsoft.Purview/accounts/$($accountName)?api-version=$($apiVersion)"

Invoke-WebRequest -URI $Uri -Method Put -Body $body -Headers $Headers -ContentType "application/json"

Next steps

• Publish and process Atlas Kafka topics messages using your Event Hubs
• Event Hubs samples in GitHub
• Event processor samples in GitHub
• An introduction to Atlas notifications