Create an Event Hubs data connection for Azure Data Explorer by using Python
Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. Azure Data Explorer offers ingestion (data loading) from event hubs, IoT hubs, and blobs written to blob containers.
In this article, you create an Event Hubs data connection for Azure Data Explorer by using Python.
- An Azure subscription. Create a Azure account.
- Create a cluster and database.
- Python 3.4+.
- Table and column mapping.
- Database and table policies (optional).
- Event Hubs with data for ingestion.
Install Python package
To install the Python package for Azure Data Explorer (Kusto), open a command prompt that has Python in its path. Run the following command:
pip install azure-common pip install azure-mgmt-kusto
To run the following example, you need an Azure Active Directory (Azure AD) application and service principal that can access resources. To create a free Azure AD application and add role assignment at the subscription level, see Create an Azure AD application. You also need the directory (tenant) ID, application ID, and client secret.
Add an Event Hubs data connection
The following example shows you how to add an Event Hubs data connection programmatically. See connect to the event hub for adding an Event Hubs data connection using the Azure portal.
from azure.mgmt.kusto import KustoManagementClient from azure.mgmt.kusto.models import EventHubDataConnection from azure.identity import ClientSecretCredential #Directory (tenant) ID tenant_id = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx" #Application ID client_id = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx" #Client Secret client_secret = "xxxxxxxxxxxxxx" subscription_id = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx" credentials = ServicePrincipalCredentials( client_id=client_id, secret=client_secret, tenant=tenant_id ) kusto_management_client = KustoManagementClient(credentials, subscription_id) resource_group_name = "myresourcegroup" #The cluster and database that are created as part of the Prerequisites cluster_name = "mycluster" database_name = "mydatabase" data_connection_name = "myeventhubconnect" #The event hub that is created as part of the Prerequisites event_hub_resource_id = "/subscriptions/xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx/resourceGroups/myresourcegroup/providers/Microsoft.EventHub/namespaces/myeventhubnamespace/eventhubs/myeventhub""; consumer_group = "$Default" location = "China East 2" #The table and column mapping that are created as part of the Prerequisites table_name = "mytable" mapping_rule_name = "mytablemappingrule" data_format = "csv" database_routing = "Multi" #Returns an instance of LROPoller, check https://docs.microsoft.com/python/api/msrest/msrest.polling.lropoller?view=azure-python poller = kusto_management_client.data_connections.create_or_update( resource_group_name=resource_group_name, cluster_name=cluster_name, database_name=database_name, data_connection_name=data_connection_name, parameters=EventHubDataConnection( event_hub_resource_id=event_hub_resource_id, consumer_group=consumer_group, location=location, table_name=table_name, mapping_rule_name=mapping_rule_name, data_format=data_format, database_routing=database_routing ) ) poller.wait() print(poller.result())
|Setting||Suggested value||Field description|
|tenant_id||xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx||Your tenant ID. Also known as directory ID.|
|subscriptionId||xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx||The subscription ID that you use for resource creation.|
|client_id||xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx||The client ID of the application that can access resources in your tenant.|
|client_secret||xxxxxxxxxxxxxx||The client secret of the application that can access resources in your tenant.|
|resource_group_name||myresourcegroup||The name of the resource group containing your cluster.|
|cluster_name||mycluster||The name of your cluster.|
|database_name||mydatabase||The name of the target database in your cluster.|
|data_connection_name||myeventhubconnect||The desired name of your data connection.|
|table_name||mytable||The name of the target table in the target database.|
|mapping_rule_name||mytablemappingrule||The name of your column mapping related to the target table.|
|data_format||csv||The data format of the message.|
|event_hub_resource_id||Resource ID||The resource ID of your event hub that holds the data for ingestion.|
|consumer_group||$Default||The consumer group of your event hub.|
|location||China East 2||The location of the data connection resource.|
|databaseRouting||Multi or Single||The database routing for the connection. If you set the value to Single, the data connection will be routed to a single database in the cluster as specified in the databaseName setting. If you set the value to Multi, you can override the default target database using the Database ingestion property. For more information, see Events routing.|
Clean up resources
To delete the data connection, use the following command:
kusto_management_client.data_connections.delete(resource_group_name=resource_group_name, cluster_name=kusto_cluster_name, database_name=kusto_database_name, data_connection_name=kusto_data_connection_name)