How to log events to Azure Event Hubs in Azure API Management

  • Article

APPLIES TO: All API Management tiers

This article describes how to log API Management events using Azure Event Hubs.

Azure Event Hubs is a highly scalable data ingress service that can ingest millions of events per second so that you can process and analyze the massive amounts of data produced by your connected devices and applications. Event Hubs acts as the "front door" for an event pipeline, and once data is collected into an event hub, it can be transformed and stored using any real-time analytics provider or batching/storage adapters. Event Hubs decouples the production of a stream of events from the consumption of those events, so that event consumers can access the events on their own schedule.

Prerequisites

Configure access to the event hub

To log events to the event hub, you need to configure credentials for access from API Management. API Management supports either of the two following access mechanisms:

  • A managed identity for your API Management instance (recommended)
  • An Event Hubs connection string

Note

Where possible, Microsoft recommends using managed identity credentials for enhanced security.

Option 1: Configure API Management managed identity

  1. Enable a system-assigned or user-assigned managed identity for API Management in your API Management instance.

    • If you enable a user-assigned managed identity, take note of the identity's Client ID.

  2. Assign the identity the Azure Event Hubs Data sender role, scoped to the Event Hubs namespace or to the event hub used for logging. To assign the role, use the Azure portal or other Azure tools.

Option 2: Configure Event Hubs connection string

To create an Event Hubs connection string, see Get an Event Hubs connection string.

  • You can use a connection string for the Event Hubs namespace or for the specific event hub you use for logging from API Management.
  • The shared access policy for the connection string must enable at least Send permissions.

Create an API Management logger

The next step is to configure a logger in your API Management service so that it can log events to the event hub.

Create and manage API Management loggers by using the API Management REST API directly or by using tools including Azure PowerShell, a Bicep template, or an Azure Resource Management template.

You can configure an API Management logger to an event hub using either system-assigned or user-assigned managed identity credentials.

Logger with system-assigned managed identity credentials

For prerequisites, see Configure API Management managed identity.

Use the API Management Logger - Create or Update REST API with the following request body.

{
  "properties": {
    "loggerType": "azureEventHub",
    "description": "Event Hub logger with system-assigned managed identity",
    "credentials": {
         "endpointAddress":"<EventHubsNamespace>.servicebus.chinacloudapi.cn",
         "identityClientId":"SystemAssigned",
         "name":"<EventHubName>"
    }
  }
}

Logger with user-assigned managed identity credentials

For prerequisites, see Configure API Management managed identity.

Use the API Management Logger - Create or Update REST API with the following request body.

{
  "properties": {
    "loggerType": "azureEventHub",
    "description": "Event Hub logger with user-assigned managed identity",
    "credentials": {
         "endpointAddress":"<EventHubsNamespace>.servicebus.chinacloudapi.cn",
         "identityClientId":"<ClientID>",
         "name":"<EventHubName>"
    }
  }
}

Option 2. Logger with connection string credentials

For prerequisites, see Configure Event Hubs connection string.

Note

Where possible, Microsoft recommends configuring the logger with managed identity credentials. See Configure logger with managed identity credentials, earlier in this article.

The following example uses the New-AzApiManagementLogger cmdlet to create a logger to an event hub by configuring a connection string.

# API Management service-specific details
$apimServiceName = "apim-hello-world"
$resourceGroupName = "myResourceGroup"

# Create logger
$context = New-AzApiManagementContext -ResourceGroupName $resourceGroupName -ServiceName $apimServiceName
New-AzApiManagementLogger -Context $context -LoggerId "ContosoLogger1" -Name "ApimEventHub" -ConnectionString "Endpoint=sb://<EventHubsNamespace>.servicebus.chinacloudapi.cn/;SharedAccessKeyName=<KeyName>;SharedAccessKey=<key>" -Description "Event hub logger with connection string"

Configure log-to-eventhub policy

Once your logger is configured in API Management, you can configure your log-to-eventhub policy to log the desired events. For example, use the log-to-eventhub policy in the inbound policy section to log requests, or in the outbound policy section to log responses.

  1. Browse to your API Management instance.

  2. Select APIs, and then select the API to which you want to add the policy. In this example, we're adding a policy to the Echo API in the Unlimited product.

  3. Select All operations.

  4. On the top of the screen, select the Design tab.

  5. In the Inbound processing or Outbound processing window, select the </> (code editor) icon. For more information, see How to set or edit policies.

  6. Position your cursor in the inbound or outbound policy section.

  7. In the window on the right, select Advanced policies > Log to EventHub. This inserts the log-to-eventhub policy statement template.

    <log-to-eventhub logger-id="logger-id">
    @{
        return new JObject(
            new JProperty("EventTime", DateTime.UtcNow.ToString()),
            new JProperty("ServiceName", context.Deployment.ServiceName),
            new JProperty("RequestId", context.RequestId),
            new JProperty("RequestIp", context.Request.IpAddress),
            new JProperty("OperationName", context.Operation.Name)
        ).ToString();
    }
</log-to-eventhub>
    1. Replace logger-id with the name of the logger that you created in the previous step.
    2. You can use any expression that returns a string as the value for the log-to-eventhub element. In this example, a string in JSON format containing the date and time, service name, request ID, request IP address, and operation name is logged.

  8. Select Save to save the updated policy configuration. As soon as it's saved, the policy is active and events are logged to the designated event hub.

Note

The maximum supported message size that can be sent to an event hub from this API Management policy is 200 kilobytes (KB). If a message that is sent to an event hub is larger than 200 KB, it will be automatically truncated, and the truncated message will be transferred to the event hub. For larger messages, consider using Azure Storage with Azure API Management as a workaround to bypass the 200KB limit.

Preview the log in Event Hubs by using Azure Stream Analytics

You can preview the log in Event Hubs by using Azure Stream Analytics queries.

  1. In the Azure portal, browse to the event hub that the logger sends events to.
  2. Under Features, select the Process data tab.
  3. On the Enable real time insights from events card, select Start.
  4. You should be able to preview the log on the Input preview tab. If the data shown isn't current, select Refresh to see the latest events.

Next steps