Send Azure resource logs to Log Analytics workspaces, Event Hubs, or Azure Storage

Azure resource logs are platform logs that provide insight into operations that are performed in an Azure resource. The content of resource logs is different for each resource type. Resource logs aren't collected by default. To collect resource logs, you must enable and configure Diagnostic Settings or use data collection rules. For more information on data collection rules, see Data collection rules in Azure Monitor. This article describes the diagnostic setting required for each Azure resource to send its resource logs to Log Analytics workspaces, Event Hubs, or Azure Storage.

Send to Log Analytics workspace

Send resource logs to a Log Analytics workspace to enable the features of Azure Monitor Logs, where you can:

• Correlate resource log data with other monitoring data collected by Azure Monitor.
• Consolidate log entries from multiple Azure resources, subscriptions, and tenants into one location for analysis together.
• Use log queries to perform complex analysis and gain deep insights on log data.
• Use log search alerts with complex alerting logic.

Create a diagnostic setting to send resource logs to a Log Analytics workspace. This data is stored in tables as described in Structure of Azure Monitor Logs. The tables used by resource logs depend on what the resource type and the type of collection the resource is using. There are two types of collection modes for resource logs:

• Azure diagnostics: All data is written to the AzureDiagnostics table.
• Resource-specific: Data is written to individual tables for each category of the resource.

Resource-specific

For logs using resource-specific mode, individual tables in the selected workspace are created for each log category selected in the diagnostic setting. Resource-specific logs have the following advantages over Azure diagnostics logs:

• Makes it easier to work with the data in log queries.
• Provides better discoverability of schemas and their structure.
• Improves performance across ingestion latency and query times.
• Provides the ability to grant Azure role-based access control rights on a specific table.

Azure diagnostics mode

In Azure diagnostics mode, all data from any diagnostic setting is collected in the AzureDiagnostics table. This legacy method is used today by a minority of Azure services. Because multiple resource types send data to the same table, its schema is the superset of the schemas of all the different data types being collected. For details on the structure of this table and how it works with this potentially large number of columns, see AzureDiagnostics reference.

The AzureDiagnostics table contains the resourceId of the resource that generated the log, the category of the log, and the time the log was generated as well as resource specific properties.

Select the collection mode

Most Azure resources write data to the workspace in either Azure diagnostics or resource-specific mode without giving you a choice. For more information, see Common and service-specific schemas for Azure resource logs.

All Azure services will eventually use the resource-specific mode. As part of this transition, some resources allow you to select a mode in the diagnostic setting. Specify resource-specific mode for any new diagnostic settings because this mode makes the data easier to manage. It also might help you avoid complex migrations later.

You can modify an existing diagnostic setting to resource-specific mode. In this case, data that was already collected remains in the AzureDiagnostics table until it's removed according to your retention setting for the workspace. New data is collected in the dedicated table. Use the union operator to query data across both tables.

Continue to watch the Azure Updates blog for announcements about Azure services that support resource-specific mode.

Send to Azure Event Hubs

Send resource logs to an event hub to send them outside of Azure. For example, resource logs might be sent to a third-party SIEM or other log analytics solutions. Resource logs from event hubs are consumed in JSON format with a records element that contains the records in each payload. The schema depends on the resource type as described in Common and service-specific schema for Azure resource logs.

The following sample output data is from Azure Event Hubs for a resource log:

{
    "records": [
        {
            "time": "2019-07-15T18:00:22.6235064Z",
            "workflowId": "/SUBSCRIPTIONS/AAAA0A0A-BB1B-CC2C-DD3D-EEEEEE4E4E4E/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA",
            "resourceId": "/SUBSCRIPTIONS/AAAA0A0A-BB1B-CC2C-DD3D-EEEEEE4E4E4E/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA/RUNS/08587330013509921957/ACTIONS/SEND_EMAIL",
            "category": "WorkflowRuntime",
            "level": "Error",
            "operationName": "Microsoft.Logic/workflows/workflowActionCompleted",
            "properties": {
                "$schema": "2016-04-01-preview",
                "startTime": "2016-07-15T17:58:55.048482Z",
                "endTime": "2016-07-15T18:00:22.4109204Z",
                "status": "Failed",
                "code": "BadGateway",
                "resource": {
                    "subscriptionId": "AAAA0A0A-BB1B-CC2C-DD3D-EEEEEE4E4E4E",
                    "resourceGroupName": "JohnKemTest",
                    "workflowId": "2222cccc-33dd-eeee-ff44-aaaaaa555555",
                    "workflowName": "JohnKemTestLA",
                    "runId": "08587330013509921957",
                    "location": "chinanorth",
                    "actionName": "Send_email"
                },
                "correlation": {
                    "actionTrackingId": "3333dddd-44ee-ffff-aa55-bbbbbbbb6666",
                    "clientTrackingId": "08587330013509921958"
                }
            }
        },
        {
            "time": "2019-07-15T18:01:15.7532989Z",
            "workflowId": "/SUBSCRIPTIONS/AAAA0A0A-BB1B-CC2C-DD3D-EEEEEE4E4E4E/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA",
            "resourceId": "/SUBSCRIPTIONS/AAAA0A0A-BB1B-CC2C-DD3D-EEEEEE4E4E4E/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA/RUNS/08587330012106702630/ACTIONS/SEND_EMAIL",
            "category": "WorkflowRuntime",
            "level": "Information",
            "operationName": "Microsoft.Logic/workflows/workflowActionStarted",
            "properties": {
                "$schema": "2016-04-01-preview",
                "startTime": "2016-07-15T18:01:15.5828115Z",
                "status": "Running",
                "resource": {
                    "subscriptionId": "AAAA0A0A-BB1B-CC2C-DD3D-EEEEEE4E4E4E",
                    "resourceGroupName": "JohnKemTest",
                    "workflowId": "dddd3333-ee44-5555-66ff-777777aaaaaa",
                    "workflowName": "JohnKemTestLA",
                    "runId": "08587330012106702630",
                    "location": "chinanorth",
                    "actionName": "Send_email"
                },
                "correlation": {
                    "actionTrackingId": "ffff5555-aa66-7777-88bb-999999cccccc",
                    "clientTrackingId": "08587330012106702632"
                }
            }
        }
    ]
}

Send to Azure Storage

Send resource logs to Azure Storage to retain them for archiving. After you've created the diagnostic setting, a storage container is created in the storage account as soon as an event occurs in one of the enabled log categories.

The blobs within the container use the following naming convention:

insights-logs-{log category name}/resourceId=/SUBSCRIPTIONS/{subscription ID}/RESOURCEGROUPS/{resource group name}/PROVIDERS/{resource provider name}/{resource type}/{resource name}/y={four-digit numeric year}/m={two-digit numeric month}/d={two-digit numeric day}/h={two-digit 24-hour clock hour}/m=00/PT1H.json

The blob for a network security group might have a name similar to this example:

insights-logs-networksecuritygrouprulecounter/resourceId=/SUBSCRIPTIONS/a0a0a0a0-bbbb-cccc-dddd-e1e1e1e1e1e1/RESOURCEGROUPS/TESTRESOURCEGROUP/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUP/TESTNSG/y=2016/m=08/d=22/h=18/m=00/PT1H.json

Each PT1H.json blob contains a JSON object with events from log files that were received during the hour specified in the blob URL. During the present hour, events are appended to the PT1H.json file as they're received, regardless of when they were generated. The minute value in the URL, m=00 is always 00 as blobs are created on a per hour basis.

Within the PT1H.json file, each event is stored in the following format. It uses a common top-level schema but is unique for each Azure service, as described in Resource logs schema.

{"time": "2016-07-01T00:00:37.2040000Z","systemId": "a0a0a0a0-bbbb-cccc-dddd-e1e1e1e1e1e1","category": "NetworkSecurityGroupRuleCounter","resourceId": "/SUBSCRIPTIONS/AAAA0A0A-BB1B-CC2C-DD3D-EEEEEE4E4E4E/RESOURCEGROUPS/TESTRESOURCEGROUP/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/TESTNSG","operationName": "NetworkSecurityGroupCounters","properties": {"vnetResourceGuid": "{12345678-9012-3456-7890-123456789012}","subnetPrefix": "10.3.0.0/24","macAddress": "000123456789","ruleName": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/testresourcegroup/providers/Microsoft.Network/networkSecurityGroups/testnsg/securityRules/default-allow-rdp","direction": "In","type": "allow","matchedConnections": 1988}}

Next steps

• Read more about resource logs.
• Create diagnostic settings to send platform logs and metrics to different destinations.