Collect events and performance counters from virtual machines with Azure Monitor Agent

  • Article

This article describes how to collect events and performance counters from virtual machines by using Azure Monitor Agent.

Prerequisites

To complete this procedure, you need:

Create a data collection rule

You can define a data collection rule to send data from multiple machines to multiple Log Analytics workspaces, including workspaces in a different region or tenant. Create the data collection rule in the same region as your Log Analytics workspace. You can send Windows event and Syslog data to Azure Monitor Logs only. You can send performance counters to both Azure Monitor Metrics and Azure Monitor Logs.

Note

At this time, Microsoft.HybridCompute (Azure Arc-enabled servers) resources can't be viewed in Metrics Explorer (the Azure portal UX), but they can be acquired via the Metrics REST API (Metric Namespaces - List, Metric Definitions - List, and Metrics - List).

Note

To send data across tenants, you must first enable Azure Lighthouse.

  1. On the Monitor menu, select Data Collection Rules.

  2. Select Create to create a new data collection rule and associations.

    Screenshot that shows the Create button on the Data Collection Rules screen.

  3. Enter a Rule name and specify a Subscription, Resource Group, Region, and Platform Type:

    • Region specifies where the DCR will be created. The virtual machines and their associations can be in any subscription or resource group in the tenant.
    • Platform Type specifies the type of resources this rule can apply to. The Custom option allows for both Windows and Linux types.

    Screenshot that shows the Basics tab of the Data Collection Rule screen.

  4. On the Resources tab:

    1. Select + Add resources and associate resources to the data collection rule. Resources can be virtual machines, Virtual Machine Scale Sets, and Azure Arc for servers. The Azure portal installs Azure Monitor Agent on resources that don't already have it installed.

      Important

      The portal enables system-assigned managed identity on the target resources, along with existing user-assigned identities, if there are any. For existing applications, unless you specify the user-assigned identity in the request, the machine defaults to using system-assigned identity instead.

      If you need network isolation using private links, select existing endpoints from the same region for the respective resources or create a new endpoint.

    2. Select Data Collection Endpoint which is an optional field that does not need to be set unless you plan to use Azure Monitor Private Links. If you need it, specifies the data collection endpoint used to collect data. This data collection endpoint must be in the same region as the Log Analytics workspace. For more information, see How to set up data collection endpoints based on your deployment.

    3. Select a data collection endpoint for each of the resources associate to the data collection rule.

    Screenshot that shows the Resources tab of the Data Collection Rule screen.

  6. On the Collect and deliver tab, select Add data source to add a data source and set a destination.

  7. Select a Data source type.

  8. Select which data you want to collect. For performance counters, you can select from a predefined set of objects and their sampling rate. For events, you can select from a set of logs and severity levels.

    Screenshot that shows the Azure portal form to select basic performance counters in a data collection rule.

  9. Select Custom to collect logs and performance counters that aren't currently supported data sources or to filter events by using XPath queries. You can then specify an XPath to collect any specific values.

    To collect a performance counter that's not available by default, use the format \PerfObject(ParentInstance/ObjectInstance#InstanceIndex)\Counter. If the counter name contains an ampersand (&), replace it with &. For example, \Memory\Free & Zero Page List Bytes.

    For examples of DCRs, see Sample data collection rules (DCRs) in Azure Monitor.

    Screenshot that shows the Azure portal form to select custom performance counters in a data collection rule.

  10. On the Destination tab, add one or more destinations for the data source. You can select multiple destinations of the same or different types. For instance, you can select multiple Log Analytics workspaces, which is also known as multihoming.

    You can send Windows event and Syslog data sources to Azure Monitor Logs only. You can send performance counters to both Azure Monitor Metrics and Azure Monitor Logs. At this time, hybrid compute (Arc for Server) resources do not support the Azure Monitor Metrics (Preview) destination.

    Screenshot that shows the Azure portal form to add a data source in a data collection rule.

  11. Select Add data source and then select Review + create to review the details of the data collection rule and association with the set of virtual machines.

  12. Select Create to create the data collection rule.

Parameter file
{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "vmName": {
      "value": "my-azure-vm"
    },
    "associationName": {
      "value": "my-windows-vm-my-dcr"
    },
    "dataCollectionRuleId": {
      "value": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/my-resource-group/providers/microsoft.insights/datacollectionrules/my-dcr"
    }
   }
}

Note

It can take up to 5 minutes for data to be sent to the destinations after you create the data collection rule.

Filter events using XPath queries

You're charged for any data you collect in a Log Analytics workspace. Therefore, you should only collect the event data you need. The basic configuration in the Azure portal provides you with a limited ability to filter out events.

Tip

For strategies to reduce your Azure Monitor costs, see Cost optimization and Azure Monitor.

To specify more filters, use custom configuration and specify an XPath that filters out the events you don't need. XPath entries are written in the form LogName!XPathQuery. For example, you might want to return only events from the Application event log with an event ID of 1035. The XPathQuery for these events would be *[System[EventID=1035]]. Because you want to retrieve the events from the Application event log, the XPath is Application!*[System[EventID=1035]]

Extract XPath queries from Windows Event Viewer

In Windows, you can use Event Viewer to extract XPath queries as shown in the screenshots.

When you paste the XPath query into the field on the Add data source screen, as shown in step 5, you must append the log type category followed by an exclamation point (!).

Screenshot that shows the steps to create an XPath query in the Windows Event Viewer.

Tip

You can use the PowerShell cmdlet Get-WinEvent with the FilterXPath parameter to test the validity of an XPath query locally on your machine first. For more information, see the tip provided in the Windows agent-based connections instructions. The Get-WinEvent PowerShell cmdlet supports up to 23 expressions. Azure Monitor data collection rules support up to 20. The following script shows an example:

$XPath = '*[System[EventID=1035]]'
Get-WinEvent -LogName 'Application' -FilterXPath $XPath
  • In the preceding cmdlet, the value of the -LogName parameter is the initial part of the XPath query until the exclamation point (!). The rest of the XPath query goes into the $XPath parameter.
  • If the script returns events, the query is valid.
  • If you receive the message "No events were found that match the specified selection criteria," the query might be valid but there are no matching events on the local machine.
  • If you receive the message "The specified query is invalid," the query syntax is invalid.

Examples of using a custom XPath to filter events:

Description XPath
Collect only System events with Event ID = 4648 System!*[System[EventID=4648]]
Collect Security Log events with Event ID = 4648 and a process name of consent.exe Security!*[System[(EventID=4648)]] and *[EventData[Data[@Name='ProcessName']='C:\Windows\System32\consent.exe']]
Collect all Critical, Error, Warning, and Information events from the System event log except for Event ID = 6 (Driver loaded) System!*[System[(Level=1 or Level=2 or Level=3) and (EventID != 6)]]
Collect all success and failure Security events except for Event ID 4624 (Successful logon) Security!*[System[(band(Keywords,13510798882111488)) and (EventID != 4624)]]

Note

For a list of limitations in the XPath supported by Windows event log, see XPath 1.0 limitations.
For instance, you can use the "position", "Band", and "timediff" functions within the query but other functions like "starts-with" and "contains" are not currently supported.

Frequently asked questions

This section provides answers to common questions.

How can I collect Windows security events by using Azure Monitor Agent?

There are two ways you can collect Security events using the new agent, when sending to a Log Analytics workspace:

  • You can use Azure Monitor Agent to natively collect Security Events, same as other Windows Events. These flow to the 'Event' table in your Log Analytics workspace.
  • If you have Microsoft Sentinel enabled on the workspace, the security events flow via Azure Monitor Agent into the SecurityEvent table instead (the same as using the Log Analytics agent). This scenario always requires the solution to be enabled first.

Will I duplicate events if I use Azure Monitor Agent and the Log Analytics agent on the same machine?

If you're collecting the same events with both agents, duplication occurs. This duplication could be the legacy agent collecting redundant data from the workspace configuration data, which is collected by the data collection rule. Or you might be collecting security events with the legacy agent and enable Windows security events with Azure Monitor Agent connectors in Microsoft Sentinel.

Limit duplication events to only the time when you transition from one agent to the other. After you've fully tested the data collection rule and verified its data collection, disable collection for the workspace and disconnect any Microsoft Monitoring Agent data connectors.

Does Azure Monitor Agent offer more granular event filtering options other than Xpath queries and specifying performance counters?

For Syslog events on Linux, you can select facilities and the log level for each facility.

If I create data collection rules that contain the same event ID and associate them to the same VM, will events be duplicated?

Yes. To avoid duplication, make sure the event selection you make in your data collection rules doesn't contain duplicate events.

Next steps