Monitor a single Azure Stack HCI cluster with Insights

Starting with the May 2023 cumulative update for Azure Stack HCI, version 22H2, you can monitor on-premises Azure Stack HCI systems with Insights for Azure Stack HCI.

Prerequisites

Here are the prerequisites for using Insights for Azure Stack HCI:

• Azure Stack HCI cluster should be registered with Azure and Arc-enabled. If you registered your cluster on or after June 15, 2021, this happens by default. Otherwise, you must enable Azure Arc integration.

• The cluster must have Azure Stack HCI, version 22H2 and the May 2023 cumulative update or later installed.

• The managed identity for the Azure resource must be enabled. For more information, see Enabled enhanced management.

Enable Insights

Enabling Insights helps you monitor all Azure Stack HCI clusters currently associated with the Log Analytics workspace by providing useful health metrics. Insights installs the Azure Monitor Agent and helps you to configure data collection rules (DCRs) for monitoring your Azure Stack HCI cluster.

To enable this capability from the Azure portal, follow these steps:

1. In the Azure portal, browse to your Azure Stack HCI cluster resource page, then select your cluster. Under the Capabilities tab, select Insights.

   

2. On the Insights page, select Get Started.

   

   Note

   The Get Started button is available only for Azure Stack HCI, version 22H2 with the May 2023 cumulative update or later installed and only after the managed identity is enabled. Otherwise, this button is disabled.

3. On the Insights configuration page, select an existing DCR from the Data collection rule dropdown. The DCR specifies the event logs and performance counters that need to be collected and stores it in a Log Analytics workspace. Insights creates a default DCR if one doesn't already exist. Only the DCRs that are enabled for Insights are included.

   

4. (Optional) You can also create a new DCR by selecting Create New on the Insights configuration page.

   Important

   We strongly recommend that you don't create your own DCR. The DCR created by Insights includes a special data stream required for its operation. You can edit this DCR to collect more data, such as Windows and Syslog events. The DCRs created through AMA installation will have a prefix AzureStackHCI- attached with the DCR name.

   1. On the New data collection rule page, specify the subscription, DCR name, and data collection endpoint (DCE) name. DCEs are used to access the configuration service to fetch associated DCRs for Azure Monitor Agent. For more information about DCE, see Data collection endpoints in Azure Monitor.

      Note

      If you're using private links on the agent, you must add DCEs. For more information about AMA network settings, see Define Azure Monitor Agent network settings.

      

   2. Select the Review + create button.

      If a DCR isn't already created for the unmonitored cluster, then one is created with performance counters enabled and the Windows event log channel enabled.

5. Review the final screen with a summary of DCR name, number of event logs, performance counters, and name of the Log Analytics workspace in which data is stored. Select Set up.

   

   After selecting Set up, you're redirected to the Extensions page, where you can see the status of your agent installation. By configuring Insights, AMA is automatically installed on all nodes of the cluster.

6. Go to your Azure Stack HCI cluster resource page, and then select your cluster. Insights now shows as Configured on the Capabilities tab:

   

Data collection rules

When you enable Insights on a machine with the Azure Monitor Agent, you must specify a DCR to use. For more information about DCRs, see Data collection rules in Azure Monitor.

Event channel

The Microsoft-windows-sddc-management/operational and Microsoft-windows-health/operational Windows event channel is added to your Log Analytics workspace under Windows event logs.

By collecting these logs, Insights shows the health status of the individual servers, drives, volumes, and VMs. By default, five performance counters are added.

Performance counters

By default, five performance counters are added:

The following table describes the performance counters that are monitored:

After you enable Insights, it can take up to 15 minutes to collect the data. When the process is finished, you are able to see a rich visualization of the health of your cluster from the Insights menu on the left pane:

Disable Insights

To disable Insights, follow these steps:

1. Select Insights under the Capabilities tab.

2. Select Disable Insights.

   

When you disable the Insights feature, the association between the data collection rule and the cluster is deleted and the Health Service and SDDC Management logs are no longer collected; however, existing data isn't deleted. If you'd like to delete that data, go into your DCR and Log Analytics workspace and delete the data manually.

Update Insights

The Insights tile shows a Needs update message in the following cases:

• A data collection rule is changed.
• A health event from the Windows events log is deleted.
• Any of the five performance counters from the Log Analytics workspace are deleted.

To enable Insights again, follow these steps:

1. Select the Insights tile under Capabilities.

2. Select Update to see the visualizations again.

   

Migrate from the Microsoft Monitoring Agent

1. To migrate from the Microsoft Monitoring Agent (MMA) to the Azure Monitoring Agent (AMA), scroll down to Insights.

   

2. Select Install AMA; the Insights configuration window opens.

   

3. Select or create a data collection rule as described previously in the Enable Insights section.

The Azure Monitor Agent and the Microsoft Monitoring Agent extension can both be installed on the same computer during migration. Running both agents might lead to duplication of data and increased cost. If a machine has both agents installed, you see a warning in the Azure portal that you might be collecting duplicate data, as shown in the following screenshot.

You must remove the Microsoft Monitoring Agent extension yourself from any computers that are using it. Before you do this step, ensure that the computer isn't relying on any other solutions that require the Microsoft Monitoring Agent. After you verify that MicrosoftMonitoringAgent isn't still connected to your Log Analytics workspace, you can remove MicrosoftMonitoringAgent manually by redirecting to the Extensions page.

Troubleshoot

This section gives guidance for resolving the issues with using Insights for Azure Stack HCI.

Troubleshoot clusters registered before November 2023

Issue. In clusters that are registered before November 2023, features that use AMA on Azure Stack HCI, such as Arc for Servers, VM Insights, Container Insights, Defender for Cloud, and Sentinel might not collect logs and event data properly.

Cause. Before November 2023, the cluster registration configured AMA to use cluster identity, while the services that use AMA on Azure Stack HCI required the cluster node's identity for proper log collection. This mismatch resulted in improper collection of logs from these services.

Solution. To address this issue, we made a change in the HCI cluster registration for AMA to use the server identity instead. To implement this change, perform the following steps on clusters that are registered before November 2023:

1. Repair cluster registration. See Repair cluster registration.
2. Repair AMA. See Repair AMA.
3. Reconfigure Insights for Azure Stack HCI. See Reconfigure Insights for Azure Stack HCI.

In the Azure portal, the Insights for Azure Stack HCI page automatically detects the change in the AMA configuration and displays a banner at the top of the page, guiding you to take necessary actions to continue using services that rely on AMA.

Repair cluster registration

Follow these steps to repair cluster registration:

1. On your cluster nodes, install the latest Az.StackHCI PowerShell module. Replace latestversion with the latest Az.StackHCI version number.

   Install-Module -Name Az.StackHCI -RequiredVersion {latestversion} -Scope CurrentUser -Repository PSGallery -Force 
   

2. Run the repair registration command to remove regkey:

   Register-AzStackHCI -TenantId {TenantID} -SubscriptionId {subscriptionID} -ComputerName {NodeName} -RepairRegistration 
   

Repair AMA for Azure Stack HCI

Choose one of the following options to repair AMA:

• Option 1: Uninstall AMA

  If AMA is already updated, uninstall it. Follow these steps to uninstall AMA:

  1. In the Azure portal, go to the Extensions page for your Azure Stack HCI cluster.

  2. Select the checkbox for AzureMonitorWindowsAgent and select Uninstall.

     

• Option 2: Update AMA

  Follow these steps to update AMA:

  1. In the Azure portal, go to the Extensions page for your Azure Stack HCI cluster.

  2. Select the checkbox for AzureMonitorWindowsAgent and select Enable automatic upgrade, if not done already.

     

• Option 3: Restart AMA

  Follow these steps on all the cluster nodes to restart AMA:

  1. Run the following command to disable AMA:

     cd C:\Packages\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent\<agent version number>
     AzureMonitorAgentExtension.exe disable
     

  2. After the executable completes and all the AMA processes stop, run the following command to restart the agent:

     AzureMonitorAgentExtension.exe enable
     

• Option 4: Reboot your cluster nodes

Reconfigure Insights for Azure Stack HCI

Follow these steps to reconfigure Insights for Azure Stack HCI:

1. In the Azure portal, the Insights page for your Azure Stack HCI cluster displays a banner at the top, as shown in the following screenshot, which helps you configure Insights again and associate DCR with the cluster nodes. Review the banner and select Configure Insights.

   

2. Reconfigure DCR. Follow the instructions to configure Insights as provided in this article. See Configure Insights for Azure Stack HCI.

   

Troubleshoot blank Workbooks page with no data populated

Issue. You see a blank Workbooks page with no data populated, as shown in the following screenshot:

Possible causes. There might be several possible causes for this issue, including recent configuration of Insights, incomplete troubleshooting steps for clusters registered before November 2023, or associated DCR not being configured properly.

Solution. To troubleshoot the issue, perform the following steps in sequence:

1. If you recently configured Insights, wait for up to one hour for AMA to gather data.
2. If there's still no data after waiting, make sure you have completed all the steps mentioned in the Troubleshoot clusters registered before November 2023 section.
3. Verify the configuration of the associated DCR. Make sure that event channels and performance counters are added as data sources to the associated DCR, as described in the Data Collection Rules section.
4. If the issue persists after performing the above steps, and you still don't see any data, contact customer support for assistance.

For more detailed troubleshooting guidance, see Troubleshooting guidance for the Azure Monitor Agent.

Insights previously known as "monitoring", enables you to monitor your resources and provide useful insights regarding cluster, servers, virtual machines, storage, and much more.

The data generated from your on-premises Azure Stack HCI cluster is collected in a Log Analytics workspace in Azure. Within that workspace, you can collect data about the health of your cluster. By default, monitoring collects the following logs every hour:

• Health Management (Microsoft-windows-health/operational)
• SDDC Management (Microsoft-Windows-SDDC-Management/Operational; Event ID: 3000, 3001, 3002, 3003, 3004)

To change the frequency of log collection, see Event Log Channel.

Prerequisites

To use Insights with the legacy Agent, make sure you complete the following prerequisites:

• Azure Stack HCI cluster should be registered with Azure and arc enabled. If you registered your cluster on or after June 15, 2021, this happens by default. Otherwise, you must enable Azure Arc integration.

• Enable Log Analytics to link the cluster to a Log Analytics workspace where the log data required for monitoring is saved.

• Enable Insights to allow Azure Monitor to start collecting the events that are required for monitoring.

Logs capability

After you register your cluster and Arc-enable the servers, you'll see the following items in the Azure portal:

• An Azure Stack HCI resource in the specified resource group.
• Server - Azure Arc resources for every server in the cluster in the <clustername>ArcInstanceResourceGroup.
• Nodes with a Server-Azure Arc resource link on the Azure Stack HCI resource page under the Nodes tab.

Now that your cluster nodes are Arc-enabled, navigate to your Azure Stack HCI cluster resource page. Under the Capabilities tab, you see the option to enable logs, which should say Not configured.

This capability is an Arc for Servers extension that simplifies installing the Microsoft Monitoring Agent. Because you're using the Arc for Servers extension to enable this workflow, if you ever add more servers to your cluster, they automatically have the Microsoft Monitoring Agent installed on them.

Configure the Microsoft Monitoring Agent extension

To configure the Microsoft Monitoring Agent (AMA) extension:

1. Under the Capabilities tab, select Logs.

2. Select Use existing to use the existing workspace for your subscription.

3. Select Add at the bottom of the page.

   

4. When the configuration is finished, Logs will appear as Configured under the Capabilities tab.

5. Select Settings > Extensions from the toolbar on the left. You should see that each of your servers successfully installed the Microsoft Monitoring Agent.

You have now successfully installed the log analytics extension.

Disable Log Analytics

If you'd like to disable the Logs capability, remove the Microsoft Monitoring Agent from the Extensions settings. This doesn't delete the Log Analytics workspace in Azure or any of the data that resides in it, so you must do that manually.

To remove the Microsoft Monitoring Agent from every server in the cluster, follow these steps:

1. Select Settings > Extensions from the toolbar on the left.
2. Select the MicrosoftMonitoringAgent checkbox.
3. Select Remove, and then Yes.

Enable Insights

Enabling Insights helps you monitor all Azure Stack HCI clusters currently associated with the Log Analytics workspace by providing useful health metrics. To enable this capability from the Azure portal, follow these steps:

1. Under the Capabilities tab, select Insights. Then select Get started.
2. Insights should now show as Configured in the Capabilities tab.

What data is collected?

The Microsoft-windows-sddc-management/operational and microsoft-windows-health/operational Windows event channel is added to your Log Analytics workspace under Windows event logs.

By collecting these logs, analytics will show the health status of the individual servers, drives, volumes, and VMs. By default, five performance counters are added:

Performance counters

The following table describes the performance counters that are monitored:

After you enable Insights, it can take up to 15 minutes to collect the data. When the process is finished, you are able to see a rich visualization of the health of your cluster, under the Insights tab on the left navigation, as shown in the following screenshot:

Disable Insights

To disable insights, follow these steps:

1. Select Insights under the Capabilities tab.

2. Select Disable Insights.

   

When you disable Insights, the Health Service and SDDC Management logs are no longer collected, but any existing data won't be deleted. If you want to delete that data, go to your Log Analytics workspace and delete the data manually.

Update Insights

The Insights tile shows a Needs update message in the following cases:

• The health event from Windows events logs is deleted.
• Any or all of the 5 performance counters from the Loganalytics workspace are deleted.

To enable Insights again,

• Select the Insights tile under Capabilities.
• Select Update to see the visualizations again.

Troubleshooting

If the Logs capability and Monitoring capability are enabled without errors but the monitoring data doesn't appear after an hour or so, you can use the Log Analytics Troubleshooting Tool.

How to use the Log Analytics Troubleshooting Tool

1. Open a PowerShell prompt as Administrator on the Azure Stack HCI host where the MicrosoftMonitoringAgent extension is installed.

2. Navigate to the directory where the tool is located.

   cd C:\Program Files\Microsoft Monitoring Agent\Agent\Troubleshooter
   

3. Execute the main script using this command:

   .\GetAgentInfo.ps1
   

4. When prompted to select a troubleshooting scenario, choose option 1: Agent not reporting data or heartbeat data missing.

5. You're prompted to select the action that you'd like to perform. Choose option 1: Diagnose.

   

6. If you encounter the error that's highlighted in the following screenshot but are still able to connect to all Log Analytics endpoints and your firewall and gateway settings are correct, you have likely encountered a time zone issue.

   

   The cause is that the local time is different than Azure time, and the workspace key couldn't be validated due to the mismatch.

   

7. To resolve the issue:

   1. Go to your Azure Stack HCI resource page in Azure portal, select [cluster name] > Extensions. Then select the tick box for the MicrosoftMonitoringAgent extension and remove the Microsoft Monitoring Agent extension.
   2. Ensure that your Azure Stack HCI host time zone is correct, and that the local time on the host is the same as Azure time for your time zone.
      1. From the Azure Stack HCI host console, select option 9: Date & Time from the Sconfig menu, then select change time zone and ensure local time is correct.
      2. Review the Active Directory PDC (Primary Domain Controller) time zone, and make sure the date and time are correct.
      3. If Active Directory PDC is correct and Azure Stack HCI local time is still incorrect, then the Active Directory domain hierarchy isn't being recognized. If so, complete steps iv - vi below. Otherwise, proceed to step c.
      4. From the Azure Stack HCI host, select option 15 to exit the Sconfig menu. Then run the following command in PowerShell as an administrator: w32tm.exe /config /syncfromflags:domhier /update - this should return a confirmation that the command completed successfully, and the time setting should now be correct.
      5. To diagnose further, run w32tm /monitor on the Azure Stack HCI host console. The active domain controller should be listed as stratum 1 server, and all other domain controllers as stratum 2.
      6. Lastly, ensure that the Windows time service and time providers aren't configured in a Group Policy Object, as this will interfere with the Active Directory domain hierarchy.
   3. Re-add the Log Analytics extension by going to your Azure Stack HCI resource page in Azure portal, select [cluster name] > Overview, then select Capabilities and configure Log Analytics and Monitoring.

8. Rerun the Log Analytics Troubleshooting Tool and you should no longer see the error. You should now see Windows agent numbers increment in your Log Analytics workspace under Agents Management to match your cluster nodes, and monitoring events begin to flow.