Monitoring Azure Virtual WAN - Best practices
This article provides configuration best practices for monitoring Virtual WAN and the different components that can be deployed with it. The recommendations presented in this article are mostly based on existing Azure Monitor metrics and logs generated by Azure Virtual WAN. For a list of metrics and logs collected for Virtual WAN, see the Monitoring Virtual WAN data reference.
Most of the recommendations in this article suggest creating Azure Monitor alerts. Azure Monitor alerts are meant to proactively notify you when there is an important event in the monitoring data to help you address the root cause quicker and ultimately reduce downtime. To learn how to create a metric alert, see Tutorial: Create a metric alert for an Azure resource.
Virtual WAN gateways
Site-to-site VPN gateway
Design checklist - metric alerts
- Create alert rule for increase in Tunnel Egress and/or Ingress packet count drop.
- Create alert rule to monitor BGP peer status.
- Create alert rule to monitor number of BGP routes advertised and learned.
- Create alert rule for VPN gateway overutilization.
- Create alert rule for tunnel overutilization.
| Recommendation | Description |
|---|---|
| Create alert rule for increase in Tunnel Egress and/or Ingress packet drop count. | An increase in tunnel egress and/or ingress packet drop count might indicate an issue with the Azure VPN gateway, or with the remote VPN device. Select the Tunnel Egress/Ingress Packet drop count metric when creating the alert rule(s). Define a static Threshold value greater than 0 and the Total aggregation type when configuring the alert logic. You can choose to monitor the Connection as a whole, or split the alert rule by Instance and Remote IP to be alerted for issues involving individual tunnels. To learn the difference between the concept of VPN connection, link, and tunnel in Virtual WAN, see the Virtual WAN FAQ. |
| Create alert rule to monitor BGP peer status. | When using BGP in your site-to-site connections, it's important to monitor the health of the BGP peerings between the gateway instances and the remote devices, as recurrent failures can disrupt connectivity. Select the BGP Peer Status metric when creating the alert rule. Using a static threshold, choose the Average aggregation type and configure the alert to be triggered whenever the value is less than 1. We recommend that you split the alert by Instance and BGP Peer Address to detect issues with individual peerings. Avoid selecting the gateway instance IPs as BGP Peer Address because this metric monitors the BGP status for every possible combination, including with the instance itself (which is always 0). |
| Create alert rule to monitor number of BGP routes advertised and learned. | BGP Routes Advertised and BGP Routes Learned monitor the number of routes advertised to and learned from peers by the VPN gateway, respectively. If these metrics drop to zero unexpectedly, it could be because there's an issue with the gateway or with on-premises. We recommend that you configure an alert for both these metrics to be triggered whenever their value is zero. Choose the Total aggregation type. Split by Instance to monitor individual gateway instances. |
| Create alert rule for VPN gateway overutilization. | A VPN gateway's aggregate throughput is determined by the number of scale units per instance. Note that all tunnels that terminate in the same gateway instance will share its aggregate throughput. It's likely that tunnel stability will be affected if an instance is working at its capacity for a long period of time. Select Gateway S2S Bandwidth when creating the alert rule. Configure the alert to be triggered whenever the Average throughput is greater than a value that is close to the maximum aggregate throughput of both instances. Alternatively, split the alert by instance and use the maximum throughput per instance as a reference. It's good practice to determine the throughput needs per tunnel in advance in order to choose the appropriate number of scale units. To learn more about the supported scale unit values for site-to-site VPN gateways, see the Virtual WAN FAQ. |
| Create alert rule for tunnel overutilization. | The maximum throughput allowed per tunnel is determined by the scale units of the gateway instance where it terminates. You might want to be alerted if a tunnel is at risk of nearing its maximum throughput, which can lead to performance and connectivity issues, and act proactively on it by investigating the root cause of the increased tunnel utilization or by increasing the gateway's scale units. Select Tunnel Bandwidth when creating the alert rule. Split by Instance and Remote IP to monitor all individual tunnels or choose specific tunnel(s) instead. Configure the alert to be triggered whenever the Average throughput is greater than a value that is close to the maximum throughput allowed per tunnel. To learn more about how a tunnel's maximum throughput is impacted by the gateway's scale units, see the Virtual WAN FAQ. |
Design checklist - log query alerts
To configure log-based alerts, you must first create a diagnostic setting for your site-to-site/point-to-site VPN gateway. A diagnostic setting is where you define what logs and/or metrics you want to collect and how you want to store that data to be analyzed later. Unlike gateway metrics, gateway logs won't be available if there's no diagnostic setting configured. To learn how to create a diagnostic setting, see Create diagnostic setting to view logs.
- Create tunnel disconnect alert rule.
- Create BGP disconnect alert rule.
| Recommendation | Description |
|---|---|
| Create tunnel disconnect alert rule. | Use Tunnel Diagnostic Logs to track disconnect events in your site-to-site connections. A disconnect event can be due to a failure to negotiate SAs, unresponsiveness of the remote VPN device, among other causes. Tunnel Diagnostic Logs also provide the disconnect reason. See the Create tunnel disconnect alert rule - log query below this table to select disconnect events when creating the alert rule. Configure the alert to be triggered whenever the number of rows resulting from running the query above is greater than 0. For this alert to be effective, select Aggregation Granularity to be between 1 and 5 minutes and the Frequency of evaluation to also be between 1 and 5 minutes. This way, after the Aggregation Granularity interval has passed, the number of rows is 0 again for a new interval. For troubleshooting tips when analyzing Tunnel Diagnostic Logs, see Troubleshoot Azure VPN gateway using diagnostic logs. Additionally, use IKE Diagnostic Logs to complement your troubleshooting, as these logs contain detailed IKE-specific diagnostics. |
| Create BGP disconnect alert rule. | Use Route Diagnostic Logs to track route updates and issues with BGP sessions. Repeated BGP disconnect events can affect connectivity and cause downtime. See the Create BGP disconnect rule alert- log query below this table to select disconnect events when creating the alert rule. Configure the alert to be triggered whenever the number of rows resulting from running the query above is greater than 0. For this alert to be effective, select Aggregation Granularity to be between 1 and 5 minutes and the Frequency of evaluation to also be between 1 and 5 minutes. This way, after the Aggregation Granularity interval has passed, the number of rows is 0 again for a new interval if the BGP sessions have been restored. For more information about the data collected by Route Diagnostic Logs, see Troubleshooting Azure VPN Gateway using diagnostic logs. |
Log queries
Create tunnel disconnect alert rule - log query: The following log query can be used to select tunnel disconnect events when creating the alert rule:
AzureDiagnostics | where Category == "TunnelDiagnosticLog" | where OperationName == "TunnelDisconnected"Create BGP disconnect rule alert- log query: The following log query can be used to select BGP disconnect events when creating the alert rule:
AzureDiagnostics | where Category == "RouteDiagnosticLog" | where OperationName == "BgpDisconnectedEvent"
Point-to-site VPN gateway
The following section details the configuration of metric-based alerts only. However, Virtual WAN point-to-site gateways also support diagnostic logs. To learn more about the available diagnostic logs for point-to-site gateways, see Virtual WAN point-to-site VPN gateway diagnostics.
Design checklist - metric alerts
- Create alert rule for gateway overutilization.
- Create alert for P2S connection count nearing limit.
- Create alert for User VPN route count nearing limit.
| Recommendation | Description |
|---|---|
| Create alert rule for gateway overutilization. | The bandwidth of a point-to-site gateway is determined by the number of scale units configured. To learn more about point-to-site gateway scale units, see Point-to-site (User VPN). Use the Gateway P2S Bandwidth metric to monitor the gateway's utilization and configure an alert rule that is triggered whenever the gateway's bandwidth is greater than a value near its aggregate throughput - for example, if the gateway was configured with 2 scale units, it will have an aggregate throughput of 1 Gbps. In this case, you could define a threshold value of 950 Mbps. Use this alert to proactively investigate the root cause of the increased utilization, and ultimately increase the number of scale units, if needed. Select the Average aggregation type when configuring the alert rule. |
| Create alert for P2S connection count nearing limit | The maximum number of point-to-site connections allowed is also determined by the number of scale units configured on the gateway. To learn more about point-to-site gateway scale units, see the FAQ for Point-to-site (User VPN). Use the P2S Connection Count metric to monitor the number of connections. Select this metric to configure an alert rule that is triggered whenever the number of connections is nearing the maximum allowed. For example, a 1-scale unit gateway supports up to 500 concurrent connections. In this case, you could configure the alert to be triggered whenever the number of connections is greater than 450. Use this alert to determine whether an increase in the number of scale units is required or not. Choose the Total aggregation type when configuring the alert rule. |
| Create alert rule for User VPN routes count nearing limit. | The maximum number of User VPN routes is determined by the protocol used. IKEv2 has a protocol-level limit of 255 routes, whereas OpenVPN has a limit of 1000 routes. To learn more about this, see VPN server configuration concepts. You might want to be alerted if you're close to hitting the maximum number of User VPN routes and act proactively to avoid any downtime. Use the User VPN Route Count to monitor this and configure an alert rule that is triggered whenever the number of routes surpasses a value close to the limit. For example, if the limit is 255 routes, an appropriate Threshold value could be 230. Choose the Total aggregation type when configuring the alert rule. |
ExpressRoute gateway
This section of the article focuses on metric-based alerts. There are no diagnostic logs currently available for Virtual WAN ExpressRoute gateways. In addition to the alerts described below, which focus on the gateway component, it's recommended to use the available metrics, logs, and tools to monitor the ExpressRoute circuit. To learn more about ExpressRoute monitoring, see ExpressRoute monitoring, metrics, and alerts.
Design checklist - metric alerts
- Create alert rule for bits received per second.
- Create alert rule for CPU overutilization.
- Create alert rule for packets per second.
- Create alert rule for number of routes advertised to peer.
- Count alert rule for number of routes learned from peer.
- Create alert rule for high frequency in route changes.
| Recommendation | Description |
|---|---|
| Create alert rule for Bits Received Per Second. | Bits Received per Second monitors the total amount of traffic received by the gateway from the MSEEs. You might want to be alerted if the amount of traffic received by the gateway is at risk of hitting its maximum throughput, as this can lead to performance and connectivity issues. This allows you to act proactively by investigating the root cause of the increased gateway utilization or increasing the gateway's maximum allowed throughput. Choose the Average aggregation type and a Threshold value close to the maximum throughput provisioned for the gateway when configuring the alert rule. Additionally, we recommend that you set an alert when the number of Bits Received per Second is near zero, as it might indicate an issue with the gateway or the MSEEs. The maximum throughput of an ExpressRoute gateway is determined by number of scale units provisioned. To learn more about ExpressRoute gateway performance, see About ExpressRoute connections in Azure Virtual WAN. |
| Create alert rule for CPU overutilization. | When using ExpressRoute gateways, it's important to monitor the CPU utilization. Prolonged high utilization can affect performance and connectivity. Use the CPU utilization metric to monitor this and create an alert for whenever the CPU utilization is greater than 80%, so you can investigate the root cause and ultimately increase the number of scale units, if needed. Choose the Average aggregation type when configuring the alert rule. To learn more about ExpressRoute gateway performance, see About ExpressRoute connections in Azure Virtual WAN. |
| Create alert rule for packets received per second. | Packets per second monitors the number of inbound packets traversing the Virtual WAN ExpressRoute gateway. You might want to be alerted if the number of packets per second is nearing the limit allowed for the number of scale units configured on the gateway. Choose the Average aggregation type when configuring the alert rule. Choose a Threshold value close to the maximum number of packets per second allowed based on the number of scale units of the gateway. To learn more about ExpressRoute performance, see About ExpressRoute connections in Azure Virtual WAN. Additionally, we recommend that you set an alert when the number of Packets per second is near zero, as it might indicate an issue with the gateway or MSEEs. |
| Create alert rule for number of routes advertised to peer. | Count of Routes Advertised to Peers monitors the number of routes advertised from the ExpressRoute gateway to the virtual hub router and to the Microsoft Enterprise Edge Devices. We recommend that you add a filter to only select the two BGP peers displayed as ExpressRoute Device and create an alert to identify when the count of advertised routes approaches the documented limit of 1000. For example, configure the alert to be triggered when the number of routes advertised is greater than 950. We also recommend that you configure an alert when the number of routes advertised to the Microsoft Edge Devices is zero in order to proactively detect any connectivity issues. To add these alerts, select the Count of Routes Advertised to Peers metric, and then select the Add filter option and the ExpressRoute devices. |
| Create alert rule for number of routes learned from peer. | Count of Routes Learned from Peers monitors the number of routes the ExpressRoute gateway learns from the virtual hub router and from the Microsoft Enterprise Edge Device. We recommend that you add a filter to only select the two BGP peers displayed as ExpressRoute Device and create an alert to identify when the count of learned routes approaches the documented limit of 4000 for Standard SKU and 10,000 for Premium SKU circuits. We also recommend that you configure an alert when the number of routes advertised to the Microsoft Edge Devices is zero. This can help in detecting when your on-premises has stopped advertising routes. |
| Create alert rule for high frequency in route changes. | Frequency of Routes changes shows the change frequency of routes being learned and advertised from and to peers, including other types of branches such as site-to-site and point-to-site VPN. This metric provides visibility when a new branch or more circuits are being connected/disconnected. This metric is a useful tool when identifying issues with BGP advertisements, such as flaplings. We recommend that you to set an alert if the environment is static and BGP changes aren't expected. Select a threshold value that is greater than 1 and an Aggregation Granularity of 15 minutes to monitor BGP behavior consistently. If the environment is dynamic and BGP changes are frequently expected, you might choose not to set an alert otherwise in order to avoid false positives. However, you can still consider this metric for observability of your network. |
Virtual hub
The following section focuses on metrics-based alerts for virtual hubs.
Design checklist - metric alerts
- Create alert rule for BGP peer status
| Recommendation | Description |
|---|---|
| Create alert rule to monitor BGP peer status. | Select the BGP Peer Status metric when creating the alert rule. Using a static threshold, choose the Average aggregation type and configure the alert to be triggered whenever the value is less than 1. This will allow you to identify when the virtual hub router is having connectivity issues with ExpressRoute, Site-to-Site VPN, and Point-to-Site VPN gateways deployed in the hub. |
Azure Firewall
This section of the article focuses on metric-based alerts. Azure Firewall offers a comprehensive list of metrics and logs for monitoring purposes. In addition to configuring the alerts described in the following section, explore how Azure Firewall Workbook can help monitor your Azure Firewall, or the benefits of connecting Azure Firewall logs to Azure Sentinel using Azure Firewall connector for Azure Sentinel.
Design checklist - metric alerts
- Create alert rule for risk of SNAT port exhaustion.
- Create alert rule for firewall overutilization.
| Recommendation | Description |
|---|---|
| Create alert rule for risk of SNAT port exhaustion. | Azure Firewall provides 2,496 SNAT ports per public IP address configured per backend virtual machine scale instance. It's important to estimate in advance the number of SNAT ports that will fulfill your organizational requirements for outbound traffic to the Internet. Not doing so increases the risk of exhausting the number of available SNAT ports on the Azure Firewall, potentially causing outbound connectivity failures. Use the SNAT port utilization metric to monitor the percentage of outbound SNAT ports currently in use. Create an alert rule for this metric to be triggered whenever this percentage surpasses 95% (due to an unforeseen traffic increase, for example) so you can act accordingly by configuring an additional public IP address on the Azure Firewall, or by using an Azure NAT Gateway instead. Use the Maximum aggregation type when configuring the alert rule. To learn more about how to interpret the SNAT port utilization metric, see Overview of Azure Firewall logs and metrics. To learn more about how to scale SNAT ports in Azure Firewall, see Scale SNAT ports with Azure NAT Gateway. |
| Create alert rule for firewall overutilization. | Azure Firewall maximum throughput differs depending on the SKU and features enabled. To learn more about Azure Firewall performance, see Azure Firewall performance. You might want to be alerted if your firewall is nearing its maximum throughput and troubleshoot the underlying cause, as this can have an impact in the firewall's performance. Create an alert rule to be triggered whenever the Throughput metric surpasses a value nearing the firewall's maximum throughput - if the maximum throughput is 30Gbps, configure 25Gbps as the Threshold value, for example. The Throughput metric unit is bits/sec. Choose the Average aggregation type when creating the alert rule. |
Resource Health Alerts
You can also configure Resource Health Alerts via Service Health for the below resources. This ensures you are informed of the availability of your Virtual WAN environment, and this allows you to troubleshoot if networking issues are due to your Azure resources entering an unhealthy state, as opposed to issues from your on-premises environment. It is recommended to configure alerts when the resource status becomes degraded or unavailable. If the resource status does become degraded/unavailable, you can analyze if there are any recent spikes in the amount of traffic processed by these resources, the routes advertised to these resources, or the number of branch/VNet connections created. Please see Azure Virtual WAN limits for additional info on limits supported in Virtual WAN.
- Microsoft.Network/vpnGateways
- Microsoft.Network/expressRouteGateways
- Microsoft.Network/azureFirewalls
- Microsoft.Network/virtualHubs
- Microsoft.Network/p2sVpnGateways
Next steps
- See Monitoring Virtual WAN data reference for a data reference of the metrics, logs, and other important values created by Virtual WAN.
- See Monitoring Azure resources with Azure Monitor for details on monitoring Azure resources.
- See Getting started with Azure Metrics Explorer for more details about Azure Monitor Metrics.
- See All resource metrics supported in Azure Monitor for a list of all supported metrics.
- See Create diagnostic settings in Azure Monitor for more information and troubleshooting when creating diagnostic settings via the Azure portal, CLI, PowerShell, etc.