Locking down an App Service Environment
Important
This article is about App Service Environment v2, which is used with Isolated App Service plans. App Service Environment v1 and v2 are retired as of 31 August 2024. There's a new version of App Service Environment that is easier to use and runs on more powerful infrastructure. To learn more about the new version, start with the Introduction to the App Service Environment. If you're currently using App Service Environment v1, please follow the steps in this article to migrate to the new version.
As of 31 August 2024, Service Level Agreement (SLA) and Service Credits no longer apply for App Service Environment v1 and v2 workloads that continue to be in production since they are retired products. Decommissioning of the App Service Environment v1 and v2 hardware has begun, and this may affect the availability and performance of your apps and data.
You must complete migration to App Service Environment v3 immediately or your apps and resources may be deleted. We will attempt to auto-migrate any remaining App Service Environment v1 and v2 on a best-effort basis using the in-place migration feature, but Microsoft makes no claim or guarantees about application availability after auto-migration. You may need to perform manual configuration to complete the migration and to optimize your App Service plan SKU choice to meet your needs. If auto-migration isn't feasible, your resources and associated app data will be deleted. We strongly urge you to act now to avoid either of these extreme scenarios.
If you need additional time, we can offer a one-time 30-day grace period for you to complete your migration. For more information and to request this grace period, review the grace period overview, and then go to Azure portal and visit the Migration blade for each of your App Service Environments.
For the most up-to-date information on the App Service Environment v1/v2 retirement, see the App Service Environment v1 and v2 retirement update.
The App Service Environment (ASE) has many external dependencies that it requires access to in order to function properly. The ASE lives in the customer Azure Virtual Network. Customers must allow the ASE dependency traffic, which is a problem for customers that want to lock down all egress from their virtual network.
There are many inbound endpoints that are used to manage an ASE. The inbound management traffic can't be sent through a firewall device. The source addresses for this traffic are known and are published in the App Service Environment management addresses document. There's also a Service Tag named AppServiceManagement, which can be used with Network Security Groups (NSGs) to secure inbound traffic.
The ASE outbound dependencies are almost entirely defined with FQDNs, which don't have static addresses behind them. The lack of static addresses means that Network Security Groups can't be used to lock down the outbound traffic from an ASE. The addresses change often enough that one can't set up rules based on the current resolution and use that to create NSGs.
The solution to securing outbound addresses lies in use of a firewall device that can control outbound traffic based on domain names. Azure Firewall can restrict outbound HTTP and HTTPS traffic based on the FQDN of the destination.
System architecture
Deploying an ASE with outbound traffic going through a firewall device requires changing routes on the ASE subnet. Routes operate at an IP level. If you aren't careful in defining your routes, you can force TCP reply traffic to source from another address. When your reply address is different from the address traffic was sent to, the problem is called asymmetric routing and it breaks TCP.
There must be routes defined so that inbound traffic to the ASE can reply back the same way the traffic came in. Routes must be defined for inbound management requests and for inbound application requests.
The traffic to and from an ASE must abide by the following conventions
- The traffic to Azure SQL, Storage, and Event Hubs aren't supported with use of a firewall device. This traffic must be sent directly to those services. The way to make that happen is to configure service endpoints for those three services.
- Route table rules must be defined that send inbound management traffic back from where it came.
- Route table rules must be defined that send inbound application traffic back from where it came.
- All other traffic leaving the ASE can be sent to your firewall device with a route table rule.
Locking down inbound management traffic
If your ASE subnet doesn't already have an NSG assigned to it, create one. Within the NSG, set the first rule to allow traffic from the Service Tag named AppServiceManagement on ports 454, 455. The rule to allow access from the AppServiceManagement tag is the only thing that is required from public IPs to manage your ASE. The addresses that are behind that Service Tag are only used to administer the Azure App Service. The management traffic that flows through these connections is encrypted and secured with authentication certificates. Typical traffic on this channel includes things like customer initiated commands and health probes.
ASEs that are made through the portal with a new subnet are made with an NSG that contains the allow rule for the AppServiceManagement tag.
Your ASE must also allow inbound requests from the Load Balancer tag on port 16001. The requests from the Load Balancer on port 16001 are keep alive checks between the Load Balancer and the ASE front ends. If port 16001 is blocked, your ASE goes unhealthy.
Configuring Azure Firewall with your ASE
The steps to lock down egress from your existing ASE with Azure Firewall are:
Enable service endpoints to SQL, Storage, and Event Hubs on your ASE subnet. To enable service endpoints, go into the networking portal > subnets and select Microsoft.EventHub, Microsoft.SQL and Microsoft.Storage from the Service endpoints dropdown. When you have service endpoints enabled to Azure SQL, any Azure SQL dependencies that your apps have must be configured with service endpoints as well.
Create a subnet named AzureFirewallSubnet in the virtual network where your ASE exists. Follow the directions in the Azure Firewall documentation to create your Azure Firewall.
From the Azure Firewall UI > Rules > Application rule collection, select Add application rule collection. Provide a name, priority, and set Allow. In the FQDN tags section, provide a name, set the source addresses to * and select the App Service Environment FQDN Tag and the Windows Update.
From the Azure Firewall UI > Rules > Network rule collection, select Add network rule collection. Provide a name, priority, and set Allow. In the Rules section under IP addresses, provide a name, select a protocol of Any, set * to Source and Destination addresses, and set the ports to 123. This rule allows the system to perform clock sync using NTP. Create another rule the same way to port 12000 to help triage any system issues.
From the Azure Firewall UI > Rules > Network rule collection, select Add network rule collection. Provide a name, priority, and set Allow. In the Rules section under Service Tags, provide a name, select a protocol of Any, set * to Source addresses, select a service tag of AzureMonitor, and set the ports to 80, 443. This rule allows the system to supply Azure Monitor with health and metrics information.
Create a route table with a route for the AppServiceManagement service tag with a next hop of Internet. This route table entry is required to avoid asymmetric routing problems. Add a Virtual Appliance route to your route table for 0.0.0.0/0 with the next hop being your Azure Firewall private IP address. You should use the service tag instead of the IP addresses to avoid having to update the route table when the management addresses change. However, if you need to use the management addresses, you can download the service tag reference file for the cloud you're using from App Service Environment management addresses and create your routes from those.
Assign the route table you created to your ASE subnet.
Deploying your ASE behind a firewall
The steps to deploy your ASE behind a firewall are the same as configuring your existing ASE with an Azure Firewall except you need to create your ASE subnet and then follow the previous steps. To create your ASE in a pre-existing subnet, you need to use a Resource Manager template as described in the document on Creating your ASE with a Resource Manager template.
Application traffic
The above steps allow your ASE to operate without problems. You still need to configure things to accommodate your application needs. There are two problems for applications in an ASE that is configured with Azure Firewall.
- Application dependencies must be added to the Azure Firewall or the route table.
- Routes must be created for the application traffic to avoid asymmetric routing issues
If your applications have dependencies, they need to be added to your Azure Firewall. Create Application rules to allow HTTP/HTTPS traffic and Network rules for everything else.
When you know the address range that your application request traffic comes from, you can add that to the route table that is assigned to your ASE subnet. If the address range is large or unspecified, then you can use a network appliance like the Application Gateway to give you one address to add to your route table. For details on configuring an Application Gateway with your ILB ASE, read Integrating your ILB ASE with an Application Gateway
This use of the Application Gateway is just one example of how to configure your system. If you did follow this path, then you would need to add a route to the ASE subnet route table so the reply traffic sent to the Application Gateway would go there directly.
Logging
Azure Firewall can send logs to Azure Storage, Event Hubs, or Azure Monitor logs. To integrate your app with any supported destination, go to the Azure Firewall portal > Diagnostic Logs and enable the logs for your desired destination. If you integrate with Azure Monitor logs, then you can see logging for any traffic sent to Azure Firewall. To see the traffic that is being denied, open your Log Analytics workspace portal > Logs and enter a query like
AzureDiagnostics | where msg_s contains "Deny" | where TimeGenerated >= ago(1h)
Integrating your Azure Firewall with Azure Monitor logs is useful when first getting an application working when you aren't aware of all of the application dependencies. You can learn more about Azure Monitor logs from Analyze log data in Azure Monitor.
Configuring third-party firewall with your ASE
The following information is only required if you wish to configure a firewall appliance other than Azure Firewall. For Azure Firewall see the section above.
Consider the following dependencies when deploying a third-party firewall with your ASE:
- Service Endpoint capable services should be configured with service endpoints.
- IP Address dependencies are for non-HTTP/S traffic (both TCP and UDP traffic)
- FQDN HTTP/HTTPS endpoints can be placed in your firewall device.
- Wildcard HTTP/HTTPS endpoints are dependencies that can vary with your ASE based on many qualifiers.
- Linux dependencies are only a concern if you're deploying Linux apps into your ASE. If you aren't deploying Linux apps into your ASE, then these addresses don't need to be added to your firewall.
Service Endpoint capable dependencies
| Endpoint |
|---|
| Azure SQL |
| Azure Storage |
| Azure Event Hubs |
IP Address dependencies
| Endpoint | Details |
|---|---|
| *:123 | NTP clock check. Traffic is checked at multiple endpoints on port 123 |
| *:12000 | This port is used for some system monitoring. If blocked, then some issues are harder to triage but your ASE continues to operate |
| 40.77.24.27:80 | Needed to monitor and alert on ASE problems |
| 40.77.24.27:443 | Needed to monitor and alert on ASE problems |
| 13.90.249.229:80 | Needed to monitor and alert on ASE problems |
| 13.90.249.229:443 | Needed to monitor and alert on ASE problems |
| 104.45.230.69:80 | Needed to monitor and alert on ASE problems |
| 104.45.230.69:443 | Needed to monitor and alert on ASE problems |
| 13.82.184.151:80 | Needed to monitor and alert on ASE problems |
| 13.82.184.151:443 | Needed to monitor and alert on ASE problems |
With an Azure Firewall, you automatically get the following configured with the FQDN tags.
FQDN HTTP/HTTPS dependencies
| Endpoint |
|---|
| microsoftgraph.chinacloudapi.cn:443 |
| login.live.com:443 |
| login.windows.com:443 |
| login.chinacloudapi.cn:443 |
| login.chinacloudapi.cn:443 |
| *.login.chinacloudapi.cn:443 |
| *.login.microsoft.com:443 |
| client.wns.windows.com:443 |
| definitionupdates.microsoft.com:443 |
| go.microsoft.com:80 |
| go.microsoft.com:443 |
| www.microsoft.com:80 |
| www.microsoft.com:443 |
| wdcpalt.microsoft.com:443 |
| wdcp.microsoft.com:443 |
| ocsp.msocsp.com:443 |
| ocsp.msocsp.com:80 |
| oneocsp.microsoft.com:80 |
| oneocsp.microsoft.com:443 |
| mscrl.microsoft.com:443 |
| mscrl.microsoft.com:80 |
| crl.microsoft.com:443 |
| crl.microsoft.com:80 |
| www.thawte.com:443 |
| crl3.digicert.com:80 |
| ocsp.digicert.com:80 |
| ocsp.digicert.com:443 |
| csc3-2009-2.crl.verisign.com:80 |
| crl.verisign.com:80 |
| ocsp.verisign.com:80 |
| cacerts.digicert.com:80 |
| azperfcounters1.blob.core.chinacloudapi.cn:443 |
| azurewatsonanalysis-prod.core.chinacloudapi.cn:443 |
| global.metrics.nsatc.net:80 |
| global.metrics.nsatc.net:443 |
| az-prod.metrics.nsatc.net:443 |
| antares.metrics.nsatc.net:443 |
| azglobal-black.azglobal.metrics.nsatc.net:443 |
| azglobal-red.azglobal.metrics.nsatc.net:443 |
| antares-black.antares.metrics.nsatc.net:443 |
| antares-red.antares.metrics.nsatc.net:443 |
| prod.microsoftmetrics.com:443 |
| maupdateaccount.blob.core.chinacloudapi.cn:443 |
| clientconfig.passport.net:443 |
| packages.microsoft.com:443 |
| schemas.microsoft.com:80 |
| schemas.microsoft.com:443 |
| management.core.chinacloudapi.cn:443 |
| management.core.chinacloudapi.cn:80 |
| management.chinacloudapi.cn:443 |
| www.msftconnecttest.com:80 |
| shavamanifestcdnprod1.azureedge.net:443 |
| validation-v2.sls.microsoft.com:443 |
| flighting.cp.wd.microsoft.com:443 |
| dmd.metaservices.microsoft.com:80 |
| admin.core.chinacloudapi.cn:443 |
| prod.warmpath.msftcloudes.com:443 |
| prod.warmpath.msftcloudes.com:80 |
| gcs.prod.monitoring.core.chinacloudapi.cn:80 |
| gcs.prod.monitoring.core.chinacloudapi.cn:443 |
| azureprofileruploads.blob.core.chinacloudapi.cn:443 |
| azureprofileruploads2.blob.core.chinacloudapi.cn:443 |
| azureprofileruploads3.blob.core.chinacloudapi.cn:443 |
| azureprofileruploads4.blob.core.chinacloudapi.cn:443 |
| azureprofileruploads5.blob.core.chinacloudapi.cn:443 |
| azperfmerges.blob.core.chinacloudapi.cn:443 |
| azprofileruploads1.blob.core.chinacloudapi.cn:443 |
| azprofileruploads10.blob.core.chinacloudapi.cn:443 |
| azprofileruploads2.blob.core.chinacloudapi.cn:443 |
| azprofileruploads3.blob.core.chinacloudapi.cn:443 |
| azprofileruploads4.blob.core.chinacloudapi.cn:443 |
| azprofileruploads6.blob.core.chinacloudapi.cn:443 |
| azprofileruploads7.blob.core.chinacloudapi.cn:443 |
| azprofileruploads8.blob.core.chinacloudapi.cn:443 |
| azprofileruploads9.blob.core.chinacloudapi.cn:443 |
| azureprofilerfrontdoor.chinacloudapp.cn:443 |
| settings-win.data.microsoft.com:443 |
| maupdateaccount2.blob.core.chinacloudapi.cn:443 |
| maupdateaccount3.blob.core.chinacloudapi.cn:443 |
| dc.services.visualstudio.com:443 |
| gmstorageprodsn1.blob.core.chinacloudapi.cn:443 |
| gmstorageprodsn1.file.core.chinacloudapi.cn:443 |
| gmstorageprodsn1.queue.core.chinacloudapi.cn:443 |
| gmstorageprodsn1.table.core.chinacloudapi.cn:443 |
| rteventservice.trafficmanager.cn:443 |
| ctldl.windowsupdate.com:80 |
| ctldl.windowsupdate.com:443 |
| global-dsms.dsms.core.chinacloudapi.cn:443 |
Wildcard HTTP/HTTPS dependencies
| Endpoint |
|---|
| gr-prod-*.chinacloudapp.cn:443 |
| gr-prod-*.azurewebsites.chinacloudapi.cn:443 |
| *.management.chinacloudapi.cn:443 |
| *.update.microsoft.com:443 |
| *.windowsupdate.microsoft.com:443 |
| *.identity.chinacloudapi.cn:443 |
| *.ctldl.windowsupdate.com:80 |
| *.ctldl.windowsupdate.com:443 |
| *.prod.microsoftmetrics.com:443 |
| *.dsms.core.chinacloudapi.cn:443 |
| *.prod.warm.ingest.monitor.core.chinacloudapi.cn:443 |
Linux dependencies
| Endpoint |
|---|
| wawsinfraprodbay063.blob.core.chinacloudapi.cn:443 |
| registry-1.dockerhub.azk8s.cn:443 |
| auth.dockerhub.azk8s.cn:443 |
| production.cloudflare.docker.com:443 |
| download.docker.com:443 |
| us.archive.ubuntu.com:80 |
| download.mono-project.com:80 |
| packages.treasuredata.com:80 |
| security.ubuntu.com:80 |
| oryx-cdn.microsoft.io:443 |
| *.cdn.mscr.io:443 |
| *.data.mcr.microsoft.com:443 |
| mcr.microsoft.com:443 |
| *.data.mcr.microsoft.com:443 |
| packages.fluentbit.io:80 |
| packages.fluentbit.io:443 |
| apt-mo.trafficmanager.cn:80 |
| apt-mo.trafficmanager.cn:443 |
| azure.archive.ubuntu.com:80 |
| azure.archive.ubuntu.com:443 |
| changelogs.ubuntu.com:80 |
| 13.74.252.37:11371 |
| 13.75.127.55:11371 |
| 13.76.190.189:11371 |
| 13.80.10.205:11371 |
| 13.91.48.226:11371 |
| 40.76.35.62:11371 |
| 104.215.95.108:11371 |