Troubleshoot Application Insights Profiler for .NET

This article provides troubleshooting steps and information to help you use Application Insights Profiler for .NET.

Are you using the appropriate .NET Profiler endpoint?

Currently the regions in Azure operated by 21Vianet require endpoint modifications.

Is your app running on the right version?

The Profiler supports .NET Framework versions later than 4.6.2.

If your web app is an ASP.NET Core application, it must run on the latest supported ASP.NET Core runtime.

Are you using the right Azure service plan?

Profiler for .NET isn't currently supported on free or shared app service plans. Upgrade to one of the basic plans for Profiler to start working.

Are you searching for .NET Profiler data within the right time frame?

If the data you're trying to view is older than two weeks, try limiting your time filter and try again. Traces are deleted after seven days.

Are you aware of the .NET Profiler sampling rate and overhead?

The .NET Profiler randomly runs two minutes per hour on each virtual machine hosting applications with Profiler enabled.

Can you access the gateway?

Check that a firewall or proxies aren't blocking your access to this webpage.

Are you seeing timeouts or do you need to check if the .NET Profiler is running?

The profiler uploads profiling data only when it can attach the data to a request that happens while the profiler is running. The .NET Profiler collects data for two minutes each hour. You can also trigger the profiler by starting a profiling session.

The profiler writes trace messages and custom events to your Application Insights resource. You can use these events to see how the profiler is running.

Search for trace messages and custom events that the .NET Profiler sends to your Application Insights resource.

1. In your Application Insights resource, select Search from the top menu.

   

2. Use the following search string to find the relevant data:

   stopprofiler OR startprofiler OR upload OR ServiceProfilerSample
   

   

   The preceding search results include two examples of searches from two AI resources:

   • If the application isn't receiving requests while the profiler is running, the message explains that the upload was canceled because of no activity.

   • The profiler starts and sends custom events when it detects requests that happen while the profiler is running. If the ServiceProfilerSample custom event is displayed, it means that a profile was captured and is available in the Application Insights Performance page.

   If no records are displayed, the profiler isn't running or took too long to respond. Make sure Profiler is enabled on your Azure service.

The .NET Profiler is on, but no traces captured

Even when the Profiler is enabled, it might not capture or upload traces, especially in these situations:

1. No incoming requests to your application:

   You can manually invoke your application or create an availability test.

2. No incoming telemetry acknowledged by Application Insights:

   • If traffic is coming to your application: validate that Application Insights Live Metrics shows incoming requests.
   • If the Incoming Requests charts are empty (no data or showing zero): troubleshoot Application Insights.
   • If you host your .NET application on Azure App Service, see Troubleshoot Application Insights integration with Azure App Service.

3. Profiler setting for Sampling is turned off:

   If no profiler traces are available, check the Profiler Sampling setting.

   1. Open Application Insights > Performance.
   2. Select Profiler.
   3. Select the Triggers button.
   4. In the Trigger Settings, ensure Sampling is on.

4. Still no traces uploaded?

   Create a support request, or ask Azure community support. You can also submit product feedback to the Azure feedback community.

Double counting in parallel threads

When two or more parallel threads associate with a request, the stack viewer's total time metric might exceed the request's duration. In this case, the total thread time surpasses the actual elapsed time.

For example, one thread might wait for the other thread to finish. The viewer tries to detect this situation and omits the uninteresting wait. In doing so, it errs on the side of displaying too much information rather than omitting what might be critical information.

When you see parallel threads in your traces, determine which threads are waiting so that you can identify the hot path for the request. Usually, the thread that quickly goes into a wait state waits on the other threads. Concentrate on the other threads and ignore the time in the waiting threads.

Troubleshoot the .NET Profiler on your specific Azure service

The following sections walk you through troubleshooting steps for using Profiler on Azure App Service.

Azure App Service

For the .NET Profiler to work properly, make sure that you:

• Enable Application Insights for your web app with the right settings.

• Run the ApplicationInsightsProfiler3 WebJob. To check the WebJob:

  1. Go to Kudu. In the Azure portal:

     1. In your App Service instance, select Advanced Tools in the left pane.

     2. Select Go.

  2. On the top menu, select Tools > WebJobs dashboard. The WebJobs pane opens.

     If ApplicationInsightsProfiler3 doesn't show up, restart your App Service application.

     

  3. To view the details of the WebJob, including the log, select the ApplicationInsightsProfiler3 link. The Continuous WebJob Details pane opens.

     

If the .NET Profiler still isn't working for you, download the log and submit an Azure support ticket.

Check the Diagnostic Services site extension status page

If you enabled the .NET Profiler through the Application Insights page in the Azure portal, the Diagnostic Services site extension manages it. You can check the status page of this extension by going to https://{site-name}.scm.chinacloudsites.cn/DiagnosticServices.

The status page shows the installation state of the .NET Profiler and Snapshot Debugger agents. If there was an unexpected error, it appears along with steps on how to fix it.

You can use the Kudu management site for App Service to get the base URL of this status page:

1. Open your App Service application in the Azure portal.

2. Select Advanced Tools.

3. Select Go.

4. On the Kudu management site:

   1. Append /DiagnosticServices to the URL.
   2. Select Enter.

It ends like https://<kudu-url>/DiagnosticServices.

A status page appears similar to the following example.

Manual installation

When you configure the .NET Profiler, the process updates the web app's settings. If necessary, you can apply the updates manually.

When you configure the .NET Profiler, updates are made to the web app's settings. If necessary, you can apply the updates manually.

Too many active profiling sessions

In Azure App Service, there's a limit of only one profiling session at a time per VM. This limit is enforced at the VM level across all applications and deployment slots running in an App Service Plan. This limit applies equally to profiling sessions started via Diagnose and solve problems, Kudu, and Application Insights Profiler for .NET.

For a single app scaled out to multiple instances, each instance runs on a separate VM and can run its own profiling session independently. The contention occurs only when multiple apps or deployment slots on the same App Service Plan share the same VM.

If the .NET Profiler tries to start a session when another is already running on the same VM, an error is logged in the Application Log and also the continuous WebJob log for ApplicationInsightsProfiler3.

You might see one of the following messages in the logs:

• Microsoft.ServiceProfiler.Exceptions.TooManyETWSessionException
• Error: StartProfiler failed. Details: System.Runtime.InteropServices.COMException (0xE111005E): Exception from HRESULT: 0xE111005E

The error code 0xE111005E indicates that a profiling session couldn't start because another session is already running.

To avoid the error and reduce noise in your deployment logs:

• Move some web apps to a different App Service Plan so they don't share VMs.
• Disable the Profiler on applications that don't need profiling.
• Stop any unused deployment slots. Each running slot has the Profiler active and competes for the profiling session on its VM.
• During deployments, consider temporarily disabling the Profiler on staging slots to prevent errors caused by slot swaps triggering concurrent sessions.

Deployment error: Directory Not Empty 'D:\home\site\wwwroot\App_Data\jobs'

If you're redeploying your web app to a Web Apps resource with the .NET Profiler enabled, you might see the following message:

"Directory Not Empty 'D:\home\site\wwwroot\App_Data\jobs'"

This error occurs if you run Web Deploy from scripts or from Azure Pipelines. Resolve it by adding the following deployment parameters to the Web Deploy task:

-skip:Directory='.*\\App_Data\\jobs\\continuous\\ApplicationInsightsProfiler.*' -skip:skipAction=Delete,objectname='dirPath',absolutepath='.*\\App_Data\\jobs\\continuous$' -skip:skipAction=Delete,objectname='dirPath',absolutepath='.*\\App_Data\\jobs$'  -skip:skipAction=Delete,objectname='dirPath',absolutepath='.*\\App_Data$'

These parameters delete the folder used by Application Insights Profiler for .NET and unblock the redeploy process. They don't affect the Profiler instance that's currently running.

Is Application Insights Profiler for .NET running?

The Profiler runs as a continuous WebJob in the web app. You can open the web app resource in the Azure portal. In the WebJobs pane, check the status of ApplicationInsightsProfiler. If it isn't running, open Logs to get more information.

Virtual machines

To see whether the .NET Profiler is configured correctly by Azure Diagnostics:

1. Verify that the deployed Azure Diagnostics configuration matches your expectations.

2. Verify that the content of the Azure Diagnostics configuration deployed is what you expect.

3. Make sure Azure Diagnostics passes the correct iKey on the Profiler command line.

4. Make sure Azure Diagnostics passes the proper iKey on the Profiler command line.

5. Review the Profiler log file to see if the .NET Profiler ran but returned an error.

To check the settings used to configure Azure Diagnostics:

1. Sign in to the virtual machine.

2. Open the log file at this location. The plug-in version might be newer on your machine.

   c:\WindowsAzure\logs\Plugins\Microsoft.Azure.Diagnostics.PaaSDiagnostics\1.11.3.12\DiagnosticsPlugin.log
   

3. Search the file for the string WadCfg to find the settings that Azure Diagnostics passes to the virtual machine to configure Azure Diagnostics.

4. Verify that the iKey used by the .NET Profiler sink is correct.

5. Check the command line that starts Profiler. The command line arguments are in the following file (the drive could be c: or d: and the directory might be hidden):

   C:\ProgramData\ApplicationInsightsProfiler\config.json
   

6. Make sure that the iKey on the Profiler command line is correct.

7. By using the path found in the preceding config.json file, check the Profiler log file, called BootstrapN.log. It displays:

   • The debug information that indicates the settings that Profiler is using.
   • Status and error messages from Profiler.

   You can find the file:

   C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.Diagnostics.IaaSDiagnostics\1.17.0.6\ApplicationInsightsProfiler
   

8. If the .NET Profiler is running while your application is receiving requests, the following message appears: "Activity detected from iKey."

9. When the trace is being uploaded, the following message appears: "Start to upload trace."

Edit network proxy or firewall rules

If your application connects to the internet by using a proxy or a firewall, you might need to update the rules to communicate with the .NET Profiler.

The IP addresses used by Application Insights Profiler for .NET are included in the Azure Monitor service tag. For more information, see Azure service tags overview.

Support

If you still need help, submit a support ticket by selecting the question mark icon in the Azure portal. Include the correlation ID from the error message.