W3WP.EXE using 100% CPU - where to start?

Question

The ASP.NET web application running on IIS6 periodically starts the processor up to 100%. This is W3WP, which is responsible for almost all CPU usage during these episodes. The processor remains 100% ranging from a few minutes to more than an hour.

This is on an intermediate server, and at the moment the site receives very light traffic from testers.

We have an ANTS profiler on the server, but it has not been clarified.

Where can we start to figure out what causes these episodes and what code holds the processor for all this time?

Answer 1

• Standard Windows performance counters (look for other correlated activity, for example, many GET requests, excessive network or disk I / O, etc.); you can read them both from code and from perfmon (to start collecting data if the CPU usage exceeds a threshold value, for example)
• Custom performance counters (in particular, time for out-of-box requests and other calls where runtime is not defined)
• Download testing using tools such as Visual Studio Team Test or WCAT
• If you can test or upgrade IIS 7, you can configure Failed Request Tracing to generate traces if requests take more than a certain time.
• Use logparser to find out what requests were received during the CPU spike.
• Code reviews / passes (in particular, find loops that may not finish properly, for example, if an error occurs, as well as locks and potential thread problems such as using statics).
• Profiling CPU and memory (may be difficult in a production system)
• Process explorer
• Windows Resource Monitor
• Detailed error logging
• Custom trace log, including runtime data (possibly conditionally based on processor performance counter)
• Errors occur when AppPool is recycled? If so, this may be the key.

Answer 2

This is not a big part of the answer, but you might need an old school and capture a snapshot of the IIS process image and debug it. You can also check out Tess Ferranddes ' blog - she is the kick of Microsoft Escalation Engineer, and her blog is focused on debugging ASP.NET windows, but the blog is about debugging Windows in general. If you select the ASP.NET tag (with which I am associated), you will see several similar elements.

Answer 3

If your processor reaches 100% and stays there, it is likely that you either have a deadlock scenario or an infinite loop. The profiler seems like a good choice for finding an infinite loop. However, dead ends are much more difficult to track.

Answer 4

Process Explorer is a great troubleshooting tool. You may try to find a high level CPU problem. This gives you an idea of ​​how your application works.

You can also try Procdump to dump the process and analyze what actually happened on the processor.

Answer 5

Also, look at your performance meters. They can tell you where a lot of time has been spent on the processor. Here is a link to the most common counters to use:

• http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/852720c8-7589-49c3-a9d1-73fdfc9126f0.mspx?mfr=true
• http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/be425785-c1a4-432c-837c-a03345f3885e.mspx?mfr=true

Answer 6

We had it on a recursive query that unloaded tons of data on the output - did you double check that everything goes out and there are no endless loops?

You can try to narrow it down with one page - we found that ANTS will not be very useful in the same case - what we ended up with is launching a site that hit the page while watching the processor - on the next page see CPU - very methodically and time-consuming, but if you don't find it with some code tracing, you might be out of luck -

We were able to use the IIS log files to track it on several suspicious pages -

Hope this helps!

Answer 7

This is an assumption at best, but perhaps your development team creates and deploys the application in debug mode instead of release mode. This will cause .pdb files to appear. The consequence of this is that your application will use additional resources to collect system status and debug information at runtime of your system, which will lead to more CPU usage.

Thus, it would be simple enough to make sure that they are building and deploying in release mode.

Answer 8

This is a very old post, I know, but this is also a common problem. All the proposed methods are very nice, but they will always point to the process, and there are many chances that we already know that our site is creating problems, but we just want to know which particular page is spending too much time on processing. The most accurate and simple tool, in my opinion, is IIS itself.

• Just click on your server in the left IIS pane.
• Click "Workflows" in the main panel. You already see which application pool is taking up too much CPU.
• Double-click on this row (eventually refresh by clicking "Show All") to see which pages consume too much CPU time ("Elapsed Time" column) in this pool

Answer 9

If you define a page that takes time to load, use the SharePoint Developer Dashboard to find out which component is taking time.