How to find v / o bottleneck in asp.net application

Question

We have a high traffic site that generates a lot of I / O. Within 10 minutes, it reads more than 10 GB of data (see W3wp in the task manager). To freeze memory and applications, I successfully used WinDbg. But I do not know how I can find the object (s) / method in the process that is responsible for the highest I / O.

Is it possible?

Edit The question arises: is there a way to profile I / O operations in a .NET assembly, say: a list of threads sorted by maximum disk I / O (or something similar that will help me where to look)

Answer 1

ANTI Performance Profiler

I used this tool with great success, dealing with the search for specific instructions that cause ~ 512 GB of memory on a large volume web farm, chewed for 5-10 minutes. It looks like a very similar situation like yours.

Now, to be realistic - it doesn’t care about the solution to your problem. It still requires a lot of customization, thorough analysis and detective work. But this tool definitely took the problem from "practically insoluble" to "solvable in a few days."

Update:

As I mentioned in the comments (and echoed Ben Emmett), we can use ANTS to monitor memory, process the file system — almost any resource consumption and expand the call stack to see the effects of certain procedures.

Answer 2

I came up with this AppDynamics Lite tool that displays your application, which requires cost and performance visually. This can help you figure out what functions perform the most expensive I / O.

Citation;

Understanding the state of the CLR with key metrics such as response time, throughput, exception rate, and garbage collection time, as well as key system resources such as processor, memory, and disk I / O.

It’s worth a shot, as it is trial / free for 30 days. Hope it helps. Ps: I have nothing to do with AppDynamics.

Answer 3

You can use (for free) the Windows Performance Toolki from Windows 8, which also runs on Windows Vista and later. There you can enable system profiling to see what happens in all processes simultaneously. No tools required. To install the secret registry key that runs WPRUI.exe, only one reboot is required.

With XPerf, you can enable the IO Init stack walk so that the call stack is taken for each initial I / O. The only problem is that the stacks will be split into 64-bit processes, which means that you will see only the first method above the BCL methods of your code, because there is a Windows 7 error in the stack capabilities for the OS.

The workaround is Ngen for your builds, or switching to Server 2012 or switching to x86 for profiling to see deeper call stacks.

You will see all the I / O and file activity even without any call stacks and file names along the length of the hard drive. This should give you good information about which part of your application is causing the IO drive. From partial call stacks, you can pinpoint your problem even without full stacks.

The tool will give you much more information than any commercially available profiler due to the fact that you need to learn how to use it. Since call stacks do not end on your code or in user mode, but in the kernel you can also determine, for example, a virus scanner causes significant I / O delays. But you need to know how your processor works. This toolkit was originally aimed at kernel developers, which explains why you see so many useless columns.

In the figure below, you see how the IO and CPU files load. When you select your tall I / O file on the disk I / O graph, it highlights in the CPU consumption all the associated call tables that were made at the same time, while IO was active. This way, you can focus on potentially blocked threads with IO.