Hints of impending failure

Question

I am looking for indicators that I can programmatically check to hint at an impending crash of my application. I am looking for common things such as "the number of free IO threads is dangerously low," "the memory available for the application pool is low," and "CPU usage is high."

This is for C # / asp.net and IIS.

Examples of checking the value programmatically would be good, but not required. Any good ideas are welcome.

Answer 1

My thoughts:

Determine the ins and outs of the performance of your web application.

• Inputs (number of users, what tasks users perform, time of day, etc.)
• Outputs (memory usage, threads, delays)

Profile your application under the data of a real request. Create a good table of inputs and their associated outputs. For instance:

4 users calling page1 -> costs 4 - 6 mb, 4 threads 5 users calling page1 -> costs 7 - 14 mb, 5 threads 2 users calling page2 -> costs 120 - 200 mb, 1 thread 

Find the outputs that often cause a failure, and find which inputs cause these outputs. Create a good model of maximum chance .

When your inputs begin to approach your failure outputs, an impending failure is likely with some probability. Record when a failure occurs and does not occur, and feed this information back into the table. Your webapp will know when it is about to fail.

UPDATED in response to comment:

Finding outs is the easiest part.

See SO questions. How to get memory used in C # , How to get CPU usage in C # and for a more general question What are the main performance monitors I should watch asp.net application .

Key points from these questions:

• GC.GetTotalMemory - tells you how much garbage collector emits.

• Processor object - will tell you about all the interesting statistics of preprocessing on the processor (processor downtime, value, etc.).

Answer 2

You can programmatically access the values ​​of some of the built-in system performance counters. If you need even more detailed information, you can also create your own counters.

Here's an article to get you started: How to calculate CPU usage programmatically

Answer 3

I really had a job trying to figure it out a few years ago. It turns out that this is a very, very difficult task - these days you can’t even trust the OS, so working in an isolated .net environment will make it almost impossible to determine what is actually happening.

Here are a few simple situations that you might want to consider if you decide to continue this.

CPU - you can determine the current CPU usage, but how do you say if your particular instance was tied to a single processor? How do you determine if you are working in a cluster or what other kernels can run? Perhaps you are running a virtual machine with a limited number of processors or threads - these restrictions can be changed on the fly.

Memory - what if you work inside a virtualized instance? What to do if suddenly an external change increases or decreases the amount of available memory?

Answer 4

It seems that you are more interested in the external causes of failure, and not in the fact that the application does not cope on its own. I think you are trying to tackle the Sisyphus task here; Instead of trying to guess possible failure factors, focus on setting up your workflow to mitigate any of these problems. IIS workflows are highly resilient - check ping, recyclying, etc.

Answer 5

Maybe not programmatically, but maintaining a measure of the number of hours of hustle or overtime that you must do in order not to fall can be a good indicator. :-)