Troubleshooting MVC4 Web API Performance

Question

I have an app.net mvc4 web api interface that receives about 54 thousand requests per day.

http://myserv.x.com/api/123/getstuff?whatstuff=thisstuff

I have 3 web servers behind a load balancer that are configured to handle HTTP requests.

Average response time ~ 300 ms. However, recently something has gone wrong (or perhaps it has always been there), since there is sporadic behavior of the response time, returning after 10-20 seconds. This will be the same request that gets to the same server directly, and not through the load balancer.

GIVEN: - System has been passed down to me so there may be gaps with IIS confiuration, etc,. - Database: SQL Server 2008R2 - Web Servers: Windows Server 2008R2 Enterprise SP1 - IIS 7.5 - Using MemoryCache aggressively with Model and Business Objects with eviction set to 2hrs - Looked at the logs but really don't see anything significantly relevant - One application pool...no other LOB applications running on this server 

Assumptions and Assignment: Somehow, I think something is processing the application pool or IIS workflows that shut down and restart, thereby triggering each new warm-up and reload request. It’s so sporadic that it’s hard to deal with the problem now. The same request to the same server returns quickly as expected (back to the back of N requests), since it was cached in about 300 ms .... but wait about 5-10-20 minutes, and the same request for the same server takes 16 seconds.

I have limited trace, as these are prod systems, so I can only expose so much registration data. Any help and information attacking this or similar behavior that someone else is encountering is appreciated. thanks

UPDATE: The w3wpe.exe process grows to ~ 3G. Somehow they destroy it, and the PID filter changes itself or something kills it every 3-4 minutes. I see tons of warnings in my web server (IIS) log:

The application pool serving the MyApplication process has been fatal with the error message using the Windows Activation Service. The process identifier was "1732". The data field contains the error number.

Answer 1

4-5 days after evaluating IIS and configuration compared to internal code problems, I finally found the problem with almost no help with IIS Windbg or debugdiag tools. These tools contain so much information, even with mini dumps or magazine stacks, that they can be red herring. The best thing was to reproduce it by setting up the copy “reasonably” instance in the production system, which we did not have at that time, and worked a little to establish something.

Needless to say, the problem is with excessive caching of business objects. There was one condition of the race, in which updates in a certain table updated the attribute of the corresponding business object (updates came from several servers), which caused the OOC stack to overflow, which largely forced caching to recursively cache itself to death, thereby causing w3wp.exe. to die and psuedo-recycle. This was one of those extreme cases that was incredibly difficult to test and reproduce in a non-production environment.