Controller actions hit multiple times - ASP.NET MVC 4 runs on Azure Emulator

Question

It may take some time to explain, but here I go :).

I created two Visual Studio 2012 projects:

• A project using a web application (Web -> ASP.NET MVC 4 Web Application) in the Visual Studio Add Project dialog box. I have not added a unit test project.
• Project using the Azure cloud services template (Cloud -> Windows Azure Cloud Service). I added one role, the ASP.NET MVC 4 web role, and again did not add the unit test project.

I left both of these projects completely intact from the way Visual Studio created them for me.

For each project, I then went to the HomeController class and set a breakpoint in each of the About and Contact action methods (drilling actions that just returned the view). For example, I set a breakpoint in a single line of the method:

 public ActionResult About() { return View(); } 

Then I started debugging the first project (a project without azure). Everything was as expected - i.e. When I use the browser to move between the About and Contact pages, the breakpoint in the corresponding action methods will be hit once for each request. It seemed to me good.

Then I started debugging the Azure project. I followed the same navigation procedure between the About and Contact pages. This time I found an interesting non-deterministic behavior. For some queries, the breakpoint in the action method will hit several times (sometimes more than two times) before the page displays. Sometimes the request simply freezes and the page will not be displayed (even after minutes of waiting).

I would like to know why this is happening because it affects more complex things in another Azure MVC application that we are working on (for example, because of this problem, the application may try to create two or more instances of the model object in my database for one request).

I am in the following environment:

• Windows 8 Pro (x64)
• Visual Studio 2012 Ultimate
• Windows Azure SDK for Visual Studio 2012
• IIS 8 Express
• Firefox browser with Firebug (also confirmed behavior using IE)
• SQL Server 2012 (not really relevant)

The only interesting thing I noticed is the following warning message appears in the output window in Visual Studio whenever I debug an Azure project:

Reconfiguring the private port 80 through 81 in the role of "My_Web_Role_Name" to avoid conflicts during emulation.

Perhaps this reassignment has something to do with it. Then, after running netstat -ano , I saw that the Azure Development Fabric process was listening on port 80, so maybe that’s why it needs to reassign ... it sounds fair enough.

In any case, I hope someone can think about what might cause this behavior. Here are some additional points and approaches that I have tried:

• Reset IIS, restart the Azure Compute / Storage emulator, restart the entire machine.
• Separate the views of all the script (i.e., the behavior still exists if the pages have only the basic HTML paragraph)
• I tried to find the appropriate IIS Express log files, but it seems that Azure projects are not starting through IIS Express, as another project does (is this correct?).
• I controlled the Azure Compute Emulator console ... there is nothing interesting there.

So ... finally, some questions:

• Can anyone else reproduce this behavior?
• Are there any additional logs created by Azure Emulator that can help me?

I really appreciate the push in the right direction here :).

Hooray!

EDIT

Here is some more info (I am updating this question while continuing to debug this issue):

I decided to run the solution without debugging, and I introduced some simple debugging messages in the About and Contact action methods, for example:

 public ActionResult About() { System.IO.File.AppendAllText(@"c:\Logs\azure.log", DateTime.Now + ": Contact, thread " + System.Threading.Thread.CurrentThread.ManagedThreadId + "\r\n"); return View(); } 

I attached Tail to this file and also ran Fiddler to find out what is happening over HTTP. Repeatedly moving between the two About and Contact links, I saw the following output:

From the tail:

• ...
• 12/09/2012 12:14:07 PM: Oh, stream 7
• 12/09/2012 12:14:08 PM: contact, stream 7
• 12/09/2012 12:14:09 PM: Oh, stream 7
• 12/9/2012 12:14:10 PM: contact, stream 7
• 12/09/2012 12:14:11 PM: Oh, stream 6
• 12/9/2012 12:14:12 PM: contact, topic 8
• 12/9/2012 12:14:31 PM: contact, topic 7
• 12/9/2012 12:14:50 PM: contact, topic 7
• 12/09/2012 12:15:03 PM: Oh, stream 7
• 12/9/2012 12:15:05 PM: contact, stream 8
• 12/09/2012 12:15:23 PM: Oh, stream 6
• 12/09/2012 12:15:42 PM: Oh, stream 14
• 12/09/2012 12:16:01 PM: Oh, stream 6
• 12/9/2012 12:16:31 PM: Contact, topic 7
• 12/09/2012 12:16:33 PM: Oh, stream 14
• ...
• 12/9/2012 12:17:08 PM: contact, topic 12
• 12/09/2012 12:17:09 PM: Oh, topic 12
• 12/09/2012 12:17:28 PM: About, thread 5
• ...

From Fiddler for the same requests (note that I also played with role endpoint settings, to no avail):

So ... return messages:

• HTTP traffic seems normal (Fiddler does not show any repeated requests)
• The action methods are called twice or three times for the same request, it would seem, non-deterministic (i.e. the original problem described above)
• It is interesting that in my situation it seems that exactly 19s passes before another “hit” occurs on the action method (that is, when action methods are hit several times, the time difference between the hits is determined)
• Thread IDs usually differ for each hit when multiple action methods are triggered.

I will continue to dig deeper ... maybe someone would be nice to offer an offer based on this updated information, though :).

EDIT 2 I decided to print the process ID along with the thread ID inside the action method in each case. For each request, the process ID corresponded to the IIS Express workflow process. Therefore, it seems that the IIS Express workflow sometimes spawns multiple threads for processing the request, each of which is divided into parts.

... and deeper I go ... :)

EDIT 3

I think I'm getting closer to the problem ... I looked at the IIS Express tray and saw the following after starting the application (without debugging):

I decided to use this URL directly (instead of 127.0.0.2:8888, as shown in Fiddler above), and the problem seems to be gone. Now it starts to make sense ... if I call the IIS application directly through port 8889, everything works fine, as it did with the original project, other than Azure. If I invoke a web role (i.e. the Azure part) through port 8888, sometimes it invokes the IIS application several times.

So now I'm starting to narrow my focus on roles on the Internet. I will try to see if I can find the relevant web role logs ...

EDIT 4

I made sure that the diagnostics were configured correctly in the application and started in debug mode. Then I switched to the Azure Compute Emulator interface, right-clicked on the instance of the web role and selected "Open Local Store". This led me to Explorer, where I managed to find the child folder "temp \ temp \ RoleTemp \ iisexpress", in which there was a log file with contents that looked like this (note that I changed my endpoint to port 80, so IIS application Express now returns to port 81):

• Request started: "GET" 127.0.0.1:81/Home/About
• The request ends: 127.0.0.1:81/Home/About with an HTTP status of 200.0
• Started request: "GET" 127.0.0.1:81/Home/Contact
• Request Complete: 127.0.0.1:81/Home/Contact with HTTP Status 200.0
• Started request: "GET" 127.0.0.1:81/Home/Contact
• Request Complete: 127.0.0.1:81/Home/Contact with HTTP Status 200.0
• Started request: "GET" 127.0.0.1:81/Home/Contact
• Request Complete: 127.0.0.1:81/Home/Contact with HTTP Status 200.0
• Started request: "GET" 127.0.0.1:81/Home/About
• The request ends: 127.0.0.1:81/Home/About with an HTTP status of 200.0
• Started request: "GET" 127.0.0.1:81/Home/About
• The request ends: 127.0.0.1:81/Home/About with an HTTP status of 200.0
• Started request: "GET" 127.0.0.1:81/Home/About
• The request ends: 127.0.0.1:81/Home/About with an HTTP status of 200.0
• Started request: "GET" 127.0.0.1:81/Home/Contact
• Request Complete: 127.0.0.1:81/Home/Contact with HTTP Status 200.0
• Started request: "GET" 127.0.0.1:81/Home/Contact
• Request Complete: 127.0.0.1:81/Home/Contact with HTTP Status 200.0
• Started request: "GET" 127.0.0.1:81/Home/Contact
• Request Complete: 127.0.0.1:81/Home/Contact with HTTP Status 200.0
• Request started: "GET" 127.0.0.1:81/Home/About
• The request ends: 127.0.0.1:81/Home/About with an HTTP status of 200.0
• Request started: "GET" 127.0.0.1:81/Home/Contact
• The request ends: 127.0.0.1:81/Home/Contact with HTTP status 200.0

Again, I simply switched between the About and Contact pages, so from the log you can see that a single request for a web role endpoint sometimes caused the web role to be called by the IIS application several times. Now to search some higher-level web role logs to find out why this is done :)!

Answer 1

So, based on what I wrote in the various editions of the original post:

• On my machine, at least the DevFC.exe process (development environment, Azure calculation emulator) sometimes generates several requests for the requested resource from IIS Express. That is, it will take one request to the endpoint of the emulator and sometimes generate several requests to the IIS Express base application one after another, even if all of them are successful with the HTTP status code 200. The time difference between these generated requests seems to be constant, but in addition, the behavior seems non-deterministic (i.e., this does not happen for every request on the emulator endpoint).
• To debug this, I need access to the source code of the Azure calculation emulator, which I assume I cannot get.

Now, my theory (and hope) is that this behavior is somehow specific to my development environment and will not reproduce in the real Azure infrastructure. However, I came up with a workaround ...

I created a simple proxy file for Firefox to redirect all traffic going to my local port of the Azure emulator to the port of the IIS Express application. This solves the problem, however I know that I am essentially bypassing the Azure emulator now, and I assume that I may have problems in the future because I do this (for example, if I decide to use the AppFabric cache?)

At the same time, I will continue this workaround for development and periodically deploy Azure in my staging environment for testing there.

Regardless, if anyone knows how I can solve a fundamental problem, let me know! :)

EDIT: I can no longer reproduce the issue using Azure SDK Tools version 2.2. If you still see this behavior, try updating it!