IIS 7: Faulting module name: mscorwks.dll Faulting application name: w3wp.exe

We experienced w3wp process crash during our testing on staging environment, with following dump in the event log.

      Faulting application name: w3wp.exe, version: 7.5.7600.16385, time stamp: 0x4a5bd0eb

      Faulting module name: mscorwks.dll, version: 2.0.50727.4952, time stamp: 0x4bebe78c

      Exception code: 0xc0000005

Environment:
   IIS 7
   ASP.NET
  .NET 3.5
  Windows Server 2008, 64 bit
 

We had following three issues crashing the asp.net w3wp process:

1. Check whether the application pool's user has permissions  ( To detect, see this post )
2. Check whether enable 32-bit applications setting is true (Go to advance settings of the application pool)
 
3. Check whether your code is in infinite loop (we had ISAPI URL rewrite module going in long recursive for particular urls. For more details,see my other post)

Once these are in place, things went smoothly.

How did we find the issue earlier:
 
          All team testing on staging environment helped us in finding these issues sooner, making the production deployment much smoother. This particular team didn't have the safety net of tests, as technical practices that enable Agile is not followed during development. By that time I joined the team, it was too late to bring the technical practices to the team. So I had to request all team testing in staging environment covering major scenarios.

This exercise proved me once again that It's never too late to test, and get the quick feedback.
In any process, I found XP practice "Test Early, Test Often, Test Automated" is very essential in delivering quality software. That to me is the core of Agile practices. It enables the Agile.

Isolating URL rewriting changes- ASP.NET IHttpModule + simple Ruby test for Quick Feedback

            I had to deal with URL rewriting very recently for a site. The old site was using ISAPI Rewrite module, so we decided to continue to use it, even though IIS7 comes with the URL rewriting module. Our SEO team came up with better URL scheme for the new site. As Search Engines indexed the old site URLS, a mapping between the old scheme and the new scheme had to be implemented, with proper redirection(301 status code) where ever necessary. 
 
Most of the URL scheme mapping involved regular expressions. I have had used regular expressions time to time, but not that extensive as required by this effort.

Following two things kept my fear factor very high, as this effort was added to the release very late.
       1)  making sure all 4000 possible old site links redirected to correct links in the new site
       2)  making sure new URL scheme works

When my fear factor goes high, I usually lean on tests.  I tried with tools used by SEO team, but they weren't quick enough.  I just wanted to focus on testing the URL rewriting in isolation with out running entire asp.net page processing.
  
So I needed to come up with a isolated testing solution that gives me quick feedback. A custom HTTP module came in handy for this situation. The idea is to have a ASP.NET HTTP module, that does nothing but writes the rewritten URL in begin request event, and ends the request.


My manual testing got faster, as the page responses were coming quickly with URL in the body.  I could finish new URL scheme working with manual testing.

Half of the battle is won. Still my fear factor is high, as I didn't have time to manually test all those 4000 old site links, while making sure nothing breaks as I add regular expressions for old scheme to new scheme mapping.

That's when I turned to Ruby. I have been playing around with Ruby on and off.  I decided to use Ruby to write tests to hit the site with all the links that I want to test.

So I got from the SEO team the list of old site links with following format in a spreadsheet.
        Old Scheme                 new scheme                         expected status code

I wrote not more than 200  lines of  ruby code which reads this test data from excel, and hit the server using net/http module, and saves the results back to the excel.It also prints the test results on the console similar to NUnit. This testing solution tested around 4600 links in about a minute.  All my fear was gone away when I looked at the test results in command window.(a sample test window is shown below.)

                    

This quick feedback helped us in finding the issues quickly. These tests in conjunction with SEO tools helped us in finding  the recursive behavior of URL rewriting which was crashing w3wp process.  SEO tools found a recursive behavior that was sending permanent redirects in loop. We had configuration file of  ISAPI rewrite engine's earlier version, which created this recursive behavior. Once it's removed,everything went smoothly.

Now with the safety net of tests, changes to url rewritings can be accomplished easily. It allows us to switch to different URL rewriting engines in future if the situation demands. Team can now embrace the change in much the XP way.

Next tasks added to my Todo list:
 1)  Add these tests to continuous integration.
 2)  Add a redirection limit to the test expectations to keep a tab on the looping behavior of URL rewrite engines.

This is my first ruby program at work. I am planning to use more ruby in writing tests.

Some of the ruby inspirations came from the book Everyday Scripting with Ruby: For Teams, Testers, and You.

This is how  XP's Test early, Test often, and Test Automated was applied.

Wish you a fearless programming.