Isolating URL rewriting changes- ASP.NET IHttpModule + simple Ruby test for Quick Feedback

            I had to deal with URL rewriting very recently for a site. The old site was using ISAPI Rewrite module, so we decided to continue to use it, even though IIS7 comes with the URL rewriting module. Our SEO team came up with better URL scheme for the new site. As Search Engines indexed the old site URLS, a mapping between the old scheme and the new scheme had to be implemented, with proper redirection(301 status code) where ever necessary. 
 
Most of the URL scheme mapping involved regular expressions. I have had used regular expressions time to time, but not that extensive as required by this effort.

Following two things kept my fear factor very high, as this effort was added to the release very late.
       1)  making sure all 4000 possible old site links redirected to correct links in the new site
       2)  making sure new URL scheme works

When my fear factor goes high, I usually lean on tests.  I tried with tools used by SEO team, but they weren't quick enough.  I just wanted to focus on testing the URL rewriting in isolation with out running entire asp.net page processing.
  
So I needed to come up with a isolated testing solution that gives me quick feedback. A custom HTTP module came in handy for this situation. The idea is to have a ASP.NET HTTP module, that does nothing but writes the rewritten URL in begin request event, and ends the request.


My manual testing got faster, as the page responses were coming quickly with URL in the body.  I could finish new URL scheme working with manual testing.

Half of the battle is won. Still my fear factor is high, as I didn't have time to manually test all those 4000 old site links, while making sure nothing breaks as I add regular expressions for old scheme to new scheme mapping.

That's when I turned to Ruby. I have been playing around with Ruby on and off.  I decided to use Ruby to write tests to hit the site with all the links that I want to test.

So I got from the SEO team the list of old site links with following format in a spreadsheet.
        Old Scheme                 new scheme                         expected status code

I wrote not more than 200  lines of  ruby code which reads this test data from excel, and hit the server using net/http module, and saves the results back to the excel.It also prints the test results on the console similar to NUnit. This testing solution tested around 4600 links in about a minute.  All my fear was gone away when I looked at the test results in command window.(a sample test window is shown below.)

                    

This quick feedback helped us in finding the issues quickly. These tests in conjunction with SEO tools helped us in finding  the recursive behavior of URL rewriting which was crashing w3wp process.  SEO tools found a recursive behavior that was sending permanent redirects in loop. We had configuration file of  ISAPI rewrite engine's earlier version, which created this recursive behavior. Once it's removed,everything went smoothly.

Now with the safety net of tests, changes to url rewritings can be accomplished easily. It allows us to switch to different URL rewriting engines in future if the situation demands. Team can now embrace the change in much the XP way.

Next tasks added to my Todo list:
 1)  Add these tests to continuous integration.
 2)  Add a redirection limit to the test expectations to keep a tab on the looping behavior of URL rewrite engines.

This is my first ruby program at work. I am planning to use more ruby in writing tests.

Some of the ruby inspirations came from the book Everyday Scripting with Ruby: For Teams, Testers, and You.

This is how  XP's Test early, Test often, and Test Automated was applied.

Wish you a fearless programming.