First, let’s configure Coypu to use the Selenium headless browser:

[Binding]
public class CoypuInitializer {
    [BeforeTestRun]
    public static void BeforeTestRun()
    {
        //I'm using the Dev Studio cassini server;
        Configuration.AppHost = "localhost";
        Configuration.Port = 64567;

        ConfigureForHeadlessBrowser();
    }

    private static void ConfigureForHeadlessBrowser()
    {
        Configuration.Driver = typeof(SeleniumHtmlUnitWebDriver);
    }

    [AfterScenario]
    public static void AfterScenario()
    {
        Coypu.Browser.EndSession();
    }
}

public class SeleniumHtmlUnitWebDriver : SeleniumWebDriver {
    public SeleniumHtmlUnitWebDriver() : base(new RemoteWebDriver(DesiredCapabilities.HtmlUnit())) {}
}

You will also need to add DLL references to Coypu.dll and WebDriver.dll (which is in the Coypu zip).

You can use this class to make other global configuration settings such as the Timeout, or the time Coypu waits between retries when looking for a HTML element. Full details can be found at the GitHub Coypu website, or the README.md file that comes with the Coypu binaries.

Now we are ready to implement the first feature step. As a reminder, the feature file is:

Scenario: Display Pi using Taylor series and Bucknall's BigNumber to many decimal places
	Given I visit the Pi.NET website
	And I have selected the 'Bucknall Big Number' algorithm
	And I have entered 500 decimal places
	When I press Go
	Then Pi should be displayed to the correct number of decimal places
	And Calculation statistics should be displayed

Let’s implement the “Given I visit the Pi.NET website” step:

[Given(@"I visit the Pi\.NET website")]
public void GivenIVisitThePiWebsite() {
    Browser.Session.Visit(string.Format("/"));
}

Using Coypu, I’m going to open the website specified in the CoypuInitializer class and visit the root of that website.

I now go to my MVC app and implement just enough to get that step passing.

Once I’ve done that, running the tests now gives:

Given I visit the Pi.NET website
-> done: PiNetSteps.GivenIVisitThePiWebsite() (15.4s)
And I have selected the 'Bucknall Big Number' algorithm
-> pending: PiNetSteps.GivenIHaveSelectedTheBucknallBigNumberAlgorithm("Bucknall Big Number")

i.e. – the first step has completed and the test has stopped at the next pending step. We’ve successfully hit the website.

I proceed like this, step by step, until I end up with the following steps file. I’ve added comments to explain the Coypu features I am using:

[Binding]
public class PiNetSteps:Steps {
    private string numberOfDecimalPlaces;

    [Given(@"I have selected the '(.+)' algorithm")]
    public void GivenIHaveSelectedAnAlgorithm(string algorithm) {
        //pass on this call to the generic radio button method (later in this class)
        Given(string.Format("I select the radio button '{0}'", algorithm));
    }

    [Given(@"I have entered (.+) decimal places")]
    public void GivenIHaveEnteredTheNumberOfRequiredDecimalPlaces(string numberOfDecimalPlaces) {
        this.numberOfDecimalPlaces = numberOfDecimalPlaces; //store for later use in an assert
        //populate a text box
        Browser.Session.FillIn("NumberOfDecimalPlaces").With(numberOfDecimalPlaces);
    }

    [When(@"I press Go")]
    public void WhenIPressGo() {
        //click a link. This fires an AJAX post to controller action
        //(I wanted to test JS capabilities of Selenium headless browser)
        Browser.Session.ClickLink("Calculate");
    }

    [Then(@"Pi should be displayed to the correct number of decimal places")]
    public void ThenPiShouldBeDisplayedToTheRequestedNumberOfDecimalPlaces() {
        //Check page HasContent I'm after - "Number of iterations" is displayed by AJAX call result
        if (Browser.Session.HasContent("Number of iterations:")) {
            //check Pi is calculated to correct number of places
            string piToNumberOfRequiredDecimalPlaces = Pi.Get(Convert.ToInt32(numberOfDecimalPlaces));
            Element piElement = Browser.Session.FindField("pi"); //find HTML element
            string calculatedPi = piElement.Text;
            Assert.That(piToNumberOfRequiredDecimalPlaces, Is.EqualTo(calculatedPi));
        }
        else {
            Assert.Fail("Could not find PI");
        }
    }

    [Then(@"Calculation statistics should be displayed")]
    public void ThenCalculationStatisticsShouldBeDisplayed() {
        //you can use actions/funcs with different retry timeouts
        Assert.IsTrue(Browser.Session.WithIndividualTimeout(TimeSpan.FromMilliseconds(1),
                       () => Browser.Session.HasContent("Number of iterations: ")),
                       "Number of Iterations taken not found");
        Assert.IsTrue(Browser.Session.WithIndividualTimeout(TimeSpan.FromMilliseconds(1),
                       () => Browser.Session.HasContent("Number of decimal places: " + numberOfDecimalPlaces)),
                       "Number of Decimal places calculated not found");
        Assert.IsTrue(Browser.Session.WithIndividualTimeout(TimeSpan.FromMilliseconds(1),
                       () => Browser.Session.HasContent("Elapsed Milliseconds: ")),
                       "Time taken not found");
    }

    [Given(@"I select the radio button '(.+)'")]
    public void SelectARadioButton(string radioButtonValue) {
        //select a radio button by value
        Browser.Session.Choose(radioButtonValue);
    }

    [Given(@"I visit the Pi.NET website")]
    public void GivenIVisitThePiPage() {
        //visit a web page at site specified in Coypu initialization.
        Browser.Session.Visit(string.Format("/"));
    }
}

I’ve used several SpecFlow and Coypu features in the class above. There are many more that you can read about at the GitHub Coypu website, or the README.md file that comes with the Coypu binaries. The documentation also shows how to use a real browser, rather than the Selenium headless one.

I’m going to use BDD to add a UI in front of the various implementations of Pi calculation algorithms I’ve been working on.

This post will describe what you need to install and how to get a SpecFlow feature file running via NUnit.

Prerequisites

Headless browser:

Firing up a real browser from a BDD test can be slow. So, I’m using the Selenium headless Java browser.

Download selenium-server-standalone-2.4.0.jar (and, of course, you need Java)

To start it up:

java -jar selenium-server-standalone-2.3.0.jar

SpecFlow:

Install SpecFlow with the full installer and it will integrate with Dev Studio.

You can keep your feature files in your solution and SpecFlow will automatically generate NUnit based code-behind files.

I also recommend you watch the SpecFlow Screen Cast – an excellent introduction to quickly get you up and running.

Coypu

To get my SpecFlow BDD tests to interact with the browser, I’m using Coypu.

Coypu is:

A more intuitive DSL for interacting with the browser in the way a human being would, inspired by the ruby framework Capybara.

Other software

You need NUnit (as SpecFlow creates NUnit tests).

Adding the first feature file

I’ve already got a few implementations of algorithms to calculate Pi. I just want a simple UI that allows you to select an algorithm, type how many decimal places you want Pi calculated to, and then hit “Calculate”. I want the “Calculate” link/button to call back to my application via AJAX (I want to see how the headless browser copes with Javascript).

Step 1: Add a feature file

A feature file is just a text file that describes the functionality you want to implement. It has a simple Given-When-Then structure that should be understandable by non-developers. Hence, they could be created by Business Analysts, etc.

I created a new class library called Spec and added a feature file using the “Add New Item” Dev Studio menu. You’ll see the option “SpecFlow Feature File”.

Feature: Calculate Pi using various algorithms
	I want to be able to view Pi to varying decimal places

Scenario: Display Pi using Taylor series and Bucknall's BigNumber to many decimal places
	Given I visit the Pi.NET website
	And I have selected the 'Bucknall Big Number' algorithm
	And I have entered 500 decimal places
	When I press Go
	Then Pi should be displayed to the correct number of decimal places
	And Calculation statistics should be displayed

You’ll also need to add DLL references to TechTalk.SpecFlow.dll and nunit.framework.dll

Once you’ve done this, compile and run all the unit tests in your Spec class library. You’ll see output like this:

Given I visit the Pi.NET website
-> No matching step definition found for the step. Use the following code to create one:
[Binding]
public class StepDefinitions {
   [Given(@"I visit the Pi\.NET website")]
   public void GivenIVisitThePi_NETWebsite()
   {
       ScenarioContext.Current.Pending();
   }
}
etc, etc.

SpecFlow has told you exactly what you need to do.

So, let’s copy and paste the code into a new C# file called PiNetSteps:

[Binding]
public class PiNetSteps {
    [Given(@"I visit the Pi\.NET website")]
    public void GivenIVisitThePi_NETWebsite() {
        ScenarioContext.Current.Pending();
    }

    [Given(@"I have selected the 'Bucknall Big Number' algorithm")]
    public void GivenIHaveSelectedTheBucknallBigNumberAlgorithm() {
        ScenarioContext.Current.Pending();
    }

    [Given(@"I have entered 500 decimal places")]
    public void GivenIHaveEntered500DecimalPlaces() {
        ScenarioContext.Current.Pending();
    }

    [When(@"I press Go")]
    public void WhenIPressGo() {
        ScenarioContext.Current.Pending();
    }

    [Then(@"Pi should be displayed to the correct number of decimal places")]
    public void ThenPiShouldBeDisplayedToTheCorrectNumberOfDecimalPlaces() {
        ScenarioContext.Current.Pending();
    }

    [Then(@"Calculation statistics should be displayed")]
    public void ThenCalculationStatisticsShouldBeDisplayed() {
        ScenarioContext.Current.Pending();
    }
}

Now when you run the unit tests you get the output:

Ignored: One or more step definitions are not implemented yet.

We can improve the steps code by parameterizing it with Regex. For example, rather than hard coding “500 decimal places”, let’s make this a parameter. And the algorithm name can also be a parameter.

Making these changes gives:

[Given(@"I have selected the '(.+)' algorithm")]
public void GivenIHaveSelectedAnAlgorithm(string algorithm) {
    ScenarioContext.Current.Pending();
}

[Given(@"I have entered '(.+)' decimal places")]
public void GivenIHaveEnteredTheRequiredNumberOfDecimalPlaces(int numberOfDecimalPlaces) {
    this.numberOfDecimalPlaces = numberOfDecimalPlaces; //store for a later assert
    ScenarioContext.Current.Pending();
}

We’re now at a stage that opens up lots of possibilities. You’ve gone from a feature file to a C# class and you now have options on how deep you want your tests to go. If you system under test is not a web application, you are ready to start implementing it straight away. If your system has lots of external dependencies that are not under your control, you could easily stub them out with a container. Or perhaps you want your tests to go through all the application layers and hit a database or webservice. You could easily introduce test database and webservices if you wish.

In the next post I’ll get Coypu up and running and use it to get my SpecFlow steps class to hit the Selenium headless browser.

Next: BDD with SpecFlow and Coypu (Part 2)

Now it is time for me to try an implementation alone!

I’m going to use the following John Machin formula:

Arctan can be calculated using the following Taylor series:

Rather than creating arbitrarily big numbers with BigInteger, I need to create numbers with arbitrarily large precision.   I need a BigRational. And one exists, but it did not make it into the .NET 4 framework. You can download BigRational form CodePlex.

So, here is my implementation of arctan:

public class ArcTanWithBigRational {
    public ArcTanWithBigRational() {
        Iterations = 0;
    }

    public int Iterations;

    public BigRational Calculate(BigRational x, int maxNumberOfIterations, int precision) {
        bool doASubtract = true;
        BigRational runningTotal = x;
        int count = 0;
        var divisor = new BigInteger(3);
        while (count < maxNumberOfIterations) {
            BigRational current = BigRational.Pow(x, divisor);
            current = current/divisor;
            if (doASubtract) {
                runningTotal = runningTotal - current;
            }
            else {
                runningTotal = runningTotal + current;
            }
            doASubtract = !doASubtract;
            count++;
            divisor = divisor + 2;
            if (WeHaveEnoughPrecision(current, precision)) {
                Iterations = count;
                break;
            }
        }
        return runningTotal;
    }

    private static bool WeHaveEnoughPrecision(BigRational current, int precision) {
        return current.GetFractionPart().ToString().Length > precision+2; //extra 2 digits to ensure enough precision
    }
}

Now to perform the calculation:

public string Calculate(int numberOfDigits) {
    var arcTanA = new ArcTanWithBigRational();
    var arcTanB = new ArcTanWithBigRational();
    var a = 16 * arcTanA.Calculate(BigRational.Divide(1, 5), 1000, numberOfDigits);
    var b = 4 * arcTanB.Calculate(BigRational.Divide(1, 239), 1000, numberOfDigits);
    var pi = a - b;
    return BigRationalPiFormatter.Format(pi, numberOfDigits);
}

And the results:

Precision = 1000 digits
Number of iterations = 923
Elapsed Milliseconds = 1192

Not bad! No where near as fast as the other implementations in my previous post, but not too bad and with far fewer iterations.

I can try one final optimization to try and get my implementation running faster. I have a dual-core laptop so let’s try some Parallel Tasks:

public string Calculate(int numberOfDigits) {
    var a = new ArcTanWithBigRational();
    var b = new ArcTanWithBigRational();
    Task<BigRational> task1 = Task<BigRational>.Factory.StartNew(
                            () => a.Calculate(BigRational.Divide(1, 5),1000,numberOfDigits));
    Task<BigRational> task2 = Task<BigRational>.Factory.StartNew(
                            () => b.Calculate(BigRational.Divide(1, 239), 1000, numberOfDigits));

    var pi = 16 * task1.Result - 4 * task2.Result;

    return BigRationalPiFormatter.Format(pi, numberOfDigits);
}

The results this time:

Precision = 1000 digits
Number of iterations = 923
Elapsed Milliseconds = 1099

A little faster, but nothing to get excited about.

For completeness, here is my routine to format the BigRational Pi into a string:

public class BigRationalPiFormatter {
    public static string Format(BigRational pi, int numberOfDigits)
    {
        BigInteger numeratorShiftedToEnoughDigits =
                       (pi.Numerator * BigInteger.Pow(new BigInteger(10), numberOfDigits));
        var bigInteger = numeratorShiftedToEnoughDigits / pi.Denominator;
        string piToBeFormatted = bigInteger.ToString();
        var builder = new StringBuilder();
        builder.Append(piToBeFormatted[0]);
        builder.Append(".");
        builder.Append(piToBeFormatted.Substring(1, numberOfDigits));
        return builder.ToString();
    }
}

Clearly I was doing something wrong.

A few Google searches later I found this implementation by Julian M Bucknall. In this implementation, Julian creates a BigNumber class to handle the fact that, before .NET 4, there was no way to handle numbers to many digits of precision. His class uses  fixed point arithmetic for storing data and performing calculations – effectively it works in base 4,294,967,296 (which is 2³²).

As .NET does now contain a BigInteger class (in .NET 4′s System.Numerics), I started searching for C# implementations that used this class. I couldn’t find anything in C#, but I did find a Java implementation that used a BigInteger class.

So, I decided to port it to C#, amending the BigInteger usages to match the .NET implementation syntax.

The formula used is the one that John Machin devised in 1706:

So, here is the .NET port of the Java implementation  - it includes a method called InverseTan to be used in the above formula, which I won’t even try to understand at the moment. I’m just doing the port!

public class PiJavaPort {
    public static BigInteger InverseTan(int denominator, int numberOfDigitsRequired) {
        int demonimatorSquared = denominator*denominator;

        int degreeNeeded = GetDegreeOfPrecisionNeeded(demonimatorSquared, numberOfDigitsRequired);

        BigInteger tenToNumberPowerOfDigitsRequired = GetTenToPowerOfNumberOfDigitsRequired(numberOfDigitsRequired);

        int c = 2*degreeNeeded + 1;
        BigInteger s = BigInteger.Divide(tenToNumberPowerOfDigitsRequired, new BigInteger(c)); // s = (10^N)/c
        for (int i = 0; i < degreeNeeded; i++) {
            c = c - 2;
            var temp1 = BigInteger.Divide(tenToNumberPowerOfDigitsRequired, new BigInteger(c));
            var temp2 = BigInteger.Divide(s, new BigInteger(demonimatorSquared));
            s = BigInteger.Subtract(temp1, temp2);
        }
        Console.WriteLine("Number of iterations=" + degreeNeeded);

        // return s/denominator, which is integer part of 10^numberOfDigitsRequired times arctan(1/k)
        return BigInteger.Divide(s, new BigInteger(denominator));

    }

    private static int GetDegreeOfPrecisionNeeded(int demonimatorSquared, int numberOfDigitsRequired) {
        //the degree of the Taylor polynomial needed to achieve numberOfDigitsRequired
        //digit accuracy of arctan(1/denominator).
        int degreeNeeded = 0;

        while ((Math.Log(2*degreeNeeded + 3) + (degreeNeeded + 1)*Math.Log10(demonimatorSquared))
                                                <= numberOfDigitsRequired*Math.Log(10)) {
            degreeNeeded++;
        }
        return degreeNeeded;
    }

    private static BigInteger GetTenToPowerOfNumberOfDigitsRequired(int numberOfDigitsRequired) {
        var tenToNumberOfDigitsRequired = new BigInteger(1);

        //  The following loop computes 10^numberOfDigitsRequired
        for (var i = 0; i < numberOfDigitsRequired; i++) {
            tenToNumberOfDigitsRequired = BigInteger.Multiply(tenToNumberOfDigitsRequired, new BigInteger(10));
        }
        return tenToNumberOfDigitsRequired;
    }
}

Finally, we need a method to put it all together – we need to implement the Machin formula:

public static string Calculate(int numberOfDigitsRequired)
{
    numberOfDigitsRequired += 8; //  To be safe, compute 8 extra digits, to be dropped at end. The 8 is arbitrary

    var a = BigInteger.Multiply(InverseTan(5, numberOfDigitsRequired), new BigInteger(16)); //16 x arctan(1/5)
    var b = BigInteger.Multiply(InverseTan(239, numberOfDigitsRequired), new BigInteger(4)); //4 x arctan(1/239)

    BigInteger pi = BigInteger.Subtract(a, b);

    var piAsString = BigInteger.Divide(pi, new BigInteger(100000000)).ToString();
    var piFormatted = piAsString[0]+"."+piAsString.Substring(1,numberOfDigitsRequired-8);
    return piFormatted;
}

Now to test the two implementations for performance – my Java port above vs. Julian M Bucknall implementation using his own BigNumber class.

First, my Java Port using .NET’s numerics:

Precision = 1000 digits
Number of iterations needed = 3659
Elapsed Milliseconds = 96

Not bad! Much better than 1 million plus iterations for 6 decimal places that my pathetic attempt took!

Now to Julian M Bucknall’s implementation (using his own BigNumber class):

Precision = 1000 digits
Number of iterations needed = 1603
Elapsed Milliseconds = 77

We have a winner.

Julian’s BigNumber class is optimised for this calculation – it can only divide a BigNumber by an integer. You cannot divide a BigNumber by another BigNumber. He states that if he had to implement such functionality he “would have discretely dropped the entire project”!

BigInteger does not have this limitation (and this functionality is used in the port).

Google took me to a post discussing using recursion to calculate Pi. Quite ironic considering the post is on Stack Overflow… Anyway, the post did provide me with plenty of links and algorithm names for me to refine my search and find a way to do it without recursion.

I soon found this: a collection of series algorithms to calculate Pi.

So, I picked an algorithm and started coding.

To get going quickly, I decided to just use the built in .NET value types (i.e. decimal, double). These clearly are not good enough if you want to calculate Pi to hundreds of decimal places, but they would do for now.

To check my algorithm implementation worked, I got Pi to a few thousand decimal places from the interweb :

public static class Pi {
    public const string PiAsString = "3.14159265358979323846264338327...etc";
    public static string Get(int numberOfDecimalPlaces) {
        return PiAsString.Substring(0, numberOfDecimalPlaces + 2);
    }
}

Leibniz-Gregory-Madhava Series

You can calculate Pi using the following:

In the code below, my base class property “PiCalculatedToRequiredPrecision” is, as the name suggests, there stop if I’ve reached my target number of digits. The “KeepGoing” property is used to bail out of the loop if I’ve done far too many iterations (which I keep track of in the Iteration property) and I have still not managed to get the required number of digits.

public class LeibnizGregoryMadhava:SeriesAlgorithmBase {
    public LeibnizGregoryMadhava(int numberOfDecimalPlaces) : base(numberOfDecimalPlaces) {}

    // Pi/4 = 1 - 1/3 + 1/5 - 1/7 + 1/9 - 1/11 .....

    public override decimal Calculate() {
        CurrentPi = 1;
        var multiplier = 1;
        var nextNumber = 3;

        decimal result = 1;
        while (KeepGoing) {
            multiplier = multiplier * -1;
            result = result + multiplier * (decimal)1 /(nextNumber);
            CurrentPi = result*4;
            if (PiCalculatedToRequiredPrecision)
            {
                break;
            }
            Iteration++;
            nextNumber += 2;
        }
        return CurrentPi;
    }
}

And now to run my algorithm a few times specifying a different precision each time:

LeibnizGregoryMadhava took 117 iterations and 6 milliseconds to calculate PI to 2 decimal places
LeibnizGregoryMadhava took 1686 iterations and 3 milliseconds to calculate PI to 3 decimal places
LeibnizGregoryMadhava took 10792 iterations and 22 milliseconds to calculate PI to 4 decimal places
LeibnizGregoryMadhava took 136119 iterations and 290 milliseconds to calculate PI to 5 decimal places
LeibnizGregoryMadhava took 1530010 iterations and 3358 milliseconds to calculate PI to 6 decimal places

Over 1.5 million iterations (and 3.4 seconds) to get to 6 decimal places!!

Time to try another algorithm.

Euler Series

public class Euler:SeriesAlgorithmBase {
    //pi^2 / 6 = 1 + 1/4 + 1/9 + ..... + 1/k^2

    public Euler(int numberOfDecimalPlaces) : base(numberOfDecimalPlaces) {}
    public override decimal Calculate() {
        decimal result = 0;
        Iteration = 1;
        while (KeepGoing)
        {
            long squared = Iteration * Iteration;
            result = result + (decimal)1/squared;
            CurrentPi = (decimal) Math.Sqrt((double) (result * 6));
            if (PiCalculatedToRequiredPrecision())
            {
                break;
            }
            Iteration++;
        }
        return CurrentPi;

    }
}

And the results:

Euler took 600 iterations and 74 milliseconds to calculate PI to 2 decimal places
Euler took 1611 iterations and 3 milliseconds to calculate PI to 3 decimal places
Euler took 10307 iterations and 22 milliseconds to calculate PI to 4 decimal places
Euler took 359863 iterations and 819 milliseconds to calculate PI to 5 decimal places
Euler took 1461054 iterations and 3319 milliseconds to calculate PI to 6 decimal places

Hmmm. Fewer iterations (just!) and 3.3 seconds to get to 6 decimal places.

I’m not going to break any world records with this code. Clearly I need to do more research.

To be continued.

I downloaded and installed the msysgit version from here: http://code.google.com/p/msysgit/downloads/list.

After installation is complete, create a directory for your source code and initialise a Git repository there:

md c:\dev\git-repo
cd c:\dev\git-repo
git init

Now, add some code/directories to c:\dev\git-repo.
You may also want to add a .gitignore file to exclude files you don’t want to check in: here is my .NET development .gitignore file.

When you’re done, add them to your Git repository and commit your changes:

git add .  <-- this adds all files beneath this level. Hence why you may want a .gitignore file.
git commit -m "Initial commit"

Now, so that you can carry your code around on a USB drive, do the following (where g: is my USB drive):

g:
md dev
cd dev
git clone --bare c:\dev\git-repo

You’ll see a message like:

Cloning into bare repository git-repo.git...
done.

The “bare” command leaves you with a Git repository without your files and directories. If you want to work on them you need to do the following (on say, your other computer):

c:
md dev
cd dev
git clone G:\dev\git-repo.git

Note: the last git clone was without the “bare” command – so your files will get extracted out of your USB repository and are available to work on.

obj
bin
_ReSharper.*
*.csproj.user
*.resharper.user
*.resharper
*.suo
*.cache
~$*

So, I’m now going to look at searching directly using Google’s RESTful API.

The URL that you hit is:

http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=MY SEARCH TEXT&rsz=large&key=MY GOOGLE KEY&start=0

You can omit the key, but Google recommends you provide it.

This returns a JSON result set like the following.

[Note: also see "Creating .NET objects from JSON using DataContractJsonSerializer"]

{"responseData":
 {"results":
 [
  {
  "GsearchResultClass":"GwebSearch",
  "unescapedUrl":"http://www.google.com/help/features.html",
  "url":"http://www.google.com/help/features.html",
  "visibleUrl":"www.google.com",
  "cacheUrl":"http://www.google.com/search?q\u003dcache:BNRWhS8EKYAJ:www.google.com",
  "title":"\u003cb\u003eSearch\u003c/b\u003e Features - \u003cb\u003eGoogle\u003c/b\u003e",
  "titleNoFormatting":"Search Features - Google",
  "content":"To find reviews and showtimes...."
  },
  {
  "GsearchResultClass":"GwebSearch",
  etc
  },
  etc
 ],
 "cursor": {
 "pages": [
  { "start": "0", "label": 1 },
  { "start": "8", "label": 2 },
  etc
  { "start": "56","label": 8 }
  ],
  "estimatedResultCount": "59600000",
  "currentPageIndex": 0,
  "moreResultsUrl": "http://www.google.com/search?oe=utf8&ie=utf8&source=uds&start=0&hl=en&q=MY SEARCH TEXT"
  }
 },
 "responseDetails": null,
 "responseStatus": 200
}

Now we’re getting somewhere. The “cursor” block at the bottom of the returned data provides paging information, a “moreResultsUrl”, an estimate of the number of results. Eveything we need.

As I have been told the number of pages (in the above example there are 8 pages) and the starting point of each page (0, 8, …, 56), I can just use my original URL again, and adjust the start parameter for each call (i.e. I don’t need to use the moreResultsUrl data provided in the cursor block).

So, I can just use:

http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=MY SEARCH TEXT&rsz=large&key=MY Google KEY&start=8

After playing with some more searches I realised that a maximum of 8 pages were being returned each time.

In my example above, each page returned 8 results. 8 pages of 8 results does not give me 59600000 results!

I tried the following URL:

http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=MY SEARCH TEXT&rsz=large&start=100

The result returned was:

{"responseData": null, "responseDetails": "out of range start", "responseStatus": 400}

Google does not allow you to search for more than 64 results!

So the only way to get more than 64 results would be to screen scrape, which is ugly (but can be made more bearable by using SGMLReader and LINQ to XML). Though I’m pretty sure Google’s T’s & C’s don’t allow screen scraping!

I needed to do some web searches and record the host names that I found, so I started investigating what APIs Google exposed.

I found the “Google APIs for .NET Framework” hosted on Google Code. There hasn’t been much activity on this project recently – the last release was April 2010 – and there is not much documentation, but it works and it is simple to use.

It provides two sets of libraries: one that wraps Google Search and one that wraps Google Translate.

Here is an example of a search:

public void SearchForSomethingUsingLib() {
  var client = new GwebSearchClient("http://mywebsite.com");
  var results = client.Search("google api for .NET", 100);
  foreach (var webResult in results) {
   Console.WriteLine("{0}, {1}, {2}", webResult.Title, webResult.Url, webResult.Content);
  }
}

The ’100′ in the above example indicates the number of search results to return.

What puzzled me was that the set of results returned didn’t appear to have a concept of paging, and the library didn’t seem to have a “Search Paged” method. What if I had asked for 1000000 results? Surely the API wasn’t going to get 1000000 results in one go and load them all into a List?

I fired up reflector. The API definitely returned a straight forward generic list. No delayed execution – one Web Request sent to Google, followed by one “new List<T>” to return the results.

I needed to investigate further, and I’ll look at that in Part 2.