Kash Farooq's software development blog

.NET Developer

Posts Tagged ‘.NET’

Getting started with MongoDB in C#

Posted by Kash Farooq on February 18, 2012

I thought I’d play with NoSQL. There are lots of implementations to choose from, but I’ve decided to start with MongoDB. Especially after seeing the impressive list of real-life production MongoDB deployments.

The MongoDB server and data structure is organised as follows:

  • A Mongo server holds a set of databases
  • A database holds a set of collections
  • A collection holds a set of documents
  • A document is a set of fields
  • A field is a key-value pair
  • A key is a string
  • A value is a
    • strings, integers, etc.
    • another document
    • an array of values

Right. Let’s get started. First, download the two things you’ll need to start developing MongoDB apps in .NET: the database software and the C# driver.

I also recommend you download a PDF of the documentation. It is available as a MongoDB docs daily build. This PDF contains everything: set-up instructions, tutorials, advice and language specific driver documentations.

Use the Windows Quick Start to get going and test your MongoDB server with the administrative JavaScript shell client. I also advise you go through the tutorial – it walks you through more complex examples using the administrative shell client. As noted above, both these guides are in the MongoDB Docs PDF.

Install the C# driver by just running the MSI. It installs to C:\Program Files (x86)\MongoDB.

Create a .NET project and add assembly references to MongoDB.Driver.dll and MongoDB.Bson.dll. I created NUnit integration tests to test MongoDB.

First, I created a class that I wanted to store as a MongoDB document:

public class Movie
{
  public BsonObjectId Id { get; set; }
  public string Title { get; set; }
  public string Year { get; set; }
  public List<string> Actors { get; set; }

  public void AddActor(string actor)
  {
    if (Actors == null)
    {
      Actors = new List<string>();
    }
    Actors.Add(actor);
  }
}

Things to note in this class:

  • I’ve used the type BsonObjectId. To get up and running quickly, I basically told the MongoDB C# driver which property to use as an ID. You don’t have to do this. To see how to remove MongoDB C# driver specific types from your domain object, see my Mapping IDs post.
  • I’ve included a generic list – I wanted to see how the C# driver copes with this, how MongoDB stores this data, and how we can search for documents using this data.
  • I had to make the Actors list public, otherwise the C# driver wouldn’t store it. Perhaps there is a way to force the driver to store it without making it public? Something to research.

Next, let’s open a connection to a database and insert some data. I’m orchestrating the prerequisites for my test through a NUnit Setup method the connects to the database, empties the collection and inserts some fresh data:

private MongoServer server;
private MongoDatabase moviesDatabase;

[SetUp]
public void SetUp()
{
  Connect();
  Clean();
  InsertData();
}

private void Connect()
{
  server = MongoServer.Create();
  moviesDatabase = server.GetDatabase("movies_db");
}

As you can see, the connect method didn’t use any information such as connection strings, usernames, etc. It just calls Create. If you don’t provide any parameters it connects to a default instance of MongoDB on localhost. If you’ve changed any of the default settings, you’ll need a connection string.

The GetDatabase method gets me an instance of the “movies” database if it exists, or lazily creates it if it doesn’t (i.e. it won’t actually be created until you save some data to it).

The Clean() method is just used to empty the collection each time a test runs. I’m calling the RemoveAll method to empty all elements from the movies_collection:

private void Clean()
{
  var moviesCollection = moviesDatabase.GetCollection<Movie>("movies_collection");
  moviesCollection.RemoveAll();
}

Now let’s add some data to the movies_collection and save it to the movies_db:

private void InsertData()
{
  //Create some data
  var movie1 = new Movie {Title = "Indiana Jones and the Raiders of the Lost Ark", Year = "1981"};
  movie1.AddActor("Harrison Ford");
  movie1.AddActor("Karen Allen");
  movie1.AddActor("Paul Freeman");

  var movie2 = new Movie {Title = "Star Wars: Episode IV - A New Hope", Year = "1977"};
  movie2.AddActor("Mark Hamill");
  movie2.AddActor("Harrison Ford");
  movie2.AddActor("Carrie Fisher");

  var movie3 = new Movie {Title = "Das Boot", Year = "1981"};
  movie3.AddActor("Jürgen Prochnow");
  movie3.AddActor("Herbert Grönemeyer");
  movie3.AddActor("Klaus Wennemann");

  //Insert the movies into the movies_collection
  var moviesCollection = moviesDatabase.GetCollection<Movie>("movies_collection");
  moviesCollection.Insert(movie1);
  moviesCollection.Insert(movie2);
  moviesCollection.Insert(movie3);
}

So, we’ve created/opened a database called movie_db and insert some data into a movies_collection. Let’s check it’s there – get the collection and check that there are 3 movies in it.

[Test]
public void ShouldFindThreeMoviesInTheCollection()
{
  var moviesCollectionRetrieved = moviesDatabase.GetCollection<Movie>("movies_collection");
  Assert.That(moviesCollectionRetrieved.Count(), Is.EqualTo(3));
}

Now let’s do a query on the movie data. Let’s look for movies that were released in 1981, i.e. perform a query based on one of the keys – in this case, the “Year” key:

[Test]
public void ShouldFindTwoMoviesThatWereMadeIn1981()
{
  var moviesCollectionRetrieved = moviesDatabase.GetCollection<Movie>("movies_collection");

  QueryComplete findMoviesMadeIn1981Query = Query.EQ("Year", "1981");
  var moviesFound = moviesCollectionRetrieved.FindAs<Movie>(findMoviesMadeIn1981Query);

  Assert.That(moviesFound.Count(), Is.EqualTo(2));
  Assert.That(moviesFound.Count(movie => movie.Title == "Das Boot"), Is.EqualTo(1));
  Assert.That(moviesFound.Count(movie => movie.Title == "Indiana Jones and the Raiders of the Lost Ark"),Is.EqualTo(1));
}

Note that I used the generic version FindAs method – to get the C# driver to create movie objects from the MongoDB documents.

You can also very easily perform queries based on values stored in lists. Here, for example, I am looking for documents that contain a certain value in the Actors list – let’s look for movies that Harrison Ford has been in:

[Test]
public void ShouldFindTwoMoviesThatHarrisonFordHasBeenIn()
{
  var moviesCollectionRetrieved = moviesDatabase.GetCollection<Movie>("movies_collection");

  var findMoviesWithHarrisonFordInThem = Query.EQ("Actors", "Harrison Ford");
  var moviesFound = moviesCollectionRetrieved.FindAs<Movie>(findMoviesWithHarrisonFordInThem);
  Assert.That(moviesFound.Count(), Is.EqualTo(2));
  Assert.That(moviesFound.Count(movie => movie.Title == "Star Wars: Episode IV - A New Hope"),Is.EqualTo(1));
  Assert.That(moviesFound.Count(movie => movie.Title == "Indiana Jones and the Raiders of the Lost Ark"),Is.EqualTo(1));
}

Very neat.

In the code above I’ve been using Count(). That does not mean that the C# driver supports delayed execution and LINQ. The official statement is: “Version 1.0 of the official C# driver does not yet support LINQ”. In fact, the documentation states that for the following code, the query is sent to the server twice (once for FirstOrDefault and once for LastOrDefault)


var query = Query.EQ("author", "Ernest Hemingway");
var cursor = books.Find(query);
var firstBook = cursor.FirstOrDefault(); //query executed twice on the server
var lastBook = cursor.LastOrDefault(); //once for each of these IEnumerable<T> extension methods.

Finally, let’s look at how the data has been stored using the administrative JavaScript shell client, mongo.exe.

With the client, I’ll connect to the server, switch to the movies database and then display all the documents in the movies_collection:


MongoDB shell version: 2.0.2
connecting to: test
> use movies_db
switched to db movies_db
> db.movies_collection.find().forEach(printjson);
{
 "_id" : ObjectId("4f3f82a1d5c8851b60430b9a"),
 "Title" : "Indiana Jones and the Raiders of the Lost Ark",
 "Year" : "1981",
 "Actors" : [
 "Harrison Ford",
 "Karen Allen",
 "Paul Freeman"
 ]
}
{
 "_id" : ObjectId("4f3f82a1d5c8851b60430b9b"),
 "Title" : "Star Wars: Episode IV - A New Hope",
 "Year" : "1977",
 "Actors" : [
 "Mark Hamill",
 "Harrison Ford",
 "Carrie Fisher"
 ]
}
{
 "_id" : ObjectId("4f3f82a1d5c8851b60430b9c"),
 "Title" : "Das Boot",
 "Year" : "1981",
 "Actors" : [
 "J├╝rgen Prochnow",
 "Herbert Gr├Ânemeyer",
 "Klaus Wennemann"
 ]
}
>

Posted in .NET, NoSQL | Tagged: , , , | Leave a Comment »

BDD with SpecFlow and Coypu (Part 2)

Posted by Kash Farooq on August 23, 2011

In my last post I introduced BDD with SpecFlow. I created a  feature file and got to a point where I now needed to hook up my BDD tests with browser integration. To do this, I’m going to use Coypu and the Selenium headless browser. See my previous post for download links.

First, let’s configure Coypu to use the Selenium headless browser:

[Binding]
public class CoypuInitializer {
    [BeforeTestRun]
    public static void BeforeTestRun()
    {
        //I'm using the Dev Studio cassini server;
        Configuration.AppHost = "localhost";
        Configuration.Port = 64567;

        ConfigureForHeadlessBrowser();
    }

    private static void ConfigureForHeadlessBrowser()
    {
        Configuration.Driver = typeof(SeleniumHtmlUnitWebDriver);
    }

    [AfterScenario]
    public static void AfterScenario()
    {
        Coypu.Browser.EndSession();
    }
}

 

public class SeleniumHtmlUnitWebDriver : SeleniumWebDriver {
    public SeleniumHtmlUnitWebDriver() : base(new RemoteWebDriver(DesiredCapabilities.HtmlUnit())) {}
}

You will also need to add DLL references to Coypu.dll and WebDriver.dll (which is in the Coypu zip).

You can use this class to make other global configuration settings such as the Timeout, or the time Coypu waits between retries when looking for a HTML element. Full details can be found at the GitHub Coypu website, or the README.md file that comes with the Coypu binaries.

Now we are ready to implement the first feature step. As a reminder, the feature file is:

Scenario: Display Pi using Taylor series and Bucknall's BigNumber to many decimal places
	Given I visit the Pi.NET website
	And I have selected the 'Bucknall Big Number' algorithm
	And I have entered 500 decimal places
	When I press Go
	Then Pi should be displayed to the correct number of decimal places
	And Calculation statistics should be displayed

Let’s implement the “Given I visit the Pi.NET website” step:

[Given(@"I visit the Pi\.NET website")]
public void GivenIVisitThePiWebsite() {
    Browser.Session.Visit(string.Format("/"));
}

Using Coypu, I’m going to open the website specified in the CoypuInitializer class and visit the root of that website.

I now go to my MVC app and implement just enough to get that step passing.

Once I’ve done that, running the tests now gives:


Given I visit the Pi.NET website
-> done: PiNetSteps.GivenIVisitThePiWebsite() (15.4s)
And I have selected the 'Bucknall Big Number' algorithm
-> pending: PiNetSteps.GivenIHaveSelectedTheBucknallBigNumberAlgorithm("Bucknall Big Number")

i.e. – the first step has completed and the test has stopped at the next pending step. We’ve successfully hit the website.

I proceed like this, step by step, until I end up with the following steps file. I’ve added comments to explain the Coypu features I am using:

[Binding]
public class PiNetSteps:Steps {
    private string numberOfDecimalPlaces;

    [Given(@"I have selected the '(.+)' algorithm")]
    public void GivenIHaveSelectedAnAlgorithm(string algorithm) {
        //pass on this call to the generic radio button method (later in this class)
        Given(string.Format("I select the radio button '{0}'", algorithm));
    }

    [Given(@"I have entered (.+) decimal places")]
    public void GivenIHaveEnteredTheNumberOfRequiredDecimalPlaces(string numberOfDecimalPlaces) {
        this.numberOfDecimalPlaces = numberOfDecimalPlaces; //store for later use in an assert
        //populate a text box
        Browser.Session.FillIn("NumberOfDecimalPlaces").With(numberOfDecimalPlaces);
    }

    [When(@"I press Go")]
    public void WhenIPressGo() {
        //click a link. This fires an AJAX post to controller action
        //(I wanted to test JS capabilities of Selenium headless browser)
        Browser.Session.ClickLink("Calculate");
    }

    [Then(@"Pi should be displayed to the correct number of decimal places")]
    public void ThenPiShouldBeDisplayedToTheRequestedNumberOfDecimalPlaces() {
        //Check page HasContent I'm after - "Number of iterations" is displayed by AJAX call result
        if (Browser.Session.HasContent("Number of iterations:")) {
            //check Pi is calculated to correct number of places
            string piToNumberOfRequiredDecimalPlaces = Pi.Get(Convert.ToInt32(numberOfDecimalPlaces));
            Element piElement = Browser.Session.FindField("pi"); //find HTML element
            string calculatedPi = piElement.Text;
            Assert.That(piToNumberOfRequiredDecimalPlaces, Is.EqualTo(calculatedPi));
        }
        else {
            Assert.Fail("Could not find PI");
        }
    }

    [Then(@"Calculation statistics should be displayed")]
    public void ThenCalculationStatisticsShouldBeDisplayed() {
        //you can use actions/funcs with different retry timeouts
        Assert.IsTrue(Browser.Session.WithIndividualTimeout(TimeSpan.FromMilliseconds(1),
                       () => Browser.Session.HasContent("Number of iterations: ")),
                       "Number of Iterations taken not found");
        Assert.IsTrue(Browser.Session.WithIndividualTimeout(TimeSpan.FromMilliseconds(1),
                       () => Browser.Session.HasContent("Number of decimal places: " + numberOfDecimalPlaces)),
                       "Number of Decimal places calculated not found");
        Assert.IsTrue(Browser.Session.WithIndividualTimeout(TimeSpan.FromMilliseconds(1),
                       () => Browser.Session.HasContent("Elapsed Milliseconds: ")),
                       "Time taken not found");
    }

    [Given(@"I select the radio button '(.+)'")]
    public void SelectARadioButton(string radioButtonValue) {
        //select a radio button by value
        Browser.Session.Choose(radioButtonValue);
    }

    [Given(@"I visit the Pi.NET website")]
    public void GivenIVisitThePiPage() {
        //visit a web page at site specified in Coypu initialization.
        Browser.Session.Visit(string.Format("/"));
    }
}

I’ve used several SpecFlow and Coypu features in the class above. There are many more that you can read about at the GitHub Coypu website, or the README.md file that comes with the Coypu binaries. The documentation also shows how to use a real browser, rather than the Selenium headless one.

Posted in .NET, Behaviour Driven Development, Test Driven Development | Tagged: , , , , | Leave a Comment »

Calculating Pi in C# part 3 – using the BigRational class

Posted by Kash Farooq on August 1, 2011

In an earlier post I ported a Java implementation of a Pi calculator – my port used the BigInteger class that lives in the System.Numerics assembly of .NET 4. I also performance tested my port against an implementation by Julian M Bucknall that used his own BigNumber class. Julian’s implementation won that performance test, coming in at an impressive 70-80 milliseconds to calculate Pi to 1000 decimal places.

Now it is time for me to try an implementation alone!

I’m going to use the following John Machin formula:

John Machin's Pi formulaArctan can be calculated using the following Taylor series:

ArcTan calculated using the Taylor SeriesRather than creating arbitrarily big numbers with BigInteger, I need to create numbers with arbitrarily large precision.   I need a BigRational. And one exists, but it did not make it into the .NET 4 framework. You can download BigRational form CodePlex.

So, here is my implementation of arctan:

public class ArcTanWithBigRational {
    public ArcTanWithBigRational() {
        Iterations = 0;
    }

    public int Iterations;

    public BigRational Calculate(BigRational x, int maxNumberOfIterations, int precision) {
        bool doASubtract = true;
        BigRational runningTotal = x;
        int count = 0;
        var divisor = new BigInteger(3);
        while (count < maxNumberOfIterations) {
            BigRational current = BigRational.Pow(x, divisor);
            current = current/divisor;
            if (doASubtract) {
                runningTotal = runningTotal - current;
            }
            else {
                runningTotal = runningTotal + current;
            }
            doASubtract = !doASubtract;
            count++;
            divisor = divisor + 2;
            if (WeHaveEnoughPrecision(current, precision)) {
                Iterations = count;
                break;
            }
        }
        return runningTotal;
    }

    private static bool WeHaveEnoughPrecision(BigRational current, int precision) {
        return current.GetFractionPart().ToString().Length > precision+2; //extra 2 digits to ensure enough precision
    }
}

Now to perform the calculation:

public string Calculate(int numberOfDigits) {
    var arcTanA = new ArcTanWithBigRational();
    var arcTanB = new ArcTanWithBigRational();
    var a = 16 * arcTanA.Calculate(BigRational.Divide(1, 5), 1000, numberOfDigits);
    var b = 4 * arcTanB.Calculate(BigRational.Divide(1, 239), 1000, numberOfDigits);
    var pi = a - b;
    return BigRationalPiFormatter.Format(pi, numberOfDigits);
}

And the results:

Precision = 1000 digits
Number of iterations = 923
Elapsed Milliseconds = 1192

Not bad! No where near as fast as the other implementations in my previous post, but not too bad and with far fewer iterations.

I can try one final optimization to try and get my implementation running faster. I have a dual-core laptop so let’s try some Parallel Tasks:

public string Calculate(int numberOfDigits) {
    var a = new ArcTanWithBigRational();
    var b = new ArcTanWithBigRational();
    Task<BigRational> task1 = Task<BigRational>.Factory.StartNew(
                            () => a.Calculate(BigRational.Divide(1, 5),1000,numberOfDigits));
    Task<BigRational> task2 = Task<BigRational>.Factory.StartNew(
                            () => b.Calculate(BigRational.Divide(1, 239), 1000, numberOfDigits));

    var pi = 16 * task1.Result - 4 * task2.Result;

    return BigRationalPiFormatter.Format(pi, numberOfDigits);
}

The results this time:

Precision = 1000 digits
Number of iterations = 923
Elapsed Milliseconds = 1099

A little faster, but nothing to get excited about.

For completeness, here is my routine to format the BigRational Pi into a string:

public class BigRationalPiFormatter {
    public static string Format(BigRational pi, int numberOfDigits)
    {
        BigInteger numeratorShiftedToEnoughDigits =
                       (pi.Numerator * BigInteger.Pow(new BigInteger(10), numberOfDigits));
        var bigInteger = numeratorShiftedToEnoughDigits / pi.Denominator;
        string piToBeFormatted = bigInteger.ToString();
        var builder = new StringBuilder();
        builder.Append(piToBeFormatted[0]);
        builder.Append(".");
        builder.Append(piToBeFormatted.Substring(1, numberOfDigits - 1));
        return builder.ToString();
    }
}

Posted in .NET, Algorithms | Tagged: , , , | 4 Comments »

Calculating Pi in C# with series algorithms

Posted by Kash Farooq on July 17, 2011

People write applications to calculate Pi to thousands of decimal places – I decided to investigate how they do that.

Google took me to a post discussing using recursion to calculate Pi. Quite ironic considering the post is on Stack Overflow… Anyway, the post did provide me with plenty of links and algorithm names for me to refine my search and find a way to do it without recursion.

I soon found this: a collection of series algorithms to calculate Pi.

So, I picked an algorithm and started coding.

To get going quickly, I decided to just use the built in .NET value types (i.e. decimal, double). These clearly are not good enough if you want to calculate Pi to hundreds of decimal places, but they would do for now.

To check my algorithm implementation worked, I got Pi to a few thousand decimal places from the interweb :

public static class Pi {
    public const string PiAsString = "3.14159265358979323846264338327...etc";
    public static string Get(int numberOfDecimalPlaces) {
        return PiAsString.Substring(0, numberOfDecimalPlaces + 2);
    }
}

Leibniz-Gregory-Madhava Series

You can calculate Pi using the following:

In the code below, my base class property “PiCalculatedToRequiredPrecision” is, as the name suggests, there stop if I’ve reached my target number of digits. The “KeepGoing” property is used to bail out of the loop if I’ve done far too many iterations (which I keep track of in the Iteration property) and I have still not managed to get the required number of digits.

public class LeibnizGregoryMadhava:SeriesAlgorithmBase {
    public LeibnizGregoryMadhava(int numberOfDecimalPlaces) : base(numberOfDecimalPlaces) {}

    // Pi/4 = 1 - 1/3 + 1/5 - 1/7 + 1/9 - 1/11 .....

    public override decimal Calculate() {
        CurrentPi = 1;
        var multiplier = 1;
        var nextNumber = 3;

        decimal result = 1;
        while (KeepGoing) {
            multiplier = multiplier * -1;
            result = result + multiplier * (decimal)1 /(nextNumber);
            CurrentPi = result*4;
            if (PiCalculatedToRequiredPrecision)
            {
                break;
            }
            Iteration++;
            nextNumber += 2;
        }
        return CurrentPi;
    }
}

And now to run my algorithm a few times specifying a different precision each time:

LeibnizGregoryMadhava took 117 iterations and 6 milliseconds to calculate PI to 2 decimal places
LeibnizGregoryMadhava took 1686 iterations and 3 milliseconds to calculate PI to 3 decimal places
LeibnizGregoryMadhava took 10792 iterations and 22 milliseconds to calculate PI to 4 decimal places
LeibnizGregoryMadhava took 136119 iterations and 290 milliseconds to calculate PI to 5 decimal places
LeibnizGregoryMadhava took 1530010 iterations and 3358 milliseconds to calculate PI to 6 decimal places

Over 1.5 million iterations (and 3.4 seconds) to get to 6 decimal places!!

Time to try another algorithm.

Euler Series

public class Euler:SeriesAlgorithmBase {
    //pi^2 / 6 = 1 + 1/4 + 1/9 + ..... + 1/k^2

    public Euler(int numberOfDecimalPlaces) : base(numberOfDecimalPlaces) {}
    public override decimal Calculate() {
        decimal result = 0;
        Iteration = 1;
        while (KeepGoing)
        {
            long squared = Iteration * Iteration;
            result = result + (decimal)1/squared;
            CurrentPi = (decimal) Math.Sqrt((double) (result * 6));
            if (PiCalculatedToRequiredPrecision())
            {
                break;
            }
            Iteration++;
        }
        return CurrentPi;

    }
}

And the results:

Euler took 600 iterations and 74 milliseconds to calculate PI to 2 decimal places
Euler took 1611 iterations and 3 milliseconds to calculate PI to 3 decimal places
Euler took 10307 iterations and 22 milliseconds to calculate PI to 4 decimal places
Euler took 359863 iterations and 819 milliseconds to calculate PI to 5 decimal places
Euler took 1461054 iterations and 3319 milliseconds to calculate PI to 6 decimal places

Hmmm. Fewer iterations (just!) and 3.3 seconds to get to 6 decimal places.

I’m not going to break any world records with this code. Clearly I need to do more research.

Related posts

Posted in .NET, Algorithms | Tagged: , , , | 4 Comments »

Creating .NET objects from JSON using DataContractJsonSerializer

Posted by Kash Farooq on January 31, 2011

Creating .NET objects from JSON using DataContract and DataMember

Note: .NET 4 has made it far easier to do this. See Creating objects from JSON in .NET 4.

In a previous post and demonstraed that the RESTful Google Search API returns data as JSON. I needed a way to convert the JSON data into .NET objects and this post shows what I ended up with.

Here is an example of the JSON result set returned:

{"responseData":
 {"results":
 [
  {
  "GsearchResultClass":"GwebSearch",
  "unescapedUrl":"http://www.google.com/help/features.html",
  "url":"http://www.google.com/help/features.html",
  "visibleUrl":"www.google.com",
  "cacheUrl":"http://www.google.com/search?q\u003dcache:BNRWhS8EKYAJ:www.google.com",
  "title":"\u003cb\u003eSearch\u003c/b\u003e Features - \u003cb\u003eGoogle\u003c/b\u003e",
  "titleNoFormatting":"Search Features - Google",
  "content":"To find reviews and showtimes...."
  },
  {
  "GsearchResultClass":"GwebSearch",
  etc
  },
  etc
 ],
 "cursor": {
 "pages": [
  { "start": "0", "label": 1 },
  { "start": "8", "label": 2 },
  etc
  { "start": "56","label": 8 }
  ],
  "estimatedResultCount": "59600000",
  "currentPageIndex": 0,
  "moreResultsUrl": "http://www.google.com/search?oe=utf8&ie=utf8&source=uds&start=0&hl=en&q=MY SEARCH TEXT"
  }
 },
 "responseDetails": null,
 "responseStatus": 200
}

We can use System.Runtime.Serialization to create .NET objects from this JSON data.

For example, the top level JSON responseData can be represented by the following .NET type:

[DataContract]
public class GoogleSearchResults {
  [DataMember(Name = "responseData")]
  public ResponseData ResponseData { get; set; }
}

Note that I’ve specified the Name attribute. Without it I would have had to have a property called “responseData” rather than “ResponseData”.

The rest of the data is extracted using the classes below. Note that I can be quite selective in what data I wanted to transfer from JSON to .NET. If I don’t need, say, the “cacheUrl”, I can just omit it from my .NET objects. I can also rename data. I have put the data from “titleNoFormatting” into a property called Title:

[DataContract]
public class ResponseData
{
  [DataMember(Name="results")]
  public IEnumerable<Result> Results { get; set; }

  [DataMember(Name = "cursor")]
  public Cursor Cursor { get; set; }
}

[DataContract]
public class Cursor
{
  [DataMember(Name = "moreResultsUrl")]
  public string MoreResultsUrl { get; set; }

  [DataMember(Name = "pages")]
  public IEnumerable<Page> Pages { get; set; }
}

[DataContract]
public class Result
{
  [DataMember(Name = "url")]
  public string Url { get; set; }

  [DataMember(Name = "titleNoFormatting")]
  public string Title { get; set; }
}

[DataContract]
public class Page
{
  [DataMember(Name = "start")]
  public int Start { get; set; }

  [DataMember(Name = "label")]
  public string Label { get; set; }
}

Finally, I need a method to actually deserialize a JSON string into my .NET objects. I can use System.Runtime.Serialization.Json.DataContractJsonSerializer to do this. I have created the following generic method that takes a JSON string and the type I want it to be deserialised into, and then returns an instantiated .NET object:

public static T Deserialise(string json) {
  var obj = Activator.CreateInstance();
  using (var memoryStream = new MemoryStream(Encoding.Unicode.GetBytes(json))) {
    var serializer = new DataContractJsonSerializer(obj.GetType());
    obj = (T) serializer.ReadObject(memoryStream);
    return obj;
  }
}

This generic method can then be used to convert a JSON string to the specified .NET type:

string reponseJson=GetJsonDataFromGoogle("MY SEARCH TERM);
results = Deserialise(responseJson);
foreach (var googleSearchResult in results.ResponseData.Results) {
  Console.WriteLine(googleSearchResult.Url);
}

And for completeness, here is GetJsonDataFromGoogle():

public static string GetJsonDataFromGoogle(string searchTerm)
{
    var url = string.Format("http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q={0}&rsz=large&start=0", searchTerm);
    var req = (HttpWebRequest) WebRequest.Create(url);
    req.Referer = "http://mywebsite.com";
    var res = (HttpWebResponse) req.GetResponse();
    string responseJson;
    using (var streamReader = new StreamReader(res.GetResponseStream())) {
        responseJson = streamReader.ReadToEnd();
    }
    return responseJson;
}

Posted in .NET | Tagged: , , , | 10 Comments »

Programmatically searching Google (Part 1): The Google API for .NET

Posted by Kash Farooq on January 24, 2011

Also see Programmatically searching Google (Part 2): Using the RESTful interface

I needed to do some web searches and record the host names that I found, so I started investigating what APIs Google exposed.

I found the “Google APIs for .NET Framework” hosted on Google Code. There hasn’t been much activity on this project recently – the last release was April 2010 – and there is not much documentation, but it works and it is simple to use.

It provides two sets of libraries: one that wraps Google Search and one that wraps Google Translate.

Here is an example of a search:

public void SearchForSomethingUsingLib() {
  var client = new GwebSearchClient("http://mywebsite.com");
  var results = client.Search("google api for .NET", 100);
  foreach (var webResult in results) {
   Console.WriteLine("{0}, {1}, {2}", webResult.Title, webResult.Url, webResult.Content);
  }
}

The ‘100’ in the above example indicates the number of search results to return.

What puzzled me was that the set of results returned didn’t appear to have a concept of paging, and the library didn’t seem to have a “Search Paged” method. What if I had asked for 1000000 results? Surely the API wasn’t going to get 1000000 results in one go and load them all into a List?

I fired up reflector. The API definitely returned a straight forward generic list. No delayed execution – one Web Request sent to Google, followed by one “new List<T>” to return the results.

I needed to investigate further, and I’ll look at that in Part 2.

Posted in .NET | Tagged: , , | Leave a Comment »

TDD with wrapper classes

Posted by Kash Farooq on January 12, 2009

Sometimes, during TDD, you come across a class that does not have an interface and therefore isn’t easy to test with Rhino Mocks. You’ll often find this is the case when you want to ensure that a class from the Base Class Library is being called correctly.
You can get around the problem using a wrapper class and an interface that describes the areas of the class you want to test.

For example, let’s say you have a method that uses System.Net.Mail.SmtpClient:

    public class SystemUnderTest {
        public void MethodUnderTest() {
            SmtpClient smtpClient=new SmtpClient();
            smtpClient.Port = 25;
            smtpClient.Host = "smtp.example.com";
            smtpClient.Send(new MailMessage{Subject = "Hello"});
        }
    }

SmtpClient is our external dependency. We want to test that the correct port and host has been set, and that Send is called with a MailMessage that has the subject “Hello”.

We start by creating an interface called ISmtpClient that includes the method and properties on SmtpClient that we want to test:

    public interface ISmtpClient {
        string Host { get; set; }
        int Port { get; set; }
        void Send(MailMessage mailMessage);
    }

We can now inject this into our system under test:

public class SystemUnderTest {      
   private readonly ISmtpClient smtpClient;

   public SystemUnderTest(ISmtpClient smtpClient) {
     this.smtpClient = smtpClient;
   }
        
   public void MethodUnderTest() {
      smtpClient.Port = 25;
      smtpClient.Host = "smtp.example.com";
      smtpClient.Send(new MailMessage{Subject = "Hello"});
   }
}

The test now becomes simple. Create a mock ISmtpClient, inject it into our system under test, and then assert that it was called correctly:

[Test]
public void MethodUnderTestSetsPortAndHostAndThenSendsMessage() {
	var mockRepository = new MockRepository();
	var smtpClient = mockRepository.DynamicMock<ISmtpClient>();

	mockRepository.ReplayAll();
	var systemUnderTest = new SystemUnderTest(smtpClient);
	systemUnderTest.MethodUnderTest();

	smtpClient.AssertWasCalled(x => x.Host = "smtp.example.com");
	smtpClient.AssertWasCalled(x => x.Port = 25);
	var mailMessage = (MailMessage)smtpClient.GetArgumentsForCallsMadeOn(x => x.Send(null))[0][0];
	Assert.AreEqual("Hello",mailMessage.Subject);
}

The final piece of the jigsaw is to wrap SmtpClient and implement ISmtpClient. The wrapper class will be used as the real implementation by the system under test:

public class SmtpClientWrapper : SmtpClient, ISmtpClient {}

Here is the final version of SystemUnderTest, using the wrapper class:

public class SystemUnderTest {
	private readonly ISmtpClient smtpClient;
	public SystemUnderTest() : this(new SmtpClientWrapper()) {}

	public SystemUnderTest(ISmtpClient smtpClient) {
		this.smtpClient = smtpClient;
	}

	public void MethodUnderTest() {
		smtpClient.Port = 25;
		smtpClient.Host = "smtp.example.com";
		smtpClient.Send(new MailMessage {Subject = "Hello"});
	}
}

Posted in Test Driven Development | Tagged: , , | Leave a Comment »

Determining the .NET version required by an assembly

Posted by Kash Farooq on December 16, 2008

As reflector doesn’t seem to tell you, here is how to determine which version of .NET an assembly was compiled for:

corflags MyAssembly.dll

On my PC, corflags it is installed as part of Microsoft Windows SDK v6.0A

Posted in .NET | Tagged: | Leave a Comment »