Feeds:
Posts
Comments

Posts Tagged ‘Unit testing’

Howdy people,

Me again with some more information of how both Code Contracts and Pex can be used together to write expressive tests (while still specifying a (partial) specification of the code). In the course of this article wel’ll also cover Assertions and Assumptions. In order to point out only the necessary topics, let’s introduce a new (very basic) sample project, and let’s perform the following steps:

  • Create a console application called MusicStore
  • Create the class Artist (as of Figure 1)
  • Create the class Album (as of Figure 2)
  • Enable Code Contracts checking (both static and runtime, as of figure 3)

clip_image002

Figure 1: The Artist class of the MusicStore project.

clip_image004

Figure 2: The Album class of the MusicStore project.

clip_image006

Figure 3: Enabling both Runtime Checking (Full), and Static Checking

Important: Do not forget to also check the Implicit Non-Null Obligations, as of Figure 3

The only critical point for CC should be the SellAlbum method of the Artist class. Why? Simply, because there we might have a parameter of type Album that could be NULL. And the Static Checker gives us the correct hints to it, as expected:

clip_image008

Figure 4: Expected CC’s Static Checker suggestion and warning.

So we have one suggestion to put a precondition, and an effective warning that we might be passing a parameter that could be NULL. This was predictable.

Running Pex over the method SellAlbum reveals the following results (If you need a starting point for Pex, please look at my article First steps with PEX: Automated White box Testing for .NET)

clip_image010

Figure 5: Pex results for 3 test cases, one providing the album parameter = NULL. Please note also the typical Pex exploration results’ table showing one column per parameter (in this case only album ).

What happens if we put an assertion to this critical parameter? Let’s simply add one, as shown in Figure 6:

clip_image012

Figure 6: Adding an assertion using a call to the static method Contract.Assert(bool condition) of the CC framework.

It makes sense to assert a number of AlbumsSold higher or equal to 1 (after 1 album has been sold), right? Does putting an assertion have an impact on Pex? Of course! Let’s run it again to see how:

clip_image014

Figure 7: Effects on Pex of using Code Contracts’ Assertions. Pex clearly identifies this as an assertion failure, hence letting the generated test case fail. Line 2 (highlighted the additionally created Pex test for the Code Contracts assertion).

So far, so good. Let’s look at Figure 6 again. We can also put such an assertion for the album object. This clearly makes sense – since selling an album should increase the album’s TimesSold count. Let’s do this in Figure 8:

clip_image016

Figure 8: Added another assertion for the album.TimesSold property.

And immediately, let’s look at the Static Checker warnings and the generated Pex test cases (Figures 9 and 10):

clip_image018

Figure 9: One more unproven assert warning for our newly added Assertion in the SellAlbum method. This was expected. Let’s run Pex again!

clip_image020

Figure 10: Line 3 shows the newly generated (and failed) test case of Pex. Again, this was expected.

Great coverage – but What About the Meaning?

This is a good question. Is this still an expressive test? Let’s recap the meaning. We could have an Album -type parameter passed, which could be theoretically NULL. Would it make any sense to then test its property TimesSold? I am sure, we can agree on that this would not be the case.

So, what we need, is the possibility to prevent Pex from generating test cases in such a case, hence producing only meaningful tests. And who is going to provide us with such a capability?

Code Contracts’ Assumptions

Important: Assumptions only work when the Full contract option is checked in the Code Contracts pane (under our project’s properties)

Figure 11 gives a brief example of how to put such an assumption into our code of the SellAlbum method.

clip_image022

Figure 11: Creating assumptions.

Figure 11 needs some explanations:

The first three assumptions simply protect us from getting the current object, the passed album, or the passed album’s artist as NULL. Makes sense, so far. The following two assumptions ensure that this.AlbumsSold and album.TimesSold both are non-negative values.

(Note also that now we put our assertions INSIDE the artist comparison’s block – makes sense, since we do not need to assert when there happened no increase of AlbumsSold and TimesSold properties).

What do We Expect?

Well, we pretty much thought at everything, didn’t we? So we expect Pex to not issue any errors or failures again. A simple re-run of Pex will show us the result, as can be seen in Figure 12.

clip_image024

Figure 12: Unexpected error on line 3, a failed assertion.

An unexpected failure? How could that happen? Let’s look at what the parameters looked like in the case of failure:

image

Did you see it? AlbumsSold is equal to int.MaxValue ! I guess, I do not need to explain what happens, if you try to increase an integer (in SellAlbum ) which is already equal to its maximum value! But why did that happen?

Simply put, this is one of the boundary test cases we explained in First steps with PEX: Automated White box Testing for .NET article. Pex guesses from the type of variables which are the boundary values (and also intermediate values).

To summarize shortly & sweetly today’s lecture:

Code Contracts & Pex: meaningful, expressive tests

(by preventing test case generation for assumed parameter values)

Let’s stop here today. There are more interesting topics regarding Code Contracts and Pex, and I am quite sure this will keep us busy for another while!

Enjoy playing around with Pex and Code Contracts, and see you next time!

Best regards,

Martin

Advertisements

Read Full Post »

Hello,

You might already guess it: Pex won’t let me go, and so I’d like to explain in short a few of Pex’ concepts.

1.1 Code Coverage

Pex is an abbreviation for Program Exploration. You guessed it: Nomen est omen. It analyzes the branches (if, then, else, etc…) of control flow and tries to cover each and every branch. This leads to a highly elevated code coverage, or, in other words: It helps to increase the amount of tested code.

1.2 The Main Idea of Parametrized Unit Testing using Pex

…is to elevate any values that should not matter (i.e. that should not influence the program’s runtime behaviour) into parameters. Pex will then care about how to deal with them, about creating test cases for all possible values they can assume. Short & sweet: Pex cares about your parameters and helps to ensure there are no cases where your program would fail because of them.

1.3 The Nuts and Bolts of a PUT

Figure 1 gives you an example of what a Parametrized Unit Test (PUT) looks like:

image

Figure 1: A simple PUT

We just use the [PexMethod] attribute to denote that the following method is a PUT.

Sometimes we need some of the parameters to be null, in order to get a method running at all. In such a case, we can give Pex the instruction not to generate tests which would assume that parameter as NULL. We simply do this by adding the attribute

[PexAssumeUnderTest]

in front of the parameter type. Sounds straightforward, doesn’t it?

Within the PUT method, you can (as suggested by Pex upon generationg of the tests) multiple assertions to ensure the correct behaviour of the method. What Pex does for you, is the following:

  • Detecting dereferencing of potential null values (and generates a test case)
  • Analysis of e.g. loop boundaries (and generates test cases for each assumable value, including boundary values)

There is only one drawback: This way of testing only works as long as we have implemented method bodies. Are there cases where we don’t have them? Of course: Interfaces. Let’s see how Pex deals with interfaces.

1.4 Extending our Example

In order to see how Pex generates test code for interfaces, we need to add an interface to our project BookStore (which we created in Pex – Automated White Box Unit Testing).

   1: using System;

   2: using System.Collections.Generic;

   3: using System.Linq;

   4: using System.Text;

   5:

   6: namespace BookStore

   7: {

   8:     public interface IStorage

   9:     {

  10:         void DeliverToAddress(String address, Book book);

  11:     }

  12: }

Snippet 1: The IStorage interface.

As a next step, we are going to use it somewhere, namely: We let our class Store implement IStorage.

public class Store : IStorage

Please note that the IStorage member method DeliverToAddress(…) has already been implemented in our previous post. Hence, the code should compile as soon as you implemented the changes.

What comes next? Pex cannot possibly know about our newly created interface, so we need to tell it somehow to generate stubs for the interface. And how do we do that?

Re-run Pex!

For that, right-click on the project BookStore and select Run Pex from the context menu. Pex generated the following class:

   1: public partial class SIStorage : IStorage

   2: {

   3:     Action<string, global::BookStore.Book> DeliverToAddressStringBook;

   4:     void DeliverToAddress(string address, Book book)

   5:     {

   6:         return DeliverToAddressStringBook(address, book);

   7:     }

   8: }

Snippet 2: The generated stub class for the IStorage interface: SIStorage.

As shown in Snippet 2, the generator creates a class named S + <interface name>, which implements the interface. For each method defined in the interface, a property (a delegate, in fact)will be added which resembles the method’s signature (method name + parameters’ types, camel case).

1.5 Customizing the stub

Now we’ll dig a little deeper. We modify our DeliverToAddress method from Snippet 1. Why do we do this? We simply double-check that the amount of ordered books is equal to the amount of the actually delivered book.

   1: // <copyright file="StoreTest.cs" company="Scientific Network">Copyright © Scientific Network 2010</copyright>

   2:

   3: using System;

   4: using BookStore;

   5: using Microsoft.Pex.Framework;

   6: using Microsoft.Pex.Framework.Validation;

   7: using Microsoft.VisualStudio.TestTools.UnitTesting;

   8: using BookStore.Stubs;

   9: using System.Collections.Generic;

  10:

  11: namespace BookStore

  12: {

  13:     [TestClass]

  14:     [PexClass(typeof(Store))]

  15:     [PexAllowedExceptionFromTypeUnderTest(typeof(ArgumentException),

  16:                                     AcceptExceptionSubtypes = true)]

  17:     [PexAllowedExceptionFromTypeUnderTest(

  18:             typeof(InvalidOperationException)

  19:     )]

  20:     public partial class StoreTest

  21:     {

  22:         [PexMethod]

  23:         public void DeliverOrders([PexAssumeUnderTest]Store target)

  24:         {

  25:             int count = 0;

  26:             var storage = new SIStorage()

  27:             {

  28:                 DeliverToAddressStringBook =

  29:                 delegate(String address, Book book)

  30:                 {

  31:                     count++;

  32:                 }

  33:             };

  34:             target.DeliverOrders();

  35:             Assert.AreEqual(CalculateExpectedDeliveries(target.Orders), count);

  36:             // TODO: add assertions to method StoreTest.DeliverOrders(Store)

  37:         }

  38:

  39:         private int CalculateExpectedDeliveries(List<Order> orders)

  40:         {

  41:             int deliveries = 0;

  42:             foreach (Order order in orders)

  43:             {

  44:                 deliveries += order.Quantity;

  45:             }

  46:             return deliveries;

  47:         }

  48:     }

  49: }

Snippet 3: The modified stub.

StoreTest explanantion, (line by line)

Don’t worry! It looks more complicated than it actually is. The real stuff starts only on line 25.

25: Declaring a counter variable which will serve our comparison

26-33: Declaring an anonymous method that will simply increase a counter each time DeliverToAddressStringBook is called.

34: Perform the actuall delivery, calling DeliverOrders, which itself will call DeliverToAddress.

35: Using an assertion in order to determine whether our expected amount of delilveries (CalculateExpectedDeliveries) equals the actual amount of delivered books (count).

39-47: The method CalculateExpectedDeliveries is only used to retrieve the correct amount of books that is expected to be delivered by the DeliverOrders method.

In order to let the tests run, simply right-click on the DeliverOrders method and choose Run Pex Explorations, as shown in Figure 2.

image

Figure 2: Run Pex on the PUT.

At this point, you will perhaps notice failing tests where the order.Quantity is equal to a negative number.

I’d like to show you a simple means to prevent Pex from generating and running test cases under certain circumstances: Code Contracts.

For this post, I just show you what you need for that purpose. We’ll probably cover Code Contracts in a future post, since they integrate very well with Pex (and vice versa).

Just drop a statement like

Contract.Assume(target.Orders.Count >= 0);

at the beginning of the DeliverOrders method. For Pex, this simply means to ignore all cases where the store would get a negative amount of orders – hence preventing Pex from generating nonsense test cases.

1.6 Why all this Work?

The advantage of all this customization should be clear: You can test an interface in general, without having to test all single methods that implement the interface.

I hope you are familiar now with (customizable) PUTs and interface tests.

Cheers, and see you at the next post!

Martin

Read Full Post »