Posts Tagged ‘white box testing’

Howdy people,

Me again with some more information of how both Code Contracts and Pex can be used together to write expressive tests (while still specifying a (partial) specification of the code). In the course of this article wel’ll also cover Assertions and Assumptions. In order to point out only the necessary topics, let’s introduce a new (very basic) sample project, and let’s perform the following steps:

  • Create a console application called MusicStore
  • Create the class Artist (as of Figure 1)
  • Create the class Album (as of Figure 2)
  • Enable Code Contracts checking (both static and runtime, as of figure 3)


Figure 1: The Artist class of the MusicStore project.


Figure 2: The Album class of the MusicStore project.


Figure 3: Enabling both Runtime Checking (Full), and Static Checking

Important: Do not forget to also check the Implicit Non-Null Obligations, as of Figure 3

The only critical point for CC should be the SellAlbum method of the Artist class. Why? Simply, because there we might have a parameter of type Album that could be NULL. And the Static Checker gives us the correct hints to it, as expected:


Figure 4: Expected CC’s Static Checker suggestion and warning.

So we have one suggestion to put a precondition, and an effective warning that we might be passing a parameter that could be NULL. This was predictable.

Running Pex over the method SellAlbum reveals the following results (If you need a starting point for Pex, please look at my article First steps with PEX: Automated White box Testing for .NET)


Figure 5: Pex results for 3 test cases, one providing the album parameter = NULL. Please note also the typical Pex exploration results’ table showing one column per parameter (in this case only album ).

What happens if we put an assertion to this critical parameter? Let’s simply add one, as shown in Figure 6:


Figure 6: Adding an assertion using a call to the static method Contract.Assert(bool condition) of the CC framework.

It makes sense to assert a number of AlbumsSold higher or equal to 1 (after 1 album has been sold), right? Does putting an assertion have an impact on Pex? Of course! Let’s run it again to see how:


Figure 7: Effects on Pex of using Code Contracts’ Assertions. Pex clearly identifies this as an assertion failure, hence letting the generated test case fail. Line 2 (highlighted the additionally created Pex test for the Code Contracts assertion).

So far, so good. Let’s look at Figure 6 again. We can also put such an assertion for the album object. This clearly makes sense – since selling an album should increase the album’s TimesSold count. Let’s do this in Figure 8:


Figure 8: Added another assertion for the album.TimesSold property.

And immediately, let’s look at the Static Checker warnings and the generated Pex test cases (Figures 9 and 10):


Figure 9: One more unproven assert warning for our newly added Assertion in the SellAlbum method. This was expected. Let’s run Pex again!


Figure 10: Line 3 shows the newly generated (and failed) test case of Pex. Again, this was expected.

Great coverage – but What About the Meaning?

This is a good question. Is this still an expressive test? Let’s recap the meaning. We could have an Album -type parameter passed, which could be theoretically NULL. Would it make any sense to then test its property TimesSold? I am sure, we can agree on that this would not be the case.

So, what we need, is the possibility to prevent Pex from generating test cases in such a case, hence producing only meaningful tests. And who is going to provide us with such a capability?

Code Contracts’ Assumptions

Important: Assumptions only work when the Full contract option is checked in the Code Contracts pane (under our project’s properties)

Figure 11 gives a brief example of how to put such an assumption into our code of the SellAlbum method.


Figure 11: Creating assumptions.

Figure 11 needs some explanations:

The first three assumptions simply protect us from getting the current object, the passed album, or the passed album’s artist as NULL. Makes sense, so far. The following two assumptions ensure that this.AlbumsSold and album.TimesSold both are non-negative values.

(Note also that now we put our assertions INSIDE the artist comparison’s block – makes sense, since we do not need to assert when there happened no increase of AlbumsSold and TimesSold properties).

What do We Expect?

Well, we pretty much thought at everything, didn’t we? So we expect Pex to not issue any errors or failures again. A simple re-run of Pex will show us the result, as can be seen in Figure 12.


Figure 12: Unexpected error on line 3, a failed assertion.

An unexpected failure? How could that happen? Let’s look at what the parameters looked like in the case of failure:


Did you see it? AlbumsSold is equal to int.MaxValue ! I guess, I do not need to explain what happens, if you try to increase an integer (in SellAlbum ) which is already equal to its maximum value! But why did that happen?

Simply put, this is one of the boundary test cases we explained in First steps with PEX: Automated White box Testing for .NET article. Pex guesses from the type of variables which are the boundary values (and also intermediate values).

To summarize shortly & sweetly today’s lecture:

Code Contracts & Pex: meaningful, expressive tests

(by preventing test case generation for assumed parameter values)

Let’s stop here today. There are more interesting topics regarding Code Contracts and Pex, and I am quite sure this will keep us busy for another while!

Enjoy playing around with Pex and Code Contracts, and see you next time!

Best regards,


Read Full Post »


You might already guess it: Pex won’t let me go, and so I’d like to explain in short a few of Pex’ concepts.

1.1 Code Coverage

Pex is an abbreviation for Program Exploration. You guessed it: Nomen est omen. It analyzes the branches (if, then, else, etc…) of control flow and tries to cover each and every branch. This leads to a highly elevated code coverage, or, in other words: It helps to increase the amount of tested code.

1.2 The Main Idea of Parametrized Unit Testing using Pex

…is to elevate any values that should not matter (i.e. that should not influence the program’s runtime behaviour) into parameters. Pex will then care about how to deal with them, about creating test cases for all possible values they can assume. Short & sweet: Pex cares about your parameters and helps to ensure there are no cases where your program would fail because of them.

1.3 The Nuts and Bolts of a PUT

Figure 1 gives you an example of what a Parametrized Unit Test (PUT) looks like:


Figure 1: A simple PUT

We just use the [PexMethod] attribute to denote that the following method is a PUT.

Sometimes we need some of the parameters to be null, in order to get a method running at all. In such a case, we can give Pex the instruction not to generate tests which would assume that parameter as NULL. We simply do this by adding the attribute


in front of the parameter type. Sounds straightforward, doesn’t it?

Within the PUT method, you can (as suggested by Pex upon generationg of the tests) multiple assertions to ensure the correct behaviour of the method. What Pex does for you, is the following:

  • Detecting dereferencing of potential null values (and generates a test case)
  • Analysis of e.g. loop boundaries (and generates test cases for each assumable value, including boundary values)

There is only one drawback: This way of testing only works as long as we have implemented method bodies. Are there cases where we don’t have them? Of course: Interfaces. Let’s see how Pex deals with interfaces.

1.4 Extending our Example

In order to see how Pex generates test code for interfaces, we need to add an interface to our project BookStore (which we created in Pex – Automated White Box Unit Testing).

   1: using System;

   2: using System.Collections.Generic;

   3: using System.Linq;

   4: using System.Text;


   6: namespace BookStore

   7: {

   8:     public interface IStorage

   9:     {

  10:         void DeliverToAddress(String address, Book book);

  11:     }

  12: }

Snippet 1: The IStorage interface.

As a next step, we are going to use it somewhere, namely: We let our class Store implement IStorage.

public class Store : IStorage

Please note that the IStorage member method DeliverToAddress(…) has already been implemented in our previous post. Hence, the code should compile as soon as you implemented the changes.

What comes next? Pex cannot possibly know about our newly created interface, so we need to tell it somehow to generate stubs for the interface. And how do we do that?

Re-run Pex!

For that, right-click on the project BookStore and select Run Pex from the context menu. Pex generated the following class:

   1: public partial class SIStorage : IStorage

   2: {

   3:     Action<string, global::BookStore.Book> DeliverToAddressStringBook;

   4:     void DeliverToAddress(string address, Book book)

   5:     {

   6:         return DeliverToAddressStringBook(address, book);

   7:     }

   8: }

Snippet 2: The generated stub class for the IStorage interface: SIStorage.

As shown in Snippet 2, the generator creates a class named S + <interface name>, which implements the interface. For each method defined in the interface, a property (a delegate, in fact)will be added which resembles the method’s signature (method name + parameters’ types, camel case).

1.5 Customizing the stub

Now we’ll dig a little deeper. We modify our DeliverToAddress method from Snippet 1. Why do we do this? We simply double-check that the amount of ordered books is equal to the amount of the actually delivered book.

   1: // <copyright file="StoreTest.cs" company="Scientific Network">Copyright © Scientific Network 2010</copyright>


   3: using System;

   4: using BookStore;

   5: using Microsoft.Pex.Framework;

   6: using Microsoft.Pex.Framework.Validation;

   7: using Microsoft.VisualStudio.TestTools.UnitTesting;

   8: using BookStore.Stubs;

   9: using System.Collections.Generic;


  11: namespace BookStore

  12: {

  13:     [TestClass]

  14:     [PexClass(typeof(Store))]

  15:     [PexAllowedExceptionFromTypeUnderTest(typeof(ArgumentException),

  16:                                     AcceptExceptionSubtypes = true)]

  17:     [PexAllowedExceptionFromTypeUnderTest(

  18:             typeof(InvalidOperationException)

  19:     )]

  20:     public partial class StoreTest

  21:     {

  22:         [PexMethod]

  23:         public void DeliverOrders([PexAssumeUnderTest]Store target)

  24:         {

  25:             int count = 0;

  26:             var storage = new SIStorage()

  27:             {

  28:                 DeliverToAddressStringBook =

  29:                 delegate(String address, Book book)

  30:                 {

  31:                     count++;

  32:                 }

  33:             };

  34:             target.DeliverOrders();

  35:             Assert.AreEqual(CalculateExpectedDeliveries(target.Orders), count);

  36:             // TODO: add assertions to method StoreTest.DeliverOrders(Store)

  37:         }


  39:         private int CalculateExpectedDeliveries(List<Order> orders)

  40:         {

  41:             int deliveries = 0;

  42:             foreach (Order order in orders)

  43:             {

  44:                 deliveries += order.Quantity;

  45:             }

  46:             return deliveries;

  47:         }

  48:     }

  49: }

Snippet 3: The modified stub.

StoreTest explanantion, (line by line)

Don’t worry! It looks more complicated than it actually is. The real stuff starts only on line 25.

25: Declaring a counter variable which will serve our comparison

26-33: Declaring an anonymous method that will simply increase a counter each time DeliverToAddressStringBook is called.

34: Perform the actuall delivery, calling DeliverOrders, which itself will call DeliverToAddress.

35: Using an assertion in order to determine whether our expected amount of delilveries (CalculateExpectedDeliveries) equals the actual amount of delivered books (count).

39-47: The method CalculateExpectedDeliveries is only used to retrieve the correct amount of books that is expected to be delivered by the DeliverOrders method.

In order to let the tests run, simply right-click on the DeliverOrders method and choose Run Pex Explorations, as shown in Figure 2.


Figure 2: Run Pex on the PUT.

At this point, you will perhaps notice failing tests where the order.Quantity is equal to a negative number.

I’d like to show you a simple means to prevent Pex from generating and running test cases under certain circumstances: Code Contracts.

For this post, I just show you what you need for that purpose. We’ll probably cover Code Contracts in a future post, since they integrate very well with Pex (and vice versa).

Just drop a statement like

Contract.Assume(target.Orders.Count >= 0);

at the beginning of the DeliverOrders method. For Pex, this simply means to ignore all cases where the store would get a negative amount of orders – hence preventing Pex from generating nonsense test cases.

1.6 Why all this Work?

The advantage of all this customization should be clear: You can test an interface in general, without having to test all single methods that implement the interface.

I hope you are familiar now with (customizable) PUTs and interface tests.

Cheers, and see you at the next post!


Read Full Post »

Hello folks,

It’s me again. During the last days I found an interesting new framework: Pex. On the Microsoft’s research labs’ page, it is summarized as follows:

Right from the Visual Studio code editor, Pex finds interesting input-output values of your methods, which you can save as a small test suite with high code coverage. Pex performs a systematic analysis, hunting for boundary conditions, exceptions and assertion failures, which you can debug right away. Pex enables Parameterized Unit Testing, an extension of Unit Testing that reduces test maintenance costs. […]

Sounds good, doesn’t it? We write code, and Pex cares about finding suitable testing values.

Pex is available for download in two flavours:

Both flavours come with a Visual Studio add-in. Only constraint: It will not work with the VS Express editions. However, in this case you can still use Pex from the command line.

I chose to install the academic version over an existing VS 2008 Pro installation.

1. Creating a test project

Let’s get started. After we (successfully) installed the Pex framework, we create a new Console Application, which, let’s say, simulates a basic bookstore. The following code snippets form our project’s skeleton:

   1: namespace BookStore

   2: {

   3:     public class Book

   4:     {

   5:         public String Title { get; set; }

   6:         public String Author { get; set; }

   7:         public String ISBN { get; set; }

   8:     }

   9: }

Snippet 1: The Book class

   1: namespace BookStore

   2: {

   3:     public class Client

   4:     {

   5:         public String Name { get; set; }

   6:         public String Surname { get; set; }

   7:         public String Address { get; set; }

   8:     }

   9: }

Snippet 2: The Client class

   1: namespace BookStore

   2: {

   3:     public class Order

   4:     {

   5:         public Client client { get; set; }

   6:         public Book book { get; set; }

   7:     }

   8: }

Snippet 3: The Order class

Up to now, our bookstore has no functionality at all. We’ll change that by implementing the Store class:

   1: namespace BookStore

   2: {

   3:     public class Store

   4:     {

   5:         public List<Order> Orders { get; set; }


   7:         public void AddOrderForDelivery(Order order)

   8:         {

   9:             Orders.Add(order);

  10:         }


  12:         public void DeliverOrders()

  13:         {

  14:             foreach (Order order in Orders)

  15:             {

  16:                 for (int i = 0; i < order.Quantity; i++)

  17:                 {

  18:                     DeliverToAddress(

  19:                         order.Client.Address,

  20:                         order.Book);

  21:                 }

  22:             }

  23:         }


  25:         public void DeliverToAddress(String address, Book book)

  26:         {

  27:             Console.WriteLine(

  28:                 "Book with ISBN " + book.ISBN +

  29:                 " delivered to address " + address);

  30:         }

  31:     }

  32: }

Snippet 4: The Store class

So far, so good, right? Now, clearly, our code is quite error-prone. It compiles, but we are not safe from runtime exceptions.

This is where unit tests come into play.

2. Running Pex:

You can run Pex by simply right-clicking on the method in question and selecting Run Pex from the context menu.


Figure 1: Right clicking on the DeliverOrders method reveals the newly added “Run Pex” entry in the context menu. Click on it.

Then we need to select a test framework. Since Pex generates tests, you can choose for which framework they should be generated. Choose Visual Studio Unit Test.


Figure 2: Choosing the test framework.

After confirmation the dialog in Figure 2, the Pex Exploration results window appears, showing the run test cases and indicating which values have been used to created the test case.


Figure 3: Pex exploration results: Failed tests are shown in bold, successfully tests using normal font weight.

On the right side of the Pex results window, you have a detail window. For a successful test, it will show the generated method, which you can save away by clicking the button Save Test below.


Figure 4: Test details for a successful test (here: the selected test case from Figure 3)

On the other hand, for a failed test case, other than seeing the generated method, you can see also a Stack trace of the failed code. Figure 5 shows the (expanded) Stack trace. Please note also that above the Stack trace you still have the possibility to expand the method implementation.


Figure 5: Stack trace of a failed test.

Did you notice the Allow Exception button at the bottom line? Click on it if you want to allow the exception that has arisen during the test run.

Important: You will need to re-run Pex after the allowance of an exception.

Let’s see what happens if we save the generated test:


Figure 6: Save test dialog window.

Confirm twice. The standard naming will do. You will immediately notice that in the background, there is an entire test project that is being created:


Figure 7: The created test project BookStore.Tests. Note also the reference to the Microsoft unit testing framework. This is due to our initial selection of the test framework when running Pex.

Why would we want to do that?

Because, this way you can actually debug each single test case.


Figure 8: The main window showing the Test case (StoreTest.DeliverOrders.g.cs, highlighted in the solution explorer on the right side). Please note also the Debug button below the stack trace (also highlighted). Simply click on it in order to debug the generated test case.

I’d say we stop here for now, for there’s enough to digest. in the meantime you can experiment with Pex and play around in order to get comfortable.

We’ll dig deeper with the next article on Pex.

Best regards,


Read Full Post »