You might already guess it: Pex won’t let me go, and so I’d like to explain in short a few of Pex’ concepts.
1.1 Code Coverage
Pex is an abbreviation for Program Exploration. You guessed it: Nomen est omen. It analyzes the branches (if, then, else, etc…) of control flow and tries to cover each and every branch. This leads to a highly elevated code coverage, or, in other words: It helps to increase the amount of tested code.
1.2 The Main Idea of Parametrized Unit Testing using Pex
…is to elevate any values that should not matter (i.e. that should not influence the program’s runtime behaviour) into parameters. Pex will then care about how to deal with them, about creating test cases for all possible values they can assume. Short & sweet: Pex cares about your parameters and helps to ensure there are no cases where your program would fail because of them.
1.3 The Nuts and Bolts of a PUT
Figure 1 gives you an example of what a Parametrized Unit Test (PUT) looks like:
Figure 1: A simple PUT
We just use the [PexMethod] attribute to denote that the following method is a PUT.
Sometimes we need some of the parameters to be null, in order to get a method running at all. In such a case, we can give Pex the instruction not to generate tests which would assume that parameter as NULL. We simply do this by adding the attribute
in front of the parameter type. Sounds straightforward, doesn’t it?
Within the PUT method, you can (as suggested by Pex upon generationg of the tests) multiple assertions to ensure the correct behaviour of the method. What Pex does for you, is the following:
- Detecting dereferencing of potential null values (and generates a test case)
- Analysis of e.g. loop boundaries (and generates test cases for each assumable value, including boundary values)
There is only one drawback: This way of testing only works as long as we have implemented method bodies. Are there cases where we don’t have them? Of course: Interfaces. Let’s see how Pex deals with interfaces.
1.4 Extending our Example
In order to see how Pex generates test code for interfaces, we need to add an interface to our project BookStore (which we created in Pex – Automated White Box Unit Testing).
1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Text;
6: namespace BookStore
8: public interface IStorage
10: void DeliverToAddress(String address, Book book);
Snippet 1: The IStorage interface.
As a next step, we are going to use it somewhere, namely: We let our class Store implement IStorage.
public class Store : IStorage
Please note that the IStorage member method DeliverToAddress(…) has already been implemented in our previous post. Hence, the code should compile as soon as you implemented the changes.
What comes next? Pex cannot possibly know about our newly created interface, so we need to tell it somehow to generate stubs for the interface. And how do we do that?
For that, right-click on the project BookStore and select Run Pex from the context menu. Pex generated the following class:
1: public partial class SIStorage : IStorage
3: Action<string, global::BookStore.Book> DeliverToAddressStringBook;
4: void DeliverToAddress(string address, Book book)
6: return DeliverToAddressStringBook(address, book);
Snippet 2: The generated stub class for the IStorage interface: SIStorage.
As shown in Snippet 2, the generator creates a class named S + <interface name>, which implements the interface. For each method defined in the interface, a property (a delegate, in fact)will be added which resembles the method’s signature (method name + parameters’ types, camel case).
1.5 Customizing the stub
Now we’ll dig a little deeper. We modify our DeliverToAddress method from Snippet 1. Why do we do this? We simply double-check that the amount of ordered books is equal to the amount of the actually delivered book.
1: // <copyright file="StoreTest.cs" company="Scientific Network">Copyright © Scientific Network 2010</copyright>
3: using System;
4: using BookStore;
5: using Microsoft.Pex.Framework;
6: using Microsoft.Pex.Framework.Validation;
7: using Microsoft.VisualStudio.TestTools.UnitTesting;
8: using BookStore.Stubs;
9: using System.Collections.Generic;
11: namespace BookStore
16: AcceptExceptionSubtypes = true)]
20: public partial class StoreTest
23: public void DeliverOrders([PexAssumeUnderTest]Store target)
25: int count = 0;
26: var storage = new SIStorage()
28: DeliverToAddressStringBook =
29: delegate(String address, Book book)
35: Assert.AreEqual(CalculateExpectedDeliveries(target.Orders), count);
36: // TODO: add assertions to method StoreTest.DeliverOrders(Store)
39: private int CalculateExpectedDeliveries(List<Order> orders)
41: int deliveries = 0;
42: foreach (Order order in orders)
44: deliveries += order.Quantity;
46: return deliveries;
Snippet 3: The modified stub.
StoreTest explanantion, (line by line)
Don’t worry! It looks more complicated than it actually is. The real stuff starts only on line 25.
25: Declaring a counter variable which will serve our comparison
26-33: Declaring an anonymous method that will simply increase a counter each time DeliverToAddressStringBook is called.
34: Perform the actuall delivery, calling DeliverOrders, which itself will call DeliverToAddress.
35: Using an assertion in order to determine whether our expected amount of delilveries (CalculateExpectedDeliveries) equals the actual amount of delivered books (count).
39-47: The method CalculateExpectedDeliveries is only used to retrieve the correct amount of books that is expected to be delivered by the DeliverOrders method.
In order to let the tests run, simply right-click on the DeliverOrders method and choose Run Pex Explorations, as shown in Figure 2.
Figure 2: Run Pex on the PUT.
At this point, you will perhaps notice failing tests where the order.Quantity is equal to a negative number.
I’d like to show you a simple means to prevent Pex from generating and running test cases under certain circumstances: Code Contracts.
For this post, I just show you what you need for that purpose. We’ll probably cover Code Contracts in a future post, since they integrate very well with Pex (and vice versa).
Just drop a statement like
Contract.Assume(target.Orders.Count >= 0);
at the beginning of the DeliverOrders method. For Pex, this simply means to ignore all cases where the store would get a negative amount of orders – hence preventing Pex from generating nonsense test cases.
1.6 Why all this Work?
The advantage of all this customization should be clear: You can test an interface in general, without having to test all single methods that implement the interface.
I hope you are familiar now with (customizable) PUTs and interface tests.
Cheers, and see you at the next post!