Feeds:
Posts
Comments

Posts Tagged ‘Pex’

Howdy people,

Me again with some more information of how both Code Contracts and Pex can be used together to write expressive tests (while still specifying a (partial) specification of the code). In the course of this article wel’ll also cover Assertions and Assumptions. In order to point out only the necessary topics, let’s introduce a new (very basic) sample project, and let’s perform the following steps:

  • Create a console application called MusicStore
  • Create the class Artist (as of Figure 1)
  • Create the class Album (as of Figure 2)
  • Enable Code Contracts checking (both static and runtime, as of figure 3)

clip_image002

Figure 1: The Artist class of the MusicStore project.

clip_image004

Figure 2: The Album class of the MusicStore project.

clip_image006

Figure 3: Enabling both Runtime Checking (Full), and Static Checking

Important: Do not forget to also check the Implicit Non-Null Obligations, as of Figure 3

The only critical point for CC should be the SellAlbum method of the Artist class. Why? Simply, because there we might have a parameter of type Album that could be NULL. And the Static Checker gives us the correct hints to it, as expected:

clip_image008

Figure 4: Expected CC’s Static Checker suggestion and warning.

So we have one suggestion to put a precondition, and an effective warning that we might be passing a parameter that could be NULL. This was predictable.

Running Pex over the method SellAlbum reveals the following results (If you need a starting point for Pex, please look at my article First steps with PEX: Automated White box Testing for .NET)

clip_image010

Figure 5: Pex results for 3 test cases, one providing the album parameter = NULL. Please note also the typical Pex exploration results’ table showing one column per parameter (in this case only album ).

What happens if we put an assertion to this critical parameter? Let’s simply add one, as shown in Figure 6:

clip_image012

Figure 6: Adding an assertion using a call to the static method Contract.Assert(bool condition) of the CC framework.

It makes sense to assert a number of AlbumsSold higher or equal to 1 (after 1 album has been sold), right? Does putting an assertion have an impact on Pex? Of course! Let’s run it again to see how:

clip_image014

Figure 7: Effects on Pex of using Code Contracts’ Assertions. Pex clearly identifies this as an assertion failure, hence letting the generated test case fail. Line 2 (highlighted the additionally created Pex test for the Code Contracts assertion).

So far, so good. Let’s look at Figure 6 again. We can also put such an assertion for the album object. This clearly makes sense – since selling an album should increase the album’s TimesSold count. Let’s do this in Figure 8:

clip_image016

Figure 8: Added another assertion for the album.TimesSold property.

And immediately, let’s look at the Static Checker warnings and the generated Pex test cases (Figures 9 and 10):

clip_image018

Figure 9: One more unproven assert warning for our newly added Assertion in the SellAlbum method. This was expected. Let’s run Pex again!

clip_image020

Figure 10: Line 3 shows the newly generated (and failed) test case of Pex. Again, this was expected.

Great coverage – but What About the Meaning?

This is a good question. Is this still an expressive test? Let’s recap the meaning. We could have an Album -type parameter passed, which could be theoretically NULL. Would it make any sense to then test its property TimesSold? I am sure, we can agree on that this would not be the case.

So, what we need, is the possibility to prevent Pex from generating test cases in such a case, hence producing only meaningful tests. And who is going to provide us with such a capability?

Code Contracts’ Assumptions

Important: Assumptions only work when the Full contract option is checked in the Code Contracts pane (under our project’s properties)

Figure 11 gives a brief example of how to put such an assumption into our code of the SellAlbum method.

clip_image022

Figure 11: Creating assumptions.

Figure 11 needs some explanations:

The first three assumptions simply protect us from getting the current object, the passed album, or the passed album’s artist as NULL. Makes sense, so far. The following two assumptions ensure that this.AlbumsSold and album.TimesSold both are non-negative values.

(Note also that now we put our assertions INSIDE the artist comparison’s block – makes sense, since we do not need to assert when there happened no increase of AlbumsSold and TimesSold properties).

What do We Expect?

Well, we pretty much thought at everything, didn’t we? So we expect Pex to not issue any errors or failures again. A simple re-run of Pex will show us the result, as can be seen in Figure 12.

clip_image024

Figure 12: Unexpected error on line 3, a failed assertion.

An unexpected failure? How could that happen? Let’s look at what the parameters looked like in the case of failure:

image

Did you see it? AlbumsSold is equal to int.MaxValue ! I guess, I do not need to explain what happens, if you try to increase an integer (in SellAlbum ) which is already equal to its maximum value! But why did that happen?

Simply put, this is one of the boundary test cases we explained in First steps with PEX: Automated White box Testing for .NET article. Pex guesses from the type of variables which are the boundary values (and also intermediate values).

To summarize shortly & sweetly today’s lecture:

Code Contracts & Pex: meaningful, expressive tests

(by preventing test case generation for assumed parameter values)

Let’s stop here today. There are more interesting topics regarding Code Contracts and Pex, and I am quite sure this will keep us busy for another while!

Enjoy playing around with Pex and Code Contracts, and see you next time!

Best regards,

Martin

Read Full Post »

Howdy,

In reference to my previous post, Pex: Parametrized Unit Tests and their Customization, I’d like to publish an update:

I received an interesting comment by Peli, that the following code piece should be replaced by a lambda expression, in order to make it more readable. Hence, I’d advice that you should just replace the following snippet:

  28:                 DeliverToAddressStringBook =
  29:                 delegate(String address, Book book)
  30:                 {
  31:                     count++;
  32:                 }
  

with the following:  

 

28:    DeliverToAddressStringBook = (address, book) => count++;

 

And in fact, using the lambda expression we could reduce the amount of code, hence making it more readable! Thanks, Peli!

Best regards,

Martin

Read Full Post »

Hello folks,

It’s me again. During the last days I found an interesting new framework: Pex. On the Microsoft’s research labs’ page, it is summarized as follows:

Right from the Visual Studio code editor, Pex finds interesting input-output values of your methods, which you can save as a small test suite with high code coverage. Pex performs a systematic analysis, hunting for boundary conditions, exceptions and assertion failures, which you can debug right away. Pex enables Parameterized Unit Testing, an extension of Unit Testing that reduces test maintenance costs. […]

Sounds good, doesn’t it? We write code, and Pex cares about finding suitable testing values.

Pex is available for download in two flavours:

Both flavours come with a Visual Studio add-in. Only constraint: It will not work with the VS Express editions. However, in this case you can still use Pex from the command line.

I chose to install the academic version over an existing VS 2008 Pro installation.

1. Creating a test project

Let’s get started. After we (successfully) installed the Pex framework, we create a new Console Application, which, let’s say, simulates a basic bookstore. The following code snippets form our project’s skeleton:

   1: namespace BookStore

   2: {

   3:     public class Book

   4:     {

   5:         public String Title { get; set; }

   6:         public String Author { get; set; }

   7:         public String ISBN { get; set; }

   8:     }

   9: }

Snippet 1: The Book class

   1: namespace BookStore

   2: {

   3:     public class Client

   4:     {

   5:         public String Name { get; set; }

   6:         public String Surname { get; set; }

   7:         public String Address { get; set; }

   8:     }

   9: }

Snippet 2: The Client class

   1: namespace BookStore

   2: {

   3:     public class Order

   4:     {

   5:         public Client client { get; set; }

   6:         public Book book { get; set; }

   7:     }

   8: }

Snippet 3: The Order class

Up to now, our bookstore has no functionality at all. We’ll change that by implementing the Store class:

   1: namespace BookStore

   2: {

   3:     public class Store

   4:     {

   5:         public List<Order> Orders { get; set; }

   6:

   7:         public void AddOrderForDelivery(Order order)

   8:         {

   9:             Orders.Add(order);

  10:         }

  11:

  12:         public void DeliverOrders()

  13:         {

  14:             foreach (Order order in Orders)

  15:             {

  16:                 for (int i = 0; i < order.Quantity; i++)

  17:                 {

  18:                     DeliverToAddress(

  19:                         order.Client.Address,

  20:                         order.Book);

  21:                 }

  22:             }

  23:         }

  24:

  25:         public void DeliverToAddress(String address, Book book)

  26:         {

  27:             Console.WriteLine(

  28:                 "Book with ISBN " + book.ISBN +

  29:                 " delivered to address " + address);

  30:         }

  31:     }

  32: }

Snippet 4: The Store class

So far, so good, right? Now, clearly, our code is quite error-prone. It compiles, but we are not safe from runtime exceptions.

This is where unit tests come into play.

2. Running Pex:

You can run Pex by simply right-clicking on the method in question and selecting Run Pex from the context menu.

image

Figure 1: Right clicking on the DeliverOrders method reveals the newly added “Run Pex” entry in the context menu. Click on it.

Then we need to select a test framework. Since Pex generates tests, you can choose for which framework they should be generated. Choose Visual Studio Unit Test.

image

Figure 2: Choosing the test framework.

After confirmation the dialog in Figure 2, the Pex Exploration results window appears, showing the run test cases and indicating which values have been used to created the test case.

image

Figure 3: Pex exploration results: Failed tests are shown in bold, successfully tests using normal font weight.

On the right side of the Pex results window, you have a detail window. For a successful test, it will show the generated method, which you can save away by clicking the button Save Test below.

image

Figure 4: Test details for a successful test (here: the selected test case from Figure 3)

On the other hand, for a failed test case, other than seeing the generated method, you can see also a Stack trace of the failed code. Figure 5 shows the (expanded) Stack trace. Please note also that above the Stack trace you still have the possibility to expand the method implementation.

image

Figure 5: Stack trace of a failed test.

Did you notice the Allow Exception button at the bottom line? Click on it if you want to allow the exception that has arisen during the test run.

Important: You will need to re-run Pex after the allowance of an exception.

Let’s see what happens if we save the generated test:

image

Figure 6: Save test dialog window.

Confirm twice. The standard naming will do. You will immediately notice that in the background, there is an entire test project that is being created:

image

Figure 7: The created test project BookStore.Tests. Note also the reference to the Microsoft unit testing framework. This is due to our initial selection of the test framework when running Pex.

Why would we want to do that?

Because, this way you can actually debug each single test case.

image

Figure 8: The main window showing the Test case (StoreTest.DeliverOrders.g.cs, highlighted in the solution explorer on the right side). Please note also the Debug button below the stack trace (also highlighted). Simply click on it in order to debug the generated test case.

I’d say we stop here for now, for there’s enough to digest. in the meantime you can experiment with Pex and play around in order to get comfortable.

We’ll dig deeper with the next article on Pex.

Best regards,

Martin

Read Full Post »