Me again with some more information of how both Code Contracts and Pex can be used together to write expressive tests (while still specifying a (partial) specification of the code). In the course of this article wel’ll also cover Assertions and Assumptions. In order to point out only the necessary topics, let’s introduce a new (very basic) sample project, and let’s perform the following steps:
- Create a console application called MusicStore
- Create the class Artist (as of Figure 1)
- Create the class Album (as of Figure 2)
- Enable Code Contracts checking (both static and runtime, as of figure 3)
Figure 1: The Artist class of the MusicStore project.
Figure 2: The Album class of the MusicStore project.
Figure 3: Enabling both Runtime Checking (Full), and Static Checking
Important: Do not forget to also check the Implicit Non-Null Obligations, as of Figure 3
The only critical point for CC should be the SellAlbum method of the Artist class. Why? Simply, because there we might have a parameter of type Album that could be NULL. And the Static Checker gives us the correct hints to it, as expected:
Figure 4: Expected CC’s Static Checker suggestion and warning.
So we have one suggestion to put a precondition, and an effective warning that we might be passing a parameter that could be NULL. This was predictable.
Running Pex over the method SellAlbum reveals the following results (If you need a starting point for Pex, please look at my article First steps with PEX: Automated White box Testing for .NET)
Figure 5: Pex results for 3 test cases, one providing the album parameter = NULL. Please note also the typical Pex exploration results’ table showing one column per parameter (in this case only album ).
What happens if we put an assertion to this critical parameter? Let’s simply add one, as shown in Figure 6:
Figure 6: Adding an assertion using a call to the static method Contract.Assert(bool condition) of the CC framework.
It makes sense to assert a number of AlbumsSold higher or equal to 1 (after 1 album has been sold), right? Does putting an assertion have an impact on Pex? Of course! Let’s run it again to see how:
Figure 7: Effects on Pex of using Code Contracts’ Assertions. Pex clearly identifies this as an assertion failure, hence letting the generated test case fail. Line 2 (highlighted the additionally created Pex test for the Code Contracts assertion).
So far, so good. Let’s look at Figure 6 again. We can also put such an assertion for the album object. This clearly makes sense – since selling an album should increase the album’s TimesSold count. Let’s do this in Figure 8:
Figure 8: Added another assertion for the album.TimesSold property.
And immediately, let’s look at the Static Checker warnings and the generated Pex test cases (Figures 9 and 10):
Figure 9: One more unproven assert warning for our newly added Assertion in the SellAlbum method. This was expected. Let’s run Pex again!
Figure 10: Line 3 shows the newly generated (and failed) test case of Pex. Again, this was expected.
Great coverage – but What About the Meaning?
This is a good question. Is this still an expressive test? Let’s recap the meaning. We could have an Album -type parameter passed, which could be theoretically NULL. Would it make any sense to then test its property TimesSold? I am sure, we can agree on that this would not be the case.
So, what we need, is the possibility to prevent Pex from generating test cases in such a case, hence producing only meaningful tests. And who is going to provide us with such a capability?
Code Contracts’ Assumptions
Important: Assumptions only work when the Full contract option is checked in the Code Contracts pane (under our project’s properties)
Figure 11 gives a brief example of how to put such an assumption into our code of the SellAlbum method.
Figure 11: Creating assumptions.
Figure 11 needs some explanations:
The first three assumptions simply protect us from getting the current object, the passed album, or the passed album’s artist as NULL. Makes sense, so far. The following two assumptions ensure that this.AlbumsSold and album.TimesSold both are non-negative values.
(Note also that now we put our assertions INSIDE the artist comparison’s block – makes sense, since we do not need to assert when there happened no increase of AlbumsSold and TimesSold properties).
What do We Expect?
Well, we pretty much thought at everything, didn’t we? So we expect Pex to not issue any errors or failures again. A simple re-run of Pex will show us the result, as can be seen in Figure 12.
Figure 12: Unexpected error on line 3, a failed assertion.
An unexpected failure? How could that happen? Let’s look at what the parameters looked like in the case of failure:
Did you see it? AlbumsSold is equal to int.MaxValue ! I guess, I do not need to explain what happens, if you try to increase an integer (in SellAlbum ) which is already equal to its maximum value! But why did that happen?
Simply put, this is one of the boundary test cases we explained in First steps with PEX: Automated White box Testing for .NET article. Pex guesses from the type of variables which are the boundary values (and also intermediate values).
To summarize shortly & sweetly today’s lecture:
Code Contracts & Pex: meaningful, expressive tests
(by preventing test case generation for assumed parameter values)
Let’s stop here today. There are more interesting topics regarding Code Contracts and Pex, and I am quite sure this will keep us busy for another while!
Enjoy playing around with Pex and Code Contracts, and see you next time!