Property Based Testing with LINQ Expressions

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

C.A.R. Hoare in his 1980 Turing Award Lecture

Although Tony Hoare's famous quote is undoubtedly correct, it is increasingly difficult nowadays to design software so simple that it is obviously bug-free by design. Software gets more and more complex every day, and usually the only practical way to assure its quality is to test it thoroughly. Design is important, as is aspiration for simplicity, but they don't guarantee error free programs.

Another famous quote by Edsger W. Dijkstra says that testing shows the presence, not the absence of bugs. However, the term testing can refer to many types of activities. We should recognize that the number and types of bugs we find varies greatly depending on the kind of testing we perform. The stricter and more systematic our approach is, the more value we get out of it.

The problem is that testing can be a very tedious and time consuming task. Even automated tests require a lot of work and discipline to be effective in discovering bugs. So, we should continuously seek ways to improve our testing strategies, and explore new methods with an open mind. The best we could hope for is a technique which reduces the amount of test code we need to write while revealing more bugs, and even new kinds of bugs. Is there an approach that could satisfy these requirements? I think there is.

Property Based Testing

Property based testing refers to a software verification method where the test cases are generated by a computer. This tehcnique was first introduced in the QuickCheck library, originally developed for the purely functional programming language Haskell. As the name "property based testing" hints, the idea is to specify properties that should hold for arbitrary input data, not just for a single test case. Typically a testing library similar to Quickcheck is used to produce random input and test the specified properties. In a sense, the testing library is trying to prove our assumptions wrong by searching for counter-examples.

If the library finds a case where one or more properties do not hold, it will reduce the case to a minimal example that still fails. This is a much better starting point for analyzing the problem than getting an exception or assertion error in the middle of program execution. The program flow that led to the error can be lengthy and hard to reproduce. Also, it might be difficult to discern which actions are relevant for reproducing the problem and which are not.

Property based testing generally works best for testing pure functions whose result depend only on their input. However, the co-inventor of QuickCheck John Hughes has demonstrated that it is possible to test also imperative programs with property based testing. Rather than generating input data directly, we create sequences of function calls that modify the program's mutable state. When a failing sequence is found, the library reduces it to a minimal example that produces the same error.

There are a lot of resources on the Internet to learn more about property based testing, but I would recommend reading the original paper by Hughes and Koen Classen. It not only introduces the concept but also explains how the library is implemented in an easy-to-follow way. The paper assumes the reader to be familiar with Haskell, which might be an obstacle. If you know the basics of Haskell, though, the paper serves as an excellent guide on how to implement the library in some other programming language.

LinqCheck

As you might have guessed LinqCheck is a port of QuickCheck for C#. It implements all of the concepts present in QuickCheck, and tries to provide an API which feels natural, not mimicking the original design too much. It also aims to demonstrate how you can program in C# like it was a functional programming language. You can write really elegant and simple code by following the principles of FP; the kind of obviously correct code that Tony Hoare was referring to in his speech. The ideas of functional programming are applicable in any programming language that has the few required constructs such as closures and function types.

Another feature you might deduce from the name is that LinqCheck is built on the language facilities provided by LINQ. It is no coincidence that LINQ expressions resemble very much monads in Haskell. Erik Meijer, who is commonly credited to be the creator of LINQ, was involved in the design of Haskell language. So, naturally he designed LINQ using the same concepts that proved to be very powerful in Haskell. Understanding the correspondence between monads and LINQ helps porting the features of QuickCheck from Haskell to C# as the orginal implementation utilizes monads in many places.

Getting Started

The easiest way to learn LinqCheck is by example. The first part of this documentation covers several test examples and introduces the most important features of the library. If you are only interested in how to use the library, you can read just those chapters. But if you want to learn how the library works under the hood, you can continue reading the following chapters which cover the whole implementation.

This documentation is generated with the LiterateCS tool, which creates HTML documentation automatically from C# projects. Literate programming is a methodology that encourages writing documentation in tandem with code. It helps you build nice documentation for programs whose operation you want to explain to others.

LiterateCS is actually a great accompanion to LinqCheck. By documenting your properties and test code you will also get live documentation which specifies in natural language how the code you are testing works.

Obtaining and Installing the Library

You can download the library as a Nuget package from https://www.nuget.org/. The library targets .NET Standard 2.0.