Implementing a Monadic Testsuite Driver
Assigned to Sebastian Voigt.
Ordinary wisdom says that the individual tests in a testsuite should be independent of each other. That is, each test should be able to run independently of any other test, and in any order. This independence allows the testsuite driver to execute tests in parallel and easily summarize the results using a tests passed / tests failed metric.
But for some applications it is much more natural to consider dependencies between tests. Consider the case of a compiler which should be tested using a set of small example programs contained in some directory. For each example program we want to know whether it parses. If it parses, we want to know whether it can be renamed. If it can be renamed, we want to know whether it typechecks, and if it typechecks we want to execute it and compare the result with some known result. It is very natural to want to express this dependency between different stages of a test somehow, but most testsuite drivers don’t support any such functionality. For this example, we expect the output of the testsuite to look something like this:
Testsuite:
Testing examples/example1.hs:
- example1.hs can be parsed. OK
- example1.hs can be renamed. OK
- example1.hs can be typechecked. OK
- example1.hs can be executed and yields expected result. OK
Testing examples/example2.hs:
- example2.hs can be parsed. OK
- example2.hs can be renamed. FAIL
- example2.hs can be typechecked. NOT EXECUTED
- example2.hs can be executed and yields expected result. NOT EXECUTED
The objective of this thesis is to develop a small testsuite driver in the programming language Haskell which implements tests depending on the results on other tests, using a monadic or applicative interface.