by Matthias Hirzel, Jonathan Immanuel Brachthäuser, and Herbert Klaeren

In Proc. of the Intl. Conference on Principles and Practices of Programming on the Java Platform: Virtual Machines, Languages, and Tools. Association for Computing Machinery, 2016.

Abstract

Regression testing can be very time expensive when running all available test cases. Test prioritization seeks to find faults early by reordering tests. Existing techniques decide in which order tests should be run based on coverage data, knowledge of code changes, historical data of prior test execution or a combination of them. Others postpone tests if similar ones are already selected for early execution. However, these approaches do not take into account that tests which appear similar still might explore different parts of the application’s state space and thus can result in different test outcome. Approaches based on structural coverage or on historical data might ignore small tests focusing on behavior that rarely changes. In this paper, we present a novel prioritization technique that is based on the frequencies with which modified code parts are executed by the tests. Our technique assumes that multiple executions of a modified code part (under different contexts) have a higher chance to reveal faults than a single execution of this code. For this purpose, we use both the output of regression test selection as well as test traces obtained during test development. We propose multiple variants of our technique, including a feedback mechanism to optimize the prioritization order dynamically, and compare them in an evaluation of Java-based applications to existing approaches using the standard APFD metric. The results show that our technique is highly competitive.