Last week, I took some days off to attend Devoxx France 2014 3rd edition. As for oysters, the largest talks do not necessarily contain the prettiest pearls. During this year’s edition, my revelation came from a 15 minutes talk by my friend Alexandre Victoor, who introduced me to the wonders of Mutation Testing. Since I’m currently writing about Integration Testing, I’m very much interested in Testing flavors I don’t know about.
Experience software developers know not to put too much faith in code coverage metrics. Reasons include:
- Some asserts may have been forgotten (purposely or not)
- Code with no value, such as getters and setters, may have been tested
- And so on…
Mutation Testing tries to go beyond code coverage metrics to increase one’s faith in tests. Here’s is how this is achieved: random code changes called mutations are introduced in the tested code. If a test still succeed despite a code change, something is definitely fishy as the test is worth nothing. As an example is worth a thousand words, here is a snippet that needs to be tested:
public class DiscountEngine {
public Double apply(Double discount, Double price) {
return (1 - discount.doubleValue()) * price.doubleValue();
}
}The testing code would be akin to:
public class DiscountEngineTest {
private DiscountEngine discounter;
@BeforeMethod
protected void setUp() {
discounter = new DiscountEngine();
}
@Test
public void should_apply_discount() {
Double price = discounter.apply(new Double(0.5), new Double(10));
assertEquals(price, new Double(5));
}
}Now, imagine line 16 was forgotten:
results from DiscountEngineTest will still pass.
In this case, however, wrong code updates in the DiscountEngine would not be detected.
That’s were mutation testing enters the arena.
By changing DiscountEngine, DiscountEngineTest will still pass and that would mean nothing is tested.
PIT is a Java tool offering mutation testing. In order to achieve this, PIT creates a number of alternate classes called mutants, where the un-mutated class is the initial source class. Those mutants will be tested against existing tests targeting the original class. If the test still pass, well, there’s a problem and the mutant is considered to have survived; if not, everything is fine as the mutant has been killed. For a single un-mutated class, this goes until the mutant gets killed or all tests targeting the class have been executed and it is still surviving.
Mutation testing in general and PIT in particular has a big disadvantage: the higher the number of mutants for a class, the higher the confidence in the results, but the higher the time required to execute tests. Therefore, it is advised to run Mutating Testing only on nightly builds. However, this cost is nothing in comparison to having trust in your tests again…
Out-of-the-box, PIT offers:
- Maven integration
- Ant integration
- Command-line
Also, Alexandre has written a dedicated plugin for Sonar.
Source code for this article can be found in IntelliJ/Maven format there.