Faster Mutation Testing

A project reference for this article is required. Let’s use the codec submodule of Netty 4.1.

At first, let’s compile the project to measure only PIT-related time. The following should be launched at the root of the Maven project hierarchy:

The -pl option let the command be applied to only specified sub-projects - the codec sub-project in this case. Now just run the tests:

For all it’s worth, 9 seconds will be the baseline. This is not really precise measurement, but good enough for the scope of this article.

Now let’s run PIT:

A whooping 14 minutes! Let’s try to reduce the time taken.

PIT offers a lot of tweaks to improve testing speed. However, most of them imply a huge drop in relevancy: faster means less feedback on code quality less than complete. As it’s the ultimate goal of Mutation Testing, it’s IMHO meaningless.

Those configuration options are:

All benefits those flags bring are negated by less information.

By default, PIT uses a single core even though most personal computers, not to mention servers have many more available.

The easiest way to run faster is to use more cores. If it takes X minutes to run PIT on a single core, and if Y additional cores are 100% available, then it should only take X / (Y + 1) minutes to run on the first and additional cores.

The number of cores is governed by the thread Java system property when launching Maven.

This yields the following results, which probably means that those additional cores were already performing some tasks during the run.

Still, 50% is good enough for such a small configuration effort.

The rationale between incremental analysis is that unchanged tests running unchanged classes will produce the same outcome as the last one.

Hence, PIT will create a hash for every test and production class. If they are similar, the test won’t run and PIT will reuse the same result. This just requires to configure where to store those hashes.

Let’s run PIT with incremental analysis. The historyInputFile system property is the file where PIT will read hashes, historyOutputFile the file where it will write them. Obviously, they should point to the same file; anyone care to enlighten me as why they can be different? Anyway:

That produces:

That’s about the time of the previous run…​ It didn’t run any faster! What could have happened? Well, that’s the first run so that hashes were not created. If PIT is ran again a second time with the exact same command-line, the output is now:

That’s way better! Just the time to startup the engine, create the hashes and compare them.

An alternative to the above incremental analysis is to delegate change checks to the VCS. However, that requires the scm section to be adequately configured in the POM and the usage of the scmMutationCoverage goal in place of the mutationCoverage one.

Note this is the reason why Maven should be launched at the root of the project hierarchy, to benefit from the SCM configuration in the root POM.

The output is the following:

This is even faster than incremental analysis, probably since there’s no hash comparison involved.

Changing files without committing will correctly send the related mutants to be tested. Hence, this goal should be used only be developers on their local repository to check their changes before commit.

Now that an authoritative figure has positively written about Mutation Testing, more and more people will be willing to use it. In that case, the right configuration might make a difference between wide adoption and complete rejection.