Ten years ago, the only tests we knew of were users acceptance tests. The last decade saw a gigantic leap forward: it brought unit testing. Unit testing was made popular with JUnit. In turn, TestNG added annotations to the test classes, making them ever easier. Then, EasyMock provided the means to mock our class dependencies in a test context while Mockito streamlined the process of doing so. One could think everything has been said and done for unit testing and we should move forward to a more worthy goal of our workforce.
I think nothing could be further from the truth. I do think that the next step will be making tests more maintainable. During my career, I had to maintain software that had no unit tests and that was bad. However, when I had to work on software with tests I had difficulties understanding, I cannot say for sure if it was better. Taking time making sure there’s no regression nor side-effect is one thing, taking time to debug test intricacies is another, and hardly expected by the management at that. One particular obstacle is in the assertion statement. I understand it may seem strange at first. How in the world could the following assertion be undecipherable?
Assert.assertFalse(myList.isEmpty());Well, I infer that some people must have had enough of always writing the same assertions and provided some common API to test them. FEST and Hamcrest are example of such testing commodities that let us boost our productivity through the use of a DSL where the domain is assertion. Left it at that, I couldn’t agree more with their use and even promote it in my teams. However, Hamcrest also provides an API to create our own custom matcher: while it’s true that creating such matcher may be a help at first glance, I’ll try to raise some points against it.
First, what about maintainability? If a test fails after I made updates, is it because of my code, the matcher or a combination of both? I’ll have to either ask the original coder, but most of the times, I’ll be on my own to decide such things. This will likely involve sweat, toil and much debugging. Taken individually, the cost is not much, but it’s exponentially proportional to the number of matchers.
Then, in order to decrease the probability of a bug in a matcher, it will have to be tested on its own! Will their respective tests need assertion matchers? It seems like it will be turtles down to the bottom. Again, costs are bound to rise.
Moreover, we are taught in school that code duplication is bad (many tools provide this metric) and should be avoided at all costs. I can only support the first axiom, just as I have to condemn the second, and more particularly the 'at all costs' part. This isn’t true! Duplication is the first level of reuse, and I hope the previous arguments made my point regarding the cost of writing and maintaining custom matchers. For example, if we duplicate two lines three times, that’s not enough in my eyes to justify the added cost of a matcher!
Finally, it may seem strange but testing is not the goal of software, only a mean to an end (something I will have to write about in a next post). We’re not tasked to make our CV shine brighter, nor to partake in the highest code coverage challenge. We’re just paid to deliver software to our customers, while respecting the holy trinity of costs, delay and quality. If writing custom matchers can help us reach this goal, fine but whether it’s effectively the case is very contextual.
At the present, it seems advantageous to me to write custom matchers under the following circumstances:
- There’s a real advantage in doing so. Search how many duplicated lines exist and whether it’s cost effective to factorize them in a matcher.
- It won’t impact delivery. What happens if we’re in the middle of writing the best matcher ever and we’re ordered to ship the product? The customer doesn’t care about matchers.
- Matchers shouldn’t ever be owned by a single developer but by the whole team. This prevents, or at least decreases the risk of having to debug a matcher to understand what its goal was.
In conclusion, I hope this article made you think about the whole custom matchers stuff. Next time you are about to create a matcher, think about it: it’s perhaps a good thing to do, but it’s not necessarily the case.