This post is the written form of one of my submission for Devoxx France 2013. As it was only chosen as backup, I lacked the necessary motivation to prepare it. The subject is important though, so I finally decided to write it down.
In 2013, if you’re a standard developer, it is practically a given that you test your code. Whether you’re a practicioner of TDD or just create them afterwards, most realize a robust automated test harness is not optional but mandatory.
Unit Test vs Integration Test
There’s some implicit behind the notion of 'test' though. Depending on the person you ask, there may be some differences of what a test is. Let us provide two definitions to prevent potential future misunderstanding.
- Unit test
-
A unit test tests a method in isolation. In this case, the unit is the method. Outside dependencies should be mocked to provide known input(s). This input can be set different values, including bordercase values, to check all code branches.
- Integration test
-
An integration test tests the collaboration of two (or more) units. In this case, the unit is the class. This means that as soon as you want to check the execution result of more than one class, it is an integration test.
On one hand, achieving 100% UT success with 100% code coverage is not enough to guarantee a bug-free application. On the other hand, a complete IT harness is not feasible in the context of real-world project since it would cost too much. The consequence is that both are complementary.
Knowing the good IT & UT ratio is part of being a successful developer. It depends on many factors, including some outside the pure technical, such as the team skills and experience, application lifetime and so on.
Fail-fast tests
Most IT are usually inherently slower than UT, as they require some kind of initialization. For example, in modern-day applications, this directly translates to some DI framework bootstrap.
It is highly unlikely that IT be successful while UT fail. Therefore, in order to have the shortest feedback loop in case of a failure, it is a good idea to execute the fastest tests first, check if they fail, and only execute slowest ones then.
Maven provides the way to achieve this with the following plugins combination:
maven-surefire-pluginto execute unit tests; convention is that their names end withTestmaven-failsafe-pluginto execute integration tests; names end withIT
By default, the former is executed during the test phase of the Maven build lifecycle.
Unfortunately, configuration of the latter is mandatory:
<project>
...
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.14.1</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>Note the Maven mojo is already bound to the integration-test phase, so that it is not necessary to configure it explicitly.
This way, UT - whose name pattern is *Test, are executed during the test phase while IT - whose name patter is *IT, are executed in the integration-test phase.
System under test boundaries
The first step before writing IT is to define the SUT boundaries. Inside lies everything that we can manage, outside the rest. For example, the database clearly belongs to the SUT because we can easily provide test data, whereas a web service does not.
Inside the SUT, mocking or initializing subsystems depend on the needed/wanted integration level:
- To UT a use-case, one could mock DAOs in the service class
- Alternatively, one could initialize a test database and use real world data. DBUnit is the framework to use in this case, providing ways to initialize data during set up and check results afterwards.
Doing both would let us test corner cases in the mocking approach (faster) and the standard case in the initialization one (slower).
Outside the SUT, there is an important decision to make regarding IT. If external dependencies are not up and stable most of the time, the build would break too often and throws too many red herrings. Those IT will probably we deactivated very quickly.
In order to still benefit from those IT but so that they do not get ignored (or even worse, removed), they should be put in a dedicated module outside the standard build. When using a continuous integration server, you should set two jobs: one for the standard build and one for those dependent IT.
In Maven, it is easily achieved by creating a module at the parent root and not referencing it in the parent POM.
This way, running mvn integration-test will only launch tests that are in the standard modules, those that are reliable enough.
Fragile tests
As seen above, fragility in IT comes from dependency on external systems. Those fragile tests are best handled as a separate module, outside the standard build process.
However, another cause of fragility is unstability. And in any application, there’s a layer that is fundamentally unstable and that is the presentation layer. Whatever the chosen technology, this layer is exposed to end-users and you can be sure there will be many changes there. Whereas your service API are your own, GUI is subject to users whims, period.
IT that uses GUI, whether dedicated GUI tests or end-to-end tests should thus also be considered fragile and isolated in a dedicated module, as above.
My personal experience, coupled by some writings by Gojko Adzic, taught me to bypass the GUI layer and start my tests at the servlet layer. Whether you use Spring Test provides a bunch of Fakes regarding the Servlet API.
Tests ordering
Developers unfamiliar with IT should probably be more than astounded by this section title. In fact, as UT should be context-independent and not be ordered in any case.
For IT, things are a little different: I like to define some UT as user stories. For example, user logs in then performs some action and then another. Those steps can of course be defined in the same test method.
The negative side of this, is that if the test fails, we have no way of easily knowing what went wrong and during which step. We could remedy to that by isolating each step in a single method and ordering those methods.
TestNG - a JUnit offshoot fully integrated in Maven and Spring, let us do that easily:
public class MyScenarioIT {
@Test
public void userLogsIn() {
...
}
@Test(dependsOnMethod = "userLogsIn")
public void someAction() {
...
}
@Test(dependsOnMethod = "someAction")
public void anotherAction() {
...
}
}This way, when an error occurs or an assert fails, log displays the faulty method: we just need to have adequate name for each method.
Framework specifics UT
Java EE in-container UT
Up until recently, automated UT were executed independently of any container. For Spring applications, this was no big deal but for Java EE intensive applications, you had to fake and mock dependencies. In the end, there was no guarantee really running the application inside the container would produce expected results.
Arquillian brought a paradigm shift, with the ability to produce UT for applications using Java EE. If you do not use it already, know that Arquillian let you automate creation of the desired archive and its deployment on one or more configured application servers.
Those can be either existing or downloaded, extracted and configured during UT execution. The former category would need to be segregated in a separate module as to prevent breaking the build while the latter can safely be assimilated to regular UT.
Spring UT
Spring Test provide a feature to assemble different Spring configuration fragments (either XML or Java classes). This means that by designing our application with enough modularity, we can use desired production-ready and tests-only configuration fragments.
As an example, by separating our data source and DAO in two different fragments, we can reuse the regular DAO fragment in UT and use a test fragment declaring an in-memory database.
With Maven, this means having the former in src/main/resources and the latter in src/test/resources.
@ContextConfiguration(locations = {"classpath:datasource.xml", "classpath:dao.xml"})
public class MyCustomUT extends AbstractTestNGSpringContextTests {
...
}Conclusion
This is the big picture regarding my personal experience regarding UT. As always, they shouldn’t be seen as hard and fast rules but be adapted to your specific project context.
However, tools listed in the article should be a real asset in all cases.