/ TESTING, ASSERTION, LIBRARY, SOFTWARE CRAFTMANSHIP

A comparison of assertion libraries

I was not a fan of assertions libraries at first. Whether assertions provided by the testing frameworks were enough is debatable. But those libraries provides the way to write custom assertions closer to the business language. While the intention is commendable, I always thought this path was a slippery slope. If one starts writing such custom assertions, then they need to be tested obviously. And then, when will it stop?

However, there’s no denying assertion libraries make writing assertions more fluent, compared to the ones offered by testing frameworks. Besides, I don’t recall any project having custom assertions in the latest years. I tend thus to assume most developers have the same reasoning and it’s pretty safe to use those assertion libraries.

The current state of assertion libraries

When I started becoming aware of assertion libraries, there were two main contenders:

  1. FEST Assert. It was part of the larger FEST suite, which included a pretty popular Swing testing library. Currently, FEST is not under active development anymore.
  2. Hamcrest. Hamcrest is an assertion library available for all major languages (Java, Python, Ruby, etc.). Some years ago, it became the reference library for assertions.
This list wouldn’t be complete without even citing Google Truth. However, I feel it never got any traction, regardless of the Google branding.

And yet, 2 years ago, the team of the project I was working on decided to use AssertJ for assertions. I’ve no idea why, and I might be wrong, but it seems AssertJ is pretty popular nowadays. Checking the respective repos on Github also reveals Hamcrest commits are bigger but more sparse compared to AssertJ’s. Finally, AssertJ provides specific assertions for Guava, Joda Time, Neo4J, Swing(!), and databases.

In this post, I’d like to compare 3 libraries:

  1. AssertJ - it will be used as the reference in this post
  2. Strikt
  3. Atrium

A sample model

In the following, I’ll use a model shamelessly taken from the AssertJ documentation:

data class TolkienCharacter(val name: String,
                            val race: Race,
                            val age: Int? = null)

enum class Race(val label: String) {
    HOBBIT("Hobbit"), MAN("Man"), ELF("Elf"), DWARF("Dwarf"), MAIA("Maia")
}

val frodo = TolkienCharacter("Frodo", HOBBIT, 33)
val sam = TolkienCharacter("Gimli", DWARF)
val sauron = TolkienCharacter("Sauron", MAIA)
val boromir = TolkienCharacter("Boromir", MAN, 37)
val aragorn = TolkienCharacter("Aragorn", MAN)
val legolas = TolkienCharacter("Legolas", ELF, 1000)
val fellowshipOfTheRing = listOf(
        boromir,
        TolkienCharacter("Gandalf", MAN),
        aragorn,
        TolkienCharacter("Sam", HOBBIT, 38),
        TolkienCharacter("Pippin", HOBBIT),
        TolkienCharacter("Merry", HOBBIT),
        frodo,
        sam,
        legolas)

Features of AssertJ

To start using AssertJ, just add the following dependency to the POM:

<dependency>
    <groupId>org.assertj</groupId>
    <artifactId>assertj-core</artifactId>
    <version>3.11.1</version>
    <scope>test</scope>
</dependency>

At the most basic level, AssertJ allows to check for equality and sameness:

@Test
fun `assert that frodo's name is equal to Frodo`() {
  assertThat(frodo.name).isEqualTo("Frodo")
}

@Test
fun `assert that frodo is not sauron`() {
    assertThat(frodo).isNotSameAs(sauron)
}
Kotlin allows a function’s name to contain space characters, provided the name is delimited by backticks. This is quite useful for assertion names.

AssertJ also offers different assertion on strings:

@Test
fun `assert that frodo's name starts with Fro and ends with do`() {
    assertThat(frodo.name)
            .startsWith("Fro")
            .endsWith("do")
            .isEqualToIgnoringCase("frodo")
}

Finally, AssertJ really shines when asserting collections:

@Test
fun `assert that fellowship of the ring members' names contains Boromir, Gandalf, Frodo and Legolas and does not contain Sauron and Elrond`() {
  assertThat(fellowshipOfTheRing).extracting<String>(TolkienCharacter::name) (1)
    .doesNotContain("Sauron", "Elrond")
}

@Test
fun `assert that fellowship of the ring members' name containing 'o' are only aragorn, frodo, legolas and boromir`() {
  assertThat(fellowshipOfTheRing).filteredOn { it.name.contains("o") }       (2)
    .containsOnly(aragorn, frodo, legolas, boromir)
}

@Test
fun `assert that fellowship of the ring members' name containing 'o' are of race HOBBIT, ELF and MAN`() {
  assertThat(fellowshipOfTheRing).filteredOn { it.name.contains("o") }       (3)
    .containsOnly(aragorn, frodo, legolas, boromir)
    .extracting<String> { it.race.label }
    .contains("Hobbit", "Elf", "Man")
}
1 extracting() is similar to map() but in the context of assertions
2 Likewise, filteredOn() is similar to filter()
3 filteredOn() and extracting() can be combined to refine assertions in an "assertion pipeline"

Failed assertion messages are pretty basic by default:

org.opentest4j.AssertionFailedError:
Expecting:
 <33>
to be equal to:
 <44>
but was not.

Such messages can be improved by using the as() function. It also allows to reference other objects, to use them in the message.

@Test
fun `assert that frodo's age is 33`() {
    assertThat(frodo.age).`as`("%s's age", frodo.name).isEqualTo(44)
}
org.opentest4j.AssertionFailedError: [Frodo's age]
Expecting:
 <33>
to be equal to:
 <44>
but was not.

Features of Strikt

Strikt is an assertion library written in Kotlin. Its documentation is pretty comprehensive and readable.

Strikt is an assertion library for Kotlin intended for use with a test runner such as JUnit or Spek.
Nothing prevents from using it with TestNG.

To start using Strikt, add this dependency snippet to the POM:

<dependency>
    <groupId>io.strikt</groupId>
    <artifactId>strikt-core</artifactId>
    <version>0.16.0</version>
    <scope>test</scope>
</dependency>

Strikt offers equivalent features to AssertJ regarding simple usages. Its API maps nearly one-to-one:

@Test
fun `assert that frodo's name is equal to Frodo`() {
    expectThat(frodo.name).isEqualTo("Frodo")
}

@Test
fun `assert that frodo is not sauron`() {
    expectThat(frodo).isNotSameInstanceAs(sauron)
}

@Test
fun `assert that frodo starts with Fro and ends with do`() {
    expectThat(frodo.name)
            .startsWith("Fro")
            .endsWith("do")
            .isEqualToIgnoringCase("frodo")
}

Strikt also offers assertions on collections:

@Test
fun `assert that fellowship of the ring has size 9, contains frodo and sam, and does not contain sauron`() {
  expectThat(fellowshipOfTheRing)
    .hasSize(9)
    .contains(frodo, sam)
    .doesNotContain(sauron)
}

However, there’s no function corresponding to extracting() nor filteredOn(): Hence, one should default back to using map() and filter():

@Test
fun `assert that fellowship of the ring members' names contains Boromir, Gandalf, Frodo and Legolas and does not contain Sauron and Elrond`() {
  expectThat(fellowshipOfTheRing).map { it.name }
    .contains("Boromir", "Gandalf", "Frodo", "Legolas")
    .doesNotContain("Sauron", "Elrond")
}

@Test
fun `assert that fellowship of the ring members' name containing 'o' are only aragorn, frodo, legolas and boromir`() {
  expectThat(fellowshipOfTheRing.filter { it.name.contains("o") })
    .containsExactlyInAnyOrder(aragorn, frodo, legolas, boromir)
}

Using the standard API doesn’t allow to chain assertions as it’s possible in AssertJ. To compensate, it’s possible to group assertions together, via the expect() function, that accepts a lambda:

@Test
fun `assert that fellowship of the ring members' name containing 'o' are of race HOBBIT, ELF and MAN`() {
  expect {
    that(fellowshipOfTheRing.filter { it.name.contains("o") })
      .containsExactlyInAnyOrder(aragorn, frodo, legolas, boromir)
    that(fellowshipOfTheRing).map { it.race.label }
      .contains("Hobbit", "Elf", "an")
  }
}

Failed assertions messages are more descriptive than AssertJ’s:

org.opentest4j.AssertionFailedError:
▼ Expect that 33:
  ✗ is equal to 44 : found 33

This really shines with collections-related asserts and grouped asserts, pointing out exactly what assertion failed:

strikt.internal.opentest4j.CompoundAssertionFailure:
▼ Expect that […]:
  ✓ contains exactly the elements […] in any order
    ✓ contains TolkienCharacter(name=Aragorn, race=MAN,…
    ✓ contains TolkienCharacter(name=Frodo, race=HOBBIT…
    ✓ contains TolkienCharacter(name=Legolas, race=ELF,…
    ✓ contains TolkienCharacter(name=Boromir, race=MAN,…
    ✓ contains no further elements
▼ Expect that […]:
  ▼ ["Man", "Man", "Man", "Hobbit"…]:
    ✗ contains the elements ["Hobbit", "Elf", "an"]
      ✓ contains "Hobbit"
      ✓ contains "Elf"
      ✗ contains "an"

Messages can also be made more descriptive:

@Test
fun `assert that frodo's age is 33`() {
    expectThat(frodo.age).describedAs("${frodo.name}'s age").isEqualTo(44)
}
org.opentest4j.AssertionFailedError:
▼ Expect that Frodo's age:
  ✗ is equal to 44 : found 33
There’s no available method signature to pass additional object, as opposed to AssertJ’s as(). However, there’s no need to because of Kotlin’s string interpolation capability.

Atrium

Atrium is another assertion library written in Kotlin.

Atrium is designed to support different APIs, different reporting styles and Internationalization (i18n). The core of Atrium as well as the builders to create sophisticated assertions are designed to be extensible and thus allow you to extend or replace components easily.

It is very powerful, but also quite complex compared to AssertJ and Strikt.

The first step is to choose which JAR(s) to depend on. Atrium is available in several flavors:

Infix-oriented

Infix allows to call the fluent API without dots:

assert(x).toBe(2)

assert(x) toBe 2
Verb

The default assertion verb is assert(). Two other verbs are available out-of-the-box: assertThat() and check(). It’s also possible to create your own verb.

Localized

Failed assertion messages are available in English and in German.

Depending on which flavors are desired, different JAR combinations need to be referenced. The following snippet will use a no-infix, assert() and English messages:

<dependency>
    <groupId>ch.tutteli.atrium</groupId>
    <artifactId>atrium-cc-en_GB-robstoll</artifactId>
    <version>0.7.0</version>
    <scope>test</scope>
</dependency>

Basic assertions look quite alike to AssertJ’s and Strikt’s':

@Test
fun `assert that frodo's name is equal to Frodo`() {
  assert(frodo.name).toBe("Frodo")
}

@Test
fun `assert that frodo is not sauron`() {
  assert(frodo).isNotSameAs(sauron)
}

However, Atrium’s API allows an alternative fully-typesafe way of writing:

@Test
fun `assert that frodo's name is equal to Frodo 2`() {
  assert(frodo) {
    property(subject::name).toBe("Frodo")
  }
}

It can adapt depending on one’s own taste. Here’s are 4 different ways on writing the same assertion on a String:

@Test
fun `assert that frodo starts with Fro and ends with do`() {
  assert(frodo.name)
    .startsWith("Fro")
    .endsWith("do")
    .isSameAs("Frodo")
}

@Test
fun `assert that frodo starts with Fro and ends with do 2`() {
  assert(frodo.name) {
    startsWith("Fro")
    endsWith("do")
    isSameAs("Frodo")
  }
}

@Test
fun `assert that frodo starts with Fro and ends with do 3`() {
  assert(frodo) {
    property(subject::name)
      .startsWith("Fro")
      .endsWith("do")
      .isSameAs("Frodo")
  }
}

@Test
fun `assert that frodo starts with Fro and ends with do 4`() {
  assert(frodo) {
    property(subject::name) {
      startsWith("Fro")
      endsWith("do")
      isSameAs("Frodo")
    }
  }
}

As AssertJ and Strikt, Atrium offers an API to execute assertions on collections:

@Test
fun `assert that fellowship of the ring has size 9, contains frodo and sam, and does not contain sauron`() {
  assert(fellowshipOfTheRing)
    .hasSize(9)
    .contains(frodo, sam)
    .containsNot(sauron)
}

@Test
fun `assert that fellowship of the ring members' names contains Boromir, Gandalf, Frodo and Legolas and does not contain Sauron and Elrond`() {
  assert(fellowshipOfTheRing.map { it.name })                             (1)
    .containsNot("Sauron", "Elrond")                                      (2) (3)
}

@Test
fun `assert that fellowship of the ring members' name containing 'o' are only aragorn, frodo, legolas and boromir`() {
  assert(fellowshipOfTheRing.filter { it.name.contains("o") })            (1)
    .contains.inAnyOrder.only.values(aragorn, frodo, legolas, boromir)    (2) (4)
}
1 As Strikt, Atrium has no specific API for map and filter. One needs to rely on Kotlin’s API.
2 Classical contain/does not contain assertions are available.
3 Shortcut assertion
4 Full-blown customizable assertion

I found no way to refine assertions in a pipeline. The only option is to call different asserts:

@Test
fun `assert that fellowship of the ring members' name containing 'o' are of race HOBBIT, ELF and MAN`() {
  val fellowshipOfTheRingMembersWhichNameContainsO = fellowshipOfTheRing.filter { it.name.contains("o") }
  assert(fellowshipOfTheRingMembersWhichNameContainsO)
    .contains.inAnyOrder.only.values(aragorn, frodo, legolas, boromir)
  assert(fellowshipOfTheRingMembersWhichNameContainsO.map { it.race.label }.distinct())
    .containsStrictly("Hobbit", "Elf", "Man")
}

With that approach, the first failed assertion will throw an exception, and short-circuit the test flow so that potentially other failing assertions won’t be executed.

Also, besides creating your own assertion, I didn’t find anything to change the failed assertion message.

Conclusion

AssertJ is a pretty well-rounded Java assertion library. It suffers from some slight limitations, some coming from Java, some from the API itself.

Strikt is very similar to AssertJ, but fixes those limitations. If using Kotlin, it can be used as a drop-in replacement.

Atrium is also written in Kotlin, but offers a lot more capabilities at the cost of quite a lot of complexity.

To go further:

This post is also available in other languages:
Nicolas Fränkel

Nicolas Fränkel

Developer Advocate with 15+ years experience consulting for many different customers, in a wide range of contexts (such as telecoms, banking, insurances, large retail and public sector). Usually working on Java/Java EE and Spring technologies, but with focused interests like Rich Internet Applications, Testing, CI/CD and DevOps. Also double as a trainer and triples as a book author.

Read More
—
A comparison of assertion libraries
Share this