Aug 27, 2017 / HIRING, RECRUITING, CERTIFICATION

You're not a compiler!

At conferences, it’s common to get gifts at booths. In general, you need to pass some some sort of challenge.

An innocent puzzle

Some of those challenges involve answering a code puzzle, like the following:

What would be the output of running the following snippet:
public class Sum {

    public static void main(String[] args) {
        System.out.println(0123 + 3210);
    }
}
3293
3333
The code doesn’t compile
The code throws an IllegalArgumentException
None of the above

My usual reaction is either to leave the booth immediately, or if I’ve some time, to write the code down on my laptop to get the result (or search on Google). You could say I’m lazy but I prefer to think I’m an expert at optimizing how I spend my time.

Simple but useless

There is a single advantage to this approach: anybody can grade the answers - if the correct results are provided. And that’s all.

Otherwise, this kind of puzzles brings absolutely nothing to the table. This wouldn’t be an issue if it stayed on conference booths, as a "fun" game during breakouts. Unfortunately, the stuff has become pretty ubiquitous, to filter out candidates during recruiting or to deliver certifications.

In both cases, skills needs to be assessed, and the easy solution is through a multiple-choice question test. Of course, if one has no access to an IDE nor Google, then it becomes a real challenge. But in the case of the above sample, what’s being checked? The knowledge of a very specific corner-case. This specific usage I would disallow on my projects anyway for this exact same reason: it’s a corner-case! And I want my code to be as free of corner cases as possible,

Another sample

This is also another kind of puzzle you can stumble upon:

In the class javax.servlet.http.HttpServlet, what’s the correct signature of the service() method?:

public void service(ServletRequest, ServletResponse)
public void service(HttpServletRequest, HttpServletResponse)
public int service(ServletRequest, ServletResponse)
public int service(HttpServletRequest, HttpServletResponse)
protected void service(ServletRequest, ServletResponse)
protected void service(HttpServletRequest, HttpServletResponse)
protected int service(ServletRequest, ServletResponse)
protected int service(HttpServletRequest, HttpServletResponse)

This kind of question asserts nothing but the capacity to remember something that is given by the most basic of IDEs or the JavaDoc.

In fact, the complete title of this post should have been "You’re not a compiler, nor a runtime, nor the documentation".

The core issue

There is no single proof of correlation between correctly answering any of the above questions and real-world proficiency of applying the skills they are meant to assess. Is there an alternative? Yes. Don’t simply check shallow knowledge, check what you want to assert, under the same conditions, including IDEs, documentation, Google, etc.

You want to check if candidates are able to read code? Make them read your own code - or similar code if your code is secret.
You want to check if candidates are able to write code? Make them write code e.g. fix an existing or already-fixed bug.

The same can be done in the context of certifications.

Of course, the downside is that it takes time. Time to prepare the material. Time to analyze what the candidate produced. Time to debrief the candidate.

Is this feasible? Unfortunately for those who would cast that aside as an utopia, it already exists. The best example is the Java EE Enterprise Architect. Though the first step is a multiple-choice questions test, other steps include a project-like assignment and an associated essay.

Another example is much more personal. I’m working as a part-time lecturer public in higher education organizations. In some courses, I evaluate students by giving them an assignment: develop a very small application. The assignment takes the form of specifications, as a business analyst would.

Conclusion

For the sake of our industry, let’s stop pretending puzzles are worth anything beyond having some fun with colleagues. You should be more concerned about the consequences of a bad hire than the about the time spent on assessing the skills of a potential hire. There are alternatives, use them. Or face the consequences.

Follow @nicolas_frankel