The Code Test Debate

Posted by Jessica Alter /January 28, 2015 / Engineering, Entrepreneurial Advice, Hot Topics

For almost the last decade I’ve worked in and with tech companies where coding tests have been an accepted part of the interview process for engineers. Even a solid collection of references and a great portfolio isn’t always enough to ensure you’re hiring right person. But a recent debate broke out on FD:Discuss about what kinds of coding tests are best and moreover whether they’re really a good tool at all. Frankly, I was surprised. Here is what I learned.

Code Test Debate - FounderDating

You’re wasting my time…

The more senior your candidates, the higher the likelihood that they will have an extensive portfolio of coding samples you can look at to piece together a picture of their skills. Before you get to testing, many members on FD: Discuss suggest taking a look at the info that’s already available. Ask about the candidate’s side projects, and ask if they have a Github account—many developers work on open source projects, writes Owen Rubel, early technical team member at Amazon and creator of API Abstraction and API Chaining. As a potential senior team member he sees coding tests as a waste of time. “Don’t waste the interviewers time is what it boils down to. Interview for primary skills needed and have an interviewer who knows what to interview for and how to interview. Rubel’s stance is that a 15-minute whiteboard test of coding skills is best. Shobhit Verma, Cofounder and CTO of EdGenie and advisor on data science and analytics says that, “Most of my engineer friends are tired of unpaid challenges.”

“Don’t rely on programmer tests. They’re not representative of real-world programming and they are often flawed.”- Tom Maiaroto

Unrepresentative of the real world

Roshan Diwakar piled on explaining, “I disagree with a 2 hour test. A developer’s yield is an exponential curve (and not linear). There are plenty of developers who struggle the first couple of hours or they are taking time to analyze the problem, collect their thoughts, sleep/shower over things and they get a brilliant insight (sometimes far superior than the ones who gave a quick solution).” Diwakar suggests a two-week independent project as a better way to test an engineer’s mettle (paid, of course).

Former SVP of Product Development at Blackboard, George Calvert, goes one step further, “It can be a waste of time to bother with clever tests that have nothing to do with your business.” Instead, he recommends that, assuming you’ve assured yourself that this person can produce code samples, start him or her as a contract-to-hire and pay by deliverable. That way you’re both sharing in the risk that the hire might not work out, and no one is relying on an arbitrary two-hour task as a ultimate way to evaluate skills that might be better demonstrated by project work. The caveat – the engineer has to be willing to do leave their current job for something less secure.

Why coding tests can help the hiring process

Across the board, members of FounderDating do agree that you need some definitive evidence of a programmer’s skills before hiring. Older code samples are important but not necessarily enough. You’re not only testing coding ability, but attention to detail, going over and above what is asked, how you problem solve. You can’t necessarily tell these characteristics from a GitHub account.

Michael Floyd, CEO of CloakLabs and MIT VLAB Board Member, encourages seeing how they think through things with someone else from the team. “One of the best techniques I’ve seen is a pair programming session with the candidate’s future manager or someone from that team,” he writes. “The interviewer sets up a problem, typically a real one they are familiar with, and then they work with the candidate to solve it over the course of a couple hours. The real value here is that the interviewer can directly follow the candidate’s thinking processes and assess their familiarity with the tools they’re going to be using…I’ve hired fantastic developers this way and my failure rate (bad hires) is essentially zero.”

To make code tests a good use of time, make sure you’re bringing t hem in to talk through their process. “Getting the right answer isn’t as important as demonstrating that they can converge on the correct answer during review with others,” asserts, Paulo Rafaelli, Senios iOS Developer and User Interface Engineer.

Veteran engineer and entrepreneur, Rob Gopper is pro code tests but adds, “the developer who follows up and says ‘I had an aha moment in the middle of the night and I’d like to show you the solution I came up with’ is the kind of developer I want.” This scenario Gropper describes is one that could slip right by you if you toss or keep a candidate based on a strict set of outcomes expected on the test. Be ready for surprising results and other indicators that your candidate responds with more than just an efficient completion of the task at hand.  You want someone who knows his or her stuff right now. But you also want one who will be able to hack around a problem when your site goes down from too much traffic right after a big launch, or learn a new coding skill when you decide to change your strategy.

The bottom line

You shouldn’t rely only on a coding test to paint the full picture of your candidate. What you’r looking for may be available in their open source projects or through references. The less history they have the less likely this is.  If you do give code tests, tailor your approach and don’t just look at raw code – there is just as much, if not more, information about that candidate in their approach and explanations.

 Agree or Disagree? Join the Code Test Debate Here or add your thoughts in comments below.