Code Anthem

I'm always trolling around the internet for cool new ideas and blogs about programming. A while back I found the blog for a new site called Code Anthem and started following it. The Code Anthem blog gives an often-refreshing look at programming in the workplace, with specific focus on hiring and getting hired.

Code Anthem isn't just a blog though. Recently they've started beta testing on their pay service, a coding test system that's designed to work something like an online portfolio. It works like this: Coders take standard tests in one of a variety of programming languages to gauge skill levels. Employers come to Code Anthem and ask "I want a programmer with X skill level in Y programming language". Code Anthem then matches up coders with that skill set in that geographic area with the prospective employer. Everybody wins. Well, not the really bad programmers, but I suspect that they never win.

Yesterday, as part of the private beta, I took the test in C#. They're only offering Java and C# for the beta test so far, and of the two I am much better with the later. I went in with pretty high hopes, since I write a lot of C# code at work and consider myself to be pretty damn good at it. I logged in, answered 5 questions, and scored a solid 3/10.

Well, that sucks in a major way. I mean, I don't think I'm the best C# coder that ever graced the language, and I maybe rushed through some of the questions faster than I should have, but I have to believe I am above the 30% mark. My pride a little wounded and having a few spare moments open up, I decided to re-take the test. Through the next 5 questions I was much more careful, took my time writing the solutions, and was sure to perform testing on each solution to verify the results. At the end I looked over everything one last time, clicked submit, and ... 3/10.

Damnit. Maybe I suck much worse at this whole programming gig than I thought I did. Maybe I'm really bad at it. Like, "time to change careers" bad, Or maybe "time to join the army" bad. It was really quite a shock, and for a few moments I got kind of upset about it. I quickly got over it, and did what I think any good programmer should do in the face of failure: start debugging.

I didn't save my answers to all my questions, but did have a few and I started rigorous testing. I also had taken some notes about the various questions: some were worded in particularly difficult or vague ways, and I had made note of those things as I went. I put together my findings into a nice email and sent it off to the powers that be. After all, the point of a beta test is to find bugs and elicit feedback.

Some of the answers I had originally gotten wrong exposed bugs in the test and the results were flipped in my favor. My score didn't change too much though because the test is adaptive: as you get questions wrong it gives you easier questions later. So even though I was getting some results changed, they were the low-tier easy questions and didn't improve my scores much. What would have been needed at that point would be for me to go back and retake the test from scratch, getting positive results at the beginning which in turn would lead to harder but more valuable questions as the test progressed, improving my score substantially.

What happened instead was that Code Anthem realized that mean scores were far too low, and they fixed a few things. First, they fixed some bugs in some tests, then re-evaluated the relative difficulty (and therefore value) of tests based on how often they were legitimately answered correctly, then they recalibrated results to move the mean score up from about 30% to 50%. The end result is that my paltry 3.0 turned into a hefty 9.0, much more in line with the result I was expecting to get from the beginning. Of course, this score will probably be wiped when the beta ends and the real test gets put in place, but it's nice to see such a high number now anyway.

In the end, what do I think about Code Anthem? For starters, I think it's a really awesome idea. I've been on a number of job interviews, mostly two years ago when I graduated and was looking for my first real job. What I remember most about them is that they were almost laughably non-technical.

One interview I had was a phone interview, and I distinctly remember because of the awkwardness of trying to "write" code to an interviewer over the phone (Actually, the thing I remember most about this particular interview was a terrible blunder I made in describing the purpose of the POSIX "fork" routine, a mistake that I might not ever forgive myself for).

A second interview I went on had a written programming test, that I remember being a very vaguely-worded "real world" question, a problem that the engineer administering the test had run into not long before the interview began and hadn't taken the time to really formulate it into a test-worthy question. I spent a little bit too much time writing out a stack class that I would have needed to solve the actual problem (the test was in embedded C, so I didn't have a fancy "stack" type to play with) and when time got short I ended up just verbally describing my solution to the guy.

Another interview I went on featured a short technical phone screen. When I got to the in-person interview, I was given a test mostly involving mathematics with very little programming. It was a company specializing in numerical computing, so it's not strange that they focused so much on my math skills.

In terms of technical interviews, these three examples are really the most rigorous demonstrations of my ability that any company saw before making a hiring decision. The only other information they would have had would be my transcripts, my resume, and the short descriptions of some past projects I gave verbally to interviewers. There would be absolutely no way for them to know, given such little reliable information, what I or any application would be really capable of. Saying that I worked on such-and-such a project when I was in school, or that I was "proficient" in languages X, Y, and Z are really uninformative and subjective personal assessments, and hardly a reliable measure of my value to a company.

So this is where Code Anthem shines. They provide answers to a very fundamental question: Can this person take a set of specifications and provide a sample of code that works? I think this is a very cool idea, and I would be very happy if the concept gained traction in the industry. It certainly is a huge improvement over the current system where professionally-crafted resumes, awkward phone screens, and off-the-cuff written tests are used to make hiring decisions which could potentially have huge effects on the company.

Of course there are some things that I think the site could improve. I'll write some critiques in another post.