Why is Skill in Software Difficult to Measure?


Programmer skill is notoriously difficult to measure.

Why should you care?

Because it makes your salary erratic.

If non-technical management can’t tell the difference between Programmer A and Programmer B, then how is B justified making 2X or even 3X in salary?

As a result, software developers can’t justify their rising salaries as their skill improves. The “skill” can’t be measured, so a non-technical manager can’t distinguish between the novice and the person with 10 years experience.

They both look like they’re just writing code. Only a trained eye can see which one is doing a better job, and even then it’s tough to tell.

So why is programming such a difficult skill to measure? Mostly because a software program is full of intangibles. The problem is further complicated by the fact that bad programmers masquerade as good ones.

Let’s look at some examples.

Enter Mike

Mike blasts through code like he was born for it. He produces working solutions for programs in 1/2 to 1/3rd the amount of time it takes his coworkers. He doesn’t know much about algorithms. Algorithms are “impractical” because he’s only had to use them twice since he left school.

Unfortunately, Mike’s code is a mess. Other programmers can’t decipher what Mike did, let alone add new features on top of it. What’s worse, Mike’s stuff is usually full of problems. Not obvious bugs, at first everything seems to be working fine. Then a week later, users run into random failures. It’s not every user, just 1 in 10, and it’s not reproducible. It’s the sort of non-deterministic nightmare that most programmers dread.

Furthermore, Mike isn’t great at debugging, so many times he can’t solve his own issues. He’ll introduce these non-deterministic quandaries and then he’s helpless to do anything about them.

Most people would consider Mike a bad programmer. He’s fast, and he produces working solutions, but the “true cost” of his code is massive.

Mike introduces intangibles into the codebase in the form of opportunity costs. He costs other people time fixing his bugs and refactoring his shitty code. These costs are almost impossible to measure. The most others can say is Mike’s code is “bad” and they wish it were “better”.

Here’s the problem: if Mike had a non-technical manager, the manager would never know about the quality of Mike’s software. In his eyes, Mike gets his shit done in less time than everyone else. The future problems may or may not ever happen, and if another programmer has trouble adding to Mike’s code, it’s the fault of the other programmer, not Mike.

The manager might figure it out after a year of other programmers complaining about Mike, but even that is unlikely.

This problem is unique to the profession of programming. If you’re a doctor and you screw up surgery, it’s immediately evident who is at fault. A mechanic – same thing most of the time. In programming, bad code causes opportunity costs for years into the future. The problems are “hidden” because they only result in lost time.

Let’s look at another example.

Enter Steve

Steve is a technical master: he got his PhD in Computer Science at the age of 23. All his code is carefully thought out and architected Perfectly™. He uses abstractions extensively, thinking up complex Object Oriented paradigms for even the simplest of tasks.

Steve’s stuff works 95% of the time. He’s good about getting all his bugs out, and he can actually do the debugging himself.

Unfortunately, Steve isn’t that fast. His solutions take 2X as long as everyone else, and Steve is lazy. He works around 28 hours a week, but he works just enough so that nobody says anything to him. He’s one of those people who will never be fired or even put on a performance review plan. He does the bare minimum, but it’s just enough so that everyone accepts that that’s the way he is.

In my mind, Steve is guilty of a couple of things:


He’s too educated and arrogant for his own good. He turns even the simplest task into a 5-class beauty that would inspire Java book writers everywhere. But again, if another programmer is reading his code, there’s an opportunity cost. That other programmer has to spend time understanding Steve’s “beautiful abstraction.”

Steve would be easy to criticize if he never used design patterns, but over-design is a much more subtle, harder to catch problem.

Not Hard Working Enough

If your whole office works 30 hours a week, then Steve would be doing fine. Or if he was just as fast as everyone else, nobody would complain. But if he doesn’t work hard AND he’s slow, the office has a problem.

Again, Steve seems like he’s a good programmer. He writes code using all the design patterns in the book, he fixes bugs, and he’s smart. He’s not so belligerent about avoiding work that anybody really notices. So I’m not sure that anything would ever be done about him.

And if it came down to the two of them, I’d much rather have Steve over Mike. But if I had the choice, I’d choose someone else entirely. Steve is undependable and Mike is borderline incompetent.

Now let’s look at this problem at the team level.

Enter Programming Team 1

A friend of mine works at a national lab. He’s forced to interact with Team 1. They’re always behind. They get distracted easily, and they ship features months late. Their stuff is low quality, and they produce tons of non-deterministic bugs that are maddening for the programmers who depend on them. And everyone depends on them because Team 1 supports a library that everyone is forced to use.

They offer no documentation or any help in debugging problems. The result is that people who use them get delayed by months. Other engineers can’t just jump into their code and debug problems because they have no way of running Team 1’s code.

But again, this poorly performing team is almost invisible to management.

According to the lab’s leaders, the team is producing software that everyone depends on. Look at how successful that team is! Look at how many people are using them! Unfortunately, the teams using them don’t have a choice.

The managers wish every team could do as much good as Team 1. Nevermind that projects are delayed, or that engineers spend weeks debugging problems because of poor documentation or non-deterministic bugs. These are hidden problems. Any delays are swept under the rug as “normal.” After all, software projects are impossible to predict anyway. It takes a very trained eye to realize things could be done more efficiently.

Those are a few examples of how bad programmers appear to be good, but it also goes the other way. Sometimes good programmers masquerade as bad. There are several psychological factors that lead to this second phenomenon.

1) Assuming that everyone else’s code is Terrible.

I’ve definitely been guilty of this. You dive into a new system, and it’s complicated. The code contains several mis-designs, and your brain hurts from trying to understand what this big, new complicated system does. You catch yourself muttering under your breath, and suddenly, the code becomes the “worst thing you’ve ever seen”, and the person who wrote it is an instant moron.

In reality, you just don’t get the code, and you don’t know what the previous programmer was thinking. Maybe the programmer knew he was leaving a shitty design, and he meant to come back and fix it, but he forgot or he didn’t have time. Maybe he had higher priorities, or he had to get something out the door. Maybe the design is actually okay, you just haven’t fully understood it yet.

2) The Tendency of a Large Codebase to turn into a Giant Mess.

After a certain number of lines of code, any system tends to turn to spaghetti. Even if it’s well architected, the code has so many layers of abstraction that it’s extremely difficult to understand.

In a sufficiently large system, let’s say a couple of hundred thousand lines of code or more, you might have a single person in the entire organization who understands large pieces of it and who can see the implications of changes on one side of the system propagating to the other.

New programmers on the project will proclaim “An idiot must have designed this thing.” But they’re not in a position to judge. Only the one person who understands the whole thing can do that.

And even if the architecture is bad, odds are that nobody actually designed the system. It just happened. The current design is the evolution of 1000 small decisions made over the last 10 years.

Nobody is at fault, even though humans like to blame each other.

Good programmers masquerading as bad ones and bad programmers masquerading as good ones. So what is an actual, highly skilled programmer?

Unfortunately the definition is largely subjective, but I will try to define it in a future post. I already started in The 2 Types of Engineer and 5 Attributes of the Engineer I want to Hire.

For now, just be aware of the reality in our industry. Managers have difficulty telling software engineers apart. It might be wise to develop other skills besides just programming that you can bring to the table.

photo credit: Made to measure via photopin (license)