There’s been a lot of chatter recently on the intertubes about whether some developers are 10x more productive than others (e.g. here, here and here). I’m not going to argue whether this or that study is valid or not; I Am Not A Scientist and I don’t play one on TV, so I’m not going to get into that argument.
However, I do think these kinds of studies are exactly what we need more of. The biggest challenges in software development are people – individual ability and how we work together; not computer science or the technical. Software development has more in common with psychology and sociology than engineering or maths. We should be studying software development as a social science.
Recently I got to wondering: where are the studies that prove that, say, TDD works; or that pair programming works. Where are the studies that conclusively prove Scrum increases project success or customer satisfaction? Ok, there are some studies – especially around TDD and some around scrum (hyper-performing teams anyone?) – but a lazy google turns up very little. I would assume that if there were credible studies into these things they’d be widely known, because it would provide a great argument for introducing these practices. Of course, its possible that I’m an ignorant arse and these studies do exist… if so, I’m happy to be educated 🙂
But before I get too distracted, Steve’s post got me thinking: if the variation between individuals really can be 10x, no methodology is going to suddenly introduce an across the board 20x difference. This means that individual variation will always significantly dwarf the difference due to methodology.
Perhaps this is why there are so few studies that conclusively show productivity improvements? Controlling for individual variation is hard. By the time you have, it makes a mockery of any methodological improvement. If “hire better developers” will be 5x more effective than your new shiny methodology, why bother developing and proving it? Ok, except the consultants who have books to sell, conferences to speak at and are looking for a gullible customer to pay them to explain their methodology – I’m interested in the non-crooked ones, why would they bother?
Methodologies and practices in software development are like fashion. The cool kid down the hall is doing XP. He gets his friends hooked. Before you know it, all the kids are doing XP. Eventually, everyone is doing XP, even the old fogies who say they were doing XP before you were born. Then the kids are talking about Scrum or Software Craftsmanship. And before you know it, the fashion has changed. But really, nothing fundamentally changed – just window dressing. Bright developers will always figure out the best, fastest way to build software. They’ll use whatever fads make sense and ignore those that don’t (DDD, I’m looking at you).
The real challenge then is the people. If simply having the right people on the team is a better predictor of productivity than choice of methodology, then surely recruitment and retention should be our focus. Rather than worrying about scrum or XP; trying to enforce code reviews or pair programming. Perhaps instead we should ensure we’ve got the best people on the team, that we can keep them and that any new hires are of the same high calibre.
And yet… recruitment is a horrible process. Anyone that’s ever been involved in interviewing candidates will have horror stories about the morons they’ve had to interview or piles of inappropriate CVs to wade through. Candidates don’t get an easier time either: dealing with recruiters who don’t understand technology and trying to decide if you really want to spend 8 hours a day in a team you know little about. It almost universally becomes a soul destroying exercise.
But how many companies bring candidates in for half a day’s pairing? How else are candidate and employer supposed to figure out if they want to work together? Once you’ve solved the gnarly problem of getting great developers and great companies together – we’ll probably discover the sad truth of the industry: there aren’t enough great developers to go round.
So rather than worrying about this technology or that; about Scrum or XP. Perhaps we should study why some developers are 10x more productive than others. Are great developers born or made? If they’re made, why aren’t we making more of them? University is obviously poor preparation for commercial software development, so should there be more vocational education – a system of turning enthusiastic hackers into great developers? You could even call it apprenticeship.
That way there’d be enough great developers to go round and maybe we can finally start having a grown up conversation about methodologies instead of slavishly following fashion.
Actively Lazy 
I’m inclined to agree. TDD is a good idea, pair programming can make programming more fun, but I find a sensible project plan and a deadline does the most to improve my productivity.
Shifting the focus to recruitment and retention is an interesting idea. We would look for some way to make the best candidates bubble up, rather than wade through the mire of cv’s and agency fodder.
During the last round of hiring, we figured the smart move was to advertise the position on stackoverflow.com. Seemed like an easy way to spot the talent.
We got zero responses.
Maybe our advert was pathologically bad, but I don’t think so. You don’t earn any hit points for replying to job adverts… maybe we should have offered a kudos bounty…
I guess the next step would be to head hunt, but i’d much rather figure out how get the network sort it out for us.
Yeah the consensus seems to be that finding good people is *hard*.
I keep coming back to this idea of something like peer-rating developers. The best indication of a good developer is whether other people rate him. However, it rapidly degenerates into linkedin style recommendations “you recommend me & I’ll recommend you” which just devalues the whole exercise.
What you really want is some kind of negative pressure. Like if an employer hated a developer you’d rated, that would count *against you*. So it’s in your interests to only rate good developers.
Of course, linkedin are best placed to do this. And why would anyone bother with such a circle-jerk unless they were job hunting?
There are a great many more studies on TDD than you might think. They’re kind of hard to find, hard to summarize but some people like Hakan Erdogmus have done a very good job of trying.
See here for a large, unfiltered set: http://www.citeulike.org/group/8263/
My own work at the moment entails collecting those articles from varied sources into a consolidated bibliography of studies relating to Agile practices. So far I’ve gathered a little under 100 of them.
Wow – awesome. Thanks, Laurent. Plenty to read through there. I shall have to have a review.
I kinda figured that somebody, somewhere must be writing and collating these things.
That sounds like a pretty interesting exercise – I’d definitely be interested in seeing the finished collection.
You say “where are the studies that prove that, say, TDD works; or that pair programming works?” Well, proof is a strong word, but if you accept the studies about individual variability as proof, then you can do the same for the studies mentioned on http://biblio.gdinwiddie.com/biblio/StudiesOfAgileEffectiveness
So, maybe you’re right about being ignorant.
To be honest, I don’t need someone’s study to prove that TDD and pair programming work. I’ve done my own tests and found that they make *me* more productive. Whether you can accomplish the same is up to you.
I’ve also found that the synergy of a good team makes a bigger difference in productivity than the variability between individuals. I don’t have proof for that, either, but that’s OK. I’ve got plenty of work helping people who *try* to improve. The people who only want a sure thing seem to get left behind.
George – thanks for the links, plenty for me to read through there. Obviously my quick google barely scratched the surface of some of this work – it makes sense that people out there would be studying this.
I absolutely agree – we all have personal experience of these practices and can see the anecdotal evidence clearly. And you’re right, its better to be actually getting on with the damned job rather than worrying about whether some academic has “proven” this or that practice works.
But I’m intrigued to what extent we can quantify that. I still think as the industry matures we need to move away from fashion and purely anecdotal evidence to practices that we “know” work. But, to be fair, TDD – for example – I can’t imagine any practising developer arguing against.
Yes, methodologies are fads. Yes the true problem is always recruitment. And yes, recruitment is hard. The tested and true interview format: get them to code on the board during the interview. There’s nothing more accurate then getting people to actually code and explain the algorithm, big-Oh, how to optimize, how to test, …etc. It’s the ultimate litmus test.
More than that, *advertise* that you will ask them to code during the interview. That will automatically filter out all those CVs of people who get intimidated by the thought of coming up with a solution / design on the spot. That’s what big tech companies do generally (goog, amzn, msft) and they seem to be doing fairly well with getting talent.