How many builds?

I’m always amazed at the seemingly high pain threshold .net developers have when it comes to tooling. I’ve written before about the poor state of tooling in .net, but just recently I hit another example of poor tooling that infuriates me: I have too many builds, and they don’t agree whether my code compiles.

One of the first things that struck me when starting to develop on .net was that compiling code was still a thing. An actual step that had to be thought about. Incremental compilers in Eclipse and the like have been around for ages – to the point where, generally, Java developers don’t actually have to instruct their IDE to compile code for them. But in Visual Studio? Oh it’s definitely necessary. And time consuming. Oh my god is it slow. Ok, maybe not as slow as Scala. But still unbelievably slow when you’re used to working at the speed of thought.

Another consequence of the closed, Microsoft way of doing things is that tools can’t share the compiler’s work. So ReSharper have implemented their own compiler, effectively. It incrementally parses source code and finds compiler errors. Sometimes it even agrees with the Visual Studio build. But all too often it doesn’t. From the spurious not-actually-an-error that I have to continually instruct ReSharper to ignore; to the warnings-as-errors build failures that ReSharper doesn’t warn me about; to the random why-does-ReSharper-not-know-about-that-NuGet-package-errors.

This can be infuriating when refactoring. E.g. if an automated refactor leaves a variable unused, I will now have a compiler warning; since all my projects run with warnings-as-errors switched on, this will fail the build. But ReSharper doesn’t know that. So I apply the refactoring, code & tests are green: commit. Push. Boom! CI is red. But it was an automated refactor for chrissakes, how’ve I broken the build?!

I also use NCrunch, an automated test runner for Visual Studio (like Infinitest in the Java world). NCrunch is awesome, by the way; better even than the continuous test runner in ReSharper 10. If you’ve never used a continuous test runner and think you’re doing TDD, sort your life out and setup Infinitest or NCrunch. It doesn’t just automate pressing the shortcut key to run all your tests. Well, actually that is exactly what it does – but the impact it has on your workflow is so much more than that. When you can type a few characters, look at the test output and see what happened you get instant feedback. This difference in degree changes the way you write code and makes it so much easier to do TDD.

Anyway I digress – NCrunch, because Microsoft, can’t use the result of the compile that Visual Studio does. So it does its own. It launches MSBuild in the background, continually re-compiling your code. This is not exactly kind on your CPU. It also introduces inconsistencies. Because NCrunch is running a slightly different MSBuild on each project to the build VisualStudio does you get subtly different results sometimes; which is different again from ReSharper with its own compiler that isn’t even using MSBuild. I now have three builds. Three compilers. It is honestly a miracle when they all agree that my code compiles.

An all-too-typical dev cycle becomes:

  • Write test
  • ReSharper is happy
  • NCrunch build is failing, force reload NCrunch project
  • NCrunch builds, test fails
  • Make test pass
  • Try to run app
  • VisualStudio build fails
  • Fix NuGet problems
  • NCrunch build is now failing
  • Force NCrunch to reload at least one project again
  • Force VisualStudio to rebuild the project
  • Then the solution
  • Run app to sanity check change
  • ReSharper now shows error
  • Re-ignore perennial ReSharper non-error
  • All three compilers agree, quick: commit!

Normally then the build fails in CI because I still screwed up the NuGet packages.

Then recently, as if this wasn’t already one of the outer circles of hell. The CI build was failing for a bizarre reason. We have a command line script which applies the same build steps that CI runs, so I thought I’d run that to replicate the problem. Unfortunately the command line build was failing on my machine for a spectacularly spurious reason that was different again than the failure in CI. Great. I now have five builds which don’t all agree on whether my code compiles.

Do you really hate computers? Do you wish you had more reasons to defenestrate every last one of them? Have you considered a career in software development?

Cutting Corners

The pressure to deliver yesterday is strong. If it’s not customers nagging you, it’s project managers breathing down your neck or your own self-doubt that this should have been simpler: the desire to get the task done quicker can often be irresistible. How do you strike the right balance between cutting corners and polishing the turd?

While working through a feature I maintain a “navigator pad” of things I want to come back to. These are refactorings I’ve spotted, tests that need cleaning up, design smells to look at or just plain questions I’m curious to know the answer to (can foo ever actually be null? is this method really used?) This list ebbs and flows as I’m working through a feature: some days I seem to do nothing but add new things to it, other days I manage to cross half the list off as some much-needed refactoring becomes critical to complete the next change. But the one constant throughout a feature is the nav pad.

Recently I was nearing the end of a feature and my nav pad didn’t seem to be getting any shorter. I’d spent a good bit of time refactoring things, but new problems kept appearing – it didn’t seem like I’d ever be “Done”. The feature was way behind schedule, my self-doubt was growing: I’m trying to do a good job, I don’t want this to take any longer but I keep spotting things I got wrong before or simply missed. Suddenly one morning, within the space of a couple of hours, I crossed 20 items off the nav pad, sat back and realised: it’s empty! I was Done.

The next thing that struck me was what a strange occurrence this was: I couldn’t remember the last time I’d actually crossed everything off the nav pad. There would always be some last refactorings on the list that on balance could wait until another time; some tidy up that could wait until another day; some question that I no longer cared to know the answer to. But for the first time in a long time, I’d crossed everything off!

Then the doubt sets in: have I over-engineered this? Could I have been done quicker? The pressure to cut corners is really strong: we’re always pushed to be done faster, to do the absolute minimum we can get away with. Yet I know what needs to be done, I know what the problems are with this code: I’ve written them all down in the nav pad. If I don’t fix them now, then when?

A pattern I see all-too-frequently when I come up against a design smell: I can see the design is wrong, the tests are a mess, the production code is a mess; there’s definitely a better way, I just can’t see it at the minute. I park the refactoring on the nav pad. I come back to it later after ticking off a few more parts of the feature, but I still can’t see a way to resolve the design smell. I spend a couple of hours refactoring back and forth – in the end I declare bankruptcy and raise an issue in the issue tracker. If I’m lucky I’ll pick up the issue again in a couple of months, have a half-hearted look at it but realise I can’t remember what I was really thinking at the time and close the issue. More likely after a few months with nobody picking up the issue I’ll quietly close it. My code guilt has been neatly dealt with. But the crap code still remains.

The pressure to cut corners is incredibly strong, that pressure is strongest when you’re facing a particularly difficult design change. You’ve identified a problem in the design, probably made obvious by other changes you’ve made. You’re struggling to correct it, which means it isn’t easy to resolve; but it’s obviously a problem because you’ve already spent time trying to resolve it. That means the next time you come through here you’re going to spot the same problem and hate the you of today for not fixing it. And yet, this moment right now is the clearest you’ve ever understood the problem. If you give up now, you’ll have to reload into memory all the context you’ve got right now – what makes you think you’ll be in less of a rush in six months time? That you’ll have time to re-learn this code? Time to do what should have been done today?

The pressure to be done yesterday is strong, but today is the best you’ve ever understood this code: so use that understanding to leave it better than you found it. If you’ve removed all the sharp edges you saw on your way through then at least you’re leaving the code better than you found it. Tomorrow when you pass this way, you’ll pass through a little quicker, with fewer sharp edges to distract you. But today? Today you have code gardening to do.

Git stash driven development

I’ve found myself using a pattern quite often recently, which I’ve been calling “git stash driven development” – that is, relying heavily on the magic of git stash as part of my development workflow.

Normally I follow what I think of as a fairly typical TDD workflow:

  • Write next test, watch it fail
  • Write code to make it pass
  • Commit
  • Refactor
  • Commit
  • Push

This cycle can repeat very frequently – as often as every couple of minutes. Sometimes this cycle gets slowed down when the next test to write isn’t obvious or the refactoring needs more thought. But generally this is the process I try and follow.

Quite often having written the next test which takes me forwards on my feature I hit a problem: I can’t actually make the test pass (easily). First I need to refactor to make the problem easy. In that situation I can mark the test as ignored, commit and come back to it later. I refactor as required, commit, push; then finally unignore my test and get back to where I was before. This is a fairly neat process.

However there are a couple of times when this process doesn’t work: what if I’m part way through writing my test and I realise I can’t finish without refactoring the test infrastructure? I can’t ignore my test, it probably isn’t even compiling. I certainly don’t want to commit it in its current state. I could just bin my test and re-write it, if I’m following the 15 minute rule I’m not going to lose much work. But, with the magic of git stash, I can stash my changes and come back once I’ve refactored the test code.

The more annoying time this happens is when I’m part way through a refactor step. This happens more commonly when I’m really going through a design-change – this isn’t really refactoring as it often happens outside of the normal TDD loop. I’m trying to evolve the design to somewhere different; sometimes this is driven by tests, sometimes its a non-feature changing refactor. But often there are non-trivial changes happening across numerous source files. At this point it is very easy to get part way through a refactor and realise that something else needed to have happened first. I could bin my change, I only stand to lose 15 minutes work – but why throw it away when I have git stash?

So I git stash my changes, go and make the change I needed to have happened first. Then, all too commonly, I get part way through this second change and realise something else needs to happen first. Well, git stash again! This stack of git stashes can get quite deep, if you’re not careful. But once I’ve bottomed the stack out, once I’ve managed to commit a refactor that frees up the step above I can git stash pop, complete the next refactor, commit, git stash pop; and so on up the stack until I’m done.

Now, arguably, I’m discovering the refactor in reverse order, but this seems to me often how I find it. I could have spent more time analysing the change in detail, of course. Spent time planning out my change on paper before embarking on it in the correct order. However, this is always time consuming and there’s still the risk that I miss something and come at a change “backwards”. I find that using git stash in this way lets me discover the refactor that I need to make one step at a time. Each commit is kept small, I try and stick to the 15 minute rule so that no single commit loses more than 15 minutes. Ultimately the design change is completed in a sequence of small commits, each of which builds logically on the one before. They’ve been discovered by exploration, the commits were just discovered in reverse order.

The danger is always that I find a refactor step I can’t complete the way I’d imagined – now I can’t unwind the stack and potentially all the previous git stashes aren’t committable. Whenever this happens I normally find going one or two levels up the stack will present a different approach, from where I can continue as before.

Dogma Driven Development

We really are an arrogant, opinionated bunch, aren’t we? We work in an industry where there aren’t any right answers. We pretend what we do is computer “science”. When in reality, its more art than science. It certainly isn’t engineering. Engineering suggests an underlying physics, mathematical models of how the world works. Is there a mathematical model of how to build software at scale? No. Do we understand the difference between what makes good software and bad software? No. Are there papers with published proofs of whether this idea or that idea has any observable difference on written software, as practised by companies the world over? No. It turns out this is a difficult field: software is weird stuff. And yet we work in an industry full of close-minded people, convinced that their way is The One True Way. It’s not science, its basically art. Our industry is dominated by fashion.

Which language we work in is fashion: should we use Ruby, or Node.js or maybe Clojure. Hey Go seems pretty cool. By which I mean “I read about it on the internet, and I’d quite like to put it on my CV so can I please f*** up your million pound project in a big experiment of whether I can figure out all the nuances of the language faster than the project can de-rail?”

If it’s not the language we’re using, its architectural patterns. The dogma attached to REST. Jesus H Christ. It’s just a bunch of HTTP requests, no need to get so picky! For a while it was SOA. Then that became the old legacy thing, so now it’s all micro-services, which are totally different. Definitely. I read it on the internet, it must be true.

Everyone has their opinions. Christ, we’ve got our opinions. Thousands of blogs and wankers on twitter telling you what they think about the world (exactly like this one) As if one person’s observations are useful for anything more than being able to replicate their past success, should you ever by mistake find yourself on their timeline from about six weeks ago.

For example: I wrote a post recently about pairing, and some fine specimen of internet based humanity felt the need to tell me that people who need to pair are an embarrassment to the profession, that we should find another line of work. Hahaha I know, don’t read the comments. Especially when it’s in reply to something you wrote. But seriously now, is it necessary to share your close minded ignorance with the world?

I shouldn’t get worked up about some asshat on the internet. But it’s not just some asshat on the internet. There are hundreds of thousands of these asshats with their closed minds and dogmatic views on the world. And not just asshats spouting off on the internet, but getting paid to build the software that increasingly runs all our lives. When will we admit that we have no idea what we’re doing. The only way to get better is to learn as many tools and techniques as we can and hopefully, along the way, we’ll learn when to apply which techniques and when not to.

For example, I’ve worked with some people that don’t get TDD. Ok, fine – some people just aren’t “test infected”. And a couple of guys that really would rather gut me and fry my liver for dinner than pair with me. Do I feel the need to evangelise to them as though I’ve just found God? No. Does it offend me that they don’t follow my religion? No. Do I feel the need to suicide bomb their project? No. Its your call. Its your funeral. When I have proof that my way is The One True Way and yours is a sham, you can damn well bet I’ll be force feeding it to you. But given that ain’t gonna happen: I think we’re all pretty safe. If you don’t wanna pair, you put your headphones on and disappear into your silent reverie. Those of us that like pairing will pair, those of us that don’t, won’t. I’m fine with that.

The trouble is, in this farcical echo chamber of an industry, where the lessons of 40 years ago still haven’t been learnt properly. Where we keep repeating the mistakes of 20 years ago. Of 10 years ago. Of 5 years ago. Of 2 years ago. Of last week. For Christ’s sake people, can we not just learn a little of what’s gone before? All we have is mindless opinion, presented as fact. Everyone’s out to flog you their new shiny products, or whatever bullshit service they’re offering this week. No, sorry, it’s all utter bollocks. We know less about building decent software now than we did 40 years ago. It’s just now we build a massive amount more of it. And it’s even more shit than it ever was. Only now, now we have those crazy bastards that otherwise would stand on street corners telling me that Jesus would save me if only I would let him; but now they’re selling me scrum master training or some other snake oil.

All of this is unfortunately entirely indistinguishable from reasoned debate, so for youngsters entering the industry they have no way to know that its just a bunch of wankers arguing which colour to paint this new square wheel they invented. Until after a few years they become as jaded and cynical as the rest of us and decide to take advantage of all the other dumb fools out there. They find their little niche, their little way of making the world a little bit worse but themselves a little bit richer. And so the cycle repeats. Fashion begets fashion. Opinion begets opinion.

There aren’t any right answers in creating software. I know what I’ve found works some of the time. I get paid to put into practice what I know. I hope you do, too. But we’ve all had a different set of experiences which means we often don’t agree on what works and what doesn’t. But this is all we have. The plural of anecdote is not data.

All we have is individual judgement, borne out of individual experience. There is no grand unified theory of Correct Software Development. The best we can hope to do is learn from each other and try as many different approaches as possible. Try and fail safely and often. The more techniques you’ve tried the better the chance you can find the right technique at the right time.

Call it craftsmanship if you like. Call it art if you like. But it certainly isn’t science. And I don’t know about you, but it’s a very long time since I saw any engineering round these parts.

TDD Against the Clock

A couple of weeks ago I ran a “TDD Against the Clock” session. The format is simple: working in pairs following a strict red-green-refactor TDD cycle we complete a simple kata. However we add one key constraint: the navigator starts a five minute timer. As soon as the five minutes is up:

  • If the code compiles and the tests are green, commit!
  • Otherwise, revert!

Either way, the pairs swap roles when the timer goes off. If the driver completes a red-green-refactor cycle faster than five minutes, they can commit ahead of time but the roles still swap.

The kata we chose for this session was the bowling kata. This is a nice, simple problem that I expected we could get a decent way through in each of the two 45 minute sessions.

Hard Time

The five minute time constraint sounds fiendish doesn’t it? How can you possibly get anything done in five minutes? Well, you can, if you tackle something small enough. This exercise is designed to force you to think in small increments of functionality.

It’s amazing how little you can type in five minutes. But if you think typing speed is the barrier, you’re not thinking hard enough about the right way to tackle the problem. There comes a point in the bowling kata where you go from dealing with single frames and simple scores to spares (or strikes) for the first time. This always requires a jump because what you had before won’t suit what you need now. How to tackle this jump incrementally is part of the challenge when working within a five minute deadline. One of our group had an idea but knew it was tough to get it done in five minutes. He typed like a demon trying to force his solution in: he still ran out of time. Typing speed is not the problem (no matter how much it seems like it is). You need a better approach, you need to think more not type more.

Good Behaviour

After a few cycles, we found hardly anybody hit the 5 minute deadline any more. It’s fascinating how quickly everyone realised that it was better to spend a 5 minute cycle discussing than to get lost half-way through a change and end up reverting. Similarly, when you find the change you wanted to make in this cycle is too hard or too time consuming, it’s better to throw away what you have, swap pairs and refactor before you try and write the failing test again.

These are all good behaviours that are useful in day-to-day life, where it’s all too easy to keep chasing down a rat hole. Learning to work in small, independent increments and making that a subconscious part of how you work will make you a better programmer.

Wrong School

The biggest trouble we found is that the bowling kata isn’t well suited to what I consider “normal”, outside-in TDD (London School TDD). Most of the time I use TDD as a design tool, to help me uncover the right roles and responsibilities. However, with the bowling kata the most elegant solution is the one Uncle Bob drives towards, which is just simple types with no object modelling.

This is fine for an algorithm like scoring a game of bowling, which has an ultimate truth and isn’t likely to change. But in the normal day-to-day world we’re designing for flexibility and constant change. This is where a good object model of the domain makes things easier to reason about and simpler to change. This is typically where outside-in TDD will help you.

A couple of the group were determined to implement an OO version of the bowling kata. It isn’t easy as it doesn’t lend itself naturally to being built incrementally towards a good object model. However, with enough stubbornness it can be done. This led to an interesting discussion of whether you can TDD algorithms and whether TDD is better suited to problems where an object model is the desired outcome.

Obviously you can TDD algorithms incrementally, whether it’s worthwhile I’m not so sure.  Typically you’re implementing an algorithm because there is a set of rules to follow. Implementing each rule one at a time might help keep you focussed, but you always need to be aware of the algorithm as a whole.

Using TDD to drive an OO design is different. There can be many, similarly correct object models that vary only by subtle nuances. TDD can help guide your design and choose between the nuances. While you still need to think of the overall system design, TDD done outside-in is very deliberately trying to limit the things you need to worry about at any given stage: focus on one pair of interactions at a time. This is where TDD is strongest: providing a framework for completing a large task in small, manageable increments.

Even if the problem we chose wasn’t ideal, overall I found the TDD against the clock session a great way to practice the discipline of keeping your commits small, with constant refactoring, working incrementally towards a better design.

How do you move a mountain? Simply move it one teaspoonful at a time.