Category: Software development

Fast Feedback

On By David GreenIn Software development4 Comments

Writing good software is all about getting feedback, quickly. Does it compile? Does it function? Does it build? Does it deploy? Does it do what the customer wanted? Does it actually work? Every step of the way we have feedback loops, to improve the software. The faster these feedback loops are, the faster the software improves.

Builds

Don’t you hate waiting for your code to compile? Or, if you use a language from this century: do you remember back in the olden days when you had to wait for the compiler? Until recently, I’d actually forgotten that incremental compilers are a relatively new invention. I remember when JBuilder (my IDE of choice back in those distant times) first introduced me to incremental compilation – it was something of a revelation! You mean, I don’t have to hit compile? It just does it? In the background? Like magic?!

A few years ago I joined a company who had something of a byzantine build process. As was the fashion at the time, they used ant for their build. Unfortunately, nobody had told them that Eclipse could also compile code. So all code was built with ant. Made a change? Run the ant build to build the relevant library (may require guesswork). Copy it (by hand) to the app server. Quickly restart WebSphere (note: not quick). Test. Lather. Rinse. Repeat. Die of boredom.

Eventually, I replaced this with an Eclipse workspace that could run the application. No more build step. No more copying things by hand. No more mistakes. No more long delays in getting feedback.

Just recently I started working with C++ again after nearly a decade in byte code / interpreted languages. I’d actually forgotten what it was like to wait for code to compile. I’d got so used to working in Eclipse where you press Go and Things Happen(tm). Now instead I have to wait for a build before I can do anything. Every little change involves minutes of my life waiting for Visual Studio.

Then, if I’m really lucky – it will even compile! Remember when your IDE didn’t give you little red squiggles or highlight broken code? How fast is that feedback loop now? Before I’ve even finished typing the IDE is telling me I’m a moron and suggesting how to fix it. Assuming my code compiles, next I run the gauntlet of linking. Normally that means some godawful error message that googling just gives decade old answers and stack overflow posts that might as well be discussing religion.

TDD

I suspect this is why TDD is less common in the C++ world. Not only does the language feel ill-suited to doing TDD (to be honest, it feels ill-suited to writing software at all), but if you have to wait minutes after each change – TDD just doesn’t work.

  • Write a failing test
  • Wait minutes for the compiler to check your work, maybe go for a cuppa
  • Write code to make the test pass
  • Wait minutes for the compiler to check your work, perhaps its lunchtime?
  • Refactor

Actually, scrap the last step – since C++ is basically entirely devoid of automated refactoring tools – just leave whatever mess you’ve created because it’s about as good as it will get.

But with a red, green, refactor cycle that takes approximately 2.6 hours – it would be impossibly slow. No wonder TDD happens less.

Pairing

I’ve been arguing recently about whether pairing or code review is the best way to improve code quality. I always used to be a huge believer in code review – I’d seen it have a massive impact on code quality and really help the team learn both the code and how to code better.

But now, after spending months pairing day in day out – I think pairing is better in every conceivable way than code review. Why? Because it’s so much more immediate. How many times have you heard feedback from code review getting left because “we don’t have time right now” or “we’ll come back to that just as soon as we’ve got this release out”.

But with pairing, you have your reviewer right there offering feedback, before you’ve even finished writing the line of code. Just like your modern IDE – giving you feedback before you’ve even had chance to get it wrong. And, just like the IDE, if the smart ass sitting next to you thinks he knows better, pass over the keyboard and let him show you.

This is just impossible with code review. The feedback cycle is necessarily much slower and, worse, it’s too easy to ignore. You can add a story to the backlog to fix code review comments. You can’t so easily add a story to the backlog to make the argumentative bloke next to you shut up! (More’s the pity, sometimes)

But either way – whether you get feedback from code review or pairing, the important thing is to get feedback. If you’re not getting feedback: you’re not learning and neither you nor your code are improving.

Pattern oriented programming

On By David GreenIn Software development10 Comments

Ever since the “Gang of Four” book, everyone and their uncle is an expert in patterns. Software is all about patterns – the trouble is, it seems very little of note has happened in the intervening 20 years.

Why patterns are good

Patterns help because they let us talk about code using a consistent language. Patterns make it easier to read code. Ultimately, reading code is all about being able to reason about it.

If I change this, what’s going to break?

This bug got reported in production, how the hell did it happen?

If I can see the patterns the code implements, it lets me reason about the code more easily, without having to carefully analyse exactly what it does. Without them, I have to carefully understand every single line of code and try and reverse engineer the patterns.

The trouble is, the patterns often get mixed up with the implementation. This makes it hard to discern what the pattern is, and whether it’s actually being followed. Just because a class is called TradeManagerVisitorFactory doesn’t mean I actually know what it does. And when you open up the rats nest, you realise it’s going to take a long time to know what on earth it’s doing.

Patterns Exist

Patterns exist in our code, whether we explicitly call them out or not – there are hundreds, probably thousands of patterns that we’re all familiar with. Wouldn’t it be great if these patterns, at least within a single code base, were consistent? Wouldn’t it be great if we could help keep developers on the straight and narrow so that when they instantiate a second instance of my “singleton” we can break the build because it’s obviously changed something that might break all sorts of assumptions.

If we could start identifying the patterns in our code, and (somehow) separate that from how they’re implemented, we might actually be able to change how certain things are implemented, consistently, mechanically, across an entire enterprise-scale code base. Let me guess, an example would help?

The Builder Pattern

Ok, it’s a really simple pattern – but it makes a nice example. I have a class, that I need to be able to create, so I use the builder pattern. However, really, a builder is a type of object construction pattern. If I start thinking about different ways objects can be instantiated I can quickly come up with a handful of alternatives:

  • A constructor, with a long parameter list
  • A builder, with a lot of setXxx methods
  • A builder, with a lot of withXxx methods, that each return the builder for method chaining
  • A builder with a list of properties for object initialisation (C#)

Now, this means I have four ways of implementing “the builder pattern”. And you know, it makes precisely zero difference, functionality-wise, which I choose. It’s an entirely stylistic choice. However, I probably want it to be consistent across my codebase.

When it boils down to it, my pattern has three elements:

  • The actual pattern – e.g. a builder with method chaining
  • The logic & configuration for the pattern – the list of fields, the types, any type conversions or parsing etc
  • The actual rendering of the first two – this is all we have today: source code

I don’t want to get rid of the last (code generation is great, until you have to, you know, actually use the generated code). This should all work from existing, real code. But maybe we could mark up the code with annotations or attributes to describe what pattern we’re following and how it’s configured. Some of the logic and configuration would have to be inferred – but that’s ok, once we know it’s a builder pattern we can take entire method bodies as “logic”.

Pattern Migration

But you know what would be really awesome? If I could change how the pattern is implemented. Maybe today I’m happy with all my objects being constructed from a long parameter list. But maybe tomorrow I decide to change my mind and use builders and withXxx method names. Wouldn’t it be awesome if I could simply change the global config and have all my code automagically refactored to match? If all my builders are annotated and identified, and all enforced to be written the same way – I can extract the configuration & logic from the implementation and re-render the pattern differently. I could programmatically replace a constructor call with a builder or vice versa.

Ok, the builder is a trivial example that probably isn’t of much use. But how many more patterns are there? Say today I have classes with lots of static methods. Well, really, they’re singletons in disguise. But none of the cool kids create singletons nowadays, they use IoC frameworks like spring to create singleton beans. Wouldn’t it be nice if I could mass-migrate a legacy codebase from static methods, to singletons, to spring beans (and back again when I realise that’s made things worse!)

Controllers

What about a more complex example – let’s compare spring-mvc and struts. Two frameworks to accomplish basically the same thing – but both with a very different philosophy. The answer isn’t to build a framework to capture the common parts of both – trust me, been there, done that: you get the worst of both worlds.

But, could you describe a pattern (probably several) that struts actions and spring-mvc controllers follow? Ultimately, both spring-mvc and struts let you respond to web requests with some code. Both give you access to the HTTP request and the session. Both give you a way of rendering a view. I wonder if you could describe how to extract the pattern-specific parts of a struts action and a spring-mvc controller? The config comes from different places, how to render views is different, how the view is selected is different, even the number of actions-per-class can be different. But, could you extract all that functionally irrelevant detail and separate the pattern configuration from the underlying source code.

If you could, maybe then you could describe a transformation between the two (sets of) patterns. This would give you a purely mechanical way of migrating a struts application to spring-mvc or vice versa.

Pattern Database

Now, the cost for me to describe, in great detail, all of the patterns for spring-mvc and struts in order to automatically migrate from one to the other is probably more than the effort to just get on and do it by hand. I could probably tackle a bunch with regexes and shell scripts, then hand finish the rest. However, what if we could define these patterns and share them? If there was some kind of open source pattern database? Then, the cost for the community is spread out – the cost for me is minimal.

Could we get to a point where our patterns are documented and codified? Where we can migrate between frameworks and even architectures with a manageable cost? An impossible dream? Or a better way of crafting software?

I can haz dependencies?

On By David GreenIn Software development9 Comments

I hate IoC containers. Spring? Evil. Guice? The devil’s own work. Why? Because it leads to such slack, lazy, thoughtless programming.

Why the hate?

Ok, perhaps I better explain myself a bit. IoC is a great idea. What annoys me, is the way IoC frameworks end up getting used by normal people. I’ve ranted previously about how IoC containers lead us to implement anaemic domain models. The trouble is, once you have a hammer, everything starts to look like a nail. Especially those pesky fingers. Once you have a dependency injection framework, everything starts to look like a dependency that needs to be injected. Need to implement some business logic? First create a new class, test drive it, then make it injectable, inject it into the class where the calling code needs it, test driving it natch, then bingo – you just hit yourself on the finger.

Now I’ve got two classes, basically closely coupled, but the IoC container hides that fact from me. I see a nice, clean interface being injected in. Aren’t I a good little OO developer? No, you’re stupid and you’re lazy.

Before you know it, your class has a dozen or more dependencies, each of which have a dozen dependencies, each of which have a dozen dependencies, each of which… you get the picture. You’ve managed to build a rats nest of a dependency graph, little by little. What you’ve TDD’d isn’t a design. The technical name for it is The Big Ball of Mud.

An Alternative

Instead, I think dependency injection works best at application seams, at architectural boundaries. Say, for example, you’re building a web app. You’ve created a TradeEntryController that allows users to, well, enter trades. The TradeEntryController naturally has loads of dependencies on the rest of the system. It needs to fetch valid assets to invest in and prices, it needs to know what your balance is so you can’t buy more shares than money in the bank etc etc. A perfect example where life without an IoC container could become really cumbersome.

But, I don’t think you need one. I think what your controller needs is a few, specific dependencies – that define the architectural boundary the controller lives within. Above the controller is a HTTP request, a session and all that blah blah. Within it, is business logic. Below it is the database. So, the dependencies we inject should represent only the architectural context in which the controller operates. For the most part, this will be common to all my controllers – not just trade entry. Controllers for managing balances, lists of assets, user accounts – these all depend on knowing stuff about their session, and to be able to talk to the next layer down: the database (or in an n-tier setup, perhaps some web services).

So, why not just inject those dependencies?

public class TradeEntryController {
    public void setSessionManager(ISessionManager sessionManager) { ... }
    public void setTradeDatabase(ITradeDatabase tradeDatabase) { ... }
    public void setAccountDatabase(IAccountDatabase accountDatabase) { ... }
    public void setAssetDatabase(IAssetDatabase assetDatabase) { ... }
}

Then in my controller, I can fetch user information from the SessionManager; I can get the list of assets from the AssetDatabase; I can check the user’s balance via the AccountDatabase; and I can record the trade via the TradeDatabase. So far, so much the same as a normal IoC container.

So what’s different?

Rather than manage these dependencies via an IoC container. I think you should push them in manually. Yes, I’m suggesting you write your own dead simple dependency injection framework. What? Am I mad? Quite probably, but bear with me.

public interface ICanHazTradeDatabase {
    void setTradeDatabase(ITradeDatabase tradeDatabase);
}

public class TradeEntryController
    implements ICanHazTradeDatabase, ICanHazAssetDatabase...
{
    ...
}

public class ControllerFactory {
    public Controller createController(Class clazz) {
        Controller c = clazz.newInstance();
        if (c instanceof ICanHazTradeDatabase)
            ((ICanHazTradeDatabase) c).setTradeDatabase(tradeDatabase);
        if (c instanceof ICanHazAssetDatabase)
            ((ICanHazAssetDatabase) c).setAssetDatabase(assetDatabase);
        if ...

        return c;
    }
}

The exact mechanics of ControllerFactory of course depend on your MVC framework, but hopefully the idea is clear: when we instantiate a controller, we check it against a known set of interfaces and push in very specific dependencies. Is it pretty? Not really. Is it easy to write? Of course. Does it push dependencies into your controller? Well, yes. Where do they come from? Well, that’s an exercise for the reader. But I’m sure you can find a way to make ControllerFactory a singleton and instantiate all your dependencies in one place.

The Point

What, exactly, is the point of all this? Well, as a developer writing a controller – I can get easy access to all the dependencies that represent the architectural context I’m running within. The databases, services, message brokers, email server, blah blah blah that the application as a whole depends on. They’re right there – I just add the interface, one method and bang – ICanHazCheeseburger.

More interesting, is what I can’t do. I can’t decide that my TradeEntryController needs a TradePricingCalculator and inject that as a dependency. Well, I could, but I’d be making TradePricingCalculator available everywhere, and I’ve got a little more work to do than I would if I was using plain old Spring or Guice – I’ve an interface to create, a couple of lines to add to some scarily named GlobalControllerFactory. Why is this important? It adds some friction. It makes something bad hard to do. I’m forced instead to think about creating a TradePrices object and adding some functionality to it. I’m forced to have a rich domain, because I can’t just move all my functionality off into a TradePriceCalculatorVisitorFactoryManagerBuilder.

The choices we make and the technologies we choose make some things easy and other things hard. We need to think carefully about whether the things we make easy should be easy. It’s always possible to do the right thing, but sometimes we need to make it easier than doing the wrong thing.

Reading code

On By David GreenIn Software development9 Comments

Writing good code is all about making it fit for human consumption. Any idiot can write code a computer can understand, it takes care to write code another human can understand. But what does it mean to make code easy to understand? Programming is a literate task – writing well requires experience of reading code and in particular reading well written code. But how do we read code? Do we start at the beginning, read line by line until the end? Hardly.

Writing Code

Let me start by asking a different question: how do you write code? You probably write a test, write a small amount of code to make the test pass, then refactor to improve the readability, design etc of your code. Little by little the functionality accumulates; little by little the design emerges from a sequence of decisions and refactoring steps. TDD is fundamentally a design activity. Although you may have an idea of what your design will look like, the actual design will emerge from a sequence of small, interdependent activities.

Reading Code

Now six months later I’m reading this same code. What do I do? Well, I might start by reading the tests – well written tests should help me understand the intent of the code in question and, if they are acceptance tests, tell me what the customer thinks is important. In practice, I find this a painful way of figuring out the important things about a system. If TDD has been followed religiously you’ll have approximately 3.6 billion tests, grouped in a number of different ways making it difficult to keep enough context in my head at once to make any sense of the system.

Instead, I probably have something specific I’m trying to do. I have a change request or a bug to track down. So I probably dive in and make some guesses about where to start. I can’t remember the code so it’s a needle in a haystack. I probably miss and have to wend my way through the code trying to figure out how it fits together. I need to get some kind of big picture – at least for the part of the code I care about, as soon as I work out what that means.

Inferring Design

What I’m trying to figure out is the design of the software. How do these 200 classes relate to each other? What are the bigger patterns that will help me figure out where I need to look. Now, if I had documentation, I could look at that. But 1. it’s probably out of date or missing 2. even if it’s there, I wouldn’t trust it anyway.

So instead I find myself scribbling in a notebook – drawing class diagrams and other doodles with boxes, arrows and lines going all over. But how do I infer the design? I’m trying to understand how some parts of the system interact with each other. I look for references to interesting methods, chasing up the call stack to see what’s interesting. I find interesting classes involved along the way and I click through into interesting method invocations to see if anything fun’s happening. All the time I’m bouncing up and down call hierarchies trying to fit the system together.

As my notes start to coalesce I’ve got some idea about the key classes I care about and their roles in the system – I might find one or two key classes and scan through the whole thing, to see what other responsibilities it has. All the time following interesting method calls and looking for references.

The problem

Reading code is frankly nothing like writing code. But if writing great code means writing code that is easy to read, it’s a damn shame that the task of writing is so fundamentally different from the task of writing. It’s not as if after I’ve written a class I can pretend to have forgotten how it works and try and infer it. The best I can do is make sure that the code itself looks superficially clean. Is the style right? Is it formatted neatly? Insanely superficial stuff that frankly won’t make any difference in 6 months when I’m cursing the idiot that decided 16 levels of method calls with similar names was a sensible design approach (me).

The solution

On a superficial level I wish we could stop dealing with code as text and instead work on the syntax tree, with automated formatting following my preferences – not yours. Then we can all have our own idiosyncratic way of having code presented without having to argue for the billionth time whether spaces are preferable to tabs (of course they are) or whether curlies should be on a new line (of course not, heathen!).

The craftsmanship ethic of writing clean, well factored code is a good step. It doesn’t tackle the fundamental problem that reading code is different from writing it – but reminds us to address some of the symptoms: a comment is a whole new method just dying to be extracted. A 100 hundred line method will be at least twice as easy to understand refactored to two 50 line methods. Make things clear and simple and maybe the code will be easy to understand. Of course, I can still write a clear and simple mess.

It’d be nice if our IDEs were less damned text based. Maybe then we could have IDEs that give us a better, visual language to describe relationships between classes in. But it’s difficult to see how that would work, or how it would avoid degenerating into all the lousy CASE tools that already litter architects desks.

Ideally writing code would be more like reading code. We would be able to describe relationships natively, at a higher level than naming methods and classes. But what does that even mean?! How can I describe a set of classes without scribbling boxes and arrows? And more importantly, how do I make that a part of the implementation language so it never gets out of sync?

Enterprise class code

On By David GreenIn Software development11 Comments

There’s a natural instinct to assume that everybody else’s code is an untidy, undisciplined mess. But, if we look objectively, some people genuinely are able to write well crafted code. Recently, I’ve come across a different approach to clean code that is unlike the code I’ve spent most of my career working with (and writing).

Enterprise-class Code

There’s a common way of writing code, perhaps particularly common in Java, but happens in C#, too – that encourages the developer to write as many classes as possible. In large enterprises this way of building code is endemic. To paraphrase: every problem in an enterprise code base can be solved by the addition of another class, except too many classes.

Why does this way of writing code happen? Is it a reaction to too much complexity? There is a certain logic in writing a small class that can be easily tested in isolation. But when everyone takes this approach, you end up with millions of classes. Trying to figure out how to break up complex systems is hard, but if we don’t and just keep on adding more classes we’re making the problem worse not better.

However, it goes deeper than that. Many people (myself included, until recently) think the best way to write well-crafted, maintainable code is to write lots of small classes. After all, a simple class is easy to explain and is more likely to have a single responsibility. But how do you go about explaining a system consisting of a hundred classes to a new developer? I bet you end up scribbling on a whiteboard with boxes and lines everywhere. I’m sure your design is simple and elegant, but when I have to get my head around 100 classes just to know what the landscape looks like – it’s going to take me a little while to understand it. Scale that up to an enterprise with thousands upon thousands of classes and it gets really complex: your average developer may never understand it.

An Example

Perhaps an example would help? Imagine I’m working on some trading middleware. We receive messages representing trades that we need to store and pass on to systems further down the line. Right now we receive these trades in a CSV feed. I start by creating a TradeMessage class.

TradeMessage

private long id;

private Date timestamp;

private BigDecimal amount;

private TradeType type;

private Asset asset;

I’m a good little functional developer so this class is immutable. Now I have two choices: i) I write a big constructor that takes a hundred parameters or ii) I create a builder to bring some sanity to the exercise. I go for option ii).

TradeMessageBuilder


public TradeMessageBuilder onDate(Date timestamp)

public TradeMessageBuilder forAmount(BigDecimal amount)

public TradeMessageBuilder ofType(TradeType type)

public TradeMessageBuilder inAsset(Asset asset)

public TradeMessage build()

Now I have a builder from which I can create TradeMessage classes. However, the builder requires the strings to have been parsed into dates, decimals etc. I also need to worry about looking up Assets, since the TradeMessage uses the Asset class, but the incoming message only has the name of the asset.

We now test-drive outside-in like good little GOOS developers. We start from a CSVTradeMessageParser (I’m ignoring the networking or whatever else feeds our parser).

We need to parse a single line of CSV, split it into its component parts from which we’ll build the TradeMessage. Now we have a few things we need to do first:

  • Parse the timestamp
  • Parse the amount
  • Parse the trade type
  • Lookup the asset in the database

Now in the most extreme enterprise-y madness, we could write one class for each of those “responsibilities”. However, that’s plainly ridiculous in this case (although add in error handling or some attempts at code reuse and you’d be amazed how quickly all those extra classes start to look like a good idea).

Instead, the only extra concern we really have here is the asset lookup. The date, amount and type parsing I can add to the parser class itself – it’s all about the single responsibility of parsing the message so it makes sense.

CSVTradeMessageParser


public TradeMessage parse(String csvline)

private Date parseTimestamp(String timestamp)

private BigDecimal parseAmount(String amount)

private TradeType parseType(String amount)
Now – there’s an issue test driving this class – how do I test all these private methods? I could make them package visible and put my tests in the same package, but that’s nasty. Or I’m forced to test from the public method, mock the builder and verify the correctly parsed values are passed to the builder. This isn’t ideal as I can’t test each parse method in isolation. Suddenly making them separate classes seems like a better idea…

Finally I need to create an AssetRepository:

AssetRepository


public Asset lookupByName(String name)

The parser uses this and passes the retrieved Asset to the TradeMessageBuilder.

And we’re done! Simple, no? So, if I’ve test driven this with interfaces for my mocked dependencies, how many classes have I had to write?

  • TradeMessage
  • TradeType
  • TradeMessageBuilder
  • ITradeMessageBuilder
  • CSVTradeMessageParser
  • Asset
  • AssetRepository
  • IAssetRepository
  • TradeMessageBuilderTest
  • CSVTradeMessageParserTest
  • AssetRepositoryTest

Oh, and since this is only unit tests, I probably need some end-to-end tests to check the whole shooting match works together:

  • CSVTradeMessageParserIntegrationTest

12 classes! Mmm enterprise-y. This is just a toy example. In the real world, we’d have FactoryFactories and BuilderVisitors to really add to the mess.

Another Way

Is there another way? Well, let’s consider TradeMessage is an API that I want human beings to use. What are the important things about this API?

TradeMessage


public Date getTimestamp()

public BigDecimal getAmount()

public TradeType getType()

public Asset getAsset()

public void parse(String csvline)

That’s really all callers care about – getting values and parsing from CSV. That’s enough for me to use in tests and production code. Here we’ve created a nice, clean, simple API that is dead easy to explain. No need for a whiteboard, or boxes and lines and long protracted explanations.

But what about our parse() method? Hasn’t this become too complex? Afterall it has to decompose the string, parse dates, amounts and trade types. That’s a lot of responsibilities for one method. But how bad does it actually look? Here’s mine, in full:

public void parse(String csvline) throws ParseException
{
    String[] parts = csvline.split(",");

    setTimestamp(fmt.parse(parts[0]));
    setTradeType(TradeType.valueOf(parts[1]));
    setAmount(new BigDecimal(parts[2]));
    setAsset(Asset.withName(parts[3]));
}

Now of course, by the time you’ve added in some real world complexity and better error handling it’s probably going to be more like 20 lines.

But, let me ask you which would you rather have: 12 tiny classes, or 4 small classes? Is it better for complex operations to be smeared across dozens of classes, or nicely fenced into a single method?