Release hell

Welcome to “release hell”. When enterprise software gets deployed to production everyone runs for cover.

How we’ve got where we are and still have such a farcical release process is beyond me. Friday was day 3 of our 4 days in staging. On Tuesday, in theory, we go-live. If only we hadn’t spent two and a half days trying to get staging working. It takes a team of 15 people 20 hours to get our as-live environment, working just like live. Incredible. Imagine how much money we’re wasting.

What I don’t understand is why in a company of so many smart, over-achievers has nobody fixed this problem already?

Is it business focus? Are we too busy delivering “business value” to fix things that actually cost us a fortune? That seems impossible – the business case for fixing these things is too obvious.

Is this the limit? Have we just hit some kind of complexity wall? It doesn’t seem right to me. Other companies seem able to manage vast, complex systems – the likes of Ebay and Amazon are still in business, but deal with huge systems.

Is it a lack of ownership? No one person or team owns the whole release process. Development manage their part, then work with the IS teams to deploy it. Each side has their own processes they follow to make their lives easier, inadvertently frustrating the other.

Luckily, we’ve created a new team to tackle this. Perhaps it will work. The general consensus is that nothing will change. Just another committee that will discuss the same old problems we’ve had for years, coming up with new grand strategies that fail to actually change anything.

How do you change processes in a large software organisation?

2 thoughts on “Release hell

  1. Lack of ownership plays a big part.

    If it’s no ones job to tidy up all the weird environmental dependencies, then you have to sit around and wait for some demented late-night refactor-itch to strike one of your team…but its typically too fiddly and thankless a task to undertake as a personal crusade on an existing codebase.

    Setting up the CI server forced me to trim a lot of the environmental toenails. It was good, but more an eating bran flakes kinda good.

    Having an entrenched divide between ‘Dev’ and ‘Ops’ doesn’t help much. It’s is too easy for each group to blame the other for failed deployments, and coddled, command-line shy developers tend to produce code that’s just simply harder to deploy.

    I guess changing process in a big organisation is hard, like maintaining a monolithic codebase. If it can’t be fixed by one person alone, then it probably wont be.

  2. I think you’re onto something there. That explains why large software organisations deteriorate – if there’s no process for fixing things that affect operational or internal customers (is there ever?) and its too large for an individual to fix then it won’t ever get fixed.

    Remind me again why I’m working for a big company? 🙂

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.