Release hell – update

Holy shit – what happened to last week?

Well our release finally made it out on Wednesday, one day late. Only caused a small outage. Oops. That’s the trouble with a release in the early hours – it might be quiet in the UK at that time, but its middle of the day in Australia. Just when is a “quiet time” when your customers are all across the globe?

I really should write up this release as a case study in how not to deploy software. Everything we could have got wrong we did: error prone manual processes, poorly documented byzantine systems, crucial business logic implemented haphazardly. You name it – we fucked it up.

I’ve gotta hand it to the team though, we stuck at it and got the release out in the end. I feel sorry for the poor IS guys that have to do the release, though – a team full of developers bitching and whining at them when they’re the only hands and feet able to do anything. That can’t be any fun at all – so I think we owe IS a few beers after this week.

Last night

Went out last night with Romi, Tim, Jo, Rosy, Jim & Sam for Tim’s birthday. Started off in a really festive bar in Camden – apparently its normally really nice, we obviously just caught them on an off night. So we left there pretty quick and headed off to Joungleurs.

The acts were really good: started off with some crazy Canadian dude and a really funny geordie lad finished up the night – not that I can remember any of their names. Ended up staying on for the disco afterwards. Much drunken dancing ensued. Hehe. Was a good night.

Ended up missing the last train home – for some reason I was certain there was a train at 1:30. Turns out it left at 1. Next train was at 5. So we ended up crashing at Tim’s.

Dragged our sorry arses back to Twyford this morning for bacon sarnies. God’s own hangover cure. So this afternoon I think I’ll sit and veg and watch the F1. Go Lewis!

Release hell

Welcome to “release hell”. When enterprise software gets deployed to production everyone runs for cover.

How we’ve got where we are and still have such a farcical release process is beyond me. Friday was day 3 of our 4 days in staging. On Tuesday, in theory, we go-live. If only we hadn’t spent two and a half days trying to get staging working. It takes a team of 15 people 20 hours to get our as-live environment, working just like live. Incredible. Imagine how much money we’re wasting.

What I don’t understand is why in a company of so many smart, over-achievers has nobody fixed this problem already?

Is it business focus? Are we too busy delivering “business value” to fix things that actually cost us a fortune? That seems impossible – the business case for fixing these things is too obvious.

Is this the limit? Have we just hit some kind of complexity wall? It doesn’t seem right to me. Other companies seem able to manage vast, complex systems – the likes of Ebay and Amazon are still in business, but deal with huge systems.

Is it a lack of ownership? No one person or team owns the whole release process. Development manage their part, then work with the IS teams to deploy it. Each side has their own processes they follow to make their lives easier, inadvertently frustrating the other.

Luckily, we’ve created a new team to tackle this. Perhaps it will work. The general consensus is that nothing will change. Just another committee that will discuss the same old problems we’ve had for years, coming up with new grand strategies that fail to actually change anything.

How do you change processes in a large software organisation?