Are integration tests worth the hassle?

Whether or not you write integration tests can be a religious argument: either you believe in them or you don’t. What we even mean by integration tests can lead to an endless semantic argument.

What do you mean?

Unit tests are easy to define they test a single unit: a single class, a single method, make a single assertion on the behaviour of that method. You probably need mocks (again, depending on your religious views on mocking).

Integration tests, as fas as I’m concerned, mean they test a deployed (or at least deployable) version of your code, outside in, as close to what your “user” will do as possible. If you’re building a website, use Selenium WebDriver. If you’re writing a web service, write a test client and make requests to a running instance of your service. Get as far outside your code as you reasonably can to mimic what your user will do, and do that. Test that your code, when integrated, actually works.

In between these two extremes exist varying degrees of mess, which some people call integration testing. E.g. testing a web service by instantiating your request handler class and passing a request to it programmatically, letting it run through to the database. This is definitely not unit testing, as it’s hitting the database. But, it’s not a complete integration test, as it misses a layer: what if HTTP requests to your service never get routed to your handler, how would you know?

What’s the problem then?

Integration tests are slow. By definition, you’re interacting with a running application which you have to spin up, setup, interact with, tear down and clean up afterwards. You’re never going to get the speed you do with unit tests. I’ve just started playing with NCrunch, a background test runner for Visual Studio – which is great, but you can’t get it running your slow, expensive integration tests all the time. If your unit tests take 30 seconds to run, I’ll bet you run them before every checkin. If your integration tests take 20 minutes to run, I bet you don’t run them.

You can end up duplicating lower level tests. If you’re following a typical two level approach of writing a failing integration test, then writing unit tests that fail then pass until eventually your integration test passes – there is an inevitable overlap between the integration test and what the unit tests cover. This is expected and by design, but can seem like repetition. When your functionality changes, you’ll have at least two tests to change.

They aren’t always easy to write. If you have a specific case to test, you’ll need to setup the environment exactly right. If your application interacts with other services / systems you’ll have to stub them so you can provide canned data. This may be non-trivial. The biggest cost, in most environments I’ve worked in, with setting up good integration tests is doing all the necessary evil of setting up test infrastructure: faking out web services, third parties, messaging systems, databases blah blah. It all takes time and maintenance and slows down your development process.

Finally integration tests can end up covering uninteresting parts of the application repeatedly, meaning some changes are spectacularly expensive in terms of updating the tests. For example, if your application has a central menu system and you change it, how many test cases need to change? If your website has a login form and you massively change the process, how many test cases require a logged in user?

Using patterns like the page object pattern you can code your tests to minimize this, but it’s not always easy to avoid this class of failure entirely. I’ve worked in too many companies where, even with the best of intentions, the integration tests end up locking in a certain way of working that you either stick with or declare bankruptcy and just delete the failing tests.

What are the advantages then?

Integration tests give you confidence that your application actually works from your user’s perspective. I’d never recommend covering every possible edge case with integration tests – but a happy-path test for a piece of functionality and a failure-case gives you good confidence that the most basic aspects of any given feature work. The complex edge cases you can unit test, but an overall integration test helps you ensure that the feature is basically integrated and you haven’t missed something obvious that unit tests wouldn’t cover.

Your integration tests can be pretty close to acceptance tests. If you’re using a BDD type approach, you should end up with quite readable test definitions that sufficiently technical users could understand. This helps you validate that the basic functionality is as the user expects, not just that it works to what you expected.

What goes wrong?

The trouble is if integration tests are hard to write you won’t write them. You’ll find another piece of test infrastructure you need to invest in, decide it isn’t worth it this time and skip it. If your approach relies on integration tests to get decent coverage of parts of your application – especially true for the UI layer – then skipping them means you can end up with a lot less coverage than you’d like.

Some time ago I was working on a WPF desktop application – I wanted to write integration tests for it. The different libraries for testing WPF applications are basically all crap. Each one of them failed to meet my needs in some annoying, critical way. What I wanted was WebDriver for WPF. So I started writing one. The trouble is, the vagaries of the Windows UI eventing system mean this is hard. After a lot of time spent investing in test infrastructure instead of writing integration tests, I still had a barely usable testing framework that left all sorts of common cases untestable.

Because I couldn’t write integration tests and unit testing WPF UI code can be hard, I’d only unit test the most core internal functionality – this left vast sections of the WPF UI layer untested. Eventually, it became clear this wasn’t acceptable and we returned to the old-school approach of writing unit tests (and unit tests alone) to get as close to 100% coverage as is practical when some of your source code is written in XML.

This brings us back full circle: we have good unit test coverage for a feature, but no integration tests to verify that all the different units are hanging together correctly and work in a deployed application. But, where the trade-off is little test coverage or decent test coverage with systematic blindspots what’s the best alternative?

Conclusion

Should you write integration tests? If you can, easily: yes! If you’re writing a web service, it’s much easier to write integration tests for than almost every other type of application. If you’re writing a relatively traditional, not too-javascript-heavy website, WebDriver is awesome (and the only practical way to get some decent cross-browser confidence). If you’re writing very complex UI code (WPF or JavaScript) it might be very hard to write decent integration tests.

This is where your test approach blurs with architecture: as much as possible, your architecture needs to make testing easy. Subtle changes to how you structure your application might make it easier to get decent test coverage: you can design the application to make it easy to test different elements in isolation (e.g. separate UI layer from a business logic service); you don’t get quite fully integrated tests, but you minimize the opportunity for bugs to slip through the cracks.

Whether or not you write integration tests is fundamentally a question of what tests your architectural choices require you to write to get confidence in your code.

Advertisements

Are comments always wrong?

A colleague asked me recently:

Why aren’t developers writing comments any more?

He’d been looking through some code his team had written, and couldn’t understand it – he was looking for comments to make sense of the mess, but there were none. Before he challenged the team, he asked my opinion: should developers be writing comments?

Excessive Comments

When I started programming some years ago, I would comment everything, and I mean everything.

    // Add four to x
    x += 4;

My logic at the time was that it was impossible to tell the difference between uncommented clear code and gnarly code you just hadn’t spotted the gnarliness in yet. So I would comment everything, where the absence of a comment meant I’d forgotten – not that it was so trivial as to not warrant mentioning.

No Comments

Eventually I started working with peers who knocked some sense into me, and I immediately halved the number of lines of code I produced. At first it was a shock, but soon I realised that clear code is easier to read if there’s just less noise. And this is all (most) comments become: noise. Sometimes they’re accurate, sometimes they’re not. But you can’t rely on them, you always have to assume they might be wrong. So if the code and the comment seem at odds, you assume it’s the comment that’s wrong and not your understanding of the code (naturally you’re infallible!)

Clean Code

Uncle Bob and the notion of clean code have taken “no comments” to an almost fanatical zeal. But every time I get into an argument with someone about how maybe this time a comment might be justified: Ctrl-Alt-M, enter your comment as the method name and it makes the code more obvious. Every. Damned. Time.

However, the trouble with a zealous argument like this is it gets taken up by asshats and lazy people. It’s too easy to say “if you’d read Clean Coder you’d know you don’t need comments. Quit living in the past, grandpa!”. Uh huh. But your code is still a muddled pile of indecipherable crap. At least with comments there’d be some signposts while I wade through your steaming mess.

Some Comments

The truth is: sometimes comments do help (squeal clean code weenies, squeal!) There are some cases where extracting a method name isn’t sufficient. Sometimes having 20 one line methods in my class does not make it easier to read. The end goal is to produce understandable code. Generally, naming things properly helps. Adding comments that get stale does not. But that doesn’t mean that writing crap code and not commenting is the answer. Don’t use “no comments” as an excuse to leave your code indecipherable by human beings.

For example, sometimes you need to document why you didn’t do something. Maybe there’s a standard way of converting between currencies in your application – which this one time you’ve deliberately not used. A comment here might help future people not refactor away your deliberate choice (even better is baking your decision into a design – some class structure that makes it really obvious). Sometimes a method name really doesn’t do a line of code justice, it’s better to be seen in the context of the lines before and after it – but it really needs some explanation of what you’re doing. This is particularly true when dealing with complex algorithms or mathematical formulae.

Getting the Balance Right

How do you get the balance right? Well, your goal is to make code that other people can understand. The best way to get feedback on that is to ask someone. Either have explicit code reviews or pair program. If another human being has read your code and they could understand it – there’s a better than average chance that most other capable people will be able to read it too.

Why shouldn’t I test private methods?

Newcomers to TDD ask some interesting questions, here’s one I was asked recently: testing private methods is bad, but why?

How did we get here?

If you’re trying to test private methods, you’re doing something wrong. You can’t get to TDD nirvana from here, you’re gonna have to go back.

It all started with an innocuous little class with an innocuous little method. It did one little job, had a nice little unit test to verify it did its thing correctly. All was right with the world. Then, I had to add an extra little piece of logic. I wrote a test for it, changed the class until the test passed. Happy place. Then I started refactoring. I realised my little method, with its handful of test cases was getting quite complicated, so I used the extract method refactoring and boom! I have a private method.

While simple when I extracted it, another couple of corner cases and this private method evolves into quite a complicated piece of code – which now I’m testing one level removed: I’m testing the overall functionality of my outer method, which indirectly tests the behaviour of the private method. At some point I’m going to hit a corner case that’s quite awkward to test from the outside, it’d be so much easier if I could just test the private method directly.

What not to do

Don’t use a test framework that let’s you test private methods. Good God, no. For the love of all that is right with the world step away from the keyboard.

What to do instead

This is a prime example of your tests speaking to you. They’re basically shouting at you. But what are they saying?

Your design stinks!

If you need to test a private method – what you’re doing wrong is design. Almost certainly, what you’ve identified in your private method is a whole new responsibility. If you think about it carefully, it probably isn’t anything to do with what your original class is. Although your original class might need renaming to make that obvious. That’s ok, too. That’s incremental design at work.

An example would help about now

Say I started off with a Portfolio class – it has a bunch of Assets in it, each of which has a Value. So I can implement a Portfolio.GetValue() to tell me how much it’s all worth. But then I start dealing with weird corner cases like opening or closing prices. And what do I mean by value, what I could sell it for, right now? Or perhaps there’s some foreign currency conversion to do, or penalty clauses for early exit, how does all that get factored in?

Before too long, GetValue() has a fairly large piece of logic, which I extract into GetSpotSalePrice(Asset). This method is then hairy enough to need testing, so it’s pretty clear that my design stinks. The deafening din of my tests, finally persuades me to extract GetSpotSalePrice(Asset) into another class, but here’s the million dollar question: which?

What not to do – part 2

For the love of SOLID, don’t put it in a AssetSalePriceCalculator, or a SalePriceManager. This is the number one easy mistake to make – you can follow TDD and ignore decent OO design and still end up with a steaming turd pile of horribleness.

NounVerber class is always a design smell. Just stop doing it. Now. I mean it. I’m watching you. I will hunt you down and feed you to the ogre of AbstractSingletonProxyFactoryBean.

What should I do then?

The answer might seem obvious, but to too many people starting out doing TDD and half-decent design – it isn’t at all obvious. The method needs to move to a class where that responsibility makes sense. In our example, it’s crushingly obvious this is really a method on Asset – it even passes one in. If your method has one class parameter and uses a bunch of data from that class, you can bet your bottom dollar you’re suffering feature envy. Sort your life out, apply the method move refactoring. Go live happily ever after.

Summary

Why shouldn’t you test private methods? Because the fact you’re asking the question means the method shouldn’t be private – it’s a piece of independent behaviour that warrants testing. The hard choice, the design decision, is where you stick it.