Code coverage with unit & integration tests

On a pet project recently I set out to build automated UI (integration) tests as well as the normal unit tests. I wanted to get all of this integrated into my maven build, with code coverage reports so I could get an idea of areas with insufficient test coverage. Rather than just publish the source code for the project, I’ve put together a simple example to demonstrate how I got all this setup; so if you’re looking to integrate maven, junit, webdriver (now selenium) and emma – read on to find out how I went about it.

First off, all the source code for this is available on github: https://github.com/activelylazy/coverage-example. I’ll show key snippets, but obviously there’s lots of detail omitted that (hopefully) isn’t relevant.

The Example App

Rather than break with tradition, the example application is a simple, if slightly contrived, hello world:

How It Works

The start page is a simple link to the hello world page:

<h1>Example app</h1>
<p>See the <a id="messageLink" href="helloWorld.html">message</a></p>

The hello world page just displays the message:

<h1>Example app</h1>
<p id="message"><c:out value="${message}"/></p>

The hello world controller renders the view, passing in the message:

public class HelloWorldController extends ParameterizableViewController {
    // Our message factory
    private MessageFactory messageFactory;
    @Override
    protected ModelAndView handleRequestInternal(HttpServletRequest request,
        HttpServletResponse response) throws Exception {
        // Get the success view
        ModelAndView mav = super.handleRequestInternal(request, response);
        // Add our message
        mav.addObject("message",messageFactory.createMessage());
        return mav;
    }
    @Autowired
    public void setMessageFactory(MessageFactory messageFactory) {
        this.messageFactory = messageFactory;
    }
}

Finally the MessageFactory simply returns the hard-coded message:

public String createMessage() {
    return "Hello world";
}

The unit test

We define a simple unit test to verify that the MessageFactory behaves as expected:

public class MessageFactoryTest {
    // The message factory
    private MessageFactory messageFactory;
    @Test
    public void testCreateMessage() {
        assertEquals("Hello world",messageFactory.createMessage());
    }
    @Autowired
    public void setMessageFactory(MessageFactory messageFactory) {
        this.messageFactory = messageFactory;
    }
}

Build

A basic maven pom file is sufficient to build this and run the unit test. At this point we have a working app, with a unit test for the core functionality (such as it is) that we can build and run.

<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.example</groupId>
    <artifactId>helloworld</artifactId>
    <packaging>war</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>helloworld Maven Webapp</name>
    <build>
        <finalName>helloworld</finalName>
    </build>
    <dependencies>
        ...omitted...
    </dependencies>
</project>

Code Coverage

Now let’s integrate Emma so we can get some code coverage reports. First, we define a new Maven profile, this allows us to control whether or not we use emma on any given build.

<profile>
    <id>with-emma</id>
    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>emma-maven-plugin</artifactId>
                <inherited>true</inherited>
                <executions>
                    <execution>
                        <id>instrument</id>
                        <phase>process-test-classes</phase>
                        <goals>
                            <goal>instrument</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</profile>

This simply invokes the “instrument” goal during the Maven “process-test-classes” phase; i.e. once we’ve compiled our class files, use emma to instrument them. We can run this by invoking maven with the new profile:

mvn clean install -Pwith-emma

Once the build has completed, we can run Emma to generate code coverage reports:

On Windows:

java -cp %USERPROFILE%/.m2/repository/emma/emma/2.0.5312/emma-2.0.5312.jar emma report -r xml,html -in coverage.ec -in target/coverage.em

On Linux:

java -cp ~/.m2/repository/emma/emma/2.0.5312/emma-2.0.5312.jar emma report -r xml,html -in coverage.ec -in target/coverage.em

We can now view the HTML coverage report in coverage/index.html. At this point, it shows we have 50% test coverage (by classes). MessageFactory is fully covered, but the HelloWorldController doesn’t have any tests at all.

Integration Test

To test our controller and JSP, we’ll use WebDriver to create a simple integration test; this is a JUnit test that happens to launch a browser.

public class HelloWorldIntegrationTest {
    // The webdriver
    private static WebDriver driver;
    @BeforeClass
    public static void initWebDriver() {
        driver = new FirefoxDriver();
    }
    @AfterClass
    public static void stopSeleniumClent() {
        try {
            driver.close();
            driver.quit();
        } catch( Throwable t ) {
            // Catch error & log, not critical for tests
            System.err.println("Error stopping driver: "+t.getMessage());
            t.printStackTrace(System.err);
        }
    }
    @Test
    public void testHelloWorld() {
        // Start from the homepage
        driver.get("http://localhost:9080/helloworld/");
        HomePage homePage = new HomePage(driver);
        HelloWorldPage helloWorldPage = homePage.clickMessageLink();
        assertEquals("Hello world",helloWorldPage.getMessage());
    }
}

Lines 4-18 simply start Web Driver before the test and shut it down (closing the browser window) once the test is finished.

On line 22 we navigate to the homepage with a hard-coded URL.

On line 23 we initialise our Web Driver page object for the homepage. This encapsulates all the details of how the page works, allowing the test to interact with the page functionally, without worrying about the mechanics (which elements to use etc).

On line 24 we use the homepage object to click the “message” link; this navigates to the hello world page.

On line 25 we confirm that the message shown on the hello world page is what we expect.

Note: I’m using page objects to separate test specification (what to do) from test implementation (how to do it). For more on why this is important see keeping tests from being brittle.

Homepage

The homepage object is pretty simple:

public HelloWorldPage clickMessageLink() {
    driver.findElement(By.id("messageLink")).click();
    return new HelloWorldPage(driver);
}

HelloWorldPage

The hello world page is equally simple:

public String getMessage() {
    return driver.findElement(By.id("message")).getText();
}

Running the Integration Test

To run the integration test during our Maven build we need to make a few changes. First, we need to exclude integration tests from the unit test phase:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    ...
    <configuration>
        ...
        <excludes>
            <exclude>**/*IntegrationTest.java</exclude>
            <exclude>**/common/*</exclude>
        </excludes>
    </configuration>
</plugin>

Then we define a new profile, so we can optionally run integration tests:

<profile>
    <id>with-integration-tests</id>
    <build>
        <plugins>
            <plugin>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>maven-jetty-plugin</artifactId>
                <version>6.1.22</version>
                <configuration>
                    <scanIntervalSeconds>5</scanIntervalSeconds>
                    <stopPort>9966</stopPort>
                    <stopKey>foo</stopKey>
                    <connectors>
                        <connector implementation="org.mortbay.jetty.nio.SelectChannelConnector">
                            <port>9080</port>
                            <maxIdleTime>60000</maxIdleTime>
                        </connector>
                    </connectors>
                </configuration>
                <executions>
                    <execution>
                        <id>start-jetty</id>
                        <phase>pre-integration-test</phase>
                        <goals>
                            <goal>run</goal>
                        </goals>
                        <configuration>
                            <daemon>true</daemon>
                        </configuration>
                    </execution>
                    <execution>
                        <id>stop-jetty</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>stop</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.5</version>
                <inherited>true</inherited>
                <executions>
                    <execution>
                        <id>integration-tests</id>
                        <phase>integration-test</phase>
                        <goals>
                            <goal>test</goal>
                        </goals>
                        <configuration>
                            <excludes>
                                <exclude>**/common/*</exclude>
                            </excludes>
                            <includes>
                                <include>**/*IntegrationTest.java</include>
                            </includes>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</profile>
<profile>
<id>with-integration-tests</id>
<build>
<plugins>
<plugin>
<groupId>org.mortbay.jetty</groupId>
<artifactId>maven-jetty-plugin</artifactId>
<version>6.1.22</version>
<configuration>
<scanIntervalSeconds>5</scanIntervalSeconds>
<stopPort>9966</stopPort>
<stopKey>foo</stopKey>
<connectors>
<connector implementation=”org.mortbay.jetty.nio.SelectChannelConnector”>
<port>${test.server.port}</port>
<maxIdleTime>60000</maxIdleTime>
</connector>
</connectors>
</configuration>
<executions>
<execution>
<id>start-jetty</id>
<phase>pre-integration-test</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<daemon>true</daemon>
</configuration>
</execution>
<execution>
<id>stop-jetty</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.5</version>
<inherited>true</inherited>
<executions>
<execution>
<id>integration-tests</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<excludes>
<exclude>**/common/*</exclude>
</excludes>
<includes>
<include>**/*IntegrationTest.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>

This may look complex, but really we’re just configuring jetty to run while we run our integration tests; then configuring how to run the integration tests themselves.

In lines 9-19 configure jetty – the port to run on and how to stop it.

Lines 21-30 configure jetty to run during the “pre-integration-test” phase of the maven build.

Lines 31-37 configure jetty to be stopped during the “post-integration-test” phase of the maven build.

In lines 40-62 we use the maven-surefire-plugin again, this time to run during the “integration-test” phase of the build, only running our integration test classes.

We can run this build with:

mvn clean install -Pwith-emma -Pwith-integration-tests

This will build everything, run the unit tests, build the war, fire up jetty to host the war, run our integration tests (you’ll see a firefox window popup while the rest runs) then shut down jetty. Because the war is built with instrumented classes, Emma also tracks code coverage while we run our integration tests.

We can now build our application, running unit tests and integration tests, gathering combined code coverage reports. If we re-run the emma report and check code coverage we now see we have 100% test coverage – since the controller is also being covered through tests.

Issues

What are the outstanding issues with this, what further extensions can be made?

  • The build produces an instrumented WAR – this means you need to run a second build, without emma, to get a production-ready build.
  • The integration test hard-codes the port that Jetty is configured to start on; meaning the tests can’t be run directly within Eclipse. It is possible to pass this port in, defaulting to say, 8080, so that integration tests can be run seemlessly within Eclipse as well via the maven build
  • When running on your build server you probably don’t want Firefox popping up at random (if X is even installed); so running xvfb is a good idea. It is possible to setup maven to start & stop xvfb before & after the integration tests.

17 thoughts on “Code coverage with unit & integration tests

  1. That’s a great, short, simple and useful tutorial in how to configure a project with all you need to start it right.

    Emphasising integration tests backed by a testing coverage tool since the beginning of a project is definitely the way forward.

    Awesome article.

  2. mada

    Im getting for only the integration-test, this error :
    GRAVE: Context initialization failed
    org.springframework.beans.factory.BeanCreationException: Error creating bean wit
    h name ‘messageFactory’ defined in ServletContext resource [/WEB-INF/conf/applic
    ationContext.xml]: Instantiation of bean failed; nested exception is java.lang.N
    oClassDefFoundError: com/vladium/emma/rt/RT

    Any help ?

    1. This is when you’re running tests, right? Check that emma.jar is on your classpath. If you’re using maven, make sure the scope for the emma dependency isn’t set to test as it needs to be built into the war you deploy and run.

  3. omarus

    Thank you that was the trick. I have edited the the pom.xml so ive removed this dependency in the hurry ;)

    I’ve just wanna say first : big thank you for this tutorial.

    Secondly, i think that there is not need to have emma on the classpath and everything can be done with maven only now.

    Right now i use one line : mvn clean installl -Pwith-integration-tests emma:emma

    And the pom.xml is lacking of portability because some of the arftifacts are not present on the maven cental. Ideally the pom should come with some repositories & be self sufficient.

    The artifact selenium is not absolutely necessary if we just use firefox because this dependency grab a lot of others artifacts (selenium-chrome .) that are unused in this tutorial.
    So i have replace it with selenium-firefox-driver.

    I have also update everything with the last version of the artifacts and compatible with maven2.

    The link of the edited pom : https://docs.google.com/document/d/18FB4UcjtDCk1ZqUg-rAtPg5nInsmgCry-USOjuH_uVc/edit?hl=en_US&authkey=CIuV9owM&pli=1

    The command to launch everything :
    mvn clean install -Pwith-integration-tests emma:emma

    1. Hey – thanks for the feedback. Glad you found it useful!

      I’ve just taken in your updated pom (thanks for that!). But, when I run “mvn clean install -Pwith-integration-tests emma:emma” I only get code coverage reported on unit tests, not integration tests. I.e. the “classes” score is only 50% – because the controller never gets covered.

      I did this from a clean workspace (to make sure there were no stray coverage data files hanging around) and it seems consistent. Shame, because it would be a much better way to run emma and saves the secondary step of having to generate the reports.

  4. omarus

    Hi, you are welcome. Concerning the code coverage issue, im sure you can fix it easily and it was a error of mine during the refactoring.

    I’m more interesting by using the last coverage tool, seems to be the most advance , Jacoco(produced by Emma).

    I would like to take your sample and apply Jacoco to it if you agree. I will give credit to your project and a link to it.

    Why do i want use Jacoco ? Because in a real world project (with a .war not a exploded war like here), there is somes issues with the Emma coverage.

    I want to dig further(multi-modules project) and link the results with Sonar in the tutorial.

    1. Oooh interesting! You’ll have to let me know how you get on. No issues at all using my example, if its helpful – go for it!

      I had a quick go at fixing the code coverage issue; when running with the “with-emma” profile (needed so coverage is applied to integration tests), I got some weird emma version conflict. I suspect there’s some weird maven dependency thing going on, I’ll have a look when I get time.

  5. omarus

    I almost sure what is your problem. Since the emma-maven-plugin depends on a specific emma artifact (see his pom) and you have add to your pom it your own emma artifact version:
    emma
    emma
    2.0.5312

    that is the problem. Make sure you are using the same emma version used by the emma maven plugin.

  6. Jose Thomas

    Thanks for this wonderful article on how to use Emma for unit and integration tests. I tried running the example, but it was not showing the coverage for integration tests. I’m not an expert in Maven :)

    One more thing I would like to know is on “how the Emma gets the coverage data during integration tests”. Is it working in the same way as that of Clover, which uses instrumented wars for running the integration tests and records the coverage?

    Thanks

    1. Hmm, you should get coverage for the integration tests. Code hasn’t changed since last time I ran this and it worked then! I’ll have a look and see if anything’s wrong. Try downloading the code from github, make sure that works. There’s some subtlety to the maven magic, if you miss bits out then the integration tests won’t run against instrumented code.

      Emma does, as you guess, work the same way as Clover – the WAR contains instrumented code which is deployed and has integration tests run against it.

  7. Surya

    Great article!!
    When I execute, mvn clean install -Pwith-emma -Pwith-integration-tests, I receive error java.lang.ClassNotFoundException: com.vladium.emma.rt.RT
    I checked the war and it doesn’t contain emma.jar. But If I execute,
    mvn clean install -Pwith-integration-tests, I see emma.jar being included to the war. Not sure, why this is happening?
    I want to see the coverage 100%, but, I am not able to execute both profiles in one go.

  8. Raghuveer

    Superb …
    I have one more question is it possible to exclude a source package during coverage (just because i dont need them as they are used for generating some documents from code)

  9. MADAN

    Hi
    Our UI is developed using HTML,JSP,JS and java . Apart from junits developed by dev team , QA team also developed some Junit selenium tests.

    can we use our selenium tests to measure the code coverage using .
    Please suggest

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s