Optimise Your Build with Faster Running JUnit Tests

Introduction
There are many techniques that you can use that can improve a build time. Here’s one that can be used when:

  • Tests break the build
  • You only care about failing tests being reported
  • You want to reuse the existing formatting utilities provided by standard Ant optional tasks
  • You don’t want the formatting to be too slow
  • The percentage of tests passing is not important

When running with the optional JUnit task, the normal strategy is to use the standard XML formatter and then style the information into something presentable with the optional JUnitReport task. Unfortunately, both the cost of spitting out XML for every single test suite executing (i.e. usually every single Test class you have) and then the cost of applying XSL is typically quite high. In my experience, it’s been several minutes running tests, let alone waiting for the report to be generated. Just try using the plain logger (<formatter type="plain"/>) and see the differences yourself.

A Better Way
The alternative to the standard XML formatter is the QuietXMLFormatter (download here). The aim of this formatter is to:

  • Only produce output on tests that fail or error;
  • Produce the XML output in the same format as the standard org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter; and
  • Do it without inheritance;

The result is a faster build (it can be quite significant depending on the number of your tests) that still reports errors and failures in the same way with only a few tweaks to the build.

Note that the QuietXMLFormatter has only been tested with:

  • Forked (once per batch) TestSuites (JUnitTask produces different behaviours depending on if you are forked or not)
  • Ant Version 1.6.2 and 1.6.5 (some of these classes change a fair bit)
  • JUnit 3.8.1

How To Add It To Your Build

  1. Download the QuietXMLFormatter distribution jar (quietXmlFormatter-0.1.jar or with source)
  2. Instead of the normal entry that looks like (<formatter type="xml"/>) use, the following: <formatter classname="com.thekua.ant.optional.junit.QuietXMLFormatter" extension=".xml"/>
  3. If you fork your build, you need to make sure that the jar is in the classpath, or ensure that ant makes it available
  4. Run your build as per normal

Known Issues
In testing this with a few of the latest versions of Ant, I found a few issues that, although not detrimental, can be slightly annoying. When I get time, I might get around to trying to see how the latest ant source handles this. The issues currently include:

  • The actual files output by the task are managed outside of a given formatter, and there is an assumption that your formatter would produce some output. This means that if you are not actually outputting anything, then you still end up with zero sized files from each individual test suite being executed.
  • At least when you run in forked mode, the extension for each output file doesn’t seem to get added to the controlling class that manages the OutputStream made available to each JUnitResultFormatter. It would be okay if JUnitReport didn’t die on files without an extension but I couldn’t work out a way. Try the following bit of code:

    &lt;move todir="${test.output.dir}" includeemptydirs="false"&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&lt;fileset dir="${test.output.dir}"&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;exclude name="**/*.xml"/&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&lt;/fileset&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&lt;mapper type="glob" from="*" to="*.xml"/&gt;<br />&lt;/move&gt;

  • JunitReport is fine for styling each of the output test reports, but because an empty file is not a valid XML document, you can end up with a fairly noisy build. The solution to this, is of course something that deletes empty files from a directory. There is also another task in the jar (DeleteEmptyFilesTask) that you can use in your build that does this. The same rules for custom ant tasks apply when you incorporate this task into your build. Integrate it like this:

    &lt;taskdef classname="com.thekua.ant.taskdefs.DeleteEmptyFilesTask" name="DeleteEmptyFiles" classpath="classes"/&gt;

    with the following code added to the target that runs your JUnitReport:

    &lt;DeleteEmptyFiles directory="${testOutputDir}"/&gt;

As always, feedback, comments and thoughts always appreciated.

5 Replies to “Optimise Your Build with Faster Running JUnit Tests”

  1. I think this is a good idea. Several people I’ve worked with have asked for this functionality for quite a while. They basically want anything that will allow them to run suites of tests quickly and still see a nice report.

    However, I like to archive nightly build reports for historical purposes. In particular I like to see the timings for each test. If you’re not saving this information to XML files for successful tests, then you’re losing the ability to track timing changes over time. But that’s a small price to pay to get the whole suite to run quicker.

  2. Hi Don,

    Thanks for your comment. I agree that collecting metrics can be useful, especially timings, but as you said I would do this with a different build target running overnight. I believe the developer build loop, which should be running many times a day should be as fast as possible.

  3. Hi Pat,

    I found a way to workaround the empty xml file problem.

    private OutputStream out;

    public void endTestSuite(JUnitTest suite) throws BuildException {
    if (errorOrFailed) {
    junitFormatter.endTestSuite(suite);
    } else {
    if (out != System.out && out != System.err) {
    FileUtils.close(out);
    File outputFile = new File(suite.getTodir(), suite.getOutfile() + “.xml”);
    outputFile.delete();
    }
    }
    }

    public void setOutput(OutputStream out) {
    this.out = out;
    junitFormatter.setOutput(out);
    }

  4. I wish I could update my entry to say Hi Pat instead of Paul. I hit the Enter key too quickly 😉
    Happy New Year !

Comments are closed.