The intersection of technology and leadership

Category: Testing (Page 2 of 3)

Beware the Spring Globaltons In Integration Tests

As I mentioned in a previous post, we’ve ben trying to fix a whole slew of tests that flicker, and we’ve spent some time fixing a number of integration tests that leave side effects.

This is the first time I’ve encountered the testing pattern (not recommended!) where SpringJUnit4ClassRunner loads up a spring context, a bean is pulled out, and then mocks used to stub out services. It’s a really evil pattern.

For one thing, mocks are about interaction based testing, not really about stubbing. I typically use them to drive out class-based roles. However, in this case, they were used to take out a portion of the application.

Using the above JUnit Runner means that there is one spring context per test run, effectively, a global pool of objects. Of course, when you pull one out and start modifying the objects, it means you have plenty of side effects across other tests. Unfortunately the don’t always manifest themselves in obvious manners.

Our solution to fix this was to use reflection to look at all beans in the spring context, and fail the test if it found any that had any mocked instances. I’d still recommend you avoid the testing pattern altogether, but if you are forced down this route, you now have a solution to detect side-effects across tests.

Testing Pattern: Exploratory API via the Web

Last year I was working with Tiest, a former colleague of mine at a bank working on a system with an internal API. It was quite an experience, though I will save that for another (set of?) blog posts for another time. The application was centrally hosted with a client API that allowed people to interact with the server side. The testers needed something to help them do more exploratory testing and my former colleague came up with a great solution to this problem that I thought would be worth recording here for posterity.

Our solution had these characteristics:

  • Provided through a web browser – Installing items onto the testers machines became a bit laborious, namely because they didn’t have rights to install, change or run anything with administrator privileges. It was easier to deploy the tool centrally, and then let the testers prod from afar. This was as simple as writing a page that submitted some code to a web server, that than executed it and returned the results back to the client.
  • Supported dynamic execution – We could have written a web layer on top of the client API but we didn’t really want to add another layer for maintenance. We simply used the javax.script package to turn our standard POJOs into a scriptable/dynamically invokable system.
  • Built in reflection functions – Using the scripting interface meant that the server would not store any state. So the server had a number of built in functions such as a listAll(object) that would show all available property/methods to the user
  • Example queries stored on a wiki – When a tester would ask for something to be checked against the test environment, I would convert it into an appropriate query they could paste into the query browser window. They could then execute it, tweaking parameters, etc and learn to compose things. They didn’t have to really understand how java (as script) worked, just build on what was there.

Although I won’t be reaching into my testing patterns toolkit for this pattern too often, I feel it worked well to enable our non-technical testers to get at something given a number of environmental constraints.

XP2010 Review

This year’s XP2010 conference was held in the great Northern parts of Norway, up in Trondheim, only about 200km south from my first XP conference (XP2006) I attended several years ago. I always find this series of conferences interesting, partly because there is always the blend between the academic (research papers and experience reports) and the industrial side as well as the infusion of the hosting country’s culture.

This year, the conference saw 425 attendees (50% of them Norwegian) across 8 different tracks and covering four days of a wide variety of conference mixtures including Lightning Talks, Panels, Workshops, Tutorials, and Open Space events.

ScottPage

The first of the two keynotes, run by Scott Page was my favourite – a lecturer and researched on complexity in different environments. He talked about the importance of diversity and its impact on building better solutions as well as a provable formula calculating what the wisdom of the crowds was. Hopefully they’ll be putting up his slides somewhere as a result.

The second of the keynotes, ran by David Anderson did a great job of condensing down a long talk into a thought-provoking discussion on the work that he’s been doing on building a Kanban movement in the places he’s been working as well as a discussion of his recently released book about Kanban. He had some nice visualisations on the slides and even finished slightly earlier with plenty of times for questions.

Overall the conference seemed to have a nice blend of both process and technical topics, although there were a few complaints about specific discussions around “XP” itself although I think plenty of discussions about specific practices quite useful within any agile context.

I ended up much busier than I thought I would be, playing host for two of the session talks, helping out with a Pecha Kucha (more on that later) session and running a workshop on my own.

Entertainment

Venue, organisation and efficiency wise, the organisers need to be congratulated on a job that will be hard to surpass. Everything seemed to run smoothly including an amazing and difficult-to-forget Conference Banquet involving interesting local foods (smoked reindeer heart, moose and local seafood), a pair of comedic jazz academics, Keep of Kalessin playing some Extreme Metal, all inside Trondheim’s amazing Student Society venue.

ExtremeMetal

The strangest part of the conference for me was participating in a “What’s in my agile suitcase?” Pecha Kucha run by Martin Heider and Bernd Schiffer. You can read more about the format here and it was great to be one of the five other prepared Pecha Kucha’s including Rachel Davies, Mary Poppendieck, Joshua Kierevsky and Jeff Patton. I found the diversity of styles and approaches fascinating as well as the specific things that people “packed” in their suitcases. Being the first time format all the speakers found it a fascinating format made thrilling by the short-time and the fact you don’t have control over the timing of the slides. If I were to do this differently, I’d definitely try to focus on a smaller range of topics (or just one).

My week ended (as did the conference) with my final workshop called “Building the Testing Pipeline.” I’d specifically targeted people who’d been working with automated tests for some time and I ended up with a surprisingly full room. I’d run this previously at ACCU with a slightly different audience. We had some great brainstorming sessions (I’ll be putting that up soon) and hopefully more people came away thinking more consciously about the balance of tests they have and whether or not they have the correct balance that maximises their confidence, and minimises their feedback cycle.

I’m also very proud that the experience report that I shepherded won best paper (experience report), and I finally got to meet the author, Jørn Ola Birkeland of Bekk Consulting, in person to congratulate him on the day.

Thanks to all the organisers, participants, and passionate people that made the conference so much fun and success. Wonderful to reconnect with old friends and make many new ones.

Build the testing pipeline at ACCU 2010

Just a short note that I’ll be running a workshop for attendees of ACCU2010 this Saturday on understanding how to get the balance right for the testing aspect to build pipelines. We’ll explore the tradeoffs and conscious decisions we should be aware of when putting these into our feedback loops.

Slide deck to come, though it’s a classic me-style, participatory workshop (you learn more by doing!) so slides won’t necessarily make a whole lot of sense by themselves.

Run automated tests as often as possible

I’ve been working with a large scale legacy system (in the Michael Feathers’ sense). Strangely enough there are a small handful of tests, unfortunately most of them falling by the way side having not been added into an Continuous Integration process. Many of them no longer look relevant, others, unable to run without a specific person’s machine (due to configuration files or special databases). Others look like a mechanism for people to simply diagnose issues with plenty of tracing statements. All of this was an inevitable system without someone running all the tests all the time.

Any serious team serious about continuous integration will think hard about their development testing strategy, considering approaches that allow you to run tests all the time. The strategy needs to consider a good way of preventing side effects from other tests (global state, global configuration, or left over state in external resources) balancing out speed with good quality feedback. Staggering build processes into smaller chunks, such as using build pipelines are a great way of achieving this.

Please don’t let leave your tests to die a sad and lonely death. It will take discipline and effort to keep them running yet the results will often pay back for themselves very quickly on any system with a reasonable lifespan.

Working in Scenarios

I’ve been busy retrofitting tests to a legacy system. We think our current best bang-for-buck are integration tests, at a service layer since there is plenty of state, lots of valueobjects and most of the interesting business behaviour cross cuts several services. We’ve been trying to build up a suite of test assitants in the form of Builder/Factories yet most unit tests don’t offer much value because the way the services interact with each other (a code smell yes, but will remain for a while given the sheer amount of it).

Instead, we’ve been wrapping service methods in a more convenient wrapper class to take the noise out of the extremely chatty APIs, wrapping away logic in something that we will call xxxServiceScenario. Just as Mark has been playing around with GetBuilderFor on his projects, we’ve hung out all of our specific service wrappers off a class called Scenario.

We then pass builders into scenarios to create the different test state that we need before calling a particular service to test its behaviour. The benefits of this approach include that it’s a bit clearer to users what we’re testing (anything associated with a Scenario is part of a GIVEN block), and we can choose to change the implementation (we’ve moved from using SQL to delegating to a service once we had more confidence in those services working properly) and our test methods are shorter without resorting to setup methods.

An example test might look like:

[Test]
public void ShouldDoSomethingInterestingGivenAParticularScenario() 
{
   User firstUser = Scenario.Users().CreateMeARealUser(new UserBuilder().Build());
   User secondUser = Scenario.Users().CreateMeARealUser(new UserBuilder().Build());
   Scenario.Settings().EstablishDefaults(firstUser);

   Event onLineEvent = EventBuilder.For(firstUser).ViewedFrom(secondUser).Build()
   new InterestingService().ReactToInterestingEvent(onLineEvent);
   
   // some assertions here.
   ...
}

Some developers seem to be liking this concept, so thought I’d share it to learn what we’ve been doing.

Improving Collaboration Between Developers and Testers

One of the biggest divides you need to cross, even on agile teams, is the chasm between testers and developers. By the nature of their different roles there will always be tension with developers focusing on creation, change, and construction, and testers focusing on breaking, destructive, and exposing questionable system behaviours. It’s essential that both organisations and, in particular, individuals realise what they can do to ensure this tension doesn’t dissolve into an exclusively confrontational relationship.

catdogfight

Photo taken from Sephiroty’s Flickr stream under the Creative Commons licence

Recently, a new QA person highlighted this very issue for me. I’d finished some functionality with my pair and the QA had been testing it after we declared it ready to test. They came along to my desk and said, “I’ve found some defects with your code.” Internally I winced as I noticed myself wanting to say, “There’s nothing wrong with my code.” It got me thinking about why.

Evaluating people puts them on the defensive
Just as giving effective feedback should focus on behaviour, poor feedback makes a person feel criticised. When the QA said that there are some “defects”, it implied a broken state that you had to fix it. Made worse is the way that they said it made it feel like they were blaming you, and it’s very hard not to feel defensive in a situation like this. A very natural outcome is to pretty much deny the “evaluation” which I’m sure anyone in our industry would have witnessed at least once.

Avoid terms like “defect”, “broken”, “bugs”
One of the biggest differences working with agile testers versus testers who come from traditional backgrounds is the terms that they use. Traditional testers constantly use the words above. Agile testers focus on discussing the behaviour of the system and what expected behaviour they would see. They only use the words above once they have agreement on both of the both the current and expected behaviours. I definitely recommend you do not start a conversation with the words above as they all carry some connotation of “evaluation” and I’m yet to meet someone who truly feels comfortable being “evaluated”

Focus on Effective Feedback
Effective feedback follows a neat and simple pattern:

  1. What behaviour did you see?
  2. What impact did it have?
  3. What do we want to change?

Testers should use a similar series of questions (in order):

  1. What behaviour did you see?
  2. What behaviours did you expect to see?
  3. What are the consequences of the current system behaviour?
  4. Is that desired or undesired?
  5. Do we need to change it?

Apply the guideline above and watch the collaboration improve!

Controlling Time: How to deal with periodic gaps

Some application logging is important enough to put regressions tests in place. For instance, when you shell out to other programs and you want to monitor the amount of time it takes. In this entry I’m going to demonstrate how you might test that you are correctly logging out the time taken to do some long running work. I’m going to assume that you already have a Clock concept in the system.

Let’s assume we’re trying to test the following block of code:

public void doSomeWork(Clock clock) {
  DateTime startTime = clock.now();
  executeSomeLongRunningWork();
  DateTime endTime = clock.now();
  long timeTaken = endTime.minus(startTime.getMillis()).getMillis();
  LOG.log(Level.INFO, "Time taken was %sms", timeTaken);
}

One approach would be refactor the above code, replacing the Clock with a StopWatch but I’m going to show a technique for testing this code without changing it. Instead, we are going to inject an AutomaticallyForwardingClock. Here’s the implementation:

public class AutomaticallyMovingForwardClock implements Clock {
  private final int periodToMoveForwardsInMs;
  private DateTime now;

  public AutomaticallyMovingForwardClock(DateTime initialStartTime,
      int periodToMoveForwardsInMs) {
    this.now = initialStartTime;
    this.periodToMoveForwardsInMs = periodToMoveForwardsInMs;
  }

  public DateTime now() {
    DateTime result = now;
    now = now.plusMillis(periodToMoveForwardsInMs);
    return result;
  }
}

So whenever anyone asks the current time from the Clock, it automatically shifts time forward by the specified amount. Now we can write a test that looks like this:

@Test
public void shouldLogTimeTakenForProgramToExecute() throws Exception {
  int periodToMoveForwards = 5;
  Clock clock = new AutomaticallyMovingForwardClock(new DateTime(), periodToMoveForwards);
  doSomeWork(clock);
        
  long expectedDuration = periodToMoveForwards;       
  assertThat(fetchLogEntry(), containsString("Time taken " + expectedDuration + "ms"));   
}

A few notes from above:

  • The test uses the Joda DateTime class.
  • The test uses the Hamcrest API
  • The test assumes you know how to retrieve the log message from a logger

Although I think that injecting a StopWatch is probably a better mechanism, I hope that this technique still proves useful for some people.

Controlling Time: Threaded Execution

Update (Feb 21): Thanks to some feedback from Taras and Lowell, the ThreadedRunner is the same as the java.util.concurrent.Executor class! I’ve updated the post to reflect this below. Thanks for the suggestion!

Unit testing threaded software can sometimes prove slightly cumbersome. Many of the examples on the net often show examples that tie a number of responsibilities together. It might be easy to see something like this:

  public class SomeExample {    
    public void doSomeWork() {
      Runnable runnable = new Runnable() {
        public void run() {
          // do some long running work
        }        
      };
      new Thread(runnable).start();
    }
  }

The thing about this class above is to realise what responsibilities it is mixing, including the new work they would like to (the item in Runnable), as well as how they would like to run it (the separate thread). It’s difficult to tease these two responsibilities apart the way it’s declared. The first thing I’d do is remove the dependency on the new thread. I’ll start by making use of the java.util.concurrent.Executor :

public interface Executor {
  void execute(Runnable runnable);
}
  
public class SingleThreadedExecutor implements Executor {
  public void execute(Runnable runnable) {
    runnable.run();
  }
}
public class NewThreadedExecutor implements Executor {
  public void execute(Runnable runnable) {
    new Thread(runnable).start();    	
  }
}

We’ll inject the SingleThreadedExecutor in our unit tests, so we only ever have one thread of execution, and we can rely on tests being consistently reproducible. We’ll inject the NewThreadedExecutor when we need to assemble our application for production. Here’s the step by step refactoring.

Wrap inside our new implementation

public class SomeExample {    
  public void doSomeWork() {
    Runnable runnable = new Runnable() {
      public void run() {
        // do some long running work
      }
    };
    new NewThreadedExecutor().start(runnable);
  }
}

Inject instance via the constructor

public class SomeExample {
  private final NewThreadedExecutor executor;
  public SomeExample(NewThreadedExecutor executor) {
    this.executor = executor;
  }  
  
  public void doSomeWork() {
    Runnable runnable = new Runnable() {
      public void run() {
        // do some long running work
      }
    };
    executor.start(runnable);
  }
}

Replace specific with interface

public class SomeExample {
  private final Executor executor;
  public SomeExample(Executor executor) {
    this.executor = executor;
  }  
  
  public void doSomeWork() {
    Runnable runnable = new Runnable() {
      public void run() {
        // do some long running work
      }
    };
    executor.start(runnable);
  }
}

In the tests, you can now inject an instance of SingleThreadedExecutor, whilst in production mode, you use the NewThreadedExecutor.

Disclaimers
Of course, this still leaves your testing strategy incomplete where you do want to test your software with the number of threads that you do expect, particularly when you need to share state across different processes.

Controlling Time: How to deal with infinity

It’s been a while since I’ve had to slice and dice legacy code. It reminds me how easily, non test driven code, slips into the abyss of tight coupling and the effort of retrofitting (unit) tests increasing in effort with time. Of course, I don’t believe TDD should be done all the time, but for most production code I do. In this entry, I’m going to use the Sprout Inner Class mechanism to demonstrate how to start making something that would have been untestable much more manageable.

The scenario
What we have below is the entry point into our program (i.e. a class with a main method). When I first encountered this class, it had too many responsibilities including, and not limited to, parsing command line options, some application logic, handling terminal signals, and wiring up all the objects for the program to start. Combined with an infinite working loop, you can imagine what a nightmare it was to unit test. Here’s the modified class, called RunningProgram representing the important parts of the class related to this walk through.

Our task: To reduce the number of responsibilities and test the body of the start method (we couldn’t simply extract method immediately due to the number of fields it modified)

package com.thekua.examples;

import sun.misc.Signal;
import sun.misc.SignalHandler;

public class RunningProgram implements SignalHandler {
	private boolean running = true;
	// some other fields
	
	public void start() {
		running = true;
		while (running) {
			// do some work (that we want to unit test)
			// it changes about ten fields depending on what condition
			// gets executed 
		}
		// Finished doing some work
	}
	
	// it also had plenty of other methods
	
	public static void main(String [] args) {
		RunningProgram program = new RunningProgram();
		Signal.handle(new Signal("TERM"), program);
		Signal.handle(new Signal("INT"), program);
		program.start();
	}

	public void handle(Signal signal) {
	    // Program terminated by a signal
		running = false;		
	}
}

Step 1: Understand the dependencies that make this test difficult to test
To get some level of functional testing around this program we had a number of timed triggers, watching for state modifications with a given timeout. Our problem was infinity itself, or rather, not sure when to interrupt infinity to watch for when the body of work inside the loop had finished its work once, twice, and not too early. We could have made it more complicated by introducing lifecycle listeners yet we hesitated at that option because we thought it would complicate the code too much.

We noticed the use of the running flag. We noticed it was the condition for whether or not we continued looping, and was also the trigger for a graceful shutdown using the sun.misc.Signal class. Notice that the running program implements the SignalHandler interface as a result. We thought that running behaviour of the RunningProgram could be extracted into a separate aspect.

Step 2: Encapsulate, encapsulate, encapsulate
Our first task, was to remove direct access to the running flag since the class modified it in two places. Sprout an inner class, and simply delegate to getters and setters and we might find we have a class that looks like:

package com.thekua.examples;

import sun.misc.Signal;
import sun.misc.SignalHandler;

public class RunningProgram implements SignalHandler {
    public static class RunningCondition {
	    private boolean running = true;
		
		boolean shouldContinue() {
		    return running;
		}
		
		public void stop() {
		    running = false;
		}
    }
	
	private RunningCondition condition = new RunningCondition();
	
	// some other fields
	
	public void start() {
		while (condition.shouldContinue()) {
			// do some work (that we want to unit test)
			// it changes about ten fields depending on what condition
			// gets executed 
		}
		// Finished doing some work
	}
	
	// it also had plenty of other methods
	
	public static void main(String [] args) {
		RunningProgram program = new RunningProgram();
		Signal.handle(new Signal("TERM"), program);
		Signal.handle(new Signal("INT"), program);
		program.start();
	}

	public void handle(Signal signal) {
	    // Program terminated by a signal
		condition.stop();
	}
}

Moving the running flag to a separate class gives us a number of benefits. It lets us hide the implementation of how we handle running, and puts us down the road of clearly teasing apart the overloaded responsibilities.

Step 3: Consolidate related behaviour
It bothered me that the main program had the shutdown hook. That behaviour definitely felt strongly related to the RunningCondition. I felt it was a good thing to move it to the that class. We now have something that looks like:

package com.thekua.examples;

import sun.misc.Signal;
import sun.misc.SignalHandler;

public class RunningProgram {
    public static class RunningCondition implements SignalHandler {
	    private boolean running = true;
		
		boolean shouldContinue() {
		    return running;
		}
		
		public void handle(Signal signal) {
	        // Program terminated by a signal
			running = false;
	    }
    }
	
	private RunningCondition condition = new RunningCondition();
	
	// some other fields
	
	public void start() {
		while (condition.shouldContinue()) {
			// do some work (that we want to unit test)
			// it changes about ten fields depending on what condition
			// gets executed 
		}
		// Finished doing some work
	}
	
	// it also had plenty of other methods
	
	public static void main(String [] args) {
		RunningProgram program = new RunningProgram();
		Signal.handle(new Signal("TERM"), program.condition);
		Signal.handle(new Signal("INT"), program.condition);
		program.start();
	}

}

Note that it is now the RunningCondition that now implements the SignalHandler interface (that we are using to register with the Signal

Step 3: Remove dependency chain
The difficulty with this class still exists. We cannot modify the RunningCondition of this program since it creates one for itself. Since I prefer Constructor Based Dependency Injection, I’m going to apply Introduce Parameter to Constructor, moving the field declaration to the constructor itself. Here it is what the class looks like now:

package com.thekua.examples;

import sun.misc.Signal;
import sun.misc.SignalHandler;

public class RunningProgram {
    public static class RunningCondition implements SignalHandler {
	    private boolean running = true;
		
		boolean shouldContinue() {
		    return running;
		}
	    
		public void handle(Signal signal) {
	        // Program terminated by a signal
			running = false;
	    }
    }
	
	private final RunningCondition condition;
	
	public RunningProgram(RunningCondition condition) {
	    this.condition = condition;
	}
	
	// some other fields
	
	public void start() {
		while (condition.shouldContinue()) {
			// do some work (that we want to unit test)
			// it changes about ten fields depending on what condition
			// gets executed 
		}
		// Finished doing some work
	}
	
	// it also had plenty of other methods
	
	public static void main(String [] args) {
		RunningProgram program = new RunningProgram(new RunningCondition());
		Signal.handle(new Signal("TERM"), program.condition);
		Signal.handle(new Signal("INT"), program.condition);
		program.start();
	}

}

Note that we are still bound to the implementation of the specific RunningCondition, so it’s time to apply Extract Interface, and understand what role that RunningCondition has. We chose the name, RunStrategy for the role name, and to help keep the names more aligned, we ended up renaming RunningCondition to RunLikeADaemonStrategy. Our code now looks like this:

package com.thekua.examples;

import sun.misc.Signal;
import sun.misc.SignalHandler;

public class RunningProgram {
    public static interface RunStrategy {
	    boolean shouldContinue();
	}
	
    public static class RunLikeADaemonStrategy implements SignalHandler, RunStrategy {
	    private boolean running = true;
		
		public boolean shouldContinue() {
		    return running;
		}
		
		public void handle(Signal signal) {
	        // Program terminated by a signal
		    running = false;
	    }
    }
	
	private final RunStrategy runStrategy;
	
	public RunningProgram(RunStrategy runStrategy) {
	    this.runStrategy = runStrategy;
	}
	
	// some other fields
	
	public void start() {
		while (runStrategy.shouldContinue()) {
			// do some work (that we want to unit test)
			// it changes about ten fields depending on what condition
			// gets executed 
		}
		// Finished doing some work
	}
	
	// it also had plenty of other methods
	
	public static void main(String [] args) {
		RunLikeADaemonStrategy strategy = new RunLikeADaemonStrategy();
		RunningProgram program = new RunningProgram(strategy);
		Signal.handle(new Signal("TERM"), strategy);
		Signal.handle(new Signal("INT"), strategy);
		program.start();
	}
}

The best thing is that our RunningProgram no longer needs to know what happens if a terminate or interrupt signal is sent to the program. With simply a dependency on the RunStrategy we can know inject a fixed run strategy for tests that we ended up calling a RunNumberOfTimesStrategy. We also promoted the specific RunLikeADaemonStrategy to a full class (not an inner class).

Thanks for getting this far! Please leave a comment if you thought this was useful.

« Older posts Newer posts »

© 2024 patkua@work

Theme by Anders NorenUp ↑