Testing According to the Capability Maturity Model

  1. Initial - You don’t test or manually test in an ad hoc manner
  2. Repeatable - You start building test suite(s) that can be run repeatably
  3. Defined - You automate your test suites and run them in a continuous integration environment
  4. Managed - You start observing metrics from the tests such as how good the coverage is, how long it takes to run and how difficult they may be to write
  5. Optimising - You start to make changes to your code to make it easier to test, and you use the tests to highlight areas that need refactoring or a redesign. You start running Test Driven Development or Behaviour Driven Development to enhance the testability of your code.

After applying the CMM to way I execute testing, I find it interesting that the practices that emerged from the agile community are heavily biased towards the higher end of the CMM. There’s definitely a message here.

Disclaimer: This list was created by myself and does not reflect any official source associated with the Software Engineering Institute (SEI). It was written by me in terms of thinking of how I have seen testing executed and then trying to describe it in terms of the CMM.

Ask EMMA If You Are Covered

Code coverage is a useful tool for helping you work out how well you are testing your code. On most projects I have been on, I have found Clover to be the most well known and most popular but there are other alternatives, and this one in particular I am going to demonstrate how to use it.

On our current project, I used EMMA, an open source code coverage tool, to monitor how well we were testing various aspects of the code base. Results highlighted areas that were not test driven or areas that the team had neglected to deal with at the time (such as certain exception handling). This library is not specifically design to work only with JUnit, so I thought it might be useful to show you how we use it in our normal build that does use JUnit.

Steps to generate code coverage using EMMA

  1. Compile your classes
  2. Instrument your classes
  3. Execute your tests
  4. Generate the report

Compile Your Classes
With EMMA there is no change to the way that you can compile your classes as it allows you to instrument them post-compilation.

Instrument Your Classes
The distribution comes with a class that you can run against a set of classes that automatically generates the instrumented classes as shown below:

java –cp emma.jar emma instr –m overwrite –cp build/classes

where:
[emma] Is the executable java class for easily running all the features from this library.
[instr] Is the command for executing the instrumentation that weaves in all the monitoring features for generate code coverage information.
[-m overwrite] Is the output mode and states to overwrite the original classes. You can also output a copy if you want but you will have to specify the output directory using the –d/-dir/-outputdir option
[-cp build/classes] tells EMMA where it should be looking for the files to instrument.

Execute Your Tests
After instrumenting your classes with the features that monitor code coverage, you only need to run your tests as normal, only ensuring that you include the EMMA’s library jar in the classpath.

After running your tests you will see two output files including coverage.em and coverage.ec that are both used as inputs into the report.

Generate The Report
We will reuse the same executable class for generating the report as below:

java –cp emma.jar emma report –r html –in ../coverage.em,../coverage.ec -sp ../src

where:
[emma] Is the executable java class for easily running all the features from this library.
[report] tells emma that you want to generate a report from the coverage readings
[-r html] tells what format the report should be in
[–in ../coverage.em,../coverage.ec] tells emma where to find the code coverage data
[-sp ../src] links the java source files to the results so that you can get some nice graphical reports.

Your should end up with a HTML report that outputs something like the following:

Summary Code Coverage Report
Summary Report
Code Coverage Report with Source
An example output file with source attached

Ant Build Integration
You can use this build file (substituting appropriate values) if you want to get up and running quickly. It assumes:

  • You have the appropriate emma jar in the specified location (see buildfile)
  • You have the jar in the classpath of the tests
  • You have compile and test targets

Okonomiyaki @ Abeno Too

Okonomiyaki is a Japanese speciality dish best described as a savoury pancake mixed with a pizza. Traditionally you cook it in front of yourself on a hot teppanyaki grill and even though the base typically includes cabbage, flour, egg and water, you can put pretty much anything else into, or on top of it. Abeno, located closest to the British Museum, is the only place I know of in London that serves this Japanese speciality. They also have a second store aptly name Abeno Too (the latter word meaning the equivalent of ‘also’ or ‘as well’ in Japanese).

Okonomiyaki

I have eaten at both of these places and both are quite good. The clean and smooth table tops and real Japanese attendants indicate that they will serve you good quality Japanese food (appropriately Westernised for taste). Its prices are decent although typical for what you pay for Japanese in London and any of the super-deluxe okonomiyaki mixes are good value for money. Both of these places are small, so I recommend you make reservations for dinner or lunch if you intend to go on busy day or night like Friday or Saturday.

The two times that I have eaten there, the service has been good although I have heard it gets more difficult if you are eating at peak times. The staff speak to you in polite and soft tones and can explain the concept if you are new to okonomiyaki. Staff will cook the food on the grill in front of you and finish it off in the traditional manner topping it with okonomiyaki sauce, mayonnaise, seaweed flakes and bonito flakes that dance entertainingly from the heat off the hotplate. You then use mini spatulas to take pieces off the grill to your plate you can then liberally douse it in additional sauces (I highly recommend the hot sauce) for additional flavour.

Details: Abeno (7 Museum Street, Bloomsbury, WC1A 1LY) or Abeno Too (7-19 Great Newport St, WC2H 7JE)
Highlights: Okonomiyaki has strong layers of flavour and there’s nothing like eating off a hot grill.
Room for improvement: The premises are quite small and can be crowded
The Kua Rating: 7 out of 10

Second Anniversary Working at Thoughtworks

This time last year I blogged about my first year of employment with Thoughtworks. Not surprisingly, one year later now marks my second year and like all good things is a nice time to reflect on what things have happened since then. The best place to start is probably to see what I wanted to change last year and what has happened since then.

Second Anniversary Working at Thoughtworks London

(From Last Year) Single-Project Consultant
Anyone who has worked in Brisbane understands that the commercial scene is not as broad as Sydney or Melbourne, and a consequence of being Brisbane-based (or is that bound?) means that I only got to work on a single project whilst I was there. I understand the commercial reasons behind it, but I think the same reasons pair rotation is important also apply to project rotation – consultants gain invaluable skills and spread knowledge, techniques and practices continually adding to the value of the organisation and to the clients they work with. Two years on I have now been on three different clients and four different projects, and I am now about to roll onto a fourth client.

(From Last Year) Lack of Travel Opportunities
Consulting brings with it the threat of constant travel, but being based in Brisbane didn’t bring anything with this. One of the programs Thoughtworks offers to its employees is an Exchange Program that enables people to transfer to the different offices (UK, Canada, India, China, USA and Australia) for a year to try living in a different country. I opted for working in London and it has been a fantastic experience since arriving in August last year. I have even visited an English seaside town for one of my projects but I do prefer experiencing all that London has to offer for the rest of the time that I will spend here.

(From Last Year) Further Feedback
One of the greatest things about working from the London office is the sheer number of people that you have access to on a regular basis. I imagine the other Australian offices would be different to Brisbane, but it’s great that you can bounce ideas around and discuss different topics with more than just a handful of people.

In terms of more formal feedback, the UK is setting great initiatives with project roll-off reviews and annual reviews. It really isn’t too hard getting other colleagues to give you feedback informally.

What Other Great Things Happened?
I am continually stunned by myself to see how my skill set continues to expand in so many different areas because I get to talk with and work with really bright people. It is especially more apparent when I pair program with people just how many more patterns, refactoring, testing and development techniques and principles I have learned.

All of these things I am now trying to share with more people. Last year I presented at an internal conference (one of our fabulous so-called “Team Hugs”), and was accepted to Agile India to present a Ping Pong Programming Challenge (but unfortunately couldn’t make it due to certain circumstances). This year I’ve submitted proposals to both XP 2006 in Finland and the Javazone conference in Norway both for later this year. I think it’s a great opportunity to share what knowledge I’ve gathered and get more feedback from industry fellows.

I am now dabbling in several new languages, some old and some new and am also now focusing on expanding on skills outside of just development to become a more well rounded consultant. My current focus is developing skills on being an agile coach because I have found clients tend not to understand a lot of the principles and I have found a team will be more effective if the team performs a number of the basics. As you may have read in previous posts, I am now currently building up a catalogue of retrospective exercises that I have found useful and what others might find useful. My personal beliefs have also significantly changed or perhaps even strengthened. I strive consistently for building self-empowered teams, not only because I think it is more successful but I think it is more fun and frankly I would rather have fun whilst working.

I did a little bit of sales work early on that was really just showing clients the way we work and the way that we do things, something I found quite interesting. It was nice that we could polish off some of the work that we did, turn it into a demo, and give it to people who were interested in doing things a different way and having them say, “Wow.” It’s such a rare opportunity that we get to do this on our projects and be appreciated by our clients.

So the good: Meeting and working with loads of other TWers, doing a variety of development and non-development roles, great amount of personal and professional growth, and all the things to improve from last year did improve.
More to improve on: Sharing this experience with more people and developing skills in even more areas.

Disappointment at Dawaat

After getting back from the big trip from Amsterdam, I caught up with Brett who was visiting from Australia for an internal work conference. We decided to head out to Brick Lane for a curry, just a little bit of that insight into the famous “curry lane” of London. We ended up at Dawaat because it was the first we stumbled across that looked seemingly alright, but turned out to be extremely disappointing.

According to its website, six brothers run the restaurant and I’m sure we probably met most of them that night. The curries are cheap and with a 20% discount offered, so I thought it was going to be a bit of a bargain night but ended up a very unsatisfying night.

It first started with poor quality poppadums served with even lower quality chutneys and sauces. Our waiter continually pushed us to order more food than we really wanted to eat throughout the night, saying we should get starters, side dishes and dessert and kept at it even when we made it clear we only wanted exactly what we ordered. Admittedly the waiter was correct about ordering two rices as well as a naan since the serves were so small and way overpriced at £2 each.

When the curries arrived they looked pretty dismal and tasted even worse. They had served my Rogon chicken dish with about two times the amount of onion than chicken, and the Korma chicken Brett had ordered was quite watery with no kick to it. They both had insubstantial amounts of meat and its sad to say but the naan bread was probably the best thing we had and it’s generally hard to get that wrong.

The evening continued to turn even more sour when we asked for the bill (several times), taking about half and hour to arrive. The waiters had even gone as far to miss the 20% discount that one brother had first offered, with a quick correction when pointed out but a noticeably lacking apology. I finally paid by credit card and it was even more difficult with the waiter standing over trying to get me to put more service.

You may think that this was a bad experience for us, but from what I observed that night, there was definitely a consistent theme there. One waiter continually insisted a particular table to leave as quickly as possible, apparently because there was a booking, and they were understandably angry because they hadn’t been told about it when they first arrived. It had escalated to a point where I heard the diners vehemently say “we will *LEAVE* when your *BOOKING* finally arrives”. One final example I saw was when another couple were shifted half-way through their meal so that the waiters could more easily accommodate a larger group of five.

Details: Dawaat
Found On: 60 - 62 Bricklane, London E1 6RF
Contactable On: 0207 375 3095
Highlights: Naan bread was alright
Room for improvement: Avoid the experience at all costs - it’s not worth your time here when there are plenty of other options on the same street. There’s plenty of place for improvement here, but frankly it’s not
The Kua Rating: 1 out of 10

A Weekend In Amsterdam

The GangI spent this last weekend with a number of people in Amsterdam to celebrate Ben’s stag weekend before he gets married in May later this year. This event had been building up for some time since we booked our flights and hotel back in January. We left on Friday morning to arrive there at just around lunchtime where the eight of us, including Ben, Karl, Nathan, Andrew, David, Matt and Damian met at the hotel.

We flew BMI, which has been one of the least painful flights I’ve flown, and it was even easier getting from the airport to the city one we arrived. A 20 minute train ride costing only €3.6 conveniently drops you right in the middle of Amsterdam Central. After navigating the treacherously narrow and winding staircase, we checked into our hotel located extremely close to the centre of town, The Orange Tulip Hotel, before we proceeded to venture forth into the city.

Ben And PatThe cityscape of Amsterdam is strew with canals and bridges crossing canals with narrow multi-storied buildings, a consequence, from what I understand, of a historical tax based on the amount of house frontage. Each building is decorated with awnings or stone and granite figures mounted at various points in typical European fashion. Many of the buildings are surprisingly but noticeably crooked, either overhanging the street from above, or slanting distinctly to the left or right into another building though I don’t really know why.

By The CanalsThe streets are wide as trams ferry large numbers of people through the city day and night. The roads also accommodate cars and pretty much most of the roads have a dedicated bike lane for the massive number of cyclists in the city. Cycling is an obviously favourable way of getting around if you’re a local, the evidence easily witnessed by the sheer number of bikes that litter the bridges, posts and sidewalks of town. Many of them appear antiquated, most of them have large handlebars, no gears and many of them appear to be rusting, but all, of course, equipped with that essential bell used to warn pedestrians that you are in their way (and not the other way around!)

Our weekend kicked off initially with a visit to the Heineken Brewery where they have a tour that is pretty much self-running. For any sort of brewery tour, they do have some more exciting things such as the Thirst DJ room (where you can mix it up with both lights, sound and video), a table-soccer table, and a number of ‘rides’ best left enjoyed to those easily entertained. It’s probably not worth the effort other than the three beers and a souvenir glass to take home.

Bike ChaosWe had dinner at an Indian restaurant and hit a number of bars/pubs to celebrate the occasion with Ben. We also walked around town including parts of the Red Light district, one of those places that you hear so much about and can’t really avoid if you visit Amsterdam. It’s not really worth describing in great detail here but let’s just say it was quite shocking to see in real life.

The next day we tried to go for a bike ride tour around the city but both bad weather and bad timing saw us miss our only opportunity. Instead we decided to embark on a ride in a glass covered canal boat around the city that proved both cheap and lengthy though unfortunately not as entertaining as originally thought. I definitely recommend taking a more personal tour if possible as the commentary on ours I found disappointing.

Overall it was a pretty fascinating city in how liberal it is, but sometimes not quite as nice with the prolific number of rowdy crowds from both the US and the UK disturbing the atmosphere. I think everyone had a great weekend anyway and that is really all that mattered in the end.

A Retrospective Timeline

The retrospective timeline is a useful exercise for gaining a better understanding and a richer context for a particular retrospective. This means that this exercise becomes less useful when you do retrospectives more frequently and especially useful when doing a retrospective spanning longer periods or you run retrospectives less frequently.

Goals
The timeline aims to give a very visual overview of the events that occurred during the period for which you are doing the retrospective. The facilitator should use distinct and bold colours to identify distinct events so that it is easier to interpret the results. The timeline is a great way of determining trends over time for periods in the project and a useful mechanism for getting a feel for how the team felt throughout the course of the project.

When To Use It
As mentioned earlier, the timeline retrospective is useful for refreshing the memories of participants and you might find it useful in the following circumstances:

  • For projects that are run without any intermediate retrospectives
  • For projects that require more formalised processes (i.e. not necessarily uber-agile)
  • If you need to communicate a more complete perspective for outside parties that may require reporting to
  • If all participants did not necessarily work on the project during the entirety of the project (people rolling on or rolling off)
  • The end of a project retrospective

The Fundamentals (Part One)
There are two major parts to this exercise. The first is noting all the events. Make sure that everyone has access to different coloured pieces of paper where:

  • Green represents good events
  • Yellow represents significant or memorable events
  • Orange represents problematic events
Retrospective Timeline Colour Chart

The types of events that you might want to consider suggesting include things like when people rolled on or off projects, when things kicked off, when someone in the business made a decision that affected the team in a great way, when the team did achieved something really amazing, etc.

Get everyone to scribble each event down on separate pieces of paper and then get them to tack it along a standard timeline along a whiteboard or several sheets of paper stuck to the wall. This is where sticky notes or lots of blu-tack helps. It should look something like the following:

Events along a timeline

It is a good idea for you to get everyone to explain events, especially if the participants may not have full knowledge of what was going on. The significant events help to put things into context and might answer a few of the puzzling questions some people might have about the project, and the good points help to identify strengths while the problematic points should lead you to discussing action items. I find this exercise is most useful for first putting context for a retrospective spanning a large time and then using other exercises to draw out further conclusions. It is a good mechanism just to get the discussions flowing within the team and allows the facilitator to try to put different perspectives on things.

The Fundamentals (Part Two)
The reason you want to reserve some space underneath the timeline is for the second part of this exercise. In this part, you split the remaining area into two sections where you get the team to fill in how they felt about each event or how they felt in general for that period. You can try to limit them by getting them to put three dots per significant period (week/month) or perhaps one per event posted, with the idea that you want to get a general feeling for the overall morale of the team. You should end up with a scattergram that looks like the following:

Team morale along a timeline

Once the team has finished plotting all their points, you can then try to draw a trend through them that helps to establish a real idea of how the project went such as below:

Trends during the timeline

Trawling The Timeline
The visual nature of the timeline allows you to easily draw out a number of trends from the project. Large concentrations of a particular colour may attract your attention to a certain period and it would be worthwhile trying to pinpoint its underlying cause so you can attempt to replicate (or avoid) the same circumstances in the future. The general trend at the bottom may or may not coincide with the events above the line so there might be another discussion point for explaining why morale might be low even though there might be lots of green or why it is high even though there were problematic events (maybe it was a good team building experience or people were growing and learning).

Agile Mythbusting

Yesterday I ran a session for another group at my current client introducing them to agile. The first part of this session helped me get a better understanding of what they thought of, or had heard about, or applied about agile and during this I noticed a few comments they touched on (and I hear numerous times from other people) that I thought might be worth sharing. I have found that when people observe and even when people practice some of the agile practices, it can be easy for them to incorrectly draw the wrong conclusions. Here’s a few of them and my thoughts on each:

Being agile means we don’t have to write documentation
I have never heard any agile coach nor practitioner ever say this, nor encourage this form of thinking. The agile manifesto states a value is “Working Software over Comprehensive Documentation”, with the thinking that people would rather some delivered instead of being given a document telling them what they could have had instead. You will find people trying to avoid writing comments in code because most of the stuff people write is redundant (it tells someone what the code does, when they can read the code right there and then – if it can’t be read, then maybe you have a bigger problem at hand). There are situations where comments are invaluable (libraries APIs given to external parties, or even commenting why something may not be obvious to a reader because of some bug/library/performance reason). I have found that sometimes having general diagrams with some descriptive phrases and a brief narrative is much more useful than any 300 page document might give. Consider that an hour spent in a conversation about what someone wants can be more productive than an hour spent writing and then another hour spent reading a document hastily put together.

Being agile means we don’t do design
Avoiding Big Design Up Front (BDUF) is a popular term but so easily abused. Not practicing BDUF is not about not doing *any* design and is even worse for a justification of a *poor* design and is a very big judgement thing. We have found that some good discussion about an approach helps (whiteboarding might be involved) can be useful, but in reality the implementation will be greatly different from the original design. It is sometimes better to just do things, and refactor to patterns or a better design (otherwise referred to emergent design) because you cannot anticipate everything. UML and class diagrams may help for *communicating* ideas, but not a great idea for trying to generate the code from it as the code tends to move faster than what the diagrams can keep up with.

Agile doesn’t work with old projects
There is no reason why inheriting someone else’s code automatically implies you cannot apply agile principles. You might find that some of the practices might be more difficult to implement such as talking to the people that originally wrote the code, or find the system difficult to test, but it shouldn’t stop you from trying. It does not mean that when you get new requirements, you cannot involve the customer more, it does not mean you cannot write your tests first and it certainly does not restrict you to release changes early and get more feedback. If you find there are problems, the old project is unlikely the cause and is more likely other processes such as the release procedures or the way your operations team works. Blaming agile or the project is not the answer.

Agile is no different from other processes, we do testing in RUP (etc)
I’ve never heard any agile practitioner say that testing is not executed in other processes. Testing in agile is different because there are different levels of testing and there is an expectation for the testing to start as early as possible. By focusing on testable, releasable and value-added units, more things can be achieved by the team such as getting better feedback, and getting more value earlier rather than in one big bang at the end (that may or may not happen).

Next Page »