It always amazes me at how many people misinterpret FIT (Framework for Integrated Testing) and I guess it’s based on the way that it’s been marketed. I’m not exactly a big fan of FIT and the biggest reason is that people don’t really understand what it’s best for, and use it all over the place where it’s not very appropriate more so than people who do know what they’re doing with it. In the last month, I’d heard statements such as “FIT is good for driving tests below the UI layer”, “FIT is great for acceptance tests”, or “We use FIT all the time”. Please stop this!
Let’s have a look at this table I put together that compares FIT and JUnit. Of course, this comparison also applies when comparing FIT to NUnit or one of its other close brethren (TestNG, etc):
Let’s get this straight: The only thing FIT is good for, is using it as a tool to help communicate with a customer. That’s it. It’s terrible for writing automated test suites, FIT encourages terrible programming habits, IDEs offer very minimal tool support and many developers don’t seem to really understand, nor want to understand how it works. I’ve seen a test suite where developers deployed FIT tests to a web container because “it was HTML” and they thought it had to be deployed for it to run. Grrrr.
The biggest barrier for FIT is that it requires a lot of education around how developers best use it, how they use it to help aid communication, and then understanding how to maintain it. Unfortunately I’ve never seen the payback you get from investing this time that’s better spent directly trying to improve the communication gap. I think it’s because the conventions used to bridge HTML to code aren’t natural and, without spending the effort to really help developers understand it, quickly leading to a whole mess of tightly coupled code and eventually, test coma.
The recommendation
I’d use FIT sporadically to help bridge gaps between the team (analysts, developers, testers) and the customer – especially for data driven examples. Acceptance level tests in JUnit should be refactored to a point where it’s understandable to customers (i.e. using as much of their language as possible). If you decide to go down the FIT route, refactor mercilessly to split out what the FIT fixture do and how they do that (some sort of driver, component, etc). In my experience, the latter should be immediately usable in another test framework, and in fact, should be what you’re striving to reuse in that other test framework.
Our team has had unfortunate flirtations with FIT, and have definitely come across the majority of your issues with it. FIT-natics will claim that the reason that most developers don’t understand it and hence misuse it – but that’s exactly the problem!
We migrated all our Customer-viewed acceptance tests to Concordion (see concordion.org) and to be honest it’s working very well – at least in terms of developer frustration! Whilst it doesn’t solve the hardcoded method-name problem, it’s definitely more junit-like and driven by the Test class instead of the HTML, hence avoiding all the custom FitRunner-editing issues without the need for an IDE plugin.
Just my 5 cents. 🙂
Hi Pat,
Excellent article! I completely agree with you. I used FIT and Fitnesse back in the days I was at TW and it was pretty painful, especially to debug problems: it was difficult to diagnose if the problem was in the application under test or our fixtures. I think that sometimes it may be necessary to create regular unit tests for those fixtures, something like “testing the tester” 🙂 . Although having these tests (for fixtures) may minimize the pain, the downside is more code to write and maintain.
Cheers,
-Alex
Nice Post,
but i think there may be some misconceptions about your comparisons (not claiming that i’m a FIT enthusiast):
– Input:
You compare the Test class on one hand (JUnit) and the representation of the test data on the other (FIT).
Even your Unit Tests may be driven by ‘data’ (that’s what you assert), though it’s represented directly within your test method (mixed with your test logic).
On the other side, the test logic within FIT lies within your fixture class, separated by the test data.
(in fact, you could abuse FIT to write unit tests and abuse JUnit to write FIT like tests – you may want to take a look at
http://gleichmann.wordpress.com/2007/11/21/jfit-a-little-brother-for-fit/
for a further elaboration on that topic)
– Output
The representation of your output (and of course the input) of a FIT run depends completely on the runner you’ll apply. Admitted, the best known Runner is fit.FileRunner, but it’s possible to use other implementations that will act as a Runner. So under this point of view, the Output of a FIT run is pluggable, too.
– Easy maintainable
I’m not quite sure what’s your point about that. Could you elaborate on this. Regarding FIT, its inner structure provides some hooks where you can place your own logic, so FIT is easily expansible (you may have a look at the book ‘Beautyful code’ for a more detailed examination)
– Easier to write OO-Code
Again, i’m not sure about your point, as unit tests and acceptance tests apply on totally different levels of abstraction having different goals:
Unit Test: do we build the code right
Acceptance Test: do we build the right code
That said, you only write a kind of glue code within your fixture that will make use of a facade which will provide the functionality under test.
– Developer support
Completely agree with you on that :o)
Greetings
Mario
If you are talking from programmer perspective, yes what you said may hold good. But FIT enables non programmers to write regression tests (One fixture are created).To my mind, kind of expressing tests in wiki format and viewing test results in HTML format is pretty nice and comes handy for non-programmer.
Tarkan – I’ve seen concordian and it looks alright so far. It seems like a better way to doing FIT.
Alex – I’m glad you agree. I also agree with your second point about having more code to write and maintain and I get more concerned by what my tests are telling me if I have to have a test to test my test fixtures. It indicates to me some other problem that needs fixing instead.
Mario – Thanks for your comment. I didn’t really do an effective job explaining my table, so here’s what I meant and I hope it helps.
Input – I’m less concerned about comparing how JUnit or FIT separate data. What I’m more concerned with is how they deal with the test case specification. As mentioned above JUnit uses code, FIT uses HTML with the overhead of a mapping layer between HTML and code, e.g. FlowFixture. This affects maintainability (see below). On another note, when you say ‘FIT like tests’, it sounds to me something like ‘acceptance test’ (please correct me if I’m wrong). If so, this is one of the biggest reasons I posted this article becuase this is one of the misconceptions I’m trying to address. FIT and JUnit are test execution frameworks, despite the ‘unit’ in the name. Both allow you to write tests at any level – unit, integration, acceptance, and if I’m going to have automated tests that require any level of coding, I’d rather I have full control so refactoring becomes easier.
Output – Sure. I guess I just don’t want to build something myself here.
Easy maintenance – I should’ve elaborated here. I see extensibility and maintainability as two different aspects to software. To me, extensibility (what you talk about) is about how much a piece of software allows you to change or extend it’s behaviour. Maintainability to me, is about how easy will it be to change and reuse to support new or changed behaviour. Lots of things affect this such as how well refactored the code, and conventions a particular library or framework encourages. I believe that FIT encourages poor practices (such as public fields, too many global variables) that lead to poor maintainability. I believe that it requires a higher level of discipline to write maintainable code because developers need to a) consider how the mapping from HTML to code works, and b) consider how to refactor it to separate concerns. JUnit still has the latter issue, though not having to worry about the former makes it easier.
Easier to write OO code – This is related back to the previous couple of points. FIT and JUnit both support writing tests for any purpose. I worry about the cost of automated tests that are code and the only way I know how to ease the burden of maintaining that is either getting rid of functionality (deleting tests) and is unlikely, or refactor my code. FIT means writing more code (to deal with the mapping layer) and just doesn’t scale in my opinion.
Selvan – I worry about non-progammers writing regression tests as I think it trades off short term and long term goals and isn’t not worth it. Normally the short term goal is we want to get tests out faster (so we’ll get non-programmers to write it). Unfortunately in my experience, this leads to slow tests or brittle fixtures (I can’t change this because so many test scripts use it), tests stop being run and their value lessons. I’d rather get non-programmers to work with programmers to generate the test scenarios, and let the programmers do what they do best and automate them (and refactoring them so they’re reusable)