Category: Development

Making jade and mustache templating work together

One our frustrations using jade and icanhaz (a javascript front end mustache implementation) was that when we were trying things that were obvious to us, jade would simply fail to template and we weren’t sure what was causing it.

Fortunately small TDD cycles and experimentation made us realise that it was the combination of new line characters and mustache code made jade work/break.

We would try something like this:

script(type="text/html", id="my_checkbox", class="partial")
  li 
    label(for="{{code}}")
      {{name}} 
    input(id="{{code}}", checked="checked", name="{{code}}", type="checkbox")

The set of statements above would be valid mustache (once converted to HTML) but jade complains because the {{name}} is on its own line. The fix was to use the pipe (|) character to force jade to recognise a line break. It looks like this now

script(type="text/html", id="my_checkbox", class="partial")
  li 
    label(for="{{code}}")
      | {{name}} 
      input(id="{{code}}", checked="checked", name="{{code}}", type="checkbox")

Simple, but not particularly obvious from the examples in their documentation.

A builder pattern implementation in Javascript

We’re using the builder pattern on our javascript project, as it useful for starting off with a set of defaults, but is clear when we want to override particular values. Although there are a few places on the net that describe who uses the builder pattern in javascript, they don’t really provide an implementation.

Here’s one that works for us:

var Builder = function() {
  var a = "defaultA";
  var b = "defaultB";
  
  return {
      withA : function(anotherA) {
        a = anotherA;
        return this;
      },
      withB : function(anotherB) {
        b = anotherB; 
        return this;
      },
      build : function() {
        return "A is: " + a +", B is: " + b;
      }
  };
};

var builder = new Builder();

console.log(builder.build());

var first = builder.withA("a different value for A").withB("a different value for B").build();

var second = builder.withB("second different value for B").build();

var third = builder.withA("now A is different again").build();

console.log(first);
console.log(second);
console.log(third);

Independent Refactoring is Irresponsible

Both Michael Feathers and Rachel Davies recently recently wrote about attempts to make refactoring a more explicit step in the development process by adding a particular refactoring task to the board.

It got me thinking about my latest project, a green-field application where, in the last week, I think we almost tripled our estimated velocity. You might think that we gamed the estimates, bloating figures to make us look good, but we did not. We still used relative complexity to estimate essential work that must be done. We did learn a little bit more than the week before, but the biggest change was actually some preparatory refactoring. Before I explain why it worked, I’m going to take a slight detour.

I remember working with one client who spun up a “refactoring team”. It sounded great at the outset – a legacy application that had plenty of code as they knew the development team cut corners to meet their milestones. Rather than completely halt new development, they split out a small team who would refactor mercilessly. This team spent one whole month renaming classes, adding tests, adding patterns here and there, fighting the big cyclomatic complexity numbers and then claiming victory on their poor opponent (the codebase). The result after the refactoring team… new feature development slowed down even more.

When investigating why feature development slowed down, we discovered the following:

  • New, and unfamiliar designs – The refactoring team did some wonderful work cleaning up certain parts of the codebase. THey made some places consistent, introduced a few patterns to tackle common smells and honestly helped reduce the codebase. What they neglected to do was to inform the other developers of the new design, where they now needed to look for the same functionality and the intention behind the newly named classes. Instead, the developers working on new functionality struggled to find the huge, very finely commented code they were familiar with and then when they tried to apply their old technique for fixes, failed to do.
  • Immediately irrelevant refactoring – With most codebases, there are parts that change a lot, parts that change a bit, and parts that almost never change. In my experience (disclaimer: not researched) those parts that change a lot and are highly complex end up as a huge source of bugs. Those parts that don’t change can remain ugly and still be perfectly fine. In the case of this client, a lot of effort spent refactoring ended up in areas where new functionality wasn’t being added.
  • A divide in cultures – I heard a few snarky comments during this time from the new feature development team about the refactoring team, basically implying most of them to be developer-divas whilst they had to do all the grunt work. The result… by outsourcing refactoring, the new feature team basically cared less about the codebase and I’m sure they weren’t helping the clean up with the code they added.

Litter
Image taken from Will Lion’s Flickr stream under the Creative Commons licence

My reading on lean thinking taught me that you need to Build Quality In and that separately quality from the product ends up costlier and results in poorer quality.

The answer… is actually quite clear in Martin Fowler’s book on Refactoring:

Refactor because you want to do something else (i.e. add function, or fix a bug)

I return to my current situation. We achieved the tripling in velocity because we spent time thinking about why adding a new feature was so cumbersome. I’ll admit that I spent a lot of time (almost a day) trying to add part of a new feature, attempting a few refactors and rolling back when they did not work. I was trying to get a feel for what steps we did most often, and attempted several approaches (most failed!) to make it simpler, clearer and added the least amount of code. We did settle on some patterns and we realised its benefits almost immediately – adding a new feature that previously took us a day to implement now took us only an hour or two with tests.

I find that sometimes the most satisfying part of software development is actually reshaping existing code so that the addition of a new feature is just a single method call, or just a single instance of a class. Unfortunately I don’t see this often enough.

Managing Ruby Development Environments

One of the principles I like is being able to set up new development environments very quickly. The java space offers many libraries for managing your environment that means that each project works in a separate space. In contrast, in the .Net space an anti-pattern is one that often requires many installs to your GAC (Global Assembly Cache) often through the use of a “mouse-driven”-only installer.

Fortunately the ruby community offers a number of tools for managing both versions of ruby and the libraries that you use. The ones that I will often reach for, now include:

  • RVM – Ruby Version Manager. Allows you to have different versions of ruby, and to quickly switch between them
  • Bundler – Management of gems.

The ultimate acceptance test for this, is can developers simply “check-out” and go. The lead time to setup a new development environment should be very quick.

Note that there is now a competing tool for managing environments called RBenv although integrated tool support (like in RubyMine) is only starting to come through

Taming the Hippo (CMS) Beast

I eluded in a previous post our struggles dealing with the HippoCMS platform. It wasn’t our preferred path, but a choice handed down from above. Enough said about that. It’s useful to understand a little bit about the environment we were using it.

I believe the pressure to choose a CMS came from a deadline that required some choices about platform choice to be made in an organisation. At this time, the extent to what the actual product was unknown. Our experience working with other clients is that you should generally work out what you want to do before you pick a product, or the platform will often dictate and limit your ability to do things. My colleague Erik Dörnenburg has been writing more about this recently.

The premises of a CMS are alluring for organisations. We have content… therefore we need a content management system. The thought ensues, “Surely we can just buy one off the shelf.” Whether or not you should use a CMS is for another blog post, and you can read some of Martin Fowler’s thoughts on the subject here.

We wanted to protect our client’s ability to evolve their website beyond the restrictions of their CMS, so we architected a system where content managed in a CMS would sit behind a content service, and a separate part of the stack focused on the rendering side. It looks a little like this:

ContentService

The issues that we faced with HippoCMS included:

A small community based on the Java Content Repository

Hippo is based on the Java Content Repository (JCR) API, a specification for standardising the storage and access of content. Even as I write this blog, putting “JCR” or “Java Content Repository” I am forced to link to the wikipedia page because I spent three minutes trying to find the official Java site (it looks like the official site is hosted by Adobe here). If the standard is small, the community surrounding the products are naturally going to be smaller. Unlike users of spring, putting a stacktrace into google will generally show the sourcecode of the file rather than how someone got over it. I’d be happy living on the bleeding edge… if the technology was actually pretty decent.

Unfortunately a lot of the gripes I write about are the fact that the product itself is based on the the JCR specification. Some simple examples include:

  • A proprietary query syntax – You query the JCR with an xpath-like query language. It’s actually less useful than xpath, such as not implementing all functions available in xpath and some weird quirks
  • Connecting to the repository via two mechanisms – Either RMI (yuck! and inefficient) or in memory. This automatically limits your deployment options to the application container model. Forget fast feedback loops of changing, starting a java process and then retesting.

Hippo CMS UI generates a huge number of exceptions
One reason Hippo was selected was for the perceived separability of the CMS editor and the website component (referred to as the Hippo Site Toolkit). We didn’t want to tightly couple the publishing/rendering side to the same technology stack as the underlying CMS. Hippo allows you to do this by having separately deployed artefacts in the application container. Unfortunately, the Wicket-based UI (maybe because we used it without the Hippo Site Toolkit) generates exceptions like nobody’s business. We spent some effort trying to understand the exceptions and fix them, but there were frankly too many to mention.

Poor taxonomy plugin implementation
One of the reasons Hippo was allegedly picked was for the taxonomy plugin. Unfortunately this gave us no world of pain both in usability and in terms of maintaining it. In terms of the specific issues we faced with the maintenance included the multi-language support (it didn’t allow that) and then just simply getting it deployed without issues.

CMS UI lack of responsiveness
Our client’s usage of the site wasn’t very big. Less than 300 articles and, at the peak, about 10 concurrent users. Let’s just say that even with three people, the UI was sluggish and unresponsive. We tried some of the suggestions on this page, but it’s a bit of a worry that it can’t responsively support more than one user out of the box with standard configuration.

Configuration inside the JCR
Most of our projects take a pretty standard approach to implementing Continuous Delivery. We want to easily source control configuration, and script deployments so that releases into different environments are repeatable, reliable, rapid and consistent. Unfortunately a lot of the configuration for new document type involves “switching a flag to capture changes”, playing around with the UI for a new document type” and then exporting a bunch of XML that you must then load with some very proprietary APIs.

After several iterations, we were able to streamline this process as best we could but that took some time (I’m guessing about a developer two weeks full time).

Lack of testability
We spent quite a bit of effort trying to work out the best automated testing strategy. Some of the developers first tried replicating the JCR structure the UI would recreate but then I pointed out that would give us no feedback of if Hippo changed the way did its mapping. We ended up with some integration tests that drove the wicket-based UI (with a wonderfully consistent but horrid set of generated IDs) and then poked our content service for expected results.

A pair of developers worked out a great strategy for dealing with this, working out the dynamically generated APIs and driving the UI via Selenium Webdriver to generate the data we would query inside the proprietary XML-based data store.

Lack of real clustering
In “enterprise” mode, you can opt to pay for clustering support although it’s a little bit strange because you aren’t recommended to upgrade a single node within a cluster when other nodes are connected to the same datastore (in case the shared state is corrupted). This kind of makes seamless upgrades without complicated DB mirror/restore and switcheroo really difficult. We ended up architecting the system for a degraded service using caches on the content service as a compromise to the “clustered” CMS.

Summary
As much as I wish success for the Hippo group, I think many of the problems are around its inherent basis on the JCR. I do think that there are a couple more things that could be done to make life easier for developers including increasing the amount of documentation and thinking about how to better streamline automated, frequent deployments around the CMS.

Cheat Sheet for Javascript Testing with Jasmine

Jasmine is the default unit testing framework that I use when writing javascript, however my poor brain can’t always remember all the different ways of getting things to work. There are quite a number of cheat sheets out on the internet including:

They don’t quite cover all the examples as well. Here’s my contributions to demonstrate some of the common uses.

describe("jasmine", function () {

    describe("basic invocations", function () {

        var SampleDependency = function () {
            return {
                usefulMethod:function (firstParameter, secondParameter) {
                },
                anotherUsefulMethod:function () {
                }
            };
        };

        var Consumer = function (dependency) {
            return {
                run:function () {
                    dependency.usefulMethod("first", "second");
                },
                runSecondMethod:function () {
                    dependency.anotherUsefulMethod();
                },
                runWithRequiredCallback:function(callback) {
                    callback("an argument");
                }
            };
        };

        it("should spy on an existing function", function () {
            // given
            var dependency = new SampleDependency();
            spyOn(dependency, "usefulMethod");
            var consumer = new Consumer(dependency);

            // when
            consumer.run();

            // then
            expect(dependency.usefulMethod).toHaveBeenCalled();
            expect(dependency.usefulMethod).toHaveBeenCalledWith("first", "second");
            expect(dependency.usefulMethod).toHaveBeenCalledWith(jasmine.any(String), jasmine.any(String));
            expect(dependency.usefulMethod.callCount).toEqual(1);
            expect(dependency.usefulMethod.mostRecentCall.args).toEqual(["first", "second"]);
        });

        it("should demonstrate resetting of the spy", function () {
            // given
            var dependency = new SampleDependency();
            spyOn(dependency, "usefulMethod");
            dependency.usefulMethod();
            dependency.usefulMethod.reset();

            // when
            dependency.usefulMethod();

            // then
            expect(dependency.usefulMethod).toHaveBeenCalled();
            expect(dependency.usefulMethod.callCount).toEqual(1);
        });

        it("should demonstrate creating a spy object with prepopulated methods", function () {
            // given
            var dependency = jasmine.createSpyObj("dependency", ["usefulMethod", "anotherUsefulMethod"]);
            var consumer = new Consumer(dependency);

            // when
            consumer.run();
            consumer.runSecondMethod();

            // then
            expect(dependency.usefulMethod).toHaveBeenCalled();
            expect(dependency.anotherUsefulMethod).toHaveBeenCalled();
        });

        it("should demonstrate creating a stub object", function () {
            // given
            var dependency = jasmine.createSpyObj("dependency", ["usefulMethod", "anotherUsefulMethod"]);
            var consumer = new Consumer(dependency);
            var stubbedCallback = jasmine.createSpy("stub callback");


            // when
            consumer.runWithRequiredCallback(stubbedCallback);

            // then
            expect(stubbedCallback).toHaveBeenCalled();
            expect(stubbedCallback).toHaveBeenCalledWith("an argument");
        });
    });


    describe("returning a value", function () {
        var Dependency = function () {
            return {
                getMultiplier:function () {
                    return 10;
                }
            };
        };

        var Consumer = function (dependency) {
            return {
                calculateSomethingWithMultiplier:function (number) {
                    return number * dependency.getMultiplier();
                }
            };
        };

        it('should be correct', function () {
            // given
            var dependency = new Dependency();
            spyOn(dependency, "getMultiplier").andReturn(40);
            var consumer = new Consumer(dependency);

            // when
            var result = consumer.calculateSomethingWithMultiplier(3);

            // then
            expect(result).toEqual(120);
        });
    });


    it("should demonstrate creating a stub that returns a value", function () {
    });

    describe("creating a stub that calls a fake", function () {
        var Dependency = function () {
            return {
                request:function (callback) {
                }
            };
        };
        var Consumer = function (dependency) {
            var capturedValue = "";
            return {
                hardAtWork:function () {
                    dependency.request(function (value) {
                        capturedValue = value;
                    });
                },
                getCapturedValue:function () {
                    return capturedValue;
                }
            };
        };

        it('should demonstrate creating a stub function that does something interesting', function () {
            // given
            var dependency = new Dependency();
            spyOn(dependency, "request").andCallFake(function (callback) {
                callback("Controlled return value from callback");
            });
            var consumer = new Consumer(dependency);

            // when
            consumer.hardAtWork();

            // then
            expect(consumer.getCapturedValue()).toEqual("Controlled return value from callback");
        });

    });
});

RequireJS is the Spring Framework of Javascript

I’ve been working on setting up the infrastructure for a mostly javascript based project, and we’ve been putting RequireJS into the codebase to help us manage the file dependencies instead of having to declare them within the page that is using them. As a concept, RequireJS is helping us keep different javascript modules apart in different files and let’s us assemble them.

RequireJS works by declaring dependencies and having the framework pull them in when you need them.

define(["aDependency"], function(theDependency) {
  // now I can do something with theDependency
  theDependency.aMethodOnIt();
})

This is pretty much how spring works, but the issue I have is that RequireJS manages the lifecycle of the javascript objects, so when you want to pass in a substitute for a test, you end up in a dilemma.

define(["aDependency"], function(theDependency) { // how do I get inject a different instance?
  // now I can do something with theDependency
  theDependency.aMethodOnIt();
})

Unsurprisingly a number of people wrote libraries such as testr which allow you to override the requirejs to inject different versions. Although very reasonable approaches, I find this approach a little bit smelly as you’re effectively patching a library you don’t own. The ruby community know the dangers of monkey patching too much, particularly those parts of a code base you cannot control and the potential issues you face when you try to upgrade.

Our current approach involves using RequireJS to manage the file/name dependencies, but for us to write javascript that allows us to control the instances of the objects that we want. Here’s an example:

dependency.js

define([], function () {
    return function () {
        return {
            doSomeWork:function () {
            }
        };
    };
});

consumer.js

define([], function () {
    return function (aDependency) {
        var dependency = aDependency;
        return {
            start:function () {
                dependency.doSomeWork();
            }
        };
    };
});

And then we control the lifecycle of the components and instances in the application using the following code.

main.js

define(["consumer", "dependency"], function (Consumer, Dependency) {
    var dependency = Dependency();
    var consumer = Consumer(dependency);
    consumer.start();
});

And our jasmine tests get to look like this:

requirejs = require('requirejs');

describe("consumer", function() {
    it("should ensure the dependency does some work", function() {
        // given
        var dependency = jasmine.createSpyObj("dependency", ["doSomeWork"]);
        var consumer = requirejs("consumer")(dependency);

        // when
        consumer.start();

        // then
        expect(dependency.doSomeWork).toHaveBeenCalled();
    });
});

This approach has been working out well, forcing us to manage the dependency and global hell that javascript global functions can quickly become. Thoughts? Please leave a comment.

Looking back at a year with a client

Over the last twelve months, I’ve worked with a client to rebuild a digital platform and a team to deliver it. It’s now time for us to leave, and a perfect time to reflect on how things went. The digital platform rebuild was part of a greater transformation programme that also involved the entire business changing alongside at almost all levels of people in the organisation. The programme also outlined, before we arrived, outlined a complete change in all technology platforms as well (CRM, CMS, website) to be rebuilt for a more integrated and holistic service offering.

Our part in this program turned into building the new web digital platform, working against a very high level roadmap, and a hard marketing deadline. We ended up building the site using Ruby on Rails serving content driven by a 3rd party decisioning platform (much like Amazon recommendations) guided by the business vision of better tailored content for end users. We didn’t have much input into the final choice of several products. I’m very proud of the end result, particularly given the tense and short-timed framed environment in which we worked. Here are some examples of constraints we worked with:

  • 4 Product Owners over the span of 11 months – From January this year, through to the end of October, the business was onto its fourth Product Owner for the digital platform. Building a consistent product is pretty much nigh impossible with changing product hands, and trying to bridge work from one Product Owner to the next was definitely a challenge.
  • Constant churn in the business – The 4 product owners is one instance, but we would often be working with people in the business to work out what should be done, only to find that the following week they were no longer with the business.
  • 3 Design Agencies engaged resulting in “reskinning” approved by the board before the 6 month public launch – We saw several “design changes” done by firms well stocked with people capable of generating beautifully-rendered PDFs that were signed off. However often these would imply new functional work, or be impractical to the web medium.
  • Marketing deadlines announced well before the development team had been engaged - A common pattern in the business was marketing launching a press release announcing a date, well before the people involved in delivering it were made aware, or even consulted on it.
  • PM Explosion – At one point, it felt like we had more Project Managers on the group planning out work with budgets and timelines that would be approved well before the development team had been approached.

Even with these constraints we’ve been able to deploy to production 37 times in the last three months and more since the original MVP go-live in July. Part of what I’m particularly proud of is the team where we were able to achieve the following:

  • Building an Evolvable Architecture – We questioned the original choice and need for a CMS but with a constraint that a decision had been made on buying these tools, we architected a solution that would hide the implementation details of the CMS via a content service. With our TW experience and pain dealing with CMSes that are shadowed by business need, we wanted something that would not constrain what the business could achieve (hence the decoupling). We even had a chance to prove this out when the business requirements quickly hit the limit of the CMS’s built in categorisation module.
  • Responding to Change – The business roadmaps seems to change on a daily basis, and our team was able to quickly tack to accommodate these business changes. We changed the team structure as the team size increased, changed the team structure as we went live, and again as people in the business changed. Whilst our process felt similar, it would look nothing like a textbook XP, Scrum or Kanban process.
  • Improving the Process – Our team has been constantly trying to change the process not only internally to the development team, but also helping people in the business find ways of improving their own way of working. Progress has been slow as the change that starts falters as people leave. Retrospectives have been a key tool but also has the ability for the team to feel empowered with recommending and pursuing improvements they see fit.
  • Setting an example of transparency – Showcases are key to the business, and we would offer fortnightly showcases to the features built to the entire organisation. Huge numbers of people came along and I found it fascinating that it was one place where people had an opportunity to talk across silos. This sometimes slowed down our ability to show what we had actually done, but I felt exposed missing communication structures that people still needed.

At a technical level, I’m really proud of some of the principles I wanted to achieve at the start and that the team lived throughout (I’d love to hear what their experience is). Some of these include:

  • Fast developer setup – Getting started on each new machine should be fast without complicated installation processes
  • Developers rotating through operations – There’s nothing like supporting the code you wrote to help developers understand the importance of logging, test cases that are missed and just experiencing what production support is like
  • DevOps culture – Everyone on the team is capable of navigating puppet, knowing where to look for configuration changes and ensuring that applications are configurable enough to be deployed without special builds across environments.
  • Continuous Delivery – Our second product owner (the first transitioned out the day we went live) actually asked for us to release less often (i.e. it is a business decision to go-live) so that they could work with the rest of the business to ensure they had their non-IT dependencies in place.
  • Devolved Authority to Feature Leads – I blogged previously about Feature Leads who could help shape the technical solution and drive the knowledge for the project.
  • Metrics Driven Requirements – Though not completely successful, we were able to stop the business from implementing some feature by showing them production metrics. In this case, we were able to avoid building a complex search algorithm to show that we could achieve the same result by adding to a list of synonyms on search.
  • Everyone grows – If I look back at the project, I think everyone on the team has experienced and grown a significant amount in different ways. I think we struck a good balance between being able to work towards individuals goals and find ways they could help the project at the same time.

Other particular things I’m proud of the team:

  • Taming the Hippo - Worthy of its own post, Hippo CMS has been one of the least developer friendly tools I’ve had to deal with for some time. The team managed to work out how to run an effective functional test around its poor UI as well as deploy and upgrade the beast in different environments without the 12 step manual process outlined on their wiki.
  • Rapid team size – Management wanted the entire team to start at the same time. Even being able to push back, we ended up with a very aggressive ramp up and we still managed to deliver well.
  • Diverse but co-operative – We had something like 17 people and 14 different nationalities and it’s one of the strongest teams I’ve seen who were able to work through their differences and forge ahead.

Things that I would like to be different:

  • Find a way to code a lot more – Due to the circumstances, many elements drew me away from coding. At best, I can remember pairing with someone for at most two days a week (for a short time) and I would like to find a way to do that more.
  • Implement more validated learning – Although dependent on a product owner willing to do this, I would have liked to work more on trying to build and experiment a lot more.
  • Have a stronger relationship with decision makers with authority – I believe development teams work best when they are connected to people who can make decisions, not just organisational proxies who provide answers. Unfortunately I felt most of this cascaded very far up the organisation and due to the constant flux and organisational change, this wasn’t possible in the timeframe. I’m hopeful that as the business matures and more permanent people find their place, this will be more possible.

Goto Aarhus 2012

This year was my first time to both attend and present at Goto Aarhus. Over the years, many of my colleagues have said that it’s one of the best conferences with topics in lots of different areas. This year focused on topics such as NoSQL, Big Data, Humans at Work, Javascript, Continuous Delivery, Cloud and many more areas.

Two of the best presentations I attended, both for content and delivery were Sam Newman and Jez Humble, author of Continuous Delivery (Disclaimer: They are my colleagues after all). What I enjoyed about their talks were both their talk about real world examples, as well as important advice as well as the delivery. Getting the balance right is really difficult to do.

I also really liked the keynote from Dirk Duellmann from CERN who talked about the big data challenges they have storing information. Although it took a while to get to the meaty part of the data, storage details I think it’s a very interesting outlook they have with architectural choices such as the view that they cannot design for hardware or devices today as these will be obsolete as time goes forward. Being able to retrieve historical information is important as it the ability to store all of the data in a format others can read. They have realised the importance of the scale of the work they are doing, so they are focusing on doing something good (storing and making available data) and working with other groups to do the analysis.

There were loads of highlights such as meeting many new people and connecting with old ones as well as some interesting side conversations.

I gave my talk (above) and was very happy with the results. The Trifork team behind the conference are awesome at getting feedback to presenters for quickly and I was very happy with the results. The conference uses a simple voting system for feedback (red, yellow, green) and they keep track of the number of walk outs. I ended up with 90 green, 26 yellow, 1 red and only 2 walkouts. I have no idea how that compares with other speakers but I’m pretty happy with the results. What I also appreciated were the people who came up afterwards to talk to me about how the topic is really important and what some people got out of it (affirmation they are doing the right thing, new ideas to take back, new books to read, more things to focus on, or a good idea of how to prepare as they step into the role).