How do I make testing faster?

Earlier this week I was the guest of a large bank in the City, OK Canary Wharf actually. They had their own little internal Agile conference. As well as myself some of the usual suspects were on parade as well as some internal speakers. It was very enjoyable and as usual the C-List speakers were some of the most interesting.

Later in the day I found myself in conversation with two people concerned about software testing. They posed the question: “How do we make testing faster?” Specifically, how do we make SIT (“System Integration Testing”) and UAT (“User Acceptance Testing”) faster?

Now of these solutions are far from unique to this particular bank. Indeed after listening to the question and situation I recalled several discussions I’d had, problems I’d seen and solutions I’d suggested before. (OK bad consultant me, I should have enquired into the problem, understood the context fully, coached her to find her own solutions etc. etc. But we only had 30 minutes and I wasn’t being paid for a full consulting session - nor am I charging you for reading this! - so lets not invent the wheel, lets fill in with assumptions and given some general answers.)

The solutions we came up with were:

1. More testing environments: it is shocking how often I hear testers say they do not have enough test environments. They have a limited number of environment and they have to time-share or co-habit. There are two answers: either increase the number of test environments or reduce the number of test and testers. I’m sorry I can’t give you a magic sentence that persuades the money people to buy you more environments. Now we can virtualise machines there is very little excuse for not creating more test environments. The usual one is cost. But even here the costs are not that high. (Unless that is you need to buy a new mainframe.)
2. Testers integrated & co-located with developers: if you want to go faster I’ve yet to find anything that beats face-to-face direct verbal communications. If you want/need people distributed then you should accept you will not be as fast as you could be. I’ve said it before and I’ll keep saying it: Testers are first class citizens, treat them as you would developers.
3. Automate SIT: simple really, automate the hell out of testing. Well easy to say, harder to do but far from impossible. Since most UAT/Beta testing is exploratory in nature it is largely SIT that need automation. And please please please, don’t spend money on tool test tools.
4. Make UAT just UAT: Recognise that some of what passes as UAT is actually SIT and automate it. In large corporates an awful lot of what is called “User Acceptance Testing” isn’t “User Testing” - although it may well be “Acceptance Testing.” Look carefully at what happens in this phase: that which is genuine Acceptance Test can - indeed should - be specified in advance (of the coding), can be moved into the SIT phase and automated. The true User bit is going to be much more explorative in nature.
5. Improve development quality with TDD and do it test first: one reason for long SIT and UAT phases is that they find bugs. One way to significantly reduce the number of bugs is to have developers practice Test Driven Development, i.e. build in quality earlier. This may mean coding takes longer in the first place but it will speed up the wider process. TDD is still quite new to many developers so you can speed up the process by giving them some help, specifically TDD coaching from someone who has done lots of this before. Until developers make the jump from “post coding automated unit tests” to “pre-coding test first development” you will not maximise the benefits of TDD, you will have more code in your system than you need and you will have a poorer overall design, and both of those make for more tests later on and potentially more problems. When tests are written before code TDD becomes a design activity.
6. Reduce batch size: i.e. reduce the size of the release, and probably do more releases. Do less, do a better job, push it out more often, reduce risk. Think dis-economies of scale. Many large corporations use a large batch size simply because any release is so painful. This is completely the wrong approach, rather than running scared of releases take them head on.
7. Done means done and tested in the iteration - at the end of the iteration it is releasable: when you reach the end of an iteration and declare work “Done” that means “Done ready for release (to customers).” Well, that should be your aim anyway and you should be working towards that point. If “Done” currently means “Done development, ready for SIT” then you should be working towards bringing SIT inside the iteration. If that looks hard right now then look again at items 1 to 6 in this list. And when you’ve moved SIT inside the iteration start on UAT. Until both those phases (and any other phases in the tail) are inside the iteration they can throw up problems (bugs) which will disrupt a future iteration which implies the work you said was Done is not Done.

Now the catch. Money.

Some of these things require money to be spent. In a rational world that would not be a problem because there is plenty of evidence that increased software quality (by which I mean specifically few bugs and higher maintainability) leads to shorter schedules and therefore reduced costs. And as a plus you will get more predictability. What’s not to like?

So rationally its a case of spend money to save money. Quality is free as Philip Crosby used to say.

But unfortunately rational engineers often find it difficult to see rationality inside big corporation operating annual budgeting cycles. One of the reoccurring themes this week - although not one I think was given official time - was the constraints of the budgeting cycle. The rational world of the engineer and the rational world of the account and budgeteer seem a long way apart.

There is a solution but it is a big solution which will scare many people. It is Beyond Budgeting, and that is why I was so keen to invite Bjarte Bogsnes - the author of Implementing Beyond Budgetting - to talk about Beyond Budgeting at this years Agile on the Beach conference. And its probably why Bjarte recently tweeted “Invitations to speak about Beyond Budgeting at Agile and HR conferences are now as frequent as Finance conference invitations. Wonderful!”

Do you know the way to Falmouth? (Agile on the Beach)

The Agile on the Beach 2014 conference is fast approaching and there is every sign that it will, again, be sold out. Which means, in the days and weeks before the conference I’m going to have several people asking me for my advice on how to get to Falmouth. So here is some advice on getting to Falmouth and the conference, not everything you could know, but the main points I tell people when they ask. (If you want a quick summary skip to the end.)

Firstly even some English people are stumped when they stop and consider how to get to Falmouth. Its in Cornwall, its not as far west as it could be (thats Penzance) but it is part of the country that many people have never been to. And if you are coming from outside the UK its even more likely that you need to stop and think.

So the first piece of advice is: plan in advance. Don’t leave it to a few days before hand. You probably need a day just to get the Falmouth.

And that leads to the second piece of advice: if you can plan to extend your stay it is well worth it. The conference is Thursday and Friday, if you can plan to stay longer. I know speakers and attendees who have brought their significant other with them, or arranged for the significant person to arrive on the Friday.

Falmouth is a great little town to spend a day or two in. Better still St Ives is close by, I have been known to go all the way to St Ives just for an exhibition at The Tate gallery there. Then there is the rest of Cornwall to explore. And, if you look carefully there are great places to eat.

Lets start with the most common case…

Falmouth by car

You can drive from London to Falmouth, its about 300 miles and five hours driving - before you take any breaks. Personally I don’t recommend it. I’ve done it a couple of times and its not my idea of fun. Especially once you leave the M5, the A30 seems to go on and on and on… If you do drive allow plenty of time and plan on a couple of breaks.

Also, you are likely to be charged for car parking at the conference itself. The conference is held at Falmouth University and despite lots of talks with the University people they refuse to remove the car parking fee. As I recall this was £3 a day but I might be wrong. Hopefully they will relent but there you go.

Flying - from London or elsewhere

You can also fly. Actually you fly to Newquay (NQY) airport from where you need to get the 25 miles or so to Falmouth. You are probably looking at a taxi which adds to the cost. Also adding to the cost is the third-world like £5 “Airport Improvement Taxi” which you need to pay on departure.

I don’t like NQY. Of all the airports I’ve been to I find the security staff the most intrusive. I know they have a job to do. I know they have to screen you. I know there are bad people in the world. But… of all the airports in the world I find the security here to be the biggest infringers of my personal space.

I once saw the security staff send once send a lady back to the checkin desk because she had a boarding pass issued by a code-share partner of the airline. The demanded a genuine FlyBe boarding pass and not a self-printed BA boarding pass. Maybe this is common but I’ve never seen it elsewhere.

Anyhow nobody flies for fun these days so maybe I’m being an grumpy old man.

If you want to fly you probably want FlyBe from Gatwick, although FlyBe also have daily flights from Manchester. FlyBe also have some flights from other places but these don’t happen everyday. At the time of writing EasyJet also list a flight from Liverpool and have operated from Southend before now but again its not everyday.

Internationally

If you are travelling internationally you may well be getting on a plane anyhow. Although Lufthansa has an occasional flight to Newquary you are probably going to have to connect somewhere. Gatwick is the obvious place but Manchester is just as good an option.

One thing to be aware of: if you are flying into Heathrow and need to get from there to Falmouth I recommend you switch to trains. Transferring between Heathrow and Gatwick is better avoided. On the other hand, getting from Heathrow to Paddington and picking up a train is really easy.

One minute please, I’ll talk about trains in a moment.

Another option is Exeter airport - Exeter also has an airport with some flights to Amsterdam and other locations in the UK and Europe. Since Amsterdam is a hub this is a viable option for anyone coming from further a field.

From Exeter airport you could continue by train. I’m not sure how you get from the airport to Exeter train station, bus or taxi is my guess.

Last year some Dutch speakers flew to Exeter and hired a car to drive the second leg.

Train

Personally I always take the train. I live in London so this means getting to London Paddington and picking up a train to Penzance. But you don’t ride all the way, a few stops before Penzance is Truro where you need to change.

Paddington to Truro is about 4.5 hours, then change for Falmouth at Truro for a little train but get off at Penryn station (Penryn live departure boards), from there it is a short walk up the hill to the University campus.

I always take the train. Once I’m on the train at Paddington I can work, read, sleep, what ever I like. Unlike the plane where you are queuing to get through security, queuing for coffee, queuing to get on the plan, squeeze in flight for 40 minutes, queuing to get off the plane you sit on the train.

If you are coming from somewhere else in the country you can pick up the Cross Country train that also goes to Penzance or pick up the train from Paddington at Reading.

If you’ve never done the journey before pay special attention to the Dawlish section after you leave Exeter, the scenery is beautiful - and yes it has been rebuilt after last winter.

My big piece of advice here is: book your train tickets in advance. If you leave it until the last minute you could end up paying £300 for Second Class. If you book in advance you can get First Class for as little as £100. The extra’s in First Class aren’t up to much but you do get free water, tea and coffee (compared to first class on East Coast Mainline or Virgin West Coast the FGW First Class is poor).

Note also: FGW don’t have wifi onboard yet and 3G reception is poor after Exeter. The further west you go the weaker it is.

Some of the First Great Western trains have a “Travelling Chef” who does a good bacon sandwich. But, determining which trains have travelling Chef’s is hard, FGW’s website makes you work hard. Second: sometimes the Chef doesn’t turn up. I’ve planned on a bacon butty before now only to find the Chef didn’t turn up for work that day.

Next advice: going home look at the sleeper - especially if you live in London. You leave Cornwall about 10pm and get to Paddington about 5am the next morning. I’ve done it for the last three year - although I’m probably not doing it this year. Again FGW don’t make the sleeper that easy to find or book (no Cross Country this time).

The sleeper isn’t really that much more expensive - I think it cost me £100 last year - and it is fun. You don’t sleep all the way but you will sleep. Its an experience.

A warning: The same trains which have the sleeper cars have regular cars where you can sit -airline style - and try and sleep. If you don’t explicitly book the sleeper you have a seat not a bed.

There you go, that is pretty much the advice I give to anyone who asks: How do I get to Agile on the Beach?

1. Avoid driving
2. Only fly if you are coming from international (or Scotland) and then try to connection somewhere other than Heathrow
3. Take the train if you possibly can
4. Book early and pay the extra for first class
5. Think about getting the sleeper back

Tailor or an image consultant?

“Gentlemen, you are getting married. You go to the best tailor in town, and you say: ‘I’m getting married, my bride is beautiful, make me look one million dollars, I don’t care what it costs.’

The tailor measures you very carefully then he stands back. He rubs his chin, he is clearly troubled by something. ‘Tell me’ you say, ‘Please tell me.’

‘Sir’, he replies, ‘I don’t think it is my place to say’

‘Tell me’ you beg, ‘I must looks fantastic for my girl’

‘Well Sir, …’ he talks slowly and cautiously, ‘. if you really want to look $1,000,000 can I suggest you lose 5kg?’

What do you say?”

This is the dilemma many a consultant find themselves in. More importantly it is the dilemma that I find again and again in corporate IT:

• Is IT a service department which is expected to build what is requested without answering back?

Or

• Is are the IT departments the experts in maximising benefit and value out IT investment?

Is the tailor a master of clothes who just makes beautiful clothes? Or are they, because of their long experience making people look beautiful in clothes, an image consultant who can help you look beautiful?

Is your IT department a tailor who is expected to produce something brilliant no matter what the request is? Or are they people who, because they work with information technology every day, have experience and knowledge to help their (internal) customers maximise their benefit? (Even if that means the original request should be reconsidered.)

I distinctly remember a senior IT manager at an American bank telling me that his department was most definitely a service department, it wasn’t his job to answer back to business requests, his job was to build what was asked for. The implication being that even if his people knew the (internal) client was approaching the issue from the wrong point, and even asking for the wrong system, then his people should not correct them. They should just build what was asked for.

Perhaps this dilemma is most accurately felt by Business Analysts who are sometimes expected to write down the orders that someone else wants to give to IT even if they know better. Fortunately some Business Analysts can go further and advise their “customers” on what the bigger questions.

The original metaphor come from Benjamin Mitchell. As I recall it was part of a very deep conversation we had in Cornwall back in 2011 and which also led to the “Abandon hope all ye who enter Agile” blog post. I’ve embellished and elaborated the story over the years.

I reminded Benjamin of this story in January - he’d forgotten about it, thats how brilliant he is, he has so many great ideas he can’t keep track of them all. Once I’d reminded him he immediately made it better.

“How would you know?” he asked, meaning, “How can you determine what is expected of you?”

For me the story is about missed opportunities, about frustration and about what you might do to change the position. For Benjamin it is about asking “Do we all expect the same thing? - If you think they consider you a service group how can you confirm this?”

Either way the core problem is about a mismatch of expectations. Your IT department might be a service department, that approach could work, its how many big corporations operate (I my experience).

Or the IT department could be the experts in technology, how to apply technology, how to frame problems so technology can help and how to maximise the return from technology.

Both are valid options.

But the real problem is when someone in the department thinks the department is should be doing more than the rest of the company does; or when the rest of the company expects more from the department but the department sees itself as a service group.

Finally, go back to my series on the economics of software development, the supply and demand curve, and apply this story. If the IT department is seen as a tailor who shouldn’t answer back they will find it very difficult, if not impossible, to change the demand curve. Such a tailor can only work on the supply side.