Christmas reading: Classic essays on software development

There are many, many books about software development. The technical ones alone (e.g. Java, C++, Apache, .NET, etc. etc.) take up meters and meters of shelf space. The ones about project management take up a bit less space but they are still plentiful, and the ones about people in the process take up even less space but are longer lasting.

It is the later category that interests me most theses days. Some books seem perennially relevant and may never go out of print, books like:

• Mythical Man Month by Fred Brooks
• Psychology of Computer Programming by Gerry Weinberg
• Peopleware by Tom DeMarco and Timothy Lister

But often you don’t need a full book to make a point. On these occasions an essay or journal article will do. Often they make the point better simply because they are shorter. Unfortunately these tend to be overlooked too often - books get all the publicity. I would like to highlight half a dozen essays which I think are perennially relevant and often overlooked. In some cases this is because the articles can be hard to get hold of or, more likely, people don’t know they exist.

1. How do committee invent? by Melvin Conway - the origin Conway’s Law. Although written in 1968 this piece is truer than ever. Arguments made by Conway explain much of what actually happens in software development. I’ve examined and looked at this subject myself elsewhere.

Conway suggests that organization that create systems (not specifically software systems) will create copies of their organizations. So, if you have one developer writing the entire system it will be all entangled, few interfaces and difficult for anyone else to work with. And if you have half a dozen developers who don’t communicate you will get six different coding styles, and six versions of most common functions.

What Conway didn’t foresee is that many of system now create the reverse force. Organizations are limited by the systems they use, thus their structures become copies of the systems in use.

2. Worse is Better by Richard ‘Dick’ Gabriel - this started as a keynote talk, became an essay and then a series of letters in which Dick argued anonymously with himself.

If you are one of these developers who think the best code will always win the day you need to read this. Dick suggests that sometimes the solution which is technically worse is economically superior. In a similar vein see Brian Foote’s Big Ball of Mud pattern.

3. No Silver Bullets from the author of Mythical Man Month, Fred Brooks. This piece came out in the mid 1980’s. The Wikipedia summary is here, otherwise you will need to by the Anniversary edition of Mythical Man Month to read it.

Brooks argues that software development is hard, and there is no ‘silver bullet’ which will make it easier. No language, no operating system, no methodology or any other tool will fix it. I have been told that Brooks has since said the if there is a Silver Bullet it is Agile Software Development but I can’t find this quoted anywhere so I don’t completely believe it.

4. A Rational Development Process and How to Fake It by Dave Parnas. Originally published by the IEEE this also appear in Software Fundamentals: The Collected Papers of David L. Parnas. You might also find some copies stashed on the Internet. Dave Parnas deserves to be more widely read by software engineers today so it you haven’t read much of his work buy the collected papers, well worth reading.

In this paper Parnas argues that while we may want to use a rational development process, and while it makes sense to do so it is impossible. This is because software development requires intuition and inspiration - or as I would put it learning. And since you can’t schedule these things or guarantee they will happen the chances of following a rational process are negligible.

However, in order to make our work accessible to those who come after us we need to fake the development process so it looks like we followed a rational process.

I like the logic, I agree with much of it but I have one disagreement. Each generation of software engineers comes along and believes the previous generation were rational and get confused by what they find. They slowly learn that they cannot be rational - causing a lot of angst. They also wonder how the previous generation made such a mess. So frustration is built up.

5. Cathedral and the Bazaar - by Eric Raymond: an explanation of Open Source software. And an explanation of why software created in corporations is frequently a mess. I don’t think Open Source is the answer to all our problems, but I think it is an answer to some and is here to stay.

6. Enrolment Management by Peter Conklin: until recently this was one you really have to hunt down, but it is worth it. Originally published in the Digital Technical Journal (Digital as in DEC or Digital Equipment) in 1992 it was republished by the IEEE in July 1996. HP now have back issues of the Digital Journal online so you can read it here.

This piece tells the story of the Alpha AXP programme in the late 1980’s and early 1999’s. In theory, on paper, on GANTT charts the project couldn’t be done. The author tells how enrollment management was used to engage those working on the project. Although written in 1992 this paper is about Agile development. And it is about Agile development in the large - over 2,000 engineers worked on the project.

I wrote about this article before, a few years go. One of the small stories in this paper always brings a smile to my face. The OpenVMS team committed to a delivery schedule and said ‘We don’t know how to achieve this, but we commit to finding a way.’

So there you have it. Six essays I recommend you read. And if you only have time for one? Well, chances are you know about Open Source and the Silver Bullet so I recommend you read Enrolment Management or A Rational Process, and if you push me more I’ll say: Enrolment Management.

Busy time - and the Software Development Practice Review

Despite the fact that I have deliberately taken a couple of months off since the end of my last assignment I’ve been keeping very busy. I’ll be glad when Christmas comes and I can leave the laptop switched off for a few days.

In addition to finishing up a couple of personal projects (the book, various patterns papers and other stuff) and giving some talks and briefings I have been involved in an attempted rescue. OK, nobody was in real danger this was more a corporate rescue.

I drafted a turn-around plan for a .com in trouble. For a while it looked like the plan would be adopted but now it looks doubtful. Seem the management have gone off the idea. Shame but so be it, it was interesting to analyse the company and make suggestions.

I have also been busy working on my own product offerings. I have today revised my website to make it clearer what I do. In addition to regrouping some of my services (training, briefings and workshops) I’ve also created a Software Development Practice Review product - SDPR for short.

I’m particularly proud of this SDPR. It has taken a few weeks to bring together the framework but the basic idea is simple. I want to be able to visit a development organization (a team or department or such) and evaluate their software practices. Notice I say practices, not processes. This is deliberate. Most of the other reviews I’ve looked at consider processes, I think practices are more important because it shows you what people are actually doing, not what they are supposed to be doing.

The framework behind the SDPR starts by considering teams (size, structure, experience, etc.), the management of the teams, how the business communicates its need to the teams and how risk is managed. After this it turns to what the team is doing, technology, routines, practices, release management, testing and a little bit about processes. Finally the framework considers what happens after a software release and how the team learns.

From this review I can create a report designed to seed team improvement and stimulate the various members to try new practices and ideas. The second key difference I see with many other types of review is that this review is intended to help the teams improve, it is not intended to measure them on ‘pass/fail’ or suitability to undertake work. It is there to help improvement.

Overall I get very excited about this review. I hear of so many development teams which could do better, some of which are almost dysfunctional and yet fixing them wouldn’t be very difficult. Hopefully this is a tool to do just this.

I’m now talking to some people about trying the framework out in January. If you think your team could benefit from this review, or you know someone who would find it useful please get in touch.

Business alignment, Agile failure and the long run

Tom Gilb sent me a link to this presentation from Ryan Shriver about the use of Scrum with Tom’s EVO techniques. Very interesting, and it returns me to the subject of IT/business alignment - which in part is also a discussion of where Agile software development is wrong.

My blog from last week touched on this but I think its worth spelling this out. There are two problems with Agile, and two answers:

Question 1: Why do Agile techniques work? - forget the technical explanations of why TDD is a good thing, why RUFD is preferable to BUFD or how value stream mapping helps. Think for a moment: there are a lot (an awful lot) of software development groups which are not effective.

Problem 1: How can it be that the same set of (Agile) practices can help all these groups?

Answer 1: Because these groups all have similar problems, probably because too many people were taught (and believed) the same set of practices that have created the problems.

Question 2: How can Agile work without business alignment? The basic Agile techniques pay little attention to what the business wants. The XP onsite customer is really lightweight, XP assumed the customer knows what they want but often they don’t. Scrum is better but the product owner is still left to their own devices, Scrum says little about how the product owner knows what’s what.

Problem 2: How can Agile be effective without business alignment?

Answer 2: It doesn’t have to be. The MIT Sloan Review piece shows that just getting to be effective is such a massive improvement over being ineffective that it is a success. You don’t have to be aligned with the business to show improvement.

In the long run, for organizations which have effective Agile teams these problems will show up. And in time, as more companies achieve Agility, the competitive advantage of Agile Software Development will diminish.

The advantage will move to the companies that address these second order problems. Such companies need to: learn how to improve Agile practices beyond what we currently know, and, get IT aligned with the business.

In an Agile world this means pushing Agile further. Tom would call this Evolutionary, I call it learning, it amounts to the same thing in the end.

Book review: Agile Project Management with SCRUM (and rant)

It is difficult to say anything bad about Agile Project Management with Scrum (Schwaber, 2004). It is a short book, lucid and easy to read. It sticks to its topic - managing projects using scrum. Rather than present Scrum in depth (which I think he has done elsewhere) Schwaber walks us through a set of small case studies and reflections on each one.

If this book has a failing it is: who will read it? - I’m not sure. There is probably not enough information here for someone to implement Scrum but I find it hard to see what someone experienced with Scrum would learn.

For me it was a late introduction to Scrum. I can’t ever recall reading a book dedicated to Scrum. But I picked up the basics of Scrum somewhere, probably websites, articles and conversations. So I would suggest this book is best for someone wanting to an introduction to Scrum, and specifically wanting an idea of how Scrum works in practice.

Which brings me to the question, Why Scrum at all?

I found that Scrum kept coming up more and more and I felt the need to get the introduction I never had so to speak. It is clear to me now (if there was ever any doubt) that Scrum is the management side that is missing from XP.

My own Blue-White-Red process is in effect an example of Scrum and XP (both modified) being used together. Looking back the roots of Blue-White-Red are difficult to work out. I think it was inspired by XP and Scrum and takes many practices form them but it was equally influenced by what I had done in the past and seen work. Perhaps more importantly it was influenced by a bunch of ideas I’d picked up on my MBA, specifically Lean manufacturing.

Unlike Scrum Blue-White-Red is there for anyone to pick up, use, abuse and modify as you see fit. Scrum isn’t, Scrum is actually Scrum (tm), and it is now surrounded by a host of Scrum certifications (Scrum Master, Product Owners, etc.), courses, books, alliances, etc. etc. Scrum has become its own little marketing platform.

In some ways this is good, by doing this Scrum can satisfy business that want to adopt some defined product, it allows Scrum to tackle CMM(I) type organizations and it allows people who want to do Agile to get on courses and learn the skills. But this is also leads to rigidity.

I recently met a Scrum Master who was having trouble with the process. I suggested a one week iteration rather than two week. She saw how this would help but knew the rest of the company used two. She expressed doubts about keeping her meeting schedule (as prescribed by Scrum) in one week. I suggested some changes, at this point she worried that she was deviating from the Scrum process too much.

At the end of this book are the rules of Scrum. Schwaber says: here are the rules, follow them, when you are good at them then (and only then) can the team think about changing them. Scrum needs this, if it was as free as Blue-White-Red it wouldn’t pass muster with CMM(I) and a certificate would be meaningless. But, it also builds in rigidity and people get worried about deviating from the rules.

For me there is an element of The Punk Ideal in Agile, some of it is just about getting our there and doing your own thing, your own way. Who cares if you get it right or wrong? The only mistake is not trying.

At this year’s XP Day conference Keith Braithwaite said something that struck a cord with me, and I’m paraphrasing here ‘cos I don’t remember word for word: ‘I’m not really part of the UK Agile set’ he said, ‘I never worked for Connextra or Thoughtworks.’ And that’s the point, neither of us waited to work for an Agile organization, or to be blessed with a Scrum certificate, we just got on with improving things.

Too many people sit around waiting for something to change, waiting for someone else to fix their problems, waiting for the magic dust that will make everything all right. That’s one reason why so many IT projects go wrong, people wait for someone else to fix it. In part our school system is to blame but that’s why the Punk Ideal is so important. Again, this takes us back to Post Modern Programming. There is no one ‘right’ way of doing this, there are many competing ideas of what right is.

So here is my advice: learn everything you can about XP and Scrum, they are good. Consider them resources to better your understanding. Then go and do Blue-White-Red.

Right now I’m adding two statements to Blue-White-Red:
• Blue-White-Red is open source, I can teach you it but there will never be a certificate
• Blue-White-Red is jazz, it is improvisation, no two performances should ever be the same

If you aren’t doing something different to what I have described you aren’t doing it properly. Like the English language, there is only one rule which is never broken, the rule is: every rule is broken at some time.

IT: Better to be effective or aligned?

In my last blog entry promised to discuss a second piece from the MIT Sloan Review. In many ways I find Avoiding the IT Alignment piece more interesting than the Rettig piece I discussed last time. The Rettig piece was a good argument but it wasn’t clear what you did next, it was insightful without being immediately useful. The second is piece is insightful and has some immediate application - the authors suggest some but I’ll add some of my own below.

Perhaps one reason why this piece is more immediately useful is that the authors (David Shpilberg, Steve Berez, Rudy Puryear and Sachin Shah) are all consultants with Bain therefore they have something to sell, consultancy. Rush out now and hire your Bain consultant! Sorry, I shouldn’t be so negative. Let me give yo a run down of the article and then draw some conclusions of my own.

The article is based on the authors’ experience and a large(ish) survey. You can argue with the underpinnings of the article but I’m quite happy to give it credibility. Like the Rettig piece they are concerned with internal IT projects rather than products for sale (my main area of interest and experience.)

From this survey they are able to divide companies into four categories - a nice 2x2 matrix based on whether a company has effective (or ineffective) IT and whether the IT is aligned with business strategy (or pursuing its own.) These four categories are:

IT Enabled Growth: These companies have highly effective IT groups who are closely aligned to the business. This is what we all want. These guys are doing it. Unfortunately only 7% of companies fall into this category. The benefit to these companies is 35% sales growth over 3 years. Pretty impressive. Perhaps surprisingly these companies spend 6% less on IT than average. So, going it right also means doing it more cheaply.

Maintenance zone: These the basket cases, IT is not aligned with the business and it is not effective. Unfortunately this accounts for 74% of all companies, really depressing. Because this is the bulk of all companies it is perhaps unsurprising that the IT spend in these companies is the average, and their 3 year sales is just slightly below average at -2%.

They are the two opposite, and perhaps we expect that. A few companies maximise value from IT but most don’t. The next two are more interesting:

Alignment trap: this is the combination the authors find most interesting. This occurs when IT operations are aligned to the business (i.e. they are doing what the business wants) but they are not effective. Only 11% of companies fall into this zone - together with the first category this means only 18% of companies actually have IT and the business aligned, quite shocking. These companies spend 13% more than average on IT but the annual growth in sales is 14% below average, a pretty poor showing really.

Such companies are doing the right thing but they are doing it badly. From the figures given this seems to be the worst category to be in, highest costs and lowest sales growth compared to average. Think about that for a minute. These companies have it half right, IT is aligned with the business but they are not effective. These companies would be return better results if they gave up on business alignment and joined the 74% in the maintenance zone.

I would suspect these companies actually make it difficult for IT people to do the right thing. They probably have plenty of processes in place to make sure they don’t do the wrong thing but doing stop people from doing very much at all. My guess is that managers in these companies don’t understand IT and are trying to run IT the same way they would run any other department. Consequently the IT people can’t do their job.

Finally we come to Well Oiled IT: this is the category I find most interesting. These companies are doing IT well, they spend 15% less than average on IT but see 11% higher sales. Only 8% of all companies fall into this group, if we add that to the 7% of companies in the IT enabled growth category we see that only 15% of companies, less than 1 in 7 actually have effective IT.

Statistically, well oiled IT companies are similar to the growth companies but less so, they have significantly lower sales growth but spend even less on IT. Possibly these companies are focusing too much on cost - my bet is they don’t have effective business analysts and product managers in place.

Now for the authors’ advice: they tell companies to focus first on operations not alignment. If companies (and remember most start in the basket case maintenance zone) focus first on alignment they are likely to end up in the alignment trap, where things are worse. Therefore, seek first to make your IT effective then to make it aligned.

I like this advice, but like so man things in the IT world the devil is in the detail. So what do the writers recommend?

• Keep it simple - we all agree in principle but we all get caught out; trouble is IT tends to get more and more complicated, battling complexity is a constant fight
• ‘Right size’ - which is a bit vague but basically they say outsource somethings, keep some things in house but do it consciously and intelligently
• End-to-end accountability - IT managers need the resources to do the job but they have to keep talking to the business, and the business needs to talk to IT

All this advice is sound and it makes for a good starting point.

So now some thoughts of my own.

First this survey supports what I’ve found in practice: most IT operations are basket cases. Few companies can do IT well. In product companies there is natural selection by market forces. If the company is too bad they will (eventually) fail and go out of business. Where IT is a service to the business failure can continue almost indefinitely. Unfortunately this means that at least 85% of people in corporate IT departments will work in failing teams.

Second I now understand why Agile development works so well in many organizations even though it is not aligned with the business. Simply it doesn’t have too. If adopting Agile development moves an organization from maintenance zone a little toward well oiled then it doesn’t matter that it is not aligned with the business. Just improving effectiveness will deliver benefits.

Third, business must take more of an interest in IT. IT can make itself more effective but to become aligned the business needs to understand and get involved with IT - back to the point I was making about the Rettig article and last year about companies that understand IT.

On the whole I find this article good news, it helps me chart a cause for improvement. I see a three stage plan for any failing IT department:
1. Apply well known techniques from the Agile toolkit: this will boost your effectiveness immediately
2. Learn what is special about your environment so you can tailor the Agile techniques and find some of your own. This will further enhance your effectiveness.
3. Continue you learning to bring about business alignment. Often this is going to be a case of removing the old perceptions about how IT departments operate (e.g. Requirements: a dialogue not a document). It also means educating managers and IT staff, and it means creating the roles of product manager and business analyst to facilitate it all.

With 93% of companies failing one way or another there is plenty of work to do.

Now if we loop back to the Rettig article on ERP. Is it any surprise that ERP deployment fails? Only 15% of companies are capable of implementing it effectively, and of these less than half will align it to the business. Of all ERP deployments we can only expect 7% to come close to fulfilling their promises.

An often heard dictum in management concerns the difference between ‘doing the right thing’ and ‘doing it right’. If this article is right, when it comes to IT then it is better to ‘do it right’ than ‘do the right thing.’ Only if you can ‘do it right’ should you even start to think about ‘doing the right thing.’ Operations over strategy. Implementation over design.

Enterprise software doomed?

Regular readers might remember me agonising last year when I subscribed to a pretentious business journal. Well my year’s subscription is about due and the last issue is IT heavy. So, I’ll probably renew for another year. Its not that I read every article in every Sloan Review but I read enough that are of sufficient quality to make it worth the money. Yes it takes time to read an article but the return on investment (financial and temporal) makes reading the journal cost effective compared to books. That is, I get more new ideas and insights from per page than from most books.

There are two articles in the latest issue that are particularly interesting. I’ll leave one for another blog entry and talk here about Cynthia Rettig’s piece ‘The Trouble With Enterprise Software’. This is a well researched and thoughtful piece. It should even win credibility with hardcore coders because it quotes Bjarne Stroustrup and James Noble and Robert Bridle. (Although she forgets the reference for Noble and Bridle;naughty, naughty, wait till I see James. I suspect it is from their notes on post modern programming report.)

Rettig uses ERP software implementations to argue her case but the same argument could be applied to many other IT implementations. ERP is a good example because it shows many of the problems large (internal) IT projects suffer from: multiple stake holders, poorly defined goals, vendor lock in, consultants everywhere, difficulty seeing value, etc. etc.

She argues that basically ERP has failed to fulfil the promise. I has proved expensive, time consuming and locked companies into software, processes and practices. Companies have lost their agility by the very thing that was supposed to enhance it. (She actually uses the word ‘agility’ but doesn’t link it to agile software development or agile anything else so don’t read too much here.)

Further she dismisses SOA as a solution. At the moment a lot of vendors are making a lot of noise, and possibly money, offering SOA as a cure all. I think Rettig is right to be sceptical, those people I know working on SOA problems are finding all the old problems (and a few new ones) just one level higher.

Part of the problem as she points out is that the Lego-brick approach doesn’t work. It is really attractive, like most software engineers I had lego as a kid. (Actually I still do, I just don’t find enough time to play with it, luckily Christmas is coming.) We extend this love of lego building into software. If we could just have simple interfaces... like the 8 studs on a lego brick...

Actually if you look at my lego bricks you will see that only 10% or so are the classic 2x8 blocks. Even before you look at the technical blocs there are 1x1, 1x2, 1x4 ... and windows, and slanted ones, and smooth on one side... although we love this metaphor to build the really cool stuff in lego you need special bricks - and you never have enough of those. The same is true in software, the interfaces get more complicated. The difference is an order of magnitude in complexity.

A further complication Rettig doesn’t mention is our tendency to work at the edge of complexity. As soon as we solve one problem we attempt something more difficult. When we invent technology to solve one problem we find we can use it to do something new. Take XML for example, we could have used it to solve file format problems, instead we are using it to implement function calls (SOAP, XML-RPC) and then SOA on top of that.

We do this because we are chasing innovation and value. If ERP was easy then everyone would have it and it would be no advantage, so you have to do more difficult ERP for it to be worthwhile.

In the end Rettig is pessimistic about our ability to fix the problem, which is nice in some ways. All too often you get into one of these articles and the author says at the end: “the solution is some new technology from my new company.”

Again I tend to agree with her, this stuff is difficult. But I’m not so pessimistic. It is valuable because it is difficult. If it was easy it wouldn’t be worth while. Still, too many companies get into new systems when they should just accept what they have and make it work better.

Rettig does come close to proposing a solution. She correctly notes executives could learn more about IT, and the business schools that teach them could do a better job of educating them about IT. I think this is the answer. Executives need to accept IT is a necessary skill. How many CEO’s would say ‘I don’t understand marketing, I leave that to the Chief Marketing Officer’ or even ‘Accountancy makes my head hurt, I just want the CFO to make it work’ ? But you they can say things like that about IT.

To my mind, more educated executives are a big part of the answer. Which, as it happens, fits with the findings of another Sloan review piece I blogged about last year - Companies who understand IT get the benefits.

This also takes us to the second Sloan piece I want to blog about, but that will have to wait a day or two...

FT piece on business and software agility

The Financial Times carried a piece on Wednesday entitled ‘Agility: Flexibility takes over from planning’. It starts off about agile business but actually spends most of piece talking about agile software development. Both the presence of the piece, and the evidence in it suggests agile is getting more prevalent.

Nice definition of company agility from Professor Donald Sull of London Business School:

“he defines [agility] as a company’s ability consistently to identify and seize opportunities more quickly and effectively than rivals.”

What is interesting about this definition is that it says nothing about what you do, or how you do it. Instead it measures agility by comparison with your competitors.

Slightly disappointing that the examples the article cites are the same examples that come up again and again, Zara, BT and perhaps slightly new, Borland.

If you have the time, this piece from the same issue is also worth a look: ‘Why do so many technology projects end in failure?’ This picks up on the EDS / BSkyB court case I commented on a couple of weeks ago.

Its a shame the editors didn’t link the two pieces. In the long term Agile is about seizing opportunities before your competitors, in the short term it is about delivering IT projects on time, on budget and without court cases.

Reflections on XP Day

I have spent the last two days at the XP Day 2008 conference here in London. A very good conference and highly recommended. Unusually for me I wasn’t speaking at this one which made for a nice change. No worries about presentations, rooms and being in the right place at the right time.

I picked up a whole bunch of ideas, and had another bunch of thoughts. Before I forget them I want to quickly record them - some of these ideas are probably worth an entire blog on their own.

Multi-media keynotes: Both keynote speakers incorporated multi-media elements in their presentation. Jeff Patton included music and Yvonne Rogers video. I think this is a sign of two things: multi-media is becoming the norm; second, people are increasingly bored with the PowerPoint (’death by’) and presenter talking at you (chalk-and-talk) type presentations.

Media companies are more receptive to Agile: A lot of the people attending XP Day had connections with media companies, the BBC and BSkyB were both represented, as were set-top box manufactures, advertising companies and others in the media. Of course there were old industry (pharma, banks, etc.) there too but they have always around.

This is a trend I’ve observed elsewhere. It seems to me that media companies are more receptive to Agile software development than others. There are perhaps two forces at work here. Firstly media companies are fairly new to software as a key part of their business. Yes they always had IT systems but not as part of the product. Agile is the current ‘best practice’ when the BBC, Sky, etc. started to development it was already the norm.

Perhaps more importantly, the second force: media companies expect just in time delivery. You can’t delay the 6 O’clock news because it isn’t ready, you know well in advance you have 30 minutes to fill, and you have to prioritise. The nature of media companies is closer to Agile development so they are more receptive.

Process as a substitute for team work? Of cause Agile software development is all about development process and practice. But scratch deeper and everyone agrees development depends on good people and team working. In fact team working has got more important as software has got bigger and more complex.

It seems to me that a lot of what we call process is actually trying to substitute for team work. We spend time talking about the process rather than the team, we devise process rather than build teams. Somehow we expect the process to substitute for a good team.

Another Agile failure mode: I forget who mentioned this but someone pointed out that Agile teams often fall apart when someone leaves, particularly if that individual is forces to leave by redundancy. This kind of fits with my experience. When an Agile team works well the team bond. The departure of one person is likely to trigger others to leave of their own accord.

Agile Business: This is a topic I’ve been thinking about a lot, so has Chris Matts and others. David Stoughton and Nick Coutts did a great session on the future of business. Not necessarily Agile but a lot of the trends they see in society and the economy suggest the successful business of tomorrow look a lot like Agile organizations.

One point that stuck in my mind above all else was the suggestion that we have moved from a supply economy to a demand economy. Rather than successful companies being those who could produce things the cheapest, and persuade the most people to buy them - a push system - the successful companies of the future will operate a pull system. They will produce what the customer wants, when the customer wants it. Rather than being cost driven they will be value driven.

For my money this was the best session at the conference.

Future of Agile: There are many many software development organizations that are a mess. A little bit of Agile can fix make a great improvement. Agile software development thinking has a long way to run and an awful lot of implementation to deliver. However, Agile has flaws.

One of these flaws is that if you examine in too closely it collapses. The Agile manifesto is actually pretty meaningless, nobody could really disagree with it. If you try and write down a high level value statement of what Agile delivers it sounds pretty much like all the other methodologies and tools that have proved to be snake-oil in the past. Even some of the Agile practices don’t stand up to examination, e.g. pair programming - most developer hate it! Retrospective? - most companies don’t do them.

Agile is still state-of-the-art, the best we have.

It is also a marketing term; a term that collects together a bunch of ideas and allows them to be labelled and grouped together. As a marketing term it is open to abuse.

Increasingly people are increasingly asking: what next?

The people asking this question most often are those with the most experience so some of this questioning is simply boredom, they are looking for the next thing. Some of it is very real and necessary.

(There is some excitement about a new Agile methodology called Kanban, I don’t know much about it and I don’t think this is the next big thing. Something here, but I haven’t had a chance to read it myself.)

So what next?

To me the next big thing will represent a break with Agile. The Agile community came from the Patterns community, Agile is not Patterns. The Patterns community came from the OO community. Patters are not OO. So we shouldn’t expect the next thing to be Agile+.

The next big thing will be People and Team. Software development always comes back to people. Each successive move (OO --> Patterns --> Agile) takes us closer to people. As I said above, sometimes we use process as a substitute, sooner or later we need to address the real issue.

The trick to improving your software development efforts is quite simple: improve your people. Expose them to new ideas, get them learning, help them put those new ideas and learning into action.

More patterns for software companies

I have posted another set of business strategy patterns on my website. These patterns were reviewed at EuroPLoP 2007. It took me a while to incorporate all the comments and then polish them, hence the delay in publishing.

I have now amassed quite a body patterns describing business. At first these were fairly generic, I was writing for any business. Over time I started to think specifically about technical companies and products and now I’ve decided to concentrate on software companies. This is a pragmatic decision, although many of these patterns are more applicable I know about software companies so I can write with knowledge.

These patterns also represent a change in target readership. At EuroPLoP Juregen Salecker pointed out that these patterns would be interesting to people on the receiving end of these strategies and tactics. Yes managers, entrepreneurs and students might find these patterns interesting but really it was those on the sharp end that these patterns are for.

I have another batch of business patterns coming up as a result of VikingPLoP 2007. Hopefully now I have some time I should be able to get these done in a week or two.

My problem, or opportunity, now is that I am amassing a lot of patterns. I increasingly feel the need to pull them all together and look at the whole. Something Cecilia Haskins and Jim Coplien also pointed out at VikingPLoP. Doing this is going to be quite an effort.

And where does that take me?

Well a few people have asked if I would consider turning these patterns into a book. And since two of these people are publishers I need to think about this. However, having just finished one book I’d like a little break... anyone out there interested in a book of business patterns?

Follow up 2: Personal retrospective and strategy

This is the second of two follow ups replying to some comments on the blog. What’s interesting about this one is that its from someone who I don’t know, or rather has chosen to hide their name. Still they have a great memory because they’ve linked something I said a few days ago with something I said last year. Now there’s paying attention! And its good for me because it helps me improve my own understanding.

Anyway this comment asked how I reconciled my comments in the personal retrospective entry where I said I should have done more quicker, with my comment last year (last thing your people need is a strategy) where I suggested managers should spend time observing before acting. Good catch who ever you are.

So, at the risk of sounding like a politician trying to wiggle out a difficult question I’ll try and explain and perhaps I’ll shift my position a little. There is no simple answer so please bear with me...

To some degree it is a question of context. One size does not fit all, and it is important to look at the problems facing an organization before acting. The trouble in the ‘solutions’ oriented world is that you risk applying the wrong solution if you are not careful. Just as it is wrong to apply a predetermined solution before looking at the problem it is wrong to use a predetermined time frame before acting.

Which leads to the question: how do you know what the context is? Well you have to look around, observe what is happening and talk to people. I still believe the people on the ground want to do the best job possible and most likely know what needs doing. So I think its more important to talk to the people doing the tasks than it is to those who nominally control what is going on.

If the people doing the tasks, and those in leadership positions, aren’t seeing the same problems, and aren’t trying to go in the same direction then they positions need to be reconciled. That might mean explaining to the leadership why their model of the world is wrong. Or it might mean explaining to the workers that the company doesn’t want that.

The key point both ways round is not to assume you know all the answers, keep an open mind yourself. Collectively the team will know the answers, and one person may know more than most. I see a manager’s role as unlocking those ideas.

Going back to my personal retrospective post, one thing I didn’t mention was that during my first week at the client I held a reverse retrospective with the development team. This was a little like a project retrospective but since there was no single project to review, and others on the team were also new we had nothing to retrospect about. I call it a “future-spective.”

I gathered the team together for an afternoon and we talked about problems we faced and the problems the management saw. We also talked about improvements we could make and created a long list of things we could do. We then collectively prioritised them and worked out what actions were needed.

One way I would go faster in future is to implement the top priorities on that list faster. I realised when I did my personal retrospective that it took us months to implement some of our ideas, and as a result it took longer to see the benefit and prevented us from moving onto other initiatives.

In future I’ll probably use the future-spective technique but I’ll implement the suggestions sooner. And I’ll go further, I allowed myself to get caught in the ‘no time to improve’ trap in places and after 3 or 4 future-spectives stopped doing them.

As my anonymous commenter pointed out there was quite a gap in the three-month period I talked about last year and the one-week period I talked about recently. To be honest that gap surprises me, I’m not sure why I said 3 months last year, it does seem quite long given what I learned in my personal retrospective. However, I’m also surprised that it only took me a week to diagnose so many issues. So perhaps one week is a minimum and 3 months a maximum, ask me again after I have some more data points.

Again its a matter of context, I’d like to think it is always less than 3 months but I’ll be surprised if its as short as 1 week. In general its is better to delay making decisions until you are sure of the situation but once those decisions are made it is better to act quickly.

One way to ensure that decisions are acted on quickly is to involve more people in making the decision. Although having more people involved may delay the decision when one is made these people share the decision and share the responsibility for acting.

I still stand by my comment that when you are new the last thing your people need is your strategy (actually stole that idea from Lou Gerstner in Who Says Elephant Can’t Dance). However, you do need your own personal strategy. You need to know what you are doing and what you are aiming for.

That personal strategy needs to have a short term element (meet, watch and listen), a medium term element (find problems and opportunities, get peoples ideas, get agreement) and a long term element (do whatever it is you were asked to). And your personal strategy needs to tell you when to wait, watch and learn - something my friend Klaus Marquardt refers to as Proactive waiting.

Follow up 1: Software as an asset

I don’t know how many readers this blog has but I don’t think it s many. In fact I tend to think I know them all. Well, that’s a illusion I can be pretty certain to dispel. I know the ones I know, because they comment on the blog, or they tell me that read it, but by definition I can’t know who I don’t know - if you get my drift. If I now find someone who reads my blog who I didn’t know read it I still don’t know how many reads I have, I just know more of them.

What is gratifying is that although the reader base may be quite small I get quite a few comments. And sometimes I get comments in e-mail not as comments on the blog. So it was that Mick Brooks e-mailed to point out an ambiguity in my blog posting, Software as an Asset, I wrote:

“Your software is worth $10,000,000. Next year, with inflation and depreciation the software may only be worth $9,000,000. So there is an incentive to invest up to $1,000,000 in improving the software, increasing functionality, or code quality, or making it easier to use.”

And Mick asked:

“I don't understand why there is an incentive to invest *up to* $1,000,000 - why
not twice that, or more?”

Well he’s right, I my thinking was sloppy. I skipped several steps in and jumped to a conclusion. The nature of blogging is that it is meant to be fast, what you think of now, not too polished. Anyway, let me think about this a bit slower...

What I was trying to say is: if your software depreciates by $1,000,000 over the course of a year there is a strong incentive to invest in it. At the moment software isn’t carried as an asset on the books, therefore it is worth $0 this year and $0 next year. Therefore, why invest in it?

Now suppose we wish to keep the software worth $10,000,000 each year. We now have to do something to add 10% value to the software just to stand still. Therefore, anything we can do that costs less than $1,000,000 is worth considering. What we really want to do is spend as little as possible to increase the value by as much as possible, i.e. maximise our rate of return.

There are two possible ways value the software. Firstly, remember I imagined a tool that could value our software? Well, what we need to know is what parameters that tool looks at, then by targeting those parameters we can increase the value.

For example, from a product perspective: if adding some new functionality to the system will add value we might do that. Or, from a programmer perspective, if improving the code quality, increasing cohesion and reducing coupling say, will increase the value we could do that. In this way the parameters the tool uses will guide us what changes we make.

The second way we might value the improvements to the software is to look at the effort expended in enhancing it. It is already possible to capitalise the cost of research and development projects, i.e. show expenditure on R&D as an investment on the balance sheet. So, we could argue that spending $100,000 on a programmer to make changes to the software increases the value of the software by $100,000.

We have to be careful with this line of arguing because we could end up counting every bug fix as an asset. WorldCom did something similar with their telephone network and look what happened to them.

Which ever way we value the software the trick is to find the way of increasing the value as much as possible by paying as little as possible - the rate of return. Once you know this you can compare such a project against others in your organization. So, if spending $100,000 to improve the software results in an extra $1,000,000 of asset on the balance sheet it has paid back 10 times.

Conversely, if spending $5,000,000 on a replacement piece of software results in an asset worth $10,000,000 (and the old software is binned) the return is only twice.

But, doing nothing, let the software stand still looks less attractive than it did when the software was not counted as an asset.

Thus, valuing software as an asset gives us a new way of thinking about how we manage the lifetimes of our software.

Back to Mick’s point: my example was a little brief. The real test is whether we can see the value of our software asset increasing faster than other assets. If it is then it is worth investing in. Whether this is a $100,000, $1m a $2m investment.

Personal retrospective

As I said in my previous blog entry I recently finished a major assignment with a start-up client. The company had a lot of problems when I started, and the development group had its share of bad problems. I think I’ve left them in a much better shape than when I arrived.

Last Friday I took some time today to reflect on what it was I did and how I did it. A personal retrospective if you like. This took place between me and my word processor. I just wrote and wrote.... I tried to understand what I did right, that is what I would do again. I tried to understand what I did wrong, things I would avoid in future. And most of all I tried to understand what I would do differently.

To help me I had my personal journal, a diary of events, thoughts and reflections I’d kept during the assignment. The entries were perhaps not as frequent as I would have liked. It started well but after the first few weeks I got very erratic until close to the end. At the time the journal was useful for thinking things through in my own head, understanding what was happening and finding courses of action.

What I found striking was that in the first week I had diagnosed most of the major problems with the development group and company. Some of these I had managed to fix, some remained problems when I left - you can’t fix everything I’m afraid. But it was just striking how much I had worked out so soon.

I normally get sceptical about the expression ‘hit the ground running.’ I’m not sure if anyone really does it. But in this case I’m glad to say I was up and running within a week of getting there.

So what were my big insights from this personal reflection? Two really. Although I consider myself a good communicator (think of this blog, think of the book, etc. etc.) I really could have communicated even more. Particularly upwards.

My second finding: I wish I’d done things faster. There are a few decisions I seemed to hold off doing for weeks but in retrospect worked out for the best.

Any note to myself for the next assignment needs to read: be fearless, go faster.

What is it I do again? (apart from blogging)

I’m at one of those points where I’m considering exactly what it is I do. On the one hand its easy: I manage software development, on the other hand its hard. Its hard because I don’t fall into any of the classic categories we find in software development.

Some of what I do is
• I architect, but I don’t code or write UML; I perform architecture by organising teams and projects. (There is a whole blog entry in that sometime.)
• I project manage, but I’m not really a project manager because I believe that delivering a project involves things beyond typical project management.
• I introduce companies to Agile software development, but I’m not an advocate of any particular methodology.

What I do do, as you can find on my revised website, and the revised Software Strategy website, is:
• Work with the most challenged software development teams to deliver successful projects
• Improve teams so they perform better and deliver more value to their employees
• Align business needs with software development

Why this period of self reflection? Because I have some time on my hands. I have been very involved with a start-up for the last six-seven months and this has now come to an end.

I’m independent, I work through my own company, Software Strategy. I either work as a consultant (I analyse companies and make suggestions) or as an interim manager (I get involved with the running of a department for some months).

I have spent the last seven months working as an Interim Software Development Manager with a company that had a lot of challenges. I help the development team deliver good stuff, I introduced a lot of agile practises. Someone even said I created order out of chaos.

But this has come to an end so I’m looking around for something new. I know the world is full of troubled software groups and I want to help! I really want to help these group improve. Its more than work, it is a passion.

I might not be going to an office each day but I’m still working hard. I’ve done three presentations in the last two weeks, I’m proof reading the final typeset proofs of my book (its almost all over!) and I’ve got some pattern papers to revise or write from scratch.

So if you think I can help your team please let me know. This could be as small as a presentation to your development group (which I did form someone yesterday) or it could be several months turning around a team or introducing Agile.

Book review: Managing the Customer Experience

I decided a couple of months ago that I needed to know more about managing customers. Not necessarily how do you sell, although some of that might be interesting, but more customer account management. Now this is a subject I’ve never really read about before so I was at a bit of a loss to know where to start.

So I hunted around the web a bit, read some Amazon reviews and eventually bought myself a copy of Managing the Customer Experience (Smith & Wheeler, 2002). This wasn’t so much buying a book by its cover as buying a book by review and publisher. Sadly this isn’t the book I wanted.

Rather than talking about how you find your customers, how you satisfy them, how you keep them happy and how you sell more to them it talks about the ‘Branded Customer Experience (R)’. I’m sure the Branded Customer Experience (R) builds to all the things I wanted but its an awfully long way around.

Perhaps the (R) has already given away part of the problem. The guys writing the book have a methodology to sell. Repeatedly uses a registered (R) term in the pages of a book comes across as really pretentious too. In fact most of the advice doesn’t seem that radical or different to what I would expect. I’m haven’t go any real insights from this book.

Plus, the Branded Customer Experience (R) is only concerned with business to business to consumer transactions, I’m more interested in business to business.

So after just two chapters I’m giving up on this book. I’m sure that its a good book if you are looking to create a Branded Customer Experience (R) but I’m not. And I still need to find a good book about customers.

Software as an asset

An interesting piece from Monday’s Financial Times - Study urges IT valuation rethink. What this study says is: IT is an asset, therefore companies should value the asset show it as an asset on the books. This isn’t such a new idea, some people have been arguing this for a while but INSEAD and Micro Focus have written a report and got some publicity for the idea. (See also the Micro Focus press release.)

I have to say I agree with the central argument being made here. IT systems are an asset to a business, they allow the business to operate, they cost money to develop and without them there would be a big hole in the organization. Companies count machines, buildings and even brands as assets but not IT systems. In most companies if the IT system disappeared tomorrow it would need to be replaced and that would cost money.

Actually companies do carry some IT elements as assets - hardware. Mainframes, PCs, printers etc. are bought, counted as assets and devalued over time. So actually, when we talk about putting IT on the balance sheet we are really talking about the software.

Now this has some interesting consequences - not all of which my younger self would be happy to hear. Suppose for the moment software is considered an asset, what happens?

Well firstly you get an increase in your balance sheet. This is good, it shows the company is worth more. It also means that software can be depreciated over time, as with hardware, we can aim to write it off over time. This might make it easier to end of life a system after a period of time. Given that software deteriorates over time this could be a good thing.

However, this also means you can’t dispose of it overnight. How many developers faced with a poor system cry: “We can’t maintain it! We need to re-write it” ? How many managers accept this argument. If software is an asset this decision is changed significantly.

Neither could you throw away failed projects so easily. Many companies bury failed projects. Rather than admit to an embarrassing failure they simply don’t and loose the costs in expenditure. However if you are building an asset and suddenly it disappears then you will have some explaining to do.

For both these reasons (and others) putting software on the balance sheet will make it more difficult to get rid of existing systems. This is the bit my younger self would hate.

When I was still programming I saw lots of really bad software. I always wanted to re-write it and sometimes I did. However rewriting is no panacea. It is often (usually) better for the business to continue with the old system. But in doing so you have to improve it. You have to move it in the right direction or else it will become impossible to work with.

This is what interests me now: how can a business maintain legacy systems? This is the challenge I now embrace as a manager.

Now, if we start counting software as an asset there will be more incentive to maintain the software and keep it alive for longer. This means we will invest in the software to keep the asset. Of course we’re going to need tools to value the software - INSEAD suggest one tool and my guess is Micro Focus can sell you a product to do so.

So what is the net effect?

Well I think this approach would allow better reasoning about the software life-cycle. If a piece of software is only going to be used once then it isn’t going to be an asset and we treat it as such. However if we are going to count it as an asset we can decide whether we are going to devalue it over time, and plan in advance for retirement or replacement. Or if we are going to maintain it and keep (or increase) the value on the books then we need to invest in it. Either way we get to reason about the future of the software more logically.

Now imagine we had a tool that assess software value. Point it at the source code control system, put in some economic numbers and bingo! Your software is worth $10,000,000. Next year, with inflation and depreciation the software may only be worth $9,000,000. So there is an incentive to invest up to $1,000,000 in improving the software, increasing functionality, or code quality, or making it easier to use. That changes the game.

(I’m sure Micro Focus, who specialise in tools for legacy (and particularly COBOL) systems are well aware of this. I for one hope we hear more about this idea.)

We tend to think our software will have a short life. But we already know that some software systems are 20, 30 or 40 years old. (Remember the millennium bug?) These are major engineering achievements.

We take it for granted than major engineering achievements are still there. The Forth Rail bridge was opened in 1890. Nearly 120 years later it still works, it is an asset to the nation. St Pancras station in London was reopened yesterday after refurbishment, originally opened in 1877 it is still an active part of the economy 130 years later and it is an asset to National Rail.

On this basis there is every reason to believe that software being written today will be in in use in 2127. Isn’t it about time we started treating it as such?

Blue White Red - a simple Agile process

My article from last months ACCU Overload is now on my website. This piece describes a simple Agile process which I have used successfully. I don’t claim it introduces anything new or radical. I don’t even claim originality, it derives from SCRUM and XP. All I claim is it worked for me, my teams and has been used in multiple organizations.

What I think is important about Blue White Red is that is shows how you can start to roll-your-own Agile-like process. Start simple with what works for you.

Read the full piece here.

Book review: The Innovators Dilmma

I’m just coming to the end of The Innovator’s Solution (Christensen and Raynor, 2003) - OK, I admit it, I can’t be bothered to read the last couple of chapters. Like so many books the first few chapters are very interesting, then the tend to get a little bit boring. I suppose this is because the first few contain the ideas while the later chapters are more about implementation. Still, I always make a point of reading the last chapter as this usually restores some of the feel of the early chapters.

That said, its a good book and interesting. I suppose more people have heard of Christensen earlier book The Innovators Dilemma. So why did I choose to read this one and not the earlier one? Well I guess because it was a) newer, and b) be seemed to offer a solution where the other offered a problem. Having read this I wonder if I would have been better reading the first book after all. That’s not to say this isn’t a good book.

Clayton Christensen is most famous for giving the world the theory of disruptive technologies. Such technologies are ones which, well, disrupt, the current market. Current market leaders fall, distribution channels change and small companies get to grow fast. All good stuff.

What gets less publicity is the alternative to disruptive technologies, namely sustaining technologies. These are technologies that can be used by the current market incumbents to maintain their market. In this book Christensen and Raynor suggest that in many instances companies can choose to use a new technology as disruptive or sustaining, it depends in part on how you package and market it, what market you are in now and which market you want to be in.

They also suggest that in an established market the incumbents will always win a straight fight with new competition. The only chance a new entrant has is to use their technology as a disruptive influence. Conversely incumbents need to use technology (maybe the same technology) to sustain their position. So if you are a new entrant don’t both trying build a better mouse trap, and if you are an incumbent don’t bother trying to disrupt the market.

What is nice about this book, although it does make it longer in place, is that the authors understand that a different context needs a different solution. Unlike many books which say: faced with problem X you should do Y; these authors say, if you face problem X and you are an incumbent than do Y, if you are a new entrant do Z.

The formula for a new entrant with a disruptive technology seems quite simply. Enter the market at the low end, offer a cheaper product than the current incumbents. Rather than compete head on target people who don’t buy the current products (non-competition) with a cheaper, less functional product. Sell to people who don’t buy the current product or are over served by it. Then as your technology improves expand this base. Quite often the current incumbents will move out of your way because they don’t want the low end of the market - that is if they notice you at all!

Of course I summarise and miss out many of the details but you get the idea. If you want to know more then read the book!

Interestingly much of the book’s key message is shared with Crossing the Chasm, they have the same idea but come at it from different perspectives. The key idea is: start with a small niche and grow it.

The advice in Crossing the Chasm is for small high-tech companies who wish to increase sales of their product. Moore suggests you define your own niche, nominate it then grow the niche. In this case the new technology is assumed to be superior - in some way - to the incumbents.

The advice in Innovator’s Solution is aimed at existing, probably large, corporations who want more innovation. Here the niche is always at the low end of the market, it is always a niche defined by cost and price. In this case the technology is superior through a lower cost structure, therefore price can be lower. Over time the niche is expanded by moving the product up market.

Most of the small software companies I see are so concerned with getting a sale, sometimes at any price, that the idea of niche marketing is foreign to them. Sales (and any marketing) tends to be a shot gun affair where any sale is what counts.

Indeed the whole idea of marketing is pretty foreign. Increasingly I see the existence of well functioning marketing as the key differentiator between successful and not successful technology companies. When I say marketing I mean both outbound marketing - telling people you have a product, advertising, etc. etc. and (the lesser known) inbound marketing - understanding your customers, knowing their problems. For both you need to define your target market.

Trouble is: you don’t need a marketing department to start a high-tech company. You do need techies, scientists and engineers to design and build the product. You do need sales people, someone to get out and sell the thing that has been built. If you lack either of these you don’t have a company. So you always find engineers and sales in any company.

But you don’t need marketing. You can carry on with the sales and engineers for quite a while without marketing. But if you want to grow you either need marketing or you need a lot of luck.

This is a mistake made by many many software companies. Which might just explain why Crossing the Chasm, The Innovators Dilemma and Solution sell so well.

Feeling sorry for EDS - business that don't know what they want

It is not often I feel sorry for EDS. Early in my career I had contact with EDS and I was not impressed. Since then nothing I have heard about them has caused me to change my opinion. So it is pretty unusual that I ever give them the benefit of the doubt, let alone feel sorry for them. But this time maybe I do...

Just at the moment EDS are being sued by BSkyB - part of News International - for failure to deliver on a project. In 2000 BSkyB agreed to pay EDS £48m to develop a new ‘state of the art’ customer service system. Unfortunately the project collapsed in 2003, BSkyB completed the project without EDS at a cost of £265m and are now suing them for £709m. The case is being covered by the FT if you want to know more.

Basically EDS claim that BSkyB didn’t know what it wanted. It seems the requirement was simply: a ‘state of the art’ customer service system to provide BSkyB competitive advantage over their competitors. BSkyB respond that EDS didn’t follow any recognisable project planning system.

I don’t know the rights and wrong of the case, what I do know is that businesses frequently don’t know what they want. I’ve written about this several times before (Software requirements and strategy and Requirements: A dialogue not a document for example). This case is about EDS and BSkyB so it is high profile but the same debate is acted out between thousands of IT providers and their customers every week. Because these names are less well know, and because these companies can’t afford to go to court we never heard about them and some agreement is reached.

This happens internally too. Companies with their own IT groups often get into this mess, the business can’t tell the IT people what they want. Perhaps it is because I’m in London but most of the stories I hear involve banks. Sometimes being part of the same company can help, and sometimes it makes things worse.

Rather than ask whose fault is it? we need to ask How can we fix it?

Well there are two parties here and both need to be involved.

The business side needs to set the overall goals - ‘build a state of the art customer service system to produce competitive advantage’. And the IT side - whether in house or outsourced - needs to implement what was asked for an no more. But in between there is a massive, massive, gap.

• What does a state of the art customer service system look like?
• What is competitive advantage in customer service?
• How do you avoid spending so much on ‘state of the art’ that you loose competitive advantage?

These aren’t easy questions to answer. Neither side can answer them alone and neither side can expect the other to answer them. You need co-operation, you need an on going dialogue.

Business needs to be able to articulate what it wants but it is wrong to think it can articulate everything up front. It can set goals but in achieving those goals there are thousands, indeed millions, of options and decisions to be made. The devil really is in the detail. Business and IT need to make choices together.

Making those decisions requires people in the role (business analysts or product managers), it requires IT people who understand business goals, it requires business people who understand how IT is created and how to recognise the benefits, and it requires a process to keep talking and making decisions.

In this case it is wrong to think BSkyB can just throw the problem ‘over the wall’ to EDS. And it is wrong of EDS to expect fully defined requirements documents up front. But, the nature of competitive business encourages people to take this approach.

I don’t know how EDS and BSkyB structured this project but here some advice:

• Start small: IT projects are often far too big, if you can find the real benefit it then stick to delivering just that, keep it small. Inside every large project is a small one struggling to get out.
• Requirements have to be driven by the customer, who needs to set business objectives and vision. However further elaboration needs to come from close collaboration between the business and the supplier.
• If you intend to build a ‘state of the art’ system, and gain competitive advantage then know your enemy, know what state of the art means and know what other systems do.

Finally, make sure you are building your ‘state of the art’ system for the right reason. Will a state of the art system really deliver competitive advantage? Maybe you don’t need ‘state of the art’, maybe you just need to use an ‘average’ system better. Maybe you don’t need to build a system at all, maybe you can just take one off the shelf. This won’t be cheap - especially if you have to customise it - but it will be a lot cheaper than starting from scratch.

Presentations in Southampton and Cambridge

I should have mentioned this before but my blogging has been erratic at late - blame the ADSL line!

I will be giving presentations to the local ACCU South Coast group this group this week (25 October) and the week after the ACCU Cambridge (1 November). To keep things simple I’m giving the same presentation to both groups.

The presentation is entitled ‘Agile Software Development - Where do I begin?’ As the name suggests it will give some ideas and thoughts on how an organization can start to adopt more Agile practises.

Now for the interesting bit...

I’ve been asked to give a talk to a large media company in London the week after. For this I’m going to give a talk entitled ‘What’s wrong with Agile.’ Talk about contradicting myself!

There is a simple explanation... Agile is good, it describes a better way of developing software, it is a powerful story and introduces some new techniques. But, you shouldn’t just adopt Agile. Ask yourself: what is the problem? Then fix the problem, you may well use some Agile tools but don’t set out to adopt Agile, set out to fix your problem.

VikingPLoP footnote - I won a prize!

A footnote to my comments on VikingPLoP the other week. I forgot to mention: I won a prize!

I was awarded the shepherd of the year prize for helping another author with their paper before the conference. For those of you who don’t know how pattern conferences work a quick word of explanation...

Pattern conferences exist to help pattern authors improve their papers. After a paper is submitted to the conference it is reviewed and, provided it is accepted, the author is assigned a shepherd. The shepherd works with the author for about three months to help improve the paper. At the end of the process it is the revised paper that is submitted to the conference.

At the conference the paper is reviewed in a workshop. The author then uses these comments to produce the third version of the paper. This version is then included in the conference proceedings.

So, shepherding is an important part of the PLoP system and culture. More information on the Hillside website.

It is an honour to receive the VikingPLoP shepherding award, or to give it it’s full title: The Neil Harrison Shepherding Award. Actually, as some readers will know, this is not the first time I’ve won this award. I won the same award at EuroPLoP 2005.

I’d just like to say thank you to the VikingPLoP committee and the author I shepherded this year for awarding me this prize.

New website

A blog is a website but it is not a website. It is a website in the sense that it is a destination on the web but the contents of a blog are by their nature different to a website. Years before I started this blog I set up my own personal website, www.allankelly.net. With the creation of this blog I have neglected my website and it has become old and dated.

So it is with great pleasure I announce the rebirth of allankelly.net. I’ve used a new (Mac) package to create a new website which is mainly an archive of my writing projects. Here you will find over five years of articles from ACCU Overload, patterns from seven or more conference, presentations from conferences and elsewhere, and various other stuff.

For the moment I’m keeping my company website, softwarestrategy.co.uk separate. I’ll be looking at this in future and may merge the two. After all, three websites is probably enough for anyone.

Something is stirring in the office software market

As a PC user there was only one game in town for office software - Microsoft Office. Yes there was other software but very few people used it. Everyone used Word, Excel, etc. and if you used anything else people thought you were strange.

Since moving to the Mac I’ve discovered the hegemony doesn’t exist here. I’ve discovered people who use NeoOffice and as Mark points out in his comment today there is Apple software too.

Funny thing is I first met Mark when we both worked for a company which produced UNIX based office automation software, Quadratron. In fact the company had made a good job of missing the PC market and realised it needed to catch up. Somehow Mark, Adrian, myself and a couple of others weren’t going to make it happen. Once it became clear the company had no future we jumped ship.

But now it appears the office automation market might be on the move again.

Yesterday I used the Google spreadsheet and word processor for the first time. In fact I was using them in collaboration mode with Till Schuemmer as we started the process or organising EuroPLoP 2008. They worked well - despite my very slow ADSL connection - and I was impressed. However shortly after we finished the ADSL connection collapsed again so while Google Apps might be the future they are not the present.

Also this week I learned that IBM is launching, or maybe re-launching, a office automation suite called Symphony. Lotus Symphony was an integrated office package in the days of 8Mhz 80286 CPUs with 640k RAM. It worked but not well enough too make much of a mark.

Why has IBM done this?
Will Google Apps (or ADSL) ever be stable enough to challenge desktop software?
Will NeoOffice or Apple displace Microsoft on the Mac?

I don’t know the answers to these questions but I do see the start of a change.

VikingPLoP and Software Engineering Radio

Two weeks ago I was at VikingPLoP in Bergen, Norway. I have never been to Norway before so I enjoyed the trip. Bergen is absolutely beautiful, it has to be one, if not the, most beautiful towns I have ever visited.

Anyway, a bunch of interesting things came up.

Personally I had another paper with business patterns work-shopped - as soon as I’ve done the changes I’ll post it online.

There was a lot of discussion about VikingPLoP 2008. All is not well in the VikingPLoP world. VikingPLoP is a small (less than 20 people) conference and it moves each year. This raises a number of problems and it is not my place to say any more. However as a result of these problems I think BritPLoP has taken a step closer.

BritPLoP is the name of a conference that doesn’t exist. It is an idea Kevlin Henney and myself sometimes discuss - usually after a few beers. If it ever takes place it might be branded as VikingPLoP, it would most probably occur in late September and would be the first true patterns conference to occur in the UK. It won’t happen in 2008 but I think it might well happen in 2009 now.

Also at VikingPLoP I was fortunate enough to meet Bob Hammer. Bob is president of the Hillside Patterns group and a really interesting guy. Like so many people in the patterns community he software engineer with a multitude of interests, one of which is book-binding.

Bob has written a new book entitled Patterns for Fault Tolerant Software which should be available in the next few weeks. I’ve seen some of these patterns before and even used a few. But until now I didn’t know Bob was behind them. I’m sure this book will be a major contribution to software engineering knowledge.

Although I don’t do much coding any more I’m going to buy this book. I know I’m going to learn a lot of new idea and techniques. And if nothing else the book includes one of my favourite patterns of all time: Leaky Bucket.

When we were in Bergen Bob was just waiting on the print run but in the meantime he had printed the final proofs and hand bound them. This makes him the first person I have ever met to have: written a book, printed a book and bound a book!

Bob also told me about Software Engineering Radio. This is an internet radio station from Markus Voelter and a few others. I haven’t had time to check out all the podcasts but the ones I have listened too are impressive. This looks like a great new way to spread software knowledge.

Still no ADSL - the state of the software industry

We are now in week 4 without ADSL. Things have improved a little, we have a 192kbp (yes kilo, not mega) link most of the time but it can disappear at any time. If I unplug the router I probably won’t see it again for several days. If we stretch the bandwidth too much it will probably collapse and it will be absent for some days.

The combination of BT, or rather their wholesale unit BT OpenReach, my ISP (Eclipse) and some unknown supplier who sits between them seems to make resolution impossible. As far as I can tell the fault is just shuffled between them. Eclipse would like to get the fault fixed I’m sure but they are, as far as I can tell, quite powerless.

I phone Eclipse just about every usually there is no progress. Sometime I have to reboot the router or switch it off altogether for a day. The it takes about 48 hours before BT do the next thing. Then we’re back to square one.

Turns out BT offer no service level agreements for broadband. It is provided on a ‘best efforts basis.’ For something that has become so essential this isn’t really acceptable.

It seems to be another sign that for all our technical progress the reliability of services is decline. Take digital television, my Philips digital box regularly crashes and requires a reboot. At other times it just goes unbelievable slow.

Fixed line phones are still as reliable as ever but mobile phones can still be poor quality, lines can get dropped and messages lost. While Skype offers superior sound quality its reliability is even worse.

Increasingly we (the population as a whole) seem prepared to trade reliability and quality of service for new features.

Unfortunately I take this as a failure of my own industry, software development. Our ability to developer fancy software is far greater than our ability to produce reliable and bug free software. Our solution to this dilemma seems to be to change the entire world, instead of fixing our quality problem we will just persuade people that this is as good as it gets and they can live with the bugs.

That is what I find so very very sad.

I’m doing what I can. Fortunately in my area I have a choice. To my disbelief I’ve now contracted with NTL to install cable broadband and telephone to replace BT and Eclipse. NTL are now branded Virgin Media but they are traditionally a by-word for poor customer service.

If it works - and it is two weeks before they can install - then I’ll get a faster broadband connection and pay less. Hard to believe, I hope it works. For once the devil I don’t know seems preferable to the devil I know.

Agile failure modes - again

Back in April and May I discussed failure modes in Agile software development. Well I’ve found another. That is to say I’ve found another way in which an Agile development practise can go wrong. I’ve seen this before but I didn’t recognise it as a failure mode, I saw it as a little local difficulty now I’ve seen it a second time I understand it a bit better. The trouble with this failure is that at first it looks a lot like what should be happening.

One of the core ideas behind Lean and Agile is improving quality of code. Improving code quality enabled several other benefits. Improved quality means fewer bugs, so future development is less disrupted because there is less rework. Improve quality means less time spent in test at the end of a development. Thus short iterative cycles become possible. And improved quality means you can deliver (or just demo) any time. So quality improvement is important.

One way in which we seek to improve quality is through automated unit testing. This is usually undertaken as Test Driven Development or Test First Development. Now there is a whole host of reasons why it can be difficult but that is another blog. With automated unit tests fine grained functionality can be tested very rapidly which provides for rapid regression testing and increased confidence in new code.

This is where the failure comes in.

We actually want the tests to fail if we change something because they prevents errors slipping into the system. So we add a test for new code, write the new code and by the time we check the code it is finished and the test pass. But, if later something changes which will break the code then we want the test to fail in order to catch the error that has been introduced. In this case we want the test to be fragile so that it catches anything which is now wrong.

The problem occurs when the test is fragile in the wrong way. In the case I’ve just observed the test was fragile when data in the database changed. Previously when I saw this the tests were fragile around COM. The test was/is not sufficiently isolated from its environment.

True dyed-in-the-wool Agile/TDD practitioners will immediately jump to say: you shouldn’t do it that way. And they are right. These fragility points should be addressed. This might be by stubbing code out, using mock objects or by tearing-down and restoring some element - such as a database.

As a consequence instead of the tests helping you validate code and move faster they become a hindrance. The need to update and change the tests regularly adds to the work load and complexity. Instead of the test showing everything is good they add to the maintenance burden, the exact opposite of what was desired.

The problem is that those trying to take the Agile/TDD approach have good intentions by they don’t have the experience, but how do you get the experience without doing it?

Well there are three answers here: first the organization should have invested in training for these people. Second the organization could have hired someone with experience to help them, perhaps as a part-time coach or perhaps as a full time co-worker. And third, the management could have helped the teams learn faster and better and encouraged them to reflect more on what they were doing.

Trouble is Agile development is often introduced bottom up, so people do just jump in. And even where management introduce it then such training and coaching can be expensive - that is if you know who to call and where to buy it in the first place. (If you don’t know who to call then let me know, I know people who do this stuff.)

So although it is well intentioned and the ‘just jump in’ / ‘just do it’ approach can get you started but it can also start building up bigger problems further down the line. If you know what to look for you can spot the problems and - at least try to - take corrective action. If you don’t then it is quite likely you’ll give the whole Agile experiment a bad name,

No ADSL and my Mac - a reminder of how it used to be

The last two and a bit weeks have been hard: our broadband ADSL failed over two weeks ago. We’ve had a little intermittent service but not much to speak of. Lots of calls to the provider and them in turn with BT.

In my opinion BT are still a big bad monopoly. Turns out they only provide ADSL on a ‘best effort’ basis. No service level agreements. Anything they need to do takes 48 or 72 hours. Most of the delay has been my ISP arguing with BT and me waiting for BT to do something.

I know realise how dependent we are on broadband. Even our fridge started to run out of food because we usually buy our groceries on line and have them delivered.

Interestingly while a lot of my neighbours have wireless access all the networks are now secured. That is a change in the last year.

To make things worse I have no modem on my Mac. When I was buying it I thought ‘when do I ever use a modem?’ And talking of my Mac....

It is nearly two months now since I switch to a Mac. I’m still glad I made the move but I’ll admit there are a few things about the Mac which are annoying me - lack of short cut keys is probably the most obvious.

The Mac is simpler to use but a lot of this simplicity comes because you simply can’t do the things you can with a PC. The Mac makes a lot of assumptions about how you want to work. That makes it simpler but when there is something you don’t like it is hard to avoid. For example, I’d really like a bigger mouse pointer but I can’t find any option to change the size of the icon.

My productivity has definitely fallen. This is largely because I have to learn, or re-learn software applications. It is also because I have to hunt down software to use. I’ve had to give up BlogJet so I had to find another piece of blogging software. After looking around the net I chose MacJournal. Its good enough but it does blogging as a secondary function. It is really journal software. The people who made BlogJet have now produced a Mac product but this too seems aimed at those keeping a journal.

I tried to avoid this problem in part by buying Microsoft Office for Mac. But it turns out the Mac version is quite different to the PC version so I’m having to relearn a lot of it. I used to use Visio on the PC but there is no Visio on the Mac so I’ve had to spend time looking for a Mac equivalent.

So I’ve been evaluating OmniGraffle and ConceptDraw. Both lack the extensive sencils of Visio and some features but both have other features. Annoyingly neither integrates to Word very well. I can’t insert a picture the way I can on a PC, and if I export a graphic picture it tends to appear blurred in Word - at least when I print it out. If I want a good image I have to use Apple PICT format which apparently is an old Apple format. Seems Microsoft haven’t updated Word.

Which is why being on the Mac feels like going back 5 or 10 years. I’ll probably buy OmniGraffle quite soon but in the mean time I keep getting e-mails from OmniGraffle and ConceptDraw making me offers and asking my views. I mentioned the integration and graphics issues to Omni Group who surprisingly replied in person. That’s how I know the image and integration problems are Microsoft and Apple issues.

And while I’m talking about Mac software I should mention RapidWeaver from RealMacSoftware. This is replacing my old copy of DreamWeaver and CityDesk.

I could solve some problems - like the image problem - by dumping Office. Apple actually have a word processor of their own, Pages - although I didn’t realise it was a word process for six weeks. But Pages doesn’t integrate with Endnote the software I use for tracking references. So you solve one problem and get another.

Perhaps the reason this all feels like going backwards is because I have to find and learn new software. With only a couple of exceptions I haven’t done this for most of the last 10 years. The other reason is because I’m finding a whole new software ecosystem.

But actually all this gives me hope.

In the Mac world Office does not dominate. Microsoft doesn’t supply everything and there are many small software companies carving out their own niche. Pages may be trying to compete with Word head on but MacJournal and BlogJet prove that there is a living to be made from producing niche specific word processes. Just when I thought word processes were a commodity I find they are not.

That all gives me hope for the future - and makes me wonder if I’m missing a niche somewhere.