Christmas reading: Classic essays on software development

There are many, many books about software development. The technical ones alone (e.g. Java, C++, Apache, .NET, etc. etc.) take up meters and meters of shelf space. The ones about project management take up a bit less space but they are still plentiful, and the ones about people in the process take up even less space but are longer lasting.

It is the later category that interests me most theses days. Some books seem perennially relevant and may never go out of print, books like:

• Mythical Man Month by Fred Brooks
• Psychology of Computer Programming by Gerry Weinberg
• Peopleware by Tom DeMarco and Timothy Lister

But often you don’t need a full book to make a point. On these occasions an essay or journal article will do. Often they make the point better simply because they are shorter. Unfortunately these tend to be overlooked too often - books get all the publicity. I would like to highlight half a dozen essays which I think are perennially relevant and often overlooked. In some cases this is because the articles can be hard to get hold of or, more likely, people don’t know they exist.

1. How do committee invent? by Melvin Conway - the origin Conway’s Law. Although written in 1968 this piece is truer than ever. Arguments made by Conway explain much of what actually happens in software development. I’ve examined and looked at this subject myself elsewhere.

Conway suggests that organization that create systems (not specifically software systems) will create copies of their organizations. So, if you have one developer writing the entire system it will be all entangled, few interfaces and difficult for anyone else to work with. And if you have half a dozen developers who don’t communicate you will get six different coding styles, and six versions of most common functions.

What Conway didn’t foresee is that many of system now create the reverse force. Organizations are limited by the systems they use, thus their structures become copies of the systems in use.

2. Worse is Better by Richard ‘Dick’ Gabriel - this started as a keynote talk, became an essay and then a series of letters in which Dick argued anonymously with himself.

If you are one of these developers who think the best code will always win the day you need to read this. Dick suggests that sometimes the solution which is technically worse is economically superior. In a similar vein see Brian Foote’s Big Ball of Mud pattern.

3. No Silver Bullets from the author of Mythical Man Month, Fred Brooks. This piece came out in the mid 1980’s. The Wikipedia summary is here, otherwise you will need to by the Anniversary edition of Mythical Man Month to read it.

Brooks argues that software development is hard, and there is no ‘silver bullet’ which will make it easier. No language, no operating system, no methodology or any other tool will fix it. I have been told that Brooks has since said the if there is a Silver Bullet it is Agile Software Development but I can’t find this quoted anywhere so I don’t completely believe it.

4. A Rational Development Process and How to Fake It by Dave Parnas. Originally published by the IEEE this also appear in Software Fundamentals: The Collected Papers of David L. Parnas. You might also find some copies stashed on the Internet. Dave Parnas deserves to be more widely read by software engineers today so it you haven’t read much of his work buy the collected papers, well worth reading.

In this paper Parnas argues that while we may want to use a rational development process, and while it makes sense to do so it is impossible. This is because software development requires intuition and inspiration - or as I would put it learning. And since you can’t schedule these things or guarantee they will happen the chances of following a rational process are negligible.

However, in order to make our work accessible to those who come after us we need to fake the development process so it looks like we followed a rational process.

I like the logic, I agree with much of it but I have one disagreement. Each generation of software engineers comes along and believes the previous generation were rational and get confused by what they find. They slowly learn that they cannot be rational - causing a lot of angst. They also wonder how the previous generation made such a mess. So frustration is built up.

5. Cathedral and the Bazaar - by Eric Raymond: an explanation of Open Source software. And an explanation of why software created in corporations is frequently a mess. I don’t think Open Source is the answer to all our problems, but I think it is an answer to some and is here to stay.

6. Enrolment Management by Peter Conklin: until recently this was one you really have to hunt down, but it is worth it. Originally published in the Digital Technical Journal (Digital as in DEC or Digital Equipment) in 1992 it was republished by the IEEE in July 1996. HP now have back issues of the Digital Journal online so you can read it here.

This piece tells the story of the Alpha AXP programme in the late 1980’s and early 1999’s. In theory, on paper, on GANTT charts the project couldn’t be done. The author tells how enrollment management was used to engage those working on the project. Although written in 1992 this paper is about Agile development. And it is about Agile development in the large - over 2,000 engineers worked on the project.

I wrote about this article before, a few years go. One of the small stories in this paper always brings a smile to my face. The OpenVMS team committed to a delivery schedule and said ‘We don’t know how to achieve this, but we commit to finding a way.’

So there you have it. Six essays I recommend you read. And if you only have time for one? Well, chances are you know about Open Source and the Silver Bullet so I recommend you read Enrolment Management or A Rational Process, and if you push me more I’ll say: Enrolment Management.

Busy time - and the Software Development Practice Review

Despite the fact that I have deliberately taken a couple of months off since the end of my last assignment I’ve been keeping very busy. I’ll be glad when Christmas comes and I can leave the laptop switched off for a few days.

In addition to finishing up a couple of personal projects (the book, various patterns papers and other stuff) and giving some talks and briefings I have been involved in an attempted rescue. OK, nobody was in real danger this was more a corporate rescue.

I drafted a turn-around plan for a .com in trouble. For a while it looked like the plan would be adopted but now it looks doubtful. Seem the management have gone off the idea. Shame but so be it, it was interesting to analyse the company and make suggestions.

I have also been busy working on my own product offerings. I have today revised my website to make it clearer what I do. In addition to regrouping some of my services (training, briefings and workshops) I’ve also created a Software Development Practice Review product - SDPR for short.

I’m particularly proud of this SDPR. It has taken a few weeks to bring together the framework but the basic idea is simple. I want to be able to visit a development organization (a team or department or such) and evaluate their software practices. Notice I say practices, not processes. This is deliberate. Most of the other reviews I’ve looked at consider processes, I think practices are more important because it shows you what people are actually doing, not what they are supposed to be doing.

The framework behind the SDPR starts by considering teams (size, structure, experience, etc.), the management of the teams, how the business communicates its need to the teams and how risk is managed. After this it turns to what the team is doing, technology, routines, practices, release management, testing and a little bit about processes. Finally the framework considers what happens after a software release and how the team learns.

From this review I can create a report designed to seed team improvement and stimulate the various members to try new practices and ideas. The second key difference I see with many other types of review is that this review is intended to help the teams improve, it is not intended to measure them on ‘pass/fail’ or suitability to undertake work. It is there to help improvement.

Overall I get very excited about this review. I hear of so many development teams which could do better, some of which are almost dysfunctional and yet fixing them wouldn’t be very difficult. Hopefully this is a tool to do just this.

I’m now talking to some people about trying the framework out in January. If you think your team could benefit from this review, or you know someone who would find it useful please get in touch.

Business alignment, Agile failure and the long run

Tom Gilb sent me a link to this presentation from Ryan Shriver about the use of Scrum with Tom’s EVO techniques. Very interesting, and it returns me to the subject of IT/business alignment - which in part is also a discussion of where Agile software development is wrong.

My blog from last week touched on this but I think its worth spelling this out. There are two problems with Agile, and two answers:

Question 1: Why do Agile techniques work? - forget the technical explanations of why TDD is a good thing, why RUFD is preferable to BUFD or how value stream mapping helps. Think for a moment: there are a lot (an awful lot) of software development groups which are not effective.

Problem 1: How can it be that the same set of (Agile) practices can help all these groups?

Answer 1: Because these groups all have similar problems, probably because too many people were taught (and believed) the same set of practices that have created the problems.

Question 2: How can Agile work without business alignment? The basic Agile techniques pay little attention to what the business wants. The XP onsite customer is really lightweight, XP assumed the customer knows what they want but often they don’t. Scrum is better but the product owner is still left to their own devices, Scrum says little about how the product owner knows what’s what.

Problem 2: How can Agile be effective without business alignment?

Answer 2: It doesn’t have to be. The MIT Sloan Review piece shows that just getting to be effective is such a massive improvement over being ineffective that it is a success. You don’t have to be aligned with the business to show improvement.

In the long run, for organizations which have effective Agile teams these problems will show up. And in time, as more companies achieve Agility, the competitive advantage of Agile Software Development will diminish.

The advantage will move to the companies that address these second order problems. Such companies need to: learn how to improve Agile practices beyond what we currently know, and, get IT aligned with the business.

In an Agile world this means pushing Agile further. Tom would call this Evolutionary, I call it learning, it amounts to the same thing in the end.

Book review: Agile Project Management with SCRUM (and rant)

It is difficult to say anything bad about Agile Project Management with Scrum (Schwaber, 2004). It is a short book, lucid and easy to read. It sticks to its topic - managing projects using scrum. Rather than present Scrum in depth (which I think he has done elsewhere) Schwaber walks us through a set of small case studies and reflections on each one.

If this book has a failing it is: who will read it? - I’m not sure. There is probably not enough information here for someone to implement Scrum but I find it hard to see what someone experienced with Scrum would learn.

For me it was a late introduction to Scrum. I can’t ever recall reading a book dedicated to Scrum. But I picked up the basics of Scrum somewhere, probably websites, articles and conversations. So I would suggest this book is best for someone wanting to an introduction to Scrum, and specifically wanting an idea of how Scrum works in practice.

Which brings me to the question, Why Scrum at all?

I found that Scrum kept coming up more and more and I felt the need to get the introduction I never had so to speak. It is clear to me now (if there was ever any doubt) that Scrum is the management side that is missing from XP.

My own Blue-White-Red process is in effect an example of Scrum and XP (both modified) being used together. Looking back the roots of Blue-White-Red are difficult to work out. I think it was inspired by XP and Scrum and takes many practices form them but it was equally influenced by what I had done in the past and seen work. Perhaps more importantly it was influenced by a bunch of ideas I’d picked up on my MBA, specifically Lean manufacturing.

Unlike Scrum Blue-White-Red is there for anyone to pick up, use, abuse and modify as you see fit. Scrum isn’t, Scrum is actually Scrum (tm), and it is now surrounded by a host of Scrum certifications (Scrum Master, Product Owners, etc.), courses, books, alliances, etc. etc. Scrum has become its own little marketing platform.

In some ways this is good, by doing this Scrum can satisfy business that want to adopt some defined product, it allows Scrum to tackle CMM(I) type organizations and it allows people who want to do Agile to get on courses and learn the skills. But this is also leads to rigidity.

I recently met a Scrum Master who was having trouble with the process. I suggested a one week iteration rather than two week. She saw how this would help but knew the rest of the company used two. She expressed doubts about keeping her meeting schedule (as prescribed by Scrum) in one week. I suggested some changes, at this point she worried that she was deviating from the Scrum process too much.

At the end of this book are the rules of Scrum. Schwaber says: here are the rules, follow them, when you are good at them then (and only then) can the team think about changing them. Scrum needs this, if it was as free as Blue-White-Red it wouldn’t pass muster with CMM(I) and a certificate would be meaningless. But, it also builds in rigidity and people get worried about deviating from the rules.

For me there is an element of The Punk Ideal in Agile, some of it is just about getting our there and doing your own thing, your own way. Who cares if you get it right or wrong? The only mistake is not trying.

At this year’s XP Day conference Keith Braithwaite said something that struck a cord with me, and I’m paraphrasing here ‘cos I don’t remember word for word: ‘I’m not really part of the UK Agile set’ he said, ‘I never worked for Connextra or Thoughtworks.’ And that’s the point, neither of us waited to work for an Agile organization, or to be blessed with a Scrum certificate, we just got on with improving things.

Too many people sit around waiting for something to change, waiting for someone else to fix their problems, waiting for the magic dust that will make everything all right. That’s one reason why so many IT projects go wrong, people wait for someone else to fix it. In part our school system is to blame but that’s why the Punk Ideal is so important. Again, this takes us back to Post Modern Programming. There is no one ‘right’ way of doing this, there are many competing ideas of what right is.

So here is my advice: learn everything you can about XP and Scrum, they are good. Consider them resources to better your understanding. Then go and do Blue-White-Red.

Right now I’m adding two statements to Blue-White-Red:
• Blue-White-Red is open source, I can teach you it but there will never be a certificate
• Blue-White-Red is jazz, it is improvisation, no two performances should ever be the same

If you aren’t doing something different to what I have described you aren’t doing it properly. Like the English language, there is only one rule which is never broken, the rule is: every rule is broken at some time.

IT: Better to be effective or aligned?

In my last blog entry promised to discuss a second piece from the MIT Sloan Review. In many ways I find Avoiding the IT Alignment piece more interesting than the Rettig piece I discussed last time. The Rettig piece was a good argument but it wasn’t clear what you did next, it was insightful without being immediately useful. The second is piece is insightful and has some immediate application - the authors suggest some but I’ll add some of my own below.

Perhaps one reason why this piece is more immediately useful is that the authors (David Shpilberg, Steve Berez, Rudy Puryear and Sachin Shah) are all consultants with Bain therefore they have something to sell, consultancy. Rush out now and hire your Bain consultant! Sorry, I shouldn’t be so negative. Let me give yo a run down of the article and then draw some conclusions of my own.

The article is based on the authors’ experience and a large(ish) survey. You can argue with the underpinnings of the article but I’m quite happy to give it credibility. Like the Rettig piece they are concerned with internal IT projects rather than products for sale (my main area of interest and experience.)

From this survey they are able to divide companies into four categories - a nice 2x2 matrix based on whether a company has effective (or ineffective) IT and whether the IT is aligned with business strategy (or pursuing its own.) These four categories are:

IT Enabled Growth: These companies have highly effective IT groups who are closely aligned to the business. This is what we all want. These guys are doing it. Unfortunately only 7% of companies fall into this category. The benefit to these companies is 35% sales growth over 3 years. Pretty impressive. Perhaps surprisingly these companies spend 6% less on IT than average. So, going it right also means doing it more cheaply.

Maintenance zone: These the basket cases, IT is not aligned with the business and it is not effective. Unfortunately this accounts for 74% of all companies, really depressing. Because this is the bulk of all companies it is perhaps unsurprising that the IT spend in these companies is the average, and their 3 year sales is just slightly below average at -2%.

They are the two opposite, and perhaps we expect that. A few companies maximise value from IT but most don’t. The next two are more interesting:

Alignment trap: this is the combination the authors find most interesting. This occurs when IT operations are aligned to the business (i.e. they are doing what the business wants) but they are not effective. Only 11% of companies fall into this zone - together with the first category this means only 18% of companies actually have IT and the business aligned, quite shocking. These companies spend 13% more than average on IT but the annual growth in sales is 14% below average, a pretty poor showing really.

Such companies are doing the right thing but they are doing it badly. From the figures given this seems to be the worst category to be in, highest costs and lowest sales growth compared to average. Think about that for a minute. These companies have it half right, IT is aligned with the business but they are not effective. These companies would be return better results if they gave up on business alignment and joined the 74% in the maintenance zone.

I would suspect these companies actually make it difficult for IT people to do the right thing. They probably have plenty of processes in place to make sure they don’t do the wrong thing but doing stop people from doing very much at all. My guess is that managers in these companies don’t understand IT and are trying to run IT the same way they would run any other department. Consequently the IT people can’t do their job.

Finally we come to Well Oiled IT: this is the category I find most interesting. These companies are doing IT well, they spend 15% less than average on IT but see 11% higher sales. Only 8% of all companies fall into this group, if we add that to the 7% of companies in the IT enabled growth category we see that only 15% of companies, less than 1 in 7 actually have effective IT.

Statistically, well oiled IT companies are similar to the growth companies but less so, they have significantly lower sales growth but spend even less on IT. Possibly these companies are focusing too much on cost - my bet is they don’t have effective business analysts and product managers in place.

Now for the authors’ advice: they tell companies to focus first on operations not alignment. If companies (and remember most start in the basket case maintenance zone) focus first on alignment they are likely to end up in the alignment trap, where things are worse. Therefore, seek first to make your IT effective then to make it aligned.

I like this advice, but like so man things in the IT world the devil is in the detail. So what do the writers recommend?

• Keep it simple - we all agree in principle but we all get caught out; trouble is IT tends to get more and more complicated, battling complexity is a constant fight
• ‘Right size’ - which is a bit vague but basically they say outsource somethings, keep some things in house but do it consciously and intelligently
• End-to-end accountability - IT managers need the resources to do the job but they have to keep talking to the business, and the business needs to talk to IT

All this advice is sound and it makes for a good starting point.

So now some thoughts of my own.

First this survey supports what I’ve found in practice: most IT operations are basket cases. Few companies can do IT well. In product companies there is natural selection by market forces. If the company is too bad they will (eventually) fail and go out of business. Where IT is a service to the business failure can continue almost indefinitely. Unfortunately this means that at least 85% of people in corporate IT departments will work in failing teams.

Second I now understand why Agile development works so well in many organizations even though it is not aligned with the business. Simply it doesn’t have too. If adopting Agile development moves an organization from maintenance zone a little toward well oiled then it doesn’t matter that it is not aligned with the business. Just improving effectiveness will deliver benefits.

Third, business must take more of an interest in IT. IT can make itself more effective but to become aligned the business needs to understand and get involved with IT - back to the point I was making about the Rettig article and last year about companies that understand IT.

On the whole I find this article good news, it helps me chart a cause for improvement. I see a three stage plan for any failing IT department:
1. Apply well known techniques from the Agile toolkit: this will boost your effectiveness immediately
2. Learn what is special about your environment so you can tailor the Agile techniques and find some of your own. This will further enhance your effectiveness.
3. Continue you learning to bring about business alignment. Often this is going to be a case of removing the old perceptions about how IT departments operate (e.g. Requirements: a dialogue not a document). It also means educating managers and IT staff, and it means creating the roles of product manager and business analyst to facilitate it all.

With 93% of companies failing one way or another there is plenty of work to do.

Now if we loop back to the Rettig article on ERP. Is it any surprise that ERP deployment fails? Only 15% of companies are capable of implementing it effectively, and of these less than half will align it to the business. Of all ERP deployments we can only expect 7% to come close to fulfilling their promises.

An often heard dictum in management concerns the difference between ‘doing the right thing’ and ‘doing it right’. If this article is right, when it comes to IT then it is better to ‘do it right’ than ‘do the right thing.’ Only if you can ‘do it right’ should you even start to think about ‘doing the right thing.’ Operations over strategy. Implementation over design.