Developers are not the only fruit

My younger self would be horrified to hear me say this but, when you develop software you need people who are not software engineers.

Don’t get me wrong, coders/engineers/programmers are the most important people because these are the people who actually produce the thing. As such they are central to any development effort, you can’t write software with a collection of managers, analysts and testers any more than you can build a ship without shipwrights.

Coding is where the metal is bent, shaped and welded. When coding there is no room for ambiguity or fudge, the code doesn’t lie and it doesn’t take to ambiguity. Thus there is a lot of details in coding which aren’t obvious before you start. So your developers need to be able to call on Product Managers for more detail and, if you have any architects, you need them at the code face coding with your developers.

See Jim Coplien and Neil Harrison’s patterns Work Flows Inward and Architect Also Implements for more discussion of this topic.

One of the worst projects I ever worked on was completely top heavy with managers (programme, project, customer, delivery, team - you name it we had a manager for it), testers, analysis, admin staff - yes you needed a lot of admin staff when the project was over 120 people. But only about 12 actual coders. Crazy. The overhead was there because... well, the ISO-9000 procedures needed a lot of managers, and it gave the consultancy implementing the project a lot of bodies to bill for, and it gave the Government (the people who were ultimately paying for it all) a lot of reassurance that it would be done one time, to budget and spec. Except, it was late, the spec trimmed left, right and centre, and if there ever was a budget we blew it.

So you see I’m no fan of carry extra weight on a project.

But, you need people who don’t code.

I’ve written before about the role of the Product Manager and why product management is important, and why you need them. So lets say we accept that Product Manager are needed.

In an idea world we wouldn’t need software testers either. And I expect one day to work somewhere where there are no testers. I don’t believe this is an impossible dream. But with the current state of play we need software testers in most organizations.

One role I don’t think we need is Software Architect. On the whole I think most architects are mislabelled. They should be Product Managers, Business Analysts, Senior Software Designers/Engineers or even Project Managers. Sometimes this is because the organization misuses the title, sometimes its because they don’t understand it (after all, what is software architecture?) but most of the time its because of title inflation. You couldn’t give someone a pay rise so you gave them a nice title.

Although I don’t read Joel Spolsky very often I do agree with him about Astronaut Architects. The moment your architects stops coding its time to fire them. Once they stop coding they are a burden not an asset.

Project Managers are another group I don’t think we need. I accept teams need some leadership but Project Managers are completely the wrong people to. Project Managers are trained to analyse, dissect, use whips (badly) and emit hot air. Their are not leaders by nature or training.

The Agile world is re-inventing project management as coaching but unfortunately the Agile world is pretty messed up about coaching. Coaches are useful when you they get it right and they are genuinely improving the team rather than acting as Project Managers with a different title.

But, and this is where my younger self will have a fit: you do need someone to oversee and manage the whole development process. The big problem is very very few people can do this job right.

By way of explanation to my younger self: done badly this role is worse than useless, done right it can really improve things. Unfortunately most people do it badly.

Its done badly for several reasons.

Firstly, most Software Managers have never been trained in the role - or management at all. They ‘make it up as they go along.’

Second, some of these managers don’t understand software development, they come from outside the domain. Such people try and make software development fit their existing model of management but it never will. Therefore they are constantly trying to make a square fit in a round hole.

Third, many of the managers who come from inside the domain, and know how to develop software can’t give up coding software. To be a good software manager you have to stop being a coder. You have to learn to look at the world differently, you have to realise you can’t intervene to fix problems, you can only arrange things so other people can fix them.

When I was younger I once worked for a bank. The code I was working on was terrible and I had no support from the organization. I really really wanted my manager, and his manager, to come and code with me. I thought their big mistake was to stop coding. I was wrong.

Their biggest mistake was not to stop coding but to stop communicating with those who did code. The big mistake was to put too much space between themselves and the building process. Had they involved themselves more they would have better understood the problems and would have been able to arrange things so I could fix them.

If you are managing a software team you should be talking to them every day. You should be improving their working conditions every day. You should be allowing work to flow to them.

When you move from coding to management you have to give up part of yourself, you have to trust the people you manage. It is their code now, not yours.

So, in conclusion: it is wrong to think you only need developers to create software. It is also wrong to think you can code and do other things - you need the separation. But it is also wrong to have too big a separation.

Make any of those mistakes and you just moved one more duck out of line.

The lining up your ducks theory of software development

Software development is easy. Don’t listen to what people say. I was 12 when I started programming and I picked it up no problem. Businesses are full of people who programme without even knowing it: Excel is programming by another name, and how many people write macros?

Yes, programming is really easy. You can buy ‘Programming for Dummies in ...’ - well just about any language you want. People can, and do, create small world changing programs without ever really being taught how to program.

That’s why software development and new businesses go hand in hand. Its one of the reasons why so much innovation today appears as software.

But....

Software development is also one of the hardest things. In fact, developing good software is so complex it might be the most complex thing man has ever attempted.

Moon landing? - it used software, compared to the complexity of the systems in the international space station I’m sure Apollo looks simple.

Nuclear power? - compared to the maths used by the finance industry today it looks simple. And that is all in software.

The thing is, writing a small piece of software can be (but not always) very easy. Problems set in when you want to grow the software, want more people to use it, want to sell it, want to package it, want to re-use the code or ideas, take bugs out of the system, add features, make it run 24x7.

It is very very easy to write simple software and have it do something useful. It is an order of magnitude harder to get it to commercial quality and keep doing it. And, to make it harder, there is no single accepted method for doing this and getting it right every time.

It sometimes amazes me that we get any of this right. More amazing is that software which is fundamentally flawed works, or at least seems to work. Stuff which by any engineering criteria is broken is used by companies every day, they even base their business on it. (There is another blog entry in this.)

Developing good software, delivering on time, producing with sufficient quality, etc. etc. - all the things you expect from ‘professional systems’ is a matter of getting your Ducks in a Row. What I mean is...

Creating good software, meeting expectations, delivering on time and the rest is not just a matter of getting one thing right. It is a matter of getting many things “right” - or at least workable. You can get some of these wrong and still delivery something - maybe a little late, or lacking features but you will do something.

Each time you do something badly, each time one of your ducks is out of position, you won’t break the whole thing. This isn’t a chain, instead of breaking you will reduce your productivity, or move away from the target, or reduce quality. With each duck that moves out of position it gets worse but it still works, somehow.

Put it another way. If creating ‘the best software ever’ means scoring 100%. Each time you move a duck out of position you remove 10%. So:
• don’t co-locate your developers, loose 10%
• start work without talking about design, loose another 10%
• work to a fixed specification or work to no specification loose 10% as well.

Down to 70%.

• Fire developers half way through the project, loose 10% for lost productivity, loose 10% for moral.
• Projects looking late, add more developers, loose 10% for violating Brooks Law.

40% doesn’t look good does it?

But there aren’t many case where you need to score 100%. 90% will do, even 80%, 70% often, some customers will be happy with 40% - or less.

The people who get upset most by this?

Not customers, they get annoyed but quite often there is nothing they can do about it.

It is the developers themselves. They see each loss, they know where all those 10% are being lost. And these guys are problem solvers, they want to fix these problems.

Now go back to my falling off a log theory of business.

I said: you do something right. Not 100%, maybe you just need 10%. But that one thing is about doing something useful, something other people want. Its about benefit and effectiveness.

Viability of your software product/company =
        benefit of your software
                x
        effectiveness of your organization

If your software doesn’t bring many benefits then you need an effective organization to stay in business. You have to be better than the rest.

If your software brings lots of benefits then it doesn’t matter how ineffective your organization - shoot the ducks! - you will stay in business.

If your software has few benefits and you can’t deliver, then Game Over.

Which leaves... lots of benefits and a highly effective organization. Then you have a world beater.

Unfortunately few companies fall into the last category which leave most people working in one of the first three.

'Changing Software Development' is published!

About 5 years after first thinking I could write this book, about four year after I first worked out what the book would look like, and 14 months after signing the contract my book, Changing Software Development is now published.

You can now buy it at Amazon, or from the publishers John Wiley.

'In search of mediocracy' - The falling off a log theory of business

When I was at business school my classmates and I read lots of good stuff about how to run a good company. Books like ‘In Search of Excellence’ and ‘Built to Last’. But I remember some of my classmates asking: ‘Why aren’t the companies I’ve worked for like these?’ To most, if not all, my classmates these books described something that was not the case.

That was five years ago. Since then I have met many people, few if any of them have worked for ‘Excellent’ companies.

Indeed most of the people I meet (particularly in software development) tell me crazy things that have happened at their companies. Decisions to close down offices just before a contract is won. Objective setting sessions were the objectives set are meaningless. Management which does the opposite of what they know is best.

One company I worked with in the last five years came closer to ‘excellence’ than any other company I know. Still, it got into trouble and has had three rounds of downsizing in the last two years. From what I hear things are pretty crazy now. Apparently in the last round they paid off four engineers, soon discovered they needed them, tried to hire them back, HR got involved and slowed the whole thing down during which time three of them got new jobs.

Another company I worked with was perpetually shooting themselves in the foot. So much so that they even let go the one person go who could make sense of the development team. Just this week I learned the same company which is letting one of their key managers go.

I end up hoping the end comes soon but these companies survive. The books say they should go out of business but they don’t.

So it is that I find myself asking ‘where are the excellent companies?’ And I feel pain when someone else tells me another story of the stupid things their employer is doing.

I’ve come up with a theory which explains it. I apply it only to the software industry because this is what I know but I suspect it might be more universal. It goes like this...

Like having a baby there is no law against starting a company. Actually there are a few laws, and in some countries it can be hard, but in most Western countries, provided you do not have a criminal record you have a right to start a company. In the UK it can take less than a day to form a company.

Most of these companies will go out of business in under a year. If I remember the statistics less than 10% survive year 1.

This 10% have done something right. Maybe they hired the right people, maybe there were in the right place at the right time. Maybe their competition was more of a mess than they were. Perhaps they really did have ‘first mover advantage.’ In other words, many of these companies survive purely on luck. They got something right. That doesn’t guarantee they got anything else right.

Some of these companies will go out of business in the years that follow but the longer a company remains in business the greater its chances of surviving even longer.

Over time these companies will gain some customers. Now customers can be surprisingly resilient, not because they are ‘loyal’ but because switching costs are great and because they have to decide to go elsewhere. Remember, these companies may be dysfunctional themselves so changing isn’t easy.

When you have a few customers, particularly big ones, you can make the money go a long way. Software companies which pull in their wings and go into maintenance mode can shed most of their staff and still stay in business for years. In fact, I would argue that almost any company you think is about to fail can last another year.

Notice there is nothing here which says the company will be ‘excellent’, or even that it will do what the management text books say. All I’m saying is that crazy mixed up companies can survive for a long time when they shouldn’t. Many of these companies have little right to exist by business school standards but they do.

As a consequence many, if not most, companies are, to some degree dysfunctional. Many people work for these companies. Many people are abused by these companies. And these companies are enough to put you off companies altogether.

I call this my ‘Falling off a Log Theory of Companies’: Starting a company is as easy as falling off a log. Most companies which come into being will fail quickly, the majority of those fail will be to some degree dysfunctional but will survive. Consequently most of the people I know will work for dysfunctional companies.

I think this theory applies especially to software companies for two reasons. Firstly there are still many many opportunities in software to be found and exploited, i.e. you can still get lucky. Secondly, the IT industry, and those who fund it, are massively optimistic. We accept failure too easily.

A few months ago I explained this theory to someone not in software. He said he’d seen the theory in action too. He also suggested the book these companies need is not ‘In Search of Excellence’ but ‘In Search of Mediocracy.’

New pieces on the website

I’ve been holding out, waiting until I have the time to write a really great start-the-year blog entry. Well I’ve not had the time and I’m about to go on holiday for a week. Still, just time to say I’ve posted some new pieces on my website.

• Session notes from Creating Awareness / Exposing Problems is the result of my ACCU 2007 conference session. These notes were published in ACCU Overload last month and are now on the website.

• My patterns from VikingPLoP 2007 (More) Patterns for Software Companies were uploaded just before Christmas but never announced here.

And what for this year?

Well I think I’m going to try and blog less. Yes less. Two reasons really.

Firstly I have plenty of ideas for blog pieces but quite a few of them would be long entries. Some of these I will write up but I am included to place the majority in ACCU Overload. When a pieces runs to several pages I think this is a better forum. I’ll still upload them to the website and announce them here but by their nature they require more time. More time on these means less time elsewhere.

Mentally I’ve accepted that I am going to write a second book. This will be concerned with, well, Business Patterns and Strategy for Software Companies (umm, that could be a good title.) Don’t expect it much before 2010, I intend to avoid signing a contract for a while. I’ll divert some of my material to this instead of the blog.

Secondly, I’ve spent most of the weeks before and during Christmas researching a new business idea and developing a business plan. Very few people know what I’m planning but this select group increased by one at lunch time.

That also means my consultancy work may have to take a back seat when things get going.

(By the way, if you know anyone who would like to invest please let me know. Current projections suggest I need about £100k).

In the meantime, if your missing this blog you can buy the book. Changing Software Development is out in a few days, I heard today that Amazon are already publicising it in e-mail.

And if that is not enough I’ll be appearing in Cambrigde later this month, at SPA and at ACCU 2008 - events list here.

Christmas reading: Classic essays on software development

There are many, many books about software development. The technical ones alone (e.g. Java, C++, Apache, .NET, etc. etc.) take up meters and meters of shelf space. The ones about project management take up a bit less space but they are still plentiful, and the ones about people in the process take up even less space but are longer lasting.

It is the later category that interests me most theses days. Some books seem perennially relevant and may never go out of print, books like:

• Mythical Man Month by Fred Brooks
• Psychology of Computer Programming by Gerry Weinberg
• Peopleware by Tom DeMarco and Timothy Lister

But often you don’t need a full book to make a point. On these occasions an essay or journal article will do. Often they make the point better simply because they are shorter. Unfortunately these tend to be overlooked too often - books get all the publicity. I would like to highlight half a dozen essays which I think are perennially relevant and often overlooked. In some cases this is because the articles can be hard to get hold of or, more likely, people don’t know they exist.

1. How do committee invent? by Melvin Conway - the origin Conway’s Law. Although written in 1968 this piece is truer than ever. Arguments made by Conway explain much of what actually happens in software development. I’ve examined and looked at this subject myself elsewhere.

Conway suggests that organization that create systems (not specifically software systems) will create copies of their organizations. So, if you have one developer writing the entire system it will be all entangled, few interfaces and difficult for anyone else to work with. And if you have half a dozen developers who don’t communicate you will get six different coding styles, and six versions of most common functions.

What Conway didn’t foresee is that many of system now create the reverse force. Organizations are limited by the systems they use, thus their structures become copies of the systems in use.

2. Worse is Better by Richard ‘Dick’ Gabriel - this started as a keynote talk, became an essay and then a series of letters in which Dick argued anonymously with himself.

If you are one of these developers who think the best code will always win the day you need to read this. Dick suggests that sometimes the solution which is technically worse is economically superior. In a similar vein see Brian Foote’s Big Ball of Mud pattern.

3. No Silver Bullets from the author of Mythical Man Month, Fred Brooks. This piece came out in the mid 1980’s. The Wikipedia summary is here, otherwise you will need to by the Anniversary edition of Mythical Man Month to read it.

Brooks argues that software development is hard, and there is no ‘silver bullet’ which will make it easier. No language, no operating system, no methodology or any other tool will fix it. I have been told that Brooks has since said the if there is a Silver Bullet it is Agile Software Development but I can’t find this quoted anywhere so I don’t completely believe it.

4. A Rational Development Process and How to Fake It by Dave Parnas. Originally published by the IEEE this also appear in Software Fundamentals: The Collected Papers of David L. Parnas. You might also find some copies stashed on the Internet. Dave Parnas deserves to be more widely read by software engineers today so it you haven’t read much of his work buy the collected papers, well worth reading.

In this paper Parnas argues that while we may want to use a rational development process, and while it makes sense to do so it is impossible. This is because software development requires intuition and inspiration - or as I would put it learning. And since you can’t schedule these things or guarantee they will happen the chances of following a rational process are negligible.

However, in order to make our work accessible to those who come after us we need to fake the development process so it looks like we followed a rational process.

I like the logic, I agree with much of it but I have one disagreement. Each generation of software engineers comes along and believes the previous generation were rational and get confused by what they find. They slowly learn that they cannot be rational - causing a lot of angst. They also wonder how the previous generation made such a mess. So frustration is built up.

5. Cathedral and the Bazaar - by Eric Raymond: an explanation of Open Source software. And an explanation of why software created in corporations is frequently a mess. I don’t think Open Source is the answer to all our problems, but I think it is an answer to some and is here to stay.

6. Enrolment Management by Peter Conklin: until recently this was one you really have to hunt down, but it is worth it. Originally published in the Digital Technical Journal (Digital as in DEC or Digital Equipment) in 1992 it was republished by the IEEE in July 1996. HP now have back issues of the Digital Journal online so you can read it here.

This piece tells the story of the Alpha AXP programme in the late 1980’s and early 1999’s. In theory, on paper, on GANTT charts the project couldn’t be done. The author tells how enrollment management was used to engage those working on the project. Although written in 1992 this paper is about Agile development. And it is about Agile development in the large - over 2,000 engineers worked on the project.

I wrote about this article before, a few years go. One of the small stories in this paper always brings a smile to my face. The OpenVMS team committed to a delivery schedule and said ‘We don’t know how to achieve this, but we commit to finding a way.’

So there you have it. Six essays I recommend you read. And if you only have time for one? Well, chances are you know about Open Source and the Silver Bullet so I recommend you read Enrolment Management or A Rational Process, and if you push me more I’ll say: Enrolment Management.

Busy time - and the Software Development Practice Review

Despite the fact that I have deliberately taken a couple of months off since the end of my last assignment I’ve been keeping very busy. I’ll be glad when Christmas comes and I can leave the laptop switched off for a few days.

In addition to finishing up a couple of personal projects (the book, various patterns papers and other stuff) and giving some talks and briefings I have been involved in an attempted rescue. OK, nobody was in real danger this was more a corporate rescue.

I drafted a turn-around plan for a .com in trouble. For a while it looked like the plan would be adopted but now it looks doubtful. Seem the management have gone off the idea. Shame but so be it, it was interesting to analyse the company and make suggestions.

I have also been busy working on my own product offerings. I have today revised my website to make it clearer what I do. In addition to regrouping some of my services (training, briefings and workshops) I’ve also created a Software Development Practice Review product - SDPR for short.

I’m particularly proud of this SDPR. It has taken a few weeks to bring together the framework but the basic idea is simple. I want to be able to visit a development organization (a team or department or such) and evaluate their software practices. Notice I say practices, not processes. This is deliberate. Most of the other reviews I’ve looked at consider processes, I think practices are more important because it shows you what people are actually doing, not what they are supposed to be doing.

The framework behind the SDPR starts by considering teams (size, structure, experience, etc.), the management of the teams, how the business communicates its need to the teams and how risk is managed. After this it turns to what the team is doing, technology, routines, practices, release management, testing and a little bit about processes. Finally the framework considers what happens after a software release and how the team learns.

From this review I can create a report designed to seed team improvement and stimulate the various members to try new practices and ideas. The second key difference I see with many other types of review is that this review is intended to help the teams improve, it is not intended to measure them on ‘pass/fail’ or suitability to undertake work. It is there to help improvement.

Overall I get very excited about this review. I hear of so many development teams which could do better, some of which are almost dysfunctional and yet fixing them wouldn’t be very difficult. Hopefully this is a tool to do just this.

I’m now talking to some people about trying the framework out in January. If you think your team could benefit from this review, or you know someone who would find it useful please get in touch.

Business alignment, Agile failure and the long run

Tom Gilb sent me a link to this presentation from Ryan Shriver about the use of Scrum with Tom’s EVO techniques. Very interesting, and it returns me to the subject of IT/business alignment - which in part is also a discussion of where Agile software development is wrong.

My blog from last week touched on this but I think its worth spelling this out. There are two problems with Agile, and two answers:

Question 1: Why do Agile techniques work? - forget the technical explanations of why TDD is a good thing, why RUFD is preferable to BUFD or how value stream mapping helps. Think for a moment: there are a lot (an awful lot) of software development groups which are not effective.

Problem 1: How can it be that the same set of (Agile) practices can help all these groups?

Answer 1: Because these groups all have similar problems, probably because too many people were taught (and believed) the same set of practices that have created the problems.

Question 2: How can Agile work without business alignment? The basic Agile techniques pay little attention to what the business wants. The XP onsite customer is really lightweight, XP assumed the customer knows what they want but often they don’t. Scrum is better but the product owner is still left to their own devices, Scrum says little about how the product owner knows what’s what.

Problem 2: How can Agile be effective without business alignment?

Answer 2: It doesn’t have to be. The MIT Sloan Review piece shows that just getting to be effective is such a massive improvement over being ineffective that it is a success. You don’t have to be aligned with the business to show improvement.

In the long run, for organizations which have effective Agile teams these problems will show up. And in time, as more companies achieve Agility, the competitive advantage of Agile Software Development will diminish.

The advantage will move to the companies that address these second order problems. Such companies need to: learn how to improve Agile practices beyond what we currently know, and, get IT aligned with the business.

In an Agile world this means pushing Agile further. Tom would call this Evolutionary, I call it learning, it amounts to the same thing in the end.

Book review: Agile Project Management with SCRUM (and rant)

It is difficult to say anything bad about Agile Project Management with Scrum (Schwaber, 2004). It is a short book, lucid and easy to read. It sticks to its topic - managing projects using scrum. Rather than present Scrum in depth (which I think he has done elsewhere) Schwaber walks us through a set of small case studies and reflections on each one.

If this book has a failing it is: who will read it? - I’m not sure. There is probably not enough information here for someone to implement Scrum but I find it hard to see what someone experienced with Scrum would learn.

For me it was a late introduction to Scrum. I can’t ever recall reading a book dedicated to Scrum. But I picked up the basics of Scrum somewhere, probably websites, articles and conversations. So I would suggest this book is best for someone wanting to an introduction to Scrum, and specifically wanting an idea of how Scrum works in practice.

Which brings me to the question, Why Scrum at all?

I found that Scrum kept coming up more and more and I felt the need to get the introduction I never had so to speak. It is clear to me now (if there was ever any doubt) that Scrum is the management side that is missing from XP.

My own Blue-White-Red process is in effect an example of Scrum and XP (both modified) being used together. Looking back the roots of Blue-White-Red are difficult to work out. I think it was inspired by XP and Scrum and takes many practices form them but it was equally influenced by what I had done in the past and seen work. Perhaps more importantly it was influenced by a bunch of ideas I’d picked up on my MBA, specifically Lean manufacturing.

Unlike Scrum Blue-White-Red is there for anyone to pick up, use, abuse and modify as you see fit. Scrum isn’t, Scrum is actually Scrum (tm), and it is now surrounded by a host of Scrum certifications (Scrum Master, Product Owners, etc.), courses, books, alliances, etc. etc. Scrum has become its own little marketing platform.

In some ways this is good, by doing this Scrum can satisfy business that want to adopt some defined product, it allows Scrum to tackle CMM(I) type organizations and it allows people who want to do Agile to get on courses and learn the skills. But this is also leads to rigidity.

I recently met a Scrum Master who was having trouble with the process. I suggested a one week iteration rather than two week. She saw how this would help but knew the rest of the company used two. She expressed doubts about keeping her meeting schedule (as prescribed by Scrum) in one week. I suggested some changes, at this point she worried that she was deviating from the Scrum process too much.

At the end of this book are the rules of Scrum. Schwaber says: here are the rules, follow them, when you are good at them then (and only then) can the team think about changing them. Scrum needs this, if it was as free as Blue-White-Red it wouldn’t pass muster with CMM(I) and a certificate would be meaningless. But, it also builds in rigidity and people get worried about deviating from the rules.

For me there is an element of The Punk Ideal in Agile, some of it is just about getting our there and doing your own thing, your own way. Who cares if you get it right or wrong? The only mistake is not trying.

At this year’s XP Day conference Keith Braithwaite said something that struck a cord with me, and I’m paraphrasing here ‘cos I don’t remember word for word: ‘I’m not really part of the UK Agile set’ he said, ‘I never worked for Connextra or Thoughtworks.’ And that’s the point, neither of us waited to work for an Agile organization, or to be blessed with a Scrum certificate, we just got on with improving things.

Too many people sit around waiting for something to change, waiting for someone else to fix their problems, waiting for the magic dust that will make everything all right. That’s one reason why so many IT projects go wrong, people wait for someone else to fix it. In part our school system is to blame but that’s why the Punk Ideal is so important. Again, this takes us back to Post Modern Programming. There is no one ‘right’ way of doing this, there are many competing ideas of what right is.

So here is my advice: learn everything you can about XP and Scrum, they are good. Consider them resources to better your understanding. Then go and do Blue-White-Red.

Right now I’m adding two statements to Blue-White-Red:
• Blue-White-Red is open source, I can teach you it but there will never be a certificate
• Blue-White-Red is jazz, it is improvisation, no two performances should ever be the same

If you aren’t doing something different to what I have described you aren’t doing it properly. Like the English language, there is only one rule which is never broken, the rule is: every rule is broken at some time.

IT: Better to be effective or aligned?

In my last blog entry promised to discuss a second piece from the MIT Sloan Review. In many ways I find Avoiding the IT Alignment piece more interesting than the Rettig piece I discussed last time. The Rettig piece was a good argument but it wasn’t clear what you did next, it was insightful without being immediately useful. The second is piece is insightful and has some immediate application - the authors suggest some but I’ll add some of my own below.

Perhaps one reason why this piece is more immediately useful is that the authors (David Shpilberg, Steve Berez, Rudy Puryear and Sachin Shah) are all consultants with Bain therefore they have something to sell, consultancy. Rush out now and hire your Bain consultant! Sorry, I shouldn’t be so negative. Let me give yo a run down of the article and then draw some conclusions of my own.

The article is based on the authors’ experience and a large(ish) survey. You can argue with the underpinnings of the article but I’m quite happy to give it credibility. Like the Rettig piece they are concerned with internal IT projects rather than products for sale (my main area of interest and experience.)

From this survey they are able to divide companies into four categories - a nice 2x2 matrix based on whether a company has effective (or ineffective) IT and whether the IT is aligned with business strategy (or pursuing its own.) These four categories are:

IT Enabled Growth: These companies have highly effective IT groups who are closely aligned to the business. This is what we all want. These guys are doing it. Unfortunately only 7% of companies fall into this category. The benefit to these companies is 35% sales growth over 3 years. Pretty impressive. Perhaps surprisingly these companies spend 6% less on IT than average. So, going it right also means doing it more cheaply.

Maintenance zone: These the basket cases, IT is not aligned with the business and it is not effective. Unfortunately this accounts for 74% of all companies, really depressing. Because this is the bulk of all companies it is perhaps unsurprising that the IT spend in these companies is the average, and their 3 year sales is just slightly below average at -2%.

They are the two opposite, and perhaps we expect that. A few companies maximise value from IT but most don’t. The next two are more interesting:

Alignment trap: this is the combination the authors find most interesting. This occurs when IT operations are aligned to the business (i.e. they are doing what the business wants) but they are not effective. Only 11% of companies fall into this zone - together with the first category this means only 18% of companies actually have IT and the business aligned, quite shocking. These companies spend 13% more than average on IT but the annual growth in sales is 14% below average, a pretty poor showing really.

Such companies are doing the right thing but they are doing it badly. From the figures given this seems to be the worst category to be in, highest costs and lowest sales growth compared to average. Think about that for a minute. These companies have it half right, IT is aligned with the business but they are not effective. These companies would be return better results if they gave up on business alignment and joined the 74% in the maintenance zone.

I would suspect these companies actually make it difficult for IT people to do the right thing. They probably have plenty of processes in place to make sure they don’t do the wrong thing but doing stop people from doing very much at all. My guess is that managers in these companies don’t understand IT and are trying to run IT the same way they would run any other department. Consequently the IT people can’t do their job.

Finally we come to Well Oiled IT: this is the category I find most interesting. These companies are doing IT well, they spend 15% less than average on IT but see 11% higher sales. Only 8% of all companies fall into this group, if we add that to the 7% of companies in the IT enabled growth category we see that only 15% of companies, less than 1 in 7 actually have effective IT.

Statistically, well oiled IT companies are similar to the growth companies but less so, they have significantly lower sales growth but spend even less on IT. Possibly these companies are focusing too much on cost - my bet is they don’t have effective business analysts and product managers in place.

Now for the authors’ advice: they tell companies to focus first on operations not alignment. If companies (and remember most start in the basket case maintenance zone) focus first on alignment they are likely to end up in the alignment trap, where things are worse. Therefore, seek first to make your IT effective then to make it aligned.

I like this advice, but like so man things in the IT world the devil is in the detail. So what do the writers recommend?

• Keep it simple - we all agree in principle but we all get caught out; trouble is IT tends to get more and more complicated, battling complexity is a constant fight
• ‘Right size’ - which is a bit vague but basically they say outsource somethings, keep some things in house but do it consciously and intelligently
• End-to-end accountability - IT managers need the resources to do the job but they have to keep talking to the business, and the business needs to talk to IT

All this advice is sound and it makes for a good starting point.

So now some thoughts of my own.

First this survey supports what I’ve found in practice: most IT operations are basket cases. Few companies can do IT well. In product companies there is natural selection by market forces. If the company is too bad they will (eventually) fail and go out of business. Where IT is a service to the business failure can continue almost indefinitely. Unfortunately this means that at least 85% of people in corporate IT departments will work in failing teams.

Second I now understand why Agile development works so well in many organizations even though it is not aligned with the business. Simply it doesn’t have too. If adopting Agile development moves an organization from maintenance zone a little toward well oiled then it doesn’t matter that it is not aligned with the business. Just improving effectiveness will deliver benefits.

Third, business must take more of an interest in IT. IT can make itself more effective but to become aligned the business needs to understand and get involved with IT - back to the point I was making about the Rettig article and last year about companies that understand IT.

On the whole I find this article good news, it helps me chart a cause for improvement. I see a three stage plan for any failing IT department:
1. Apply well known techniques from the Agile toolkit: this will boost your effectiveness immediately
2. Learn what is special about your environment so you can tailor the Agile techniques and find some of your own. This will further enhance your effectiveness.
3. Continue you learning to bring about business alignment. Often this is going to be a case of removing the old perceptions about how IT departments operate (e.g. Requirements: a dialogue not a document). It also means educating managers and IT staff, and it means creating the roles of product manager and business analyst to facilitate it all.

With 93% of companies failing one way or another there is plenty of work to do.

Now if we loop back to the Rettig article on ERP. Is it any surprise that ERP deployment fails? Only 15% of companies are capable of implementing it effectively, and of these less than half will align it to the business. Of all ERP deployments we can only expect 7% to come close to fulfilling their promises.

An often heard dictum in management concerns the difference between ‘doing the right thing’ and ‘doing it right’. If this article is right, when it comes to IT then it is better to ‘do it right’ than ‘do the right thing.’ Only if you can ‘do it right’ should you even start to think about ‘doing the right thing.’ Operations over strategy. Implementation over design.