Events for next year

I have some speaking events booked for next year which might interest readers:

• 23 February, Jax conference London: I’m updating The Future of Agile presentation for the conference
• 23 March, BCS PROMS-G group London are running a repeat (with updates) of BCS Bristol’s Agile Spring School, if you miss The Future of Agile at Jax then catch it here
• April 2010, ACCU Conference, Oxford: New presentation bringing together some ideas into 10 Steps to Agile Requirements
• 20 May, Software East in Cambridge: Just a date in the dairy so far, details are still being sorted out

I hope to see you at one of these.

Happy New Year and all the other seasons greetings!

Banks spend a lot on IT but its a mess

Living and working in London means a lot of the software people I meet and talk to work in financial services. It amazes me the number of programmers etc. that banks employ. And you hear stories, some really awful stories about the state of IT in banks. And I’ve seen a few in my time.

So, for all those readers out there who have to work with banks and other financial services companies I recommend you get hold of a copy of this weeks Economist and read “Silo but deadly.” This article lends some weight to those stories I hear in pubs, at conferences and during training courses.

Some highlights:

• Financial services companies are estimated to spend $503bn on IT this year, thats more than Governments, more than manufacturing and more than any other sector
• The chief risk officer at Deutsche Bank says: “Banks are essentially technology firms”
• SOA architecture is a problem, that’s Spreadsheet Oriented Architecture
• Data quality is a recognised problem

The articles suggests that the woeful state of IT in banks may have contributed to the recent problems. This is something I’ve suggested other links before now, e.g. the Moody’s bug.

Personally, I think one of the reasons why banks and other financial companies have such poor IT goes back to that quote above: Banks are technology firms but they don’t view themselves that way. As a result the way they don’t necessarily take the right approach to people or projects, and they don’t take the right short-term v. long-term view and so on.

Lest we get too negative about our industry you might also want to read this recent survey from McKinsey which suggests IT isn’t doing such a bad job, and outside the IT department corporations are pretty happy with IT on the whole.

Requirements, requirements everywhere; no clue on what to do

Twice this year I have visited companies which want to try a more Agile way of working but who have requirements coming of their ears. I’m talking hundreds of pages of requirements. Sometimes as use cases and scenarios. Sometimes as big meat documents written by expensive consultants and bearing names like “Requirements Specification” or “Business Design.”

Its hard to argue with such requirements documents, in part because its hard to read them and understand what they are about. The thing they have in common is a level of detail. They go into great detail on what needs doing and how you might go about doing it.

Its not the first time I’ve seen such documents. I’ve been on similar projects before. One variation is a project I saw last year were the documents were not requirements documents but strategy documents. Several big name consultancies had been into this (media) company and do strategic work around a new product. Lots of presentations, lots of SWOT analysis and so on.

What all these documents have in common, apart from being expensive, is that they are worse than useless. They give the impression that things are know but they aren’t know. They knowledge walked out the door, what you have is a snapshot of someone’s understanding. But its a snapshot that nobody can really take in because its so big. And it is a snapshot that is rapidly aging.

As I’ve said before: the bigger a document the less likely it is to be read, and if it is read, then the bigger it is the less that will be remembered of it.

To which we might add: the bigger the document is the more there is to go out of date.

Yet because these documents exist people feel compelled to use them. They may even be mandated to use them. They don’t feel they can say to their boss: this was a useful learning exercise, no we will do it for real. So we then embark on a process that try to use documents which only get in the way.

Having someone investigate requirements or strategy can be useful. But think of it as a learning and feasibility exercise. And above all: don’t let that person leave the project.

At best these documents are historical reference source; as worst they are an obstruction.

Set a clear goal for the work and work out what you need to do to meet it. As Tom Gilb says, this should fit on one page of A4.

As you work towards this goal you will uncover requirements. You might want to note them down but remember there is no point in running far ahead of the development team.

Once you have something up and running - even in demo - what people think they want will change. So start small, get something working and get a team that delivers.

Just say No to big requirements documents. Say Yes to short, concise, goal to aim at.

Pattern writing in Sofia

Continuing about patterns... At the beginning of the month I went out to Bulgaria to teach pattern writing to some academics and students at the University of Sofia. It was the first time I’d been to Bulgaria so I was interested to see the country which physically, historically and culturally sited between Europe and Russia.

We made the class very practical, no PowerPoint, in fact apart from a 5 minute video it was computer free. Yes we talked, yes we used flip charts but most of all it was practice, this was about writing after all. Only about an hour in and we had our participants putting pen to paper to write some patterns.

For this we used a video about milk packaging from TetraPak, there is an English version on You Tube, as it happens we used the short German version but it was the pictures not the words which were important. After watching this we did some brainstorming to identify what was happening in the video, i.e. what solutions were being used to solve which problems.

Initially we asked the participants to think about three things: the state before the solution, what the solution was and how things were different after the solution. Once the groups had grasped this we expended these into the traditional parts of a pattern: context, problem forces; solution and implementation; consequences and all the other bits (known uses, etc.)

During the writing myself and my co-presenters (EuroPLoP regulars Klaus Marquardt and Didi Schueltz) circulated between the groups to answer questions, provide advice and deliver a bit of shepherding.

And then we repeated the exercise with different subject material. We did some brainstorming to identify ideas which the participants would like to write patterns about, problems in their own domain. We repeated the writing process and finished the day with a writers workshop.

So in the space of one day we covered all the pattern basics. Although not in as much depth as maybe we could have. We carried on into a second day but only used half of the day. Next time I’ll be sure to use two whole days.

On the second day we covered the ideas behind shepherding, talked some more about pattern form, answered some questions and ran a second writers workshop, this time on an existing pattern by an experienced writer.

More than ever I left feeling that patterns are a generally applicable knowledge management technique. Not everything belongs as a pattern but the form is useful in many domains. And pattern thinking (or perhaps pattern analysis is a better term) is useful even when the thing you are writing (or just analyzing) is not a pattern.

And what did I learn? Well...

• I really enjoyed teaching pattern writing and hope to do it again
• Using a video to provide source material worked well
• Not using slides worked well, it requires confidence but the result was better
• You still need to give the attendees something to take away so having support material is necessary. Still, I’m always disappointed with slides which are little more than bullet points after the class. We did hand out some of my writing about patterns and patterns form, some patterns about patterns and some links to websites with more information and patterns.
• Having co-presenters was good. This was an intensive class so it needed more than one person
• You really need two days to do this properly
• Having attendees experience what you do was far more valuable than just lecturing them

Next time I would like to include a pattern reading exercise too. Take a well written pattern and read it through as a group. I’m going to be delivering a software patterns course this week and I plan to try this exercise in miniature.

Gartner talks up Pattern Based Strategy

For those who don’t know, Gartner group are a business/technology research outfit who produce reports on things technical and business related. They have a pretty good reputation and they have a couple of sidelines in consulting and event organizing (or is it the other way round?). Sometimes they are leading opinion and sometimes following. They sell their reports, their conferences are expensive and their consulting must be more so.

Anyway, the reason for saying this is that Gartner have decided “Pattern Based Strategy(tm)” is the next big thing: “Companies Must Implement a Pattern-Based Strategy to Increase Their Competitive Advantage.”

(Notice the “(tm)”. Somebody, somewhere owns the trademark “Pattern Based Strategy.” Hillside own the trademark “PLoP” but not “pattern”. Could the owner of this one be...?)

Now as many of my regular readers will know, I’ve been talking about Business Strategy Patterns for over five years - and you can read my Business Strategy Pattern papers for free. Since posting up the 2009 EuroPLoP paper I’ve spent some time on the whole, I know have a 230 page draft of a book and I’m starting to approach publishers. There is still a lot to do but I hope to have something out by the end of 2010.

Getting back to Gartner. I’d love to tell you how their patterns and mine link up but I can’t actually read the Gartner report. Call me penny-pinching if you will but $500 seems pretty tall. Actually, they have a bunch more papers on strategy patterns but again, its $500 a throw.

I asked Gartner for a copy of their research and they said: “Unfortunately, our research is only available to our clients, but I can provide you with the press releases we have done on the topic which you may find helpful.”

So there you go, a closed shop. They aren’t willing for other people to review their work, definately not the culture we in the patterns community have.

It also means I can’t see their sources. I’m naturally suspicious of organization that won’t disclose their sources. For all I know they could be be building on sand - they could even be referencing me left right and center. Never mind me, I’d like to see some third party research to support their ideas.

From the press release (the first link I gave) and some other downloads from the site it doesn’t look like they are talking about Patterns in the same sense that anyone familiar with Christopher Alexander’s work would understand it. And by extension, they are not linking their patterns work up with work from the software engineering community - Design Patterns (GoF, POSA, etc.). Neither are the talking about patterns as knowledge management from the likes of Mary Lynn-Manns and Daniel Mays, or the work on Business Strategy Patterns I and others have been doing.

At the basic level there are similarities about the patterns they see and the underlying ideas of Alexander. For both Gartner and Alexander it is about a reoccurring sequence of events in some setting - Alexander would say place, software folks would say context and Gartner say market.

I think Gartners work is more related to data mining and business intelligence than it is to Alexander. As such it may well have more to do with bird spotting. Some years ago Harvard Business Review ran a piece on how spotting patterns in birds could help with business strategy - see Spotting Patterns on the Fly from HBR November 2002.

Although we are all dealing with the same thing: reoccurring events, sequences, constructions and the forces that bring them about, I consider Gartner’s patterns as “patterns in the small.” Reoccurrence without the repetition, analysis and explanation that make up “pattern in the large”. Its the same word but used differently.

Gartner also discuss “weak signals” and scenario planning. This is interesting because I’ve always believed pattern thinking has a lot in common with scenario planning. It is about detecting the forces at work and seeing how they play out.

Gartner are mixing a number of different themes into their “Pattern Based Strategy(tm)”: IT systems, event monitoring and data mining. They talk about finding “new patterns” but how can you find new patterns if you don’t know the old ones? It would be nice to think Garner have some patterns they might share but I suspect they don’t.

OK, lets give it 10 years and see if anyone remembers Gartner’s “Pattern Based Strategy (tm).” Alexander and Design Patterns will still be around - timeless.

Finally, I would imagine that Gartner are one of those companies with a reputation management system so someone in Gartner will shortly be alerted to this blog. I’d like to hear your comments.

Speaking at Jax London conference

The highly successful Jax conference is coming to London next year - 22 to 24 February. And I’m pleased to say I’m going to be speaking there.

I’ve been asked to present my Future of Agile talk, but, of course, this is going to need updating. The problem with the future, is it keeps arriving, so it keeps changing. The good news is this means the talk will probably always have a future!

Creating change through IT projects

In my last blog (Magic happens here - ERP, CRM, SAP, BPM) I mentioned change programmes in passing but didn’t elaborate. So I will now. This topic isn’t unique to the BPM work I have discussed in the last few blog entries but it is related.

When I think of “Change Programmes” I am thinking of the scenario where an organization decided something must change. Someone - consultants, managers, senior folk - decide what need sot change and how, e.g. introduce a new IT system, change the business processes, etc. - and then set about making it happen.

From time to time I run across articles online, in blogs and in prints journals and books which address the question like:

• how to you motivate people to change?
• how do you overcome resistance to change?
• why do so many change projects fail?
• why do people resist change?

And so on. The common theme is that a group of people, lets call then Workers, are required to change their behavior in response to some initiative from other people, lets call them Managers - although they often go by the name of Consultants.

This is a problem many IT projects of all shapes and sizes face at times. Its also a problem faced within IT when Managers want to change the way the team works.

I don’t believe people resist change. I think people, for Workers are always People, are happy to embrace change when they see the advantage, when they have choice and when they are involved actively in the change.

For example: many people are only too happy to get a new mobile phone when their supplier tells them they can have a free upgrade.

Change problems only exist when one group of people want to impose change on another group. Its not resistance to change, its it resistance to being changed. Been told what to do differently.

As long as one group of people think they know best and try to impose change on another group people will continue resist change, articles like I mention above will continue to be written and change initiatives will continue to fail.

The solution is to change the way you deal with the problem. Those who want change need to involve those who need to change. If you are a Manager who wants a team to improve their performance then ask team what they could do to improve. Ask the team what they would do differently. Take the teams ideas and help the team turn them into action.

If you are an IT expert charged with delivering a new IT system - to support business change or otherwise - then ask the people who will use the new system what it should do, give them demonstrations as the system developers, ask their opinion, deploy the system early and incorporate the feedback of the Workers.

I’m not saying you ignore long term issues, strategy, corporate objectives and such. Tell your Workers about these things and harness their ideas. Integrate the ideas and goals into a shared understanding.

This advice applies to all change initiatives and all IT change projects. (My “Agile approach to BPM” entry a few weeks ago put this in the context of Agile and BPM.)

Too often companies use IT as a blunt instrument to impose a change in process, behavior and working practices. I sometimes imagine it as a dice game. Faced with imagined resistance to change and complex systems Managers commission an IT projects in the hope that it will create change. They grab the dice - very expensive dice I should say - and roll them in the hope that they will roll a six and win.

Surely we can do better and move beyond luck for success.

Magic happens here - ERP, CRM, SAP, BPM

I want to draw a line under the BPM/SAP mini-series of blogs. I am more aware than ever about the need to learn more in this area and actually I already know more thanks to people’s responses to these blogs - a few comments and a few e-mails. I hope to spend time in future finding out more about this arena and why it presents so many difficulties.

(And I’m hoping to see Jez Higgins talk at ACCU 2010, Wrestling with Giants.)

However, I continue to have this feeling that the SAP folks just don’t explain themselves. It feels like people are constantly telling me: “you don’t understand...”. What I would like is someone to explain please? Dont just tell me I don’t understand. Please, tell me: what is it that is different? What am I missing here?

Until that happens the BPM/SAP world will continue to feel like a secret society of people who “understand it” but will not explain it.. Until then all this is a mystery, SAP will be a box marked “magic happens here.” I’ve looked a some websites on SAP and I’ve looked at some books. The emphasis on results rather than how only deepens the mystery.

So, let me close by posing a hypothesis, maybe some researcher can pick this up and explore it.

I suggest companies buy big application suites one the assumption that:

Cost of Purchase + Cost of Customisation (programming) + Cost of Change Programme

is less than <

the Cost of Developing a Product from scratch

And that:

Cost of ongoing licencing + Cost of ongoing Maintenance

is less than <

Cost of Maintaining a Development Team to maintain a bespoke product

You might also add in a saving in management time, headaches from managing software developers, speed to market and a few other variables but that is the basic equation.

My belief - and I have no evidence for this so it is conjecture (don’t you just love blogging?) - is that companies underestimate the cost of installing and customising these big packages, and simultaneously overestimate the cost of developing their own solution.

That second part is probably the most controversial bit. I suggest that when run well these costs are not excessive and not run away. The current crop of development techniques (yes, the Agile and Lean ones) together with modern tools and existing libraries can help contain these costs.

A couple of cases might demonstrate my point....

As it happens, I do know of a company in London which has taken this route. They have four developer using PHP to continually develop their CRM system and other business functions.

A few years ago a company I worked with introduced SalesForce to help the sales team and started to extend SalesForce to marketing and support. At first it looked like a large part of my project would be thrown away as SalesForce replaced it. After many weeks of talking to the third party integrator it became clear there was something we needed and the system wouldn’t do. We developed the functionality ourselves for a fraction of the cost and time it would have taken to have the third party customise (programme) SalesForce. but then, it was a software company which knew how to do these things.

Other things about SAP (which might block Agile)

To finish off my BPM/BPE/SAP mini-series here are some other things I learned about SAP and BPM. These might be specific to the team I encountered or they might be generic. I think all of them present obstacles to doing SAP implementation Agile. Not impossible, just difficult.

Please, if I have any of this wrong correct me. I’d love to be wrong on most of these things. I’m also acknowledge my dataset is small so anything you can add to these notes would be great.

First a word on TLAs, it gets confusing:

• BPR - Business Process Re-Engineering; where it all began (sort of), comes from a book by Hammer and Champy and was very popular in the 1990’s. Largely discredited now, a blunt instrument used by many consultants which caused a lot of damage
• BPM - Business Process Management; and
• BPE - Business Process Engineering; I don’t know the difference between these two terms, they seem to be the same thing. They describe the thing that rose from the ashes of BPR. Still focused on business processes but removing some the neutron bomb like aspects
• BPM (again) - Business Process Modeling; the activity of examining your business processes, a step in applying BPM (Business Process Management), something you can buy tools for
• CRM - Customer relationship management; and
• ERP - Enterprise resource planning; two of the most common applications for which BPM, and SAP in particular, are used.

As you can see, it get confusing.

OK, things I learned about SAP:

• Actually understanding what SAP does, and does not do, is very difficult. Despite masses of literature very little of it contains hard data. Most of it is hand-waving generalist, marking speak or so focused on outcome that it is impossible to tell which bit SAP did all by itself, and which bit was a custom bit bolted on the side.
• Nobody is a developer, there are lots of designers, even the people who you might think did configuration and table updates are designers. They all expect to have someone below them to actually get their hands dirty.

• SAP culture seems decided un-Agile. There is an expectation that business processes are be defined up front, software design up front, etc. etc.
• At the end of the development everything is rolled out, i.e. change is imposed on others.

• SAP people are plug compatible, well so I was told. Apparently you can give the SAP HR person a week off, call up the consultancy they work for and have another SAP HR consultant for that week.
• SAP consultants have a lot of frequent flyer miles. Don’t expect them to attend a 9am Monday stand-up meting or hang around on Friday afternoons.
• SAP consultants tend to work for big name consultancies. These consultancies seem to have an attachment to waterfall development. Perhaps because it gives them some nice sign-off (i.e. billing) points. Strategy, requirements, design, implementation, testing, change management, ....
• SAP is all encompassing. It is a development environment, a programming system/langauage, a run time environment, a source code control system and many other things too.
• SAP “configrability” seems to be at a GUI level, cannot textualise the configuations. Thus very hard to automate tests, compare differences and version control.
• Similarly code reviews are difficult but pair-programming should work. But given the prices consultancies charge for their consultants I don’t see many takers.
• SAP source code control does not provide for branching or merging - and probably not importing or exporting.
• Some SAP configuration can only be done by hand. So no automated builds or unit tests. (SAP doesn’t even try to support them.).
• SAP has a programming language, ABAP. Its supposed to be a little like Cobol. There is an ABAP Unit test framework. It won’t do you much good because most SAP “programming” is done through configuration and there is no way to do unit testing on that configuration.
• (These and other aspects of the SAP tool set very old. This will make life difficult to Agile teams who may need to create new technology to address these issues.)
• Once you have finished your SAP application you pay SAP to “maintain” it. Or rather, pay them not to break it with a future SAP release.
• Nothing from SAP works out of the box. One of the first things you do in an SAP project is switch on the bits you want and switch off the bits you don’t. Some of these appear to be one time only decisions, you don’t get to change your mind.

Overall I got the impression that when you buy SAP you buy a legacy system. You start from nothing and you move immediately to legacy maintenance. That should be ideal but in buying SAP you buy lots and lots of functionality you don’t need and that imposes an horrendous maintenance burden.

I hope that doesn’t come across as SAP bashing. I’ve spoken to some people this year about other ERP and CRM type systems and it appears they are all largely the same. For example: there is no way to do unit testing in Microsoft Dynamics.

SAP present their product as a suit of unified tools. This seems to be pure marketing. It seems that the system has been developed by so many different companies, teams and at different times its a miracle that any of it works together. The Microsoft and Oracle equivalents seem to be in even more divergent as these have been assembled by buying the components in.

I also learned recently that a lot of these fancy “business process” applications are actually just a series of forms on screen. People fill in forms on the screen and it gets stored in a database. Or it triggers a dispatch. Or some such. Business Process Management is often just forms and workflow. Nowhere near as exciting at as “re-engineering” or the other fancy terms that get used in this space.

For the moment I think “Business Processes” is a very a grand title for what is basically: flow charts.

Whether it is exclusive locking on a source code control systems or flow charts, I kept feeling like I was back in the 1970’s, or maybe the 1980’s. Of course, I’m too young to remember those times in IT so I may be wrong. But this is a very different world from the internet world I’m use to.

In my research, I also learned that SAP have tried Agile, specifically Scrum, internally. However corporate anti-bodies killed most of the attempts. Some infection might remain. Having seen what I saw this year I think SAP corporate culture probably can’t stand the idea of Agile in reality.

Still, my experience of using Agile on SAP projects is limited. As you can see from the list above I think there are many obstacles. I think it can be done but I think you need to go back to basics, values and principles.

Agile approach to BPM / BPE / BPR

Continuing on the Business Process theme I want to think about what an Agile approach to BPM/BPR would be.

For me there are two scenarios: Business Process Management for new processes and the “re-engineering” version which looks improve existing processes.

In the first case the answer is: start small.
Don’t just look to develop a small piece of software;; look at the business you want to enter, look at the product or service you plan to offer. Find small pieces which can exist in their own right. Maybe you can start by offering it in a small geographic area. Maybe you can limit your offering to start off. Maybe you can shave off some of the frills to the offering.

Find the smallest possible product or service that you could enter the smallest possible market with. Build just enough (if any) technology to support that and enter the market. Then set about expanding your offering and you sales. As you do add the bare minimum of technology to do it.

Yes, its incremental growth. Yes, it is bootstrapping yourself to organic growth. That doesn’t look good on a business plan, doesn’t look good when you are GE and you need big new markets and big new products to make any difference to your balance sheet. Sorry about that.

It does mean the risk is less. It does mean you could fund a start-up to do it. It does mean you could make a lot of experiments to see which markets are attractive.

So it does mean you need to change the way you enter new businesses.

Second case: you have some existing processes at the moment.

Here you want to set in place a process of improvement. Not a big bank, sweep everything away but a million small changes that build to big change. Involve the people who are doing the work. Find out what they think would improve matters.

Perhaps you want to leave the IT to one side entirely to get get started with. See what changes and improvements you can make without any IT change. When you’ve made them use IT to cement the changes.

Again it doesn’t have the big bang mega-bucks appeal of the big project but its far more likely to lead to long term success. This is how you create and sustain a continuous improvement culture and a learning organization.

The traditional “Business process” mindset seems to view the business as a gaint state machine. It is the SAP teams job to help the people in that organization move through the states and transitions of that machine. This approach assumes “the big brains know best.” Which is exactly what has been discredited in the software development world but there is still an illusion that this can happen on the business side. Thats why we need an Agile approach to business processes even more than we need it in software development.

I keep thinking of Naked Objects. It seems to the Naked Objects approach could be a great approach to bootstrap a an Agile BPM programme.

Agile and Business Processes

I want to stay with the theme of Business Processes and Agile software development for a couple of entries. I’ll break it up into bite-sized chunks rather than one rambling entry. And they may come in quick succession.

Quite a lot has come out of my last entry (BPM - is it just programming the corporation? (And the SAP Irony)), comments on the entry and conversations I’ve had recently. I’m grateful to Hans for adding some more comments to the entry which I found very interesting.

Hans suggested that the way the term “agile” is used is different between the BPM/SAP type of people and the software development community. He suggests - if I understand him right - that Agile software people value the ability to change so “agile” means “can change (software) to respond to situations”; while BPM/SAP folk consider “agile” to mean “the technology works - and we can do stuff with it quickly - which free us up to think about other (more important) things.”

There is a subtle distinction here, I wouldn’t say it is two meanings of the word, I’d say it is two ways of looking at the problem. Two different aspects view, or opportunities to exploit.

The Agile software folk say: “look how quickly we can change this software, look how quickly you can get results, now think of the business opportunities that opens up.” In other words: being Agile in your software technology opens business opportunities itself.

To the BPM/SAP folk “agile software development” means: better.
So: software will be delivered sooner, with the features we actually asked for, within the budget we first thought of an maybe with less bugs. “With the hassle of software development out of the way we can use all the free time to think about other business opportunities.”

To me this is the difference in:

• Agile software development as a better way of developing software

and

• Agile software development as a way of creating new business opportunities

Ultimately its a question of perspective and what you want.

For me the first sees IT, and development in particular, as a cost and neglects the benefits. For some business that will be true. The ones that interest me are the ones that use technology to change the way they do business and gain advantage.

BPM - is it just programming the corporation? (And the SAP Irony)

On and off I’ve run across Business Process Management - the descendent of Business Process Re-Engineering. Earlier this year I did a little work with a SAP team who wanted to be Agile, and a few weeks I was at the BPM conference in London.

What was noticeable at the BPM conference was both how dependent on IT it is, and how many of the techniques used by BPM practitioners look like IT techniques, specifically, the practices in software development.

Whenever I encounter BPM - or when I looked at BPR years ago - I get this feeling that it is an attempt to apply software development methodologies to the corporation. The BPM folks seem to regard the corporation as a machine to be programmed.

Processes are the sub-routines. These are programmed and linked together. Processes are re-usable - at least in theory. Under taking BPM seems to involve analysis, design and implementation (but no testing, strangely.) It sounds like Waterfall software development.

BPM folks want to create “the agile business.” They are talking the language of Agile and borrowing a few techniques. I even met a company at the BPM conference who have their own proprietary BPM process which looks, sounds and smells like Scrum.

Which leads me to ask: Is BPM just programming at the corporate level?

I think BPM folk would reject this suggestion simply because they view “programming” as dirty, cheap, low-end work for unwashed young people. What they do is expensive, strategic and practiced by ex-programmers (who would prefer not to talk about the past).

The other reason I think BPM people would rush to reject the idea of programming is because - as they are always keen to tell you - the software systems they use are not programmed. They may be configured or they may be customized. In the extreme you might write a customer module (or rather pay someone else to).

But, as all programmers know: programming is not just about code and its not just procedural. The configuration and customization are a form of programming. Some of the configuration tables in SAP look like table driven programming to me - not unlike a Turning machine, actually.

Now the BIG irony....

BPM folk are keen to talk about end-to-end activities and processes. Yet getting BPM tools - I’m thinking of SAP specifically - to do something means working with functional specialists who know their module and little else. An SAP Finance person would never think of working on SAP HR, an SAP HR person would never work on SAP Logistics.

A standard techniques in Agile teams is to have people work on difference parts of the system to do end-to-end. But this seems to be one thing SAP people can’t do. Ironic isn’t it, the people who advocate end-to-end processes can’t do it themselves.

Is Kanban complicated? Is Kanban Agile?

An interesting comment from Rob Bowley to my “Hardest part of Agile” blog entry. Rob asked “A lot of this discussion around Lean and Kanban Software Development seems complicated and is getting quite scientific. Does that mean it's probably not agile?”

Lets try and answer that.

When I have introduced Kanban to teams I’ve actually found it easier to get the team started on Kanban than other Agile approaches. I definately think it is easier to get started with Kanban than say Scrum.

I say this because Scrum asks you adopt Scrum perfectly before you change it. Even if you don’t adopt Scrum perfectly you need to make a lot of changes to get close to Scrum. Many teams which claim to be doing Scrum would not pass the Nokia test or any checklist of Scrum practices.

Kanban on the other hand isn’t so much a software development method but a method for process improvement. When I introduce Kanban I start by mapping out the teams process and using that to create a Kanban board. The Kanban board shows were the issues are and we work to improve them.

Why Kanban sounds so difficult is because is because people are still working out why it works, what works best and how to implement it. You need this kind of understanding to validate the approach and to apply it widely. But most people don’t. They can delve into the Kanban toolbox as and when they need to.

For example: visualizing the process on the board will help you improve flow. You don’t need to know the word flow or what’s behind it.

But, and this is the big But with Kanban, and its my big fear with it....

Kanban can be badly or poorly applied even more easily than Scrum. You do need someone on the team, or available to the team, to get the process improvement process going. It is easy to map out the board and leave it at that. Getting the process improvement bit going is the difficult bit. Whether you are doing Scrum, Kanban or anything else creating a culture of improvement is hard.

And a Kanban board shows up your problems more quickly than anything else. Which means things might look like trouble very quickly. If you don’t have someone who knows what the board is telling you, and someone who can help the team address the core problems then it can look scary.

So even more than Scrum I advise Kanban teams to have a coach. Someone who’s been there before and someone who can drive team based improvement.

Back Rob’s question. “Is Kanban Agile?”. Well hopefully I’ve explained why it is simple enough to be considered Agile so on that basis I think it is. Whether the Kanban folk want it to be considered Agile is another question. I think many of them would prefer it was considered Lean not Agile.

As I’ve said before, I consider Agile to be a form of Lean or even exactly the same (see Agile and Lean - the same but different). Kanban’s roots are more clearly in Lean than other Agile methods. I also wonder if we would have had Kanban without 10 years of learning from XP, Scrum, etc. So Kanban might be the first “post Agile” method, thats why I put it on the side of my triangle:



See Agile and Lean - the same but different for an explanation of the triangle.

More Business Strategy patterns for software companies

As you might guess, I’m lightly loaded this week so I’m catching up with a number of writing projects.

The latest set of patterns in my Business Strategy Patterns for Software Companies series are now online.

This set were reviewed at EuroPLoP 2009 and describe a Pattern Language for Product Distribution. The patterns are:

• Branded Shops
• Named Sales People
• Internet Store
• Independent Retail
• Local Guide
• White Label
• Wholesaler

And while we are talking about patterns, did you know: there is a Google Pattern Search engine? Thanks Gregor.

FAQ: What is the hardest part getting Agile?

I made a passing comment in the last blog entry to “the hardest bit” of Agile. I keep feeling I should elaborate rather than leaving the comment hanging.

A couple of weeks ago I was teaching an Agile class for DevelopMentor in London and someone asked “What is the hardest part of learning Agile?” It was a natural question and one I realised I‘d heard several times before in different forms.

So it is a frequently asked question. And the answer is...

Its not so much what you have to learn to do Agile its what you have to unlearn.

Most Agile methods are fairly easy to learn, remember they were originally termed “lightweight” and emphasis simplicity. Any method that gets overly complicated and difficult to learn probably isn’t Agile.

(One of the reasons I don’t consider RUP to be an Agile method is the complexity and size. Although you can make RUP work Agile that doesn’t make it an Agile method, but we digress...)

Generally I deliver an Agile training class in two days. We cover a lot of ground but we can’t cover all the corner cases, rough edges and unexpected situations you come across. When I work embedded in a team, as an Agile Coach, I brief the team as we go along. I might start with a couple of hours teaching to lay out some basics but thats about it. Since I’m there for longer I can teach team what they need to know when they need to know it. Or better still, I can help them find their own solutions.

(In both cases I like to use plenty of exercises. Its one thing to talk about something, its another to experience it. But we digress again...)

But what is more difficult is not what you need to learn, but what you need to unlearn. People need to unlearn previous beliefs and the behaviors that are rooted in those beliefs. For example:

• Unlearn the belief that work can be specified up front. Instead learn that needs emerge, evolve and change.
• Unlearn the idea that work can be planned in advance, specifically unlearn the idea that Gantt, Pert or CPA charts hold the solution. Learn that work is planned reviewed and replanned with blinkers on so we only look at the near future.
• Unlearn that estimates should be padded and schedules include slack time to buffer. Learn that estimates are just estimated, not commitments, and schedule slack hides problems.
• Unlearn that bugs are inevitable and we needs lots and lots of testing to find them and remove them. Learn that quality is free and we can eliminate most rework by prevention.
• Unlearn a faith in tools to solve our problems. Learn that problems are solved by people and our own initiative.
• Unlearn that “thats the way we do it around here, you can’t change the system”. Learn that if things are going to get better thing have to change around here.

I could go on but it would be a very long list. Maybe you’d like to add some of your own.

(I wrote about unlearning in a guest editorial for ACCU Overload a few years back, The Need to Unlearn.)

Unlearning means we stop doing things. We take things away from the way we work today. In the short run we might make some compromises and add Agile practices before we remove current ones, but that can only be a short lived situation otherwise we drown in contradictory practices and excessive work.

The implications are quite scary for some people. For example:

• Adopting Agile project management means you need to change the way you plan at the moment. Continuing your PRINCE2 based project management while running an Agile project management process in tandem is self defeating. Yes you can use the two together but you need to change your current practices not just add more.
• Agile development work can, and should, start before requirements are fully defined. Work is goal and value driven rather than document and feature driven. Again, you can start your transition to Agile with a big requirements document in place but if you maintain strong resistance to change you will not see the benefits of Agile working and you will not really be Agile.
• You need to trust your staff, your developers, testers, business analysts and everyone else to do the right thing. If you can’t trust them why do you employ them?

Again I could go on.

A few months ago I took a stab at guess-stimating how many people are “doing Agile” (How many people are doing Agile?). Since then I’ve come to the conclusion that a far higher number of organizations and teams claim to be Agile - after all who wouldn’t? Agile is good for you, like Apple-pie. But I think most of these places are taking the additive approach. They are applying some elements of Agile to what they have already. They are not unlearning and they are therefore not seeing the improvements they hoped for.

Magic Agile Fairy Dust and the hardest bit of Agile

There is no such thing as Magic Agile Fairy Dust. Neither me or anyone else can come to your company and sprinkle my magic dust and fix all your problems. There, I said it.

Its more complicated than that, there is no out-of-the-box solution. I can help you, you can use some tools, some methods but it involves change.

That change involves not doing some things. Changing to Agile involves subtraction as well as addition. It is not enough to just add Sprints, stand-up meetings and what not, you have to stop doing some things.

And thats the hard bit. I can teach you the basics of “Agile” in a few hours if I need to. But it takes a lot longer to help you unlearn what you did before. Unlearn behaviors, assumptions, ideas and ways of working. You can’t do that in a few hours in a classroom, it has to be done in the workplace and over time. Thats what Agile Coaching is for.

The only thing that comes close to Agile Fairy Dust is Test Driven Development (automated) and Automated Acceptance Testing. You can add those things and start to see a difference. Not immediately but in less time than you think.

Memo to self: things I learned at the UK Lean Conference

A few of points I picked up at the UK Lean Conference the other week I made a note of and think are worth recording.

Point 1: There are two kinds of Kanban limits, i.e. or work in progress limits.

The first is what we usually think of when we talk about work limits: how many tasks are we working on at any one time. This might be an limitation of the machinery and process we are using or it might be a limit we impose to manage the work.

The second type of limit is a limit on events. For example, you may decide the supply truck is only going to visit the factory once a day.

Or, to put it another way: decide what you are optimising. What is it you see to minimise? Maximise? or optimise some other way?

Point 2: Sometimes, flow trumps waste, and sometimes, value trumps flow. Or, to put it another way: Value > Flow > Waste

This was mentioned several times, and someone gave the original reference but it was too quick and I didn’t note it down. If anyone has it please let me know, I’d like to read it. Until then, here is my understanding.

When people learn about Lean there is a lot of focus on Waste. The Waste idea is easy to get and clearly makes sense. Then there are seven wastes to think about reducing - or eight if you read Liker in “The Toyota Way” and some talk about a ninth waste and I’ve even heard talk of ten. Anyway, 7, 8, 9 or 10 wastes, its something to get your teeth into and start doing.

But a more important concept in lean is flow. The idea that the path work takes through a system needs to be uninterrupted, smooth, and we should work to reduce the time it takes for work to move from one end of the system to the other.

So, sometimes, waste is acceptable if having waste improves flow through a system.

Then there is value. The idea that all work should deliver value to customers. And sometimes, we can deliver more value if we break flow.

For example: There was once a software development team who would throw away all their most recent work if the company made a new sale. The team worked in short iterations so they were never throwing away more than a couple of weeks work but, if they could make a sale by promising a quick delivery then the team reasoned they would add more value by just aborting the current iteration and starting over on the customers work.

Point 3: Waste breaks down in knowledge work
There seemed to be an emerging consensus that while the waste idea made a lot of sense that it could be carried too far; and in software work, and perhaps knowledge work in general, it lead us down some dead-ends and got in the way sometimes.

Because: knowledge work requires some experimentation and exploration, from a production point of view this could be seen as waste. However in this context it is knowledge creation.

I’m sure I learned other things at the conference but right now these are the things that stick in my mind.

As usual at conferences these points were made and learned more between sessions than in sessions. That said you need the sessions not only to give the conference form but to attract the kind of people with whom you will have these conversations. The sessions and speakers act as a kind of filter (Point 4).

A tale of two conferences (Lean & BA)

I spent the first part of the week attending the UK Lean conference swiftly followed by the IRM BA conference. Heres a rough cut at some thoughts from the two conferences.

The first day of the Lean conference was my favorite - and not just because I was speaking. It was “practitioners day” which means it was light on big names. The second day was packed full of big name speakers and oddly that was a downside. While the first day was a two way conversation the second day felt more corporate.

I didn’t make it to the third day of the Lean conference because I was at the BA conference. Now this was a corporate conference. Yet the speaker line up was not so strong. OK, thats unfair, most of those at the Lean conference were already pretty advanced at practicing some form of Lean or Agile and the conference was about improving these activities. The BA conference seemed to contain more introductory material. So perhaps I’m not the right person to comment.

Actually, although I was officially attending, and speaking at, the BA conference I actually attended several sessions at the Business Process Management (BPM) conference. The two conference were running in parallel at the same location and on the same schedule.

Although I was a little unsure how I would be received by the Business Analysis community I need to report the reception was very positive. I was talking about Agile and I found that most Business Analysts were both interested in Agile and welcoming Agile way of working.

I’m happy to report that on Twit on Twitter declared I was the highlight of the conference!

As I said in my last blog entry, the presentation is now online - More important than ever, the role of the Business Analyst in Agile transition.

So what did I learn? What were my highlights?

John Seddon at the Lean conference. He took a few pot shots at Lean and raised some valid concerns. His alternative is systems thinking. Personally I think the two approaches are complementary. I am always keen to relate Agile and Lean back or organizational learning, and I believe that system thinking is an essential element of such learning.

John introduced me to a new term: “Failure demand” - that is the demand created on a organization or system because of earlier failure. He sited examples of companies and organization where the apparent demand was 60, 70 or even 80% related to earlier failures.

One common theme that seemed to span both conferences was the dangers of using inappropriate tools. Whether it was Lean in service organizations (from John) or using Six Sigma black-belt tools when simply asking users what the problem was everyone seemed to warn against excessive reliance on a tools.

One insight that will stick with me came from Jeff Patton (at the Lean conference) who pointed out that a Minimally Marketable Feature (MMF) is not the same as a marketable product. He introduced the expression “Minimally Viable Product” (MVP), an MVP is a collection of MMFs.

Over at the BA/BPM conference I was exposed to the Business Process Management world - speakers, exhibitors, attendees. It seemed to me that BPM far more intwined IT and technology than I ever imagined.

Mary Poppendieck (at Lean) was, as ever, a very entertaining speak. She introduced a new case study: the Empire State building. Built between 1929 and 1931 the main building was completed in 8 months, end to end it was a little over 18 months.

My notebook is crowded with ideas and comments but I don’t have time to document them all here. I am sure the ideas will find there way here eventually.

Business Analysts in Agile development

My presentation from the IRM IIBA Business Analysis conference 2009 is now online

More important than ever, the role of the Business Analyst in Agile transition

Some people quite liked it, see Twitter.

The Learning View in Tools and Methods

Another article by myself, this time in Tools and Methods. Anyone familiar with my work will probably recognize the theme: The Learning View.

For anyone who’s not familiar with these ideas this is probably my shortest yet introduction to the topic.

The Quick & Dirty Myth

There is no such thing as “Quick and Dirty”. All my experience tells me “Quick and Dirty” actually means “Slow and Dirty” or “Slow, Dirty and everyone is unhappy.”

It came up again last week. I was asked: “When there is pressure to deliver something and you have the choice between getting it done, doing it right or getting it done and right, where do you come down?” It was a fancy way of dressing up the “Will you do quick and dirty?”.

Or rather, since we were talking about the management of software development the question was “Will you make your developers do it quick and dirty?”

This question, in its various forms, is used as a kind of test for us. The person asking the question wants to know if you have a similar set of values to them. In order to answer the question “right” you need to look into the soul of the person sitting opposite you and determine what they value.

All my experience tells me that when someone chooses a “quick and dirty” path a) it takes far longer than is expected, b) the quality is so low we go around the loop a few times making things slower, c) every one involved will feel bad, even “dirty” about the soltiion and d) it causes more problems very soon.

Perhaps (c) and (d) are where the name comes from: “A choice of actions which makes you feel dirty and quickly cause more problems.”

The “quick and dirty” or “slow and right” dilemma is only a problem because of the way we approach the question. There is an assumption that there are two, and only two, options. One will take a while and one can be done quickly, one is “good” and one is “dirty.” These two options are then juxtaposed and connected by the “tyranny of the OR.”

Everything about the question smells of winner and loose, or zero-sum game to give it a fancy name. “Quick and dirty” means “customer wins, developer looses”. And the alternative is “customer looses, developer wins.”

As someone who manages software development teams I find the question particularly naive. When I’m managing the team I can’t actually make anyone do anything. I may tell a developer to “do it quick and dirty” but I have no way of knowing what they will actually do. Indeed, asking developers to repeatedly do things “quick and dirty” not only damages the code base but means I’ll probably p***-off my developers. Before long they will be heading for the exit.

My solution: don’t accept this framing of the problem. Get more options on the table. (If you can stomach the tired old cliche: find a win-win solution.)

Define what you mean by the “quick and dirty” solution. What will it involve doing? Why do we think it is “dirty” ? Why do we think it is “quick” ? What will be the outcome? Will it meet our needs? Indeed, what is the real need? Is it what we think it is?

Repeat the exercise for your “slow and clean” option. Now if these two are opposite end of the spectrum what are all the thing in-between? What other options are available? What questions do you have which you should clarify first? Often there are questions over what is actually needed, when these are clarified the problem and solution may look very different.

Now of the options in front of you can you combine any of them?

Usually when I do this exercise more options open up. There are now “clean and quick”, “minimal” and other answers available. Sometimes “do nothing” becomes an option.

Unfortunately you also find that sometimes you options are very limited. Everything looks “dirty”. The “right” solution is not even on the table so there never such an option in the first place. Doing it this way also makes people feel better, if you do have to compromise somewhere then at least everyone involved has had a chance to have their say and look at the options.

If this all sounds like it will take too long then try it and see. Taking longer in the “what are we doing” phase usually saves time in the “do it” phase.

This is not to say there are not times when you should not take fast action. Sometimes the right cause of action is to “damn the torpedoes,” push on, take the risk and get something done. However those occasions are about getting stuff done, not about making choices.

So, next time someone asks you to do “quick and dirty” remember: there is no such thing as “quick and dirty, only dirty and slow.” Then get some options on the table.

Public training course next week

While I’m on the subject of training courses. I’m teaching two public courses next week fro DeveloMentor in London - Foundations and Requirements.

I believe there are places left, if you (or someone you know) calls DevelopMentor quickly and mentions my name you might even get a discount.

What is Advanced Agile?

Suggestions please.

I few weeks ago I received a request to create and deliver an “Advanced Agile” training course for a Scandinavian client. My first reaction was: that is more of a consulting or coaching assignment. But when I thought it through and considered the kind of material I covered in my own “Agile Foundations” course, and the introductory courses I know from other people the more I realized how much more there was I could cover.

I’m not most of the way through creating the course and I’ll deliver it in a couple of week. I’ll report back on what I put in and what people thing.

In the meantime, I was wondering: what do readers of this blog consider Advanced Agile?

Suggestions please, on this blog or e-mail me and I’ll summarize them.

Defining Agile in e-Technology Management

I have a piece by me in the latest issue of e-Technology Management entitled “Agile agile everywhere - not a definition to speak of”. Believe it or not, for all the stuff I write in this blog about “agile” I’ve been trying to define what “agile is” for a long time. When I deliver a training course I always want to say “Agile is...” but there is no simple/short definition I can come up with.

In the e-Technology Management article I distinguish between Agile methods, the Agile toolkit and Agility or, to put it another way “the sate of being Agile”. The term “Agile” is really a label for all of these ideas. Its short hand, its useful in conversation but if you want to really understand it you have to go beyond the label.

One of the problems with “Agile” is that it are often defined as “not the waterfall.” Someone says “What is Agile?” and the answer starts out “Well you know traditionally we did requirements, then analysis, then development, then.... ?”

Yes its laziness, yes I’m guilty, but I’m not the only one.

When we define Agile by what it is not then the scope is unlimited. In my experience, very little development work happens the way it is supposed to in the “waterfall”. So most development is “not waterfall”. Thus, if you define Agile as as what it is not (i.e. not the waterfall), and most development is not waterfall (at the point the work is done) then most development is Agile.

Anyway, I’m sure this is a theme I’ll return to.

97 Things Every Programmer should know

Behind the scenes Kevlin Henney has been busy the last few month soliciting and editing contributions to a new O’Reilly web publication entitled 97 Things Every Programmer Should Know.

I’m proud to say a couple of my contributions have made it into the list:

• Check your code first before looking to blame others

and

• Two Wrongs Can Make a Right (and Are Difficult to Fix)

Other contributions are from names which will be familiar to ACCU’ers: Michael Feathers, “Uncle” Bob Martin, Pete Sommerlad, Pete Goodliffe, Klaus Marquardt, Giovanni Apsroni, Jon Jagger and of Kevlin himself. Plus there are are contributions from many others.

Just don’t ask me why it is 97 and not 98, 99 or 96 things. 97 it is.

Not quite the doom of Agile

Just in the nick of time I resolved my baby sitting dilemma and made it to Tom Gilb’s SPA London talk, only missing the first 10 minutes. Here are a few notes I took - I don’t think I missed anything radical in the first few minutes but if I did I’m sure someone will correct me.

Quantification
As always Tom was provocative and seeded a lively debate with the audience. As always Tom pushed for quantification and measurement of what software work is trying to achieve and how successful it is. And, as always, many in the audience pushed back on this.

Without going into too much detail: Tom says you can quantify just about anything, and for a software development you should. Many other people question whether you can quantify everything, whether it is necessary to quantify and if it is cost-effective to quantify.

I see both sides of the debate. I tend towards Tom’s point of view but I understand the reservations and share many of them. I also think there are some things it is extremely hard to quantify. Tom has this skill. Many others don’t and one of the results is a lot of bad quantification, bad metrics and consequently lots damage done.

Agile
Agile is not doomed the way the talk synopsis made out. Indeed the conclusion is much more that Tom’s methods - Evo and value management - are complementary t o Agile and specifically Scrum.

He didn’t address the “XP is already dead” quote until he was questioned at the end. This assertion seems to be based on the assumption that XP was mainly hype and now the hype has gone XP has gone.

Where Tom sees Agile methods going wrong is in the lack of understanding about what needs doing, for who it needs doing and a focus on “features” over “performance.” I agree with him here, I’ve blogged and written elsewhere about this problem myself (e.g. Requirements: the next challenge). I even gave a talk at the BBC a few years ago entitled “What’s wrong with Agile”.

Tom does believe teams need managing. Although he didn’t say as much that implicitly rejects one of the pillars of Scrum: the self organizing team. I don’t think Tom wants to see a return to command and control management - far from it - but he does see a need for active management. And that brings us to the next point.

Stakeholders and Product Owners
One of Tom’s key point is the role of stakeholders. That is, the need for all software development work to identify the stakeholders in the work and what they value from the work. Different stakeholders have different values and want different things, and these things change over time.

Tom asserted that stakeholder identification and management is not covered by Agile methods. Last night I thought “I’m sure it is in...” but now I look through my collection of Agile books I see its pretty thin on the ground. That said, there aren’t many references in Tom’s own Competitive Engineering book either.

As far as I am concerned, stakeholder identification, communication and management falls firmly within the Product Owner role. This is part of what Product Owners need to do when they are off the Agile stage and is part of the reason this role is so important.

At one point Tom stated that the Product Owner role dooms the project. I challenged him on this and it turns out we are in agreement. The doom occurs when the Product Owner is not managing the stakeholders.

Value over stories
Stakeholders have needs. Agile teams often interpret those needs as features. Tom suggests that User Stories feeds this assumption. I’ve blogged about this before: Requirements not functionality.

Tom proposes that by understand who the stakeholders are and what they value we can start to deliver value rather than features. Now I believe User Stories can be used to communicate this if they are written to communicate this. Tom’s own alternative to User Stories - Planguage - emphasizes the quantification, and thus value, more then User Stories. That said, Planguage is more difficult to pick up and use than User Stories.

Value driven planning
When Tom puts all this together you get Value Driven Planning. Here the idea is that critical stakeholders determine the value of software work.

Naturally there is conflict here as stakeholders argue about which work has the greatest value. The important thing is: conflict is natural. It is not resolving the conflict which is the problem. Once conflict is identified it can be resolved.

Potential v. Delivered value
Another key point to understand is the difference between potential value and delivered value. Too many projects have potential value which is not delivered because the software is not used, the users understand how to use it to the full or something else gets in the way.

It is not enough for the development team to measure the value they think they have delivered - e.g. functionality that can be used. They need to measure what value is created.

When there is a difference between these two numbers then it is time to act. If potential value is 1024 and actual delivered value is 500 then there is a lot of value being lost.

Scaling
For me a lot of this debate comes down to scaling. In small teams, with few stakeholders and small(ish) budgets doing a lot of stakeholder analysis, quantification and measurement may well be more expensive then just doing it. As was pointed out by someone, the trial-and-error approach of iteration works well, and cheaply.

When there is a big team, many stakeholders and this a big budget then not only is it worth making these investments it is necessary to ensure the team is doing the right thing, that everyone agrees on what is being done and understands why, and that the budget is justified.

Perhaps that my biggest “take away” lesson from last night.

In the end...
As you might tell, and is often the case with Tom, he hits you with hundreds of ideas many of which are left hanging there for you to follow up or thing about later.

The conclusion to my previous blog entries this week (The Doom of Agile, Thursday and Tuesday. ) is: Tom doesn’t really disagree with Agile, he is highlighting a issue others have seen and wants to shift the focus.

If you are interested Tom’s slides can be downloaded (be warned, this is a big presentation, 23Mb in PPTX format). Mark Stringer at AgileLab has some comments on the presentation too.

Postscript: The doom of Agile

A post script to my last blog entry (“SPA London, Tom Gilb and the doom of Agile”) generated some comments and a couple of private e-mails.

I think Mark is right when he commented: “they are 'fads', the hope is they will just become nameless best practices”. Many of the Agile practices were already best practices in places, but until they were named, publicized and, yes, hyped, few people knew of them.

Have a look at EPISODES from 1995, thats 4 years before “XP” and 6 years before “Agile”. Its documenting best practices. Or one of the earliest attempts to document Scrum, The Scrum Pattern Language in 1998.

Both of these are patterns, which means they document what is being done, not what is suggested. they are documented because people think these are good practices.

One day, the Agile “hype” will fall down and what we are left with is the practices. Thats what I was trying to say when I wrote “ If XP is dead then it died so Agile might live.” The hype around XP is less, much of XP has already become best practices - think of TDD, think of daily stand up meetings. Obsessing about XP is wrong but doing the XP things is right.

Thats why I also agree with Rob Bowley’s comment when he wrote: “XP is here to stay (as much as Agile is) because through time it has proven to be effective.”

We may dislike hype but its part of the way the world works. Object oriented programming, UML, ISO 9000 and CMM have all been hyped and left their mark. SOA is hyped today, it means a lot to some but technically its just a continuation of modularisation (first there were static libraries, then there were objects, then components and now services.) Without hype people wouldn’t get to hear of these things. That’s why Simula and many other technologies have fallen by the way.

We might argue over which Agile method is best: XP, Scrum, Kanban?
We might disagree over which practices are applicable: TDD, BDD, stand-ups?
Or how to implement these ideas in unusual environments: distributed teams?

But I don’t see anyone seriously arguing for a return to to “waterfall”.

I still hope to make it to Tom’s presentation on Wednesday but we seem to have a baby sitting clash so I might be grounded. Won’t know until the last minute unfortunately.

SPA London, Tom Gilb and the doom of Agile

I’m planning on going to SPA London session next week, Tom Gilb is giving a talk entitled “Does real Software Practice Advancement need yet another 'Manifesto'? ” He’s being quite provocative, look at this from the synopsis:

"AGILE HAS DOOMED ITSELF - TO BECOME YET ANOTHER FAD: XP IS ALREADY DEAD. What is Seriously Wrong with Agile practices and interpretations - why AGILE, AS CURRENTLY PRACTICED, is PROJECT-failure-prone as a culture."

I think Tom is both right and wrong. He is right that projects need more of a value driven approach, he is right that Agile is a fad but he is wrong to say XP is dead or Agile is doomed. Personally I’ve been expecting an “Agile backlash” for at least two years now, and to some degree I think it deserves it.

I too have concerns about “Agile” both the marketing of it and what lies below. The term is very vague, covers a multitude of things and is often abused. Its fair to expect it to come down a peg. But, now I look too faced, and you may be asking “Why does Allan talk a lot about Agile?” the answer is simple. Its the best tag we have at the moment, its the best term for grouping all these ideas together.

Lets take the synopsis one part at a time.

“Agile has doomed itself”: perhaps, the term is poorly defined. As a result a lot of bad practice hides behind the Agile label and gives Agile a bad name. Still there is a lot of good in there.

“[Agile] is another Fad”: certainly there is a lot of hype, much undeserved and certainly this will blow over. As bits of Agile become mainstream some of that will go naturally. But some hype is necessary, its called “marketing” and its all around us. The Agile bubble will deflate, it might get pricked but thats not a reason to attack it alone.

“XP is already dead”: I’m sorry to see Tom adopting this view. XP’ers were the storm troopers of the Agile movement. The very word “Extreme” meant it would repulse some people. But XP got people listening, got people excited and now XP takes a back seat.

Look at what XP has left us with: Test Driven Development, Continuous Integration, Refactoring, simplicity in design - the technical practices which underpin Agile development.

XP’s project management side was always a bit weak but it isn’t that different to Scrum. Scrum finesses it a bit, makes it more acceptable to management, adds in some bits that were missing and tightens it up but the two are the same. If XP’s project management is deeply flawed then so too is Scrum.

So XP is still with us, it just split in two. If XP is dead then it died so Agile might live.

Hopefully the talk will explain the rest of the synopsis.

Now why Agile is a success and is here to stay. There are three parts to this, one especially for Tom.

First, as I’ve argued again and again, Agile software development is a branch of Lean thinking, in particular Lean product development. Lean has been around for a long time and for about 20 years has been documented, studied and been something of a “fad” itself. As such I don’t think Agile is going away, but I think we will see more use of the word Lean.

Second is the name factor. “Agile” is a solution, it is a solution to slow software projects, it is a solution to poorly run software projects, it is a solution to poor quality software, it improves team performance and customer satisfaction. Yes “Agile” is a vague term but it is a label for all those things.

Go searching on Google for any of those things. You’ll find what you want, eventually. You will need to dig through lots of stuff first. Yes “Agile” is something of a marketing term but it is a marketing term that labels a whole set of desirable outcomes and ways of achieving them. As such it is short hand for finding ways to make software work better.

Finally, there are lots of different pieces make up the “Agile” toolkit. Scrum brings Project Management. XP brought better quality software - a crown inherited now by the Software Craftsmanship movement. Crystal (remember that one?) and Lean bring continual improvement. Elsewhere I’ve written about Agile governance, and so the list goes.

Tom and Tom’s Evo method brings important tools to that toolkit. Evo pinoneered short iterations. Evo emphasizes value delivery. Which leads us to better requirements understanding and governance. Evo emphasizes measurement. Planguage gives us another way of documenting requirements. And so the list goes on. If you are not familiar with Toms work then its well worth a look, lots of good ideas.

But, Tom’s ideas aren’t the whole toolkit. XP, Scrum, Lean, Kanban, etc. all bring something to the party, its up to you to decide which guests you want.

On the one hand I find it ironic that Tom appears to be attacking Agile. For me Evo is an Agile method. Evo shares common roots with Agile, namely the work of Edwards Deming. On the other hand, I enjoy Tom’s ideas, he’s thought provoking and I always learn something. Maybe he’s just getting some of the Agile hype for himself.

Events: Business Analysts & UK Lean conference

As the summer comes to an end conferences start again.

At the end of September I’m going to be at the UK Lean Conference (27-29 September) on a panel discussing Kanban. A couple of days later I’m going to be across town at the IRM Business Analysis conference (28-30 September).

Monday the 28 is going to be a tough call. Should I go to the BA pre-conference session? Or the second day of the Lean conference?

If your going to be at either conference please say Hello.

Agile failure modes from othe

I have been preparing some training material for a company which wants to know more about how Agile projects fail. Fortunately I’ve made a habit of recording Agile failure modes in this blog so I have much of the material to hand. (Use the search box above and search on “Agile failure modes.”)

But, I’ve not got exclusive rights to this topic and others have done the same thing, often with catchier names. So, for completeness:

• Wagile from Jason Gorman
• ScrummerFall from Mitch Lacey
• ScrumBut from Eric Gunnerson
• WaterScrum from Scrum Community

As far as I can tell these are the original sources of these terms but I may be wrong. Others have also used the term and some have Wikipedia entries so I could be wrong.

The Product Owner role

The Agile Journal this month has an article from me this month entitled Dissecting the Product Owner role.

This complements my pieces on the Business Analyst and Product Manager roles which appeared in ACCU Overload earlier this year.

CMM & Agile

I’ve finally, about 6 months after everyone else, got around to reading the Software Engineering Institutes report on CMMI and Agile. Entitled “CMMI or Agile: Why Not Embrace Both!” the report argues that CMMI and Agile are not only compatible but complement one another. Actually, you can guess at what the report says from the fact that the title ends in a exclamation mark (“!”) rather than the question mark (“?”) you might expect.

The report argues that the two are compatible and complementary. They make a strong case and I for one am glad. CMM initiatives have too often made the words “process improvement” dirty words. They shouldn’t be, it is what we should be doing.

The report argues that CMMI, and its predecessor CMM, have been misunderstood and misused by the industry. Firstly CMM(I) was taken to be a methodology when it was a model. CMM(I) doesn’t tell you what to do, it allows you to evaluate your capabilities.

The second mistake compounds the first by imposing CMM(I) on organizations. Rather that improving an existing process and organization CMM(I) initiatives have replaced what already exists with something else - and in the process destroyed the good that existed. This is the scenario I wrote about a few weeks ago in “A true story about CMM”.

There are a few points were I could take issue with the report.

My first issue is that the report does not define what it means by ”Agile.” I agree this is a problem, and one I’ve been thinking about addressing in this blog. But, CMM and CMMI are very well defined while Agile is not. How can one critique a method without defining what is mean by the term?

Second, the report seems determined to identify an “Agile Institute” to mirror the Software Engineering Institute. It seems to spend a lot of time talking about the authors of the Agile manifesto and founders of the Agile Project Leadership Network. Neither of these groups owns the term “Agile” or speaks for the whole Agile community. There is much under the Agile umbrella which, rightly or wrongly, falls outside these two groups.

Comparing Agile and CMMI is an asymmetrical comparison. One is clearly defined, with a well funded champion to issue standards, revisions and issue certificates. The other is the ill-defined product of an ill-defined community which owes as much to the punk ideal as anything else.

A bigger issue with the report concerns why you might want to bring CMM(I) and Agile together. The report spends several pages early on explaining that the CMM grew out of one type of environment with one set of problems. Namely large projects, often military in nature, often with life at risk, and with a low-trust relationship between supplier and client.

Conversely, Agile came from a different environment: mainly corporate IT groups, small projects (no more than a dozen or two people), with high-trust within the team. Failure to understand these origins and contexts has led to much misunderstanding between advocates of the two.

Given that these two environments still exist, and that Agile has demonstrated success in its native environment and CMM(I) has demonstrated success in its native environment, then why would people in either environment want to adopt elements of the other?

I don’t think this question is really answered. This is a shame because the authors have spent a lot of time setting up a story to explain why we got where we are today but don’t explain how that story ends.

To my mind the story only tells part of the truth. Both CMM(I) and Agile have been sold into many organizations outside their native environments. There are many people who want the respectability of CMMI, and there are others who want the fashion of Agile. Nor am I aware of anyone in the Agile or CMMI camps saying “Don’t do that sort of project with our model.”

I’ll defend Agile here because I believe Agile has, and is, a journey of exploration. The underlying principles are universally applicable but that doesn’t mean all the details are worked out in any given field yet. So if you ask me “Can I do Agile on an X type project?” my answer is “Yes but I may don’t know how immediately.”

However the SEI folks who wrote this report claim CMM(I) has been misapplied. So I ask: which of them was saying “Don’t use CMMI on an X type project” ?

My final niggle is one of clarity. The report has five authors. Several of those authors are quoted and referenced throughout the text to support the arguments made. Yet nowhere in the report, when one of the authors is quoted is there a disclaimer. That is, the report does not say “Smith, one the authors of this report showed the CMMI is effective (Smith 2003).”

The names of the authors aren’t hidden, there is no subterfuge but its not clear and I wish they had been.

On the more positive side there are several quotes in the report worth calling out. About Agile and CMMI:

• “In today’s increasingly dynamic world, CMMI-based organizational process improvement approaches cannot rely exclusively on traditional project management approaches, waterfall-based project lifecycles, and heavyweight analysis methods.”
• “CMMI and Agile can complement each other by creating synergies that benefit the organization using them. Agile methods provide software development how-to’s that are missing from CMMI best practices that work well”

And about CMM(I) and the approach companies have taken in the past:

• “Too often, CMMI has been applied rather than implemented”
• “SCAMPI appraisals are not audits or tests”
• “the complexities of large-scale projects require a more involved infrastructure, this fact is not a license to create unnecessary bureaucracies and unbalanced production at the expense of productivity”
• “Using process experts to create and deploy process improvement activities is not unusual, and often has the advantage of allowing these experts (and the project teams) to focus on their respective tasks. But what makes this approach risky is that project personnel are frequently left out of process design activities and are disinclined or openly skeptical toward the adoption of process improvement activities. This situation is typical of some Six Sigma style approaches to process improvement as well. Quality process teams are created and given the responsibility for quality. As a result, developers abdicate their responsibility for improvement (or worse, for quality).”

Interestingly I also noticed that the SEI’s own Team Software Process (TSP) is now listed as an Agile method. I’ll have to look into this some more.

I am also please that the report highlights the role of organizational learning in software development. Regular readers know this is something I’m very keen on (heck, I wrote a book on the subject!) so I’m pleased to see it explicitly considered.

Even with my reservations I welcome the report. I believe it is a roundabout endorsement of Agile (in its various forms) by the Software Engineering Institute. It is also a very big sign saying “Agile and CMM(I) are compatible.”

A favour to ask of my readers

Dear readers,
There are getting to be quite a few of you. I know who some of you are but not all of you. And I know a few of you have bought, and even read my book, Changing Software Development: Learning to Be Agile.

I have a little request to ask of you: would any of you care to write a review of the book on Amazon?

Yes, I’m looking to improve my sales and the more (positive) reviews the better.

Thank you very much.

allan
ps Little bit of trivia for you. Last year I discovered the book I thought was called “Changing Software Development: Learning to Be Agile” was actually called “Changing Software Development: Learning to Become Agile.”

Well, a few weeks ago I discovered the book has both titles. If you have a copy look at the cover, the title if “Learning to become Agile”, not turn inside and look at the title on the inside page, here it is “Learning to Be Agile” - the way I wanted.

Reprise: Leave your laptops, Blackberries and phones out of meetings

At the end of last year, or was it the start of this year, I wrote a blog entry which got a lot of attention: Leave your laptops out of meetings. I was talking specifically about laptops but I could have easily included Blackberries, iPhones and other portable devices.

In todays Financial Times two professors from Toronto Rotman School of Management make the same call: Why e-mail must disappear from the boardroom. Their piece specifically addresses e-mail and boardrooms but its the same argument I was making.

Professors Beaty and Weber wonder what how directors can concentrate on discussions when they are doing something else. And the question what will happen when someone decides to sue a company for decisions taken by directors who are not concentrating.

So there are three good reasons to leave your laptop out of the meeting, switch off your Blackberrry and ignore your SMS messages:

1. You cannot do your job properly is you trying to do two things at once
2. You are opening your company to negligence claims
3. It is rude; you are insulting the people who are in the room asking for your attention

For the record I very really take a laptop to a meeting, only really if I have something on it I want to share. I don’t own a Blackberry, iPhone or other portable e-mail device. And I set my mobile phone to silent when I arrive at an office.

I am guilty of occasionally viewing an SMS during a meeting, and even more occasionally sending an SMS during a meeting. I apologize to everyone in a meeting where I have ever done so.

I fully support Beaty and Weber’s call: think of the message and example it would send if a company board stated that Blackberries and laptops were not to be used in meetings.

The Scrum Wall (another Agile failure mode)

I recently came across the expression “The Scrum Wall”, as in the expression “Hitting the Scrum Wall”. Its akin to “the pain barrier”, or “feel the burn” in aerobics workouts of old.

Once I heard it I knew what it was, I’ve talked about this before, now I know it has a name.

The Scrum Wall is the thing Bob Martin described at the ACCU conference. And it is probably why Jeff Sutherland endorses Test Driven Development and other technical practices from XP.

You hit the Scrum wall when you adopt Scrum and everything goes well, then, after a few Sprints things don’t work any more - to use an English expression, they go pear shaped. You can’t keep your commitments, you can’t release software, your customers get annoyed and angry, it looks like Scrum is broken.

This is what happens when you adopt Scrum without technical practices such as Test Driven Development, continuous integration and refectoring. When teams adopt the Scrum process, they go faster, show progress, things look good... and then the quality becomes a problem. Now the team are fighting through quick sand.

The code quality is poor and developers are expected to continue to make progress. Maybe the Scrum Master/Project Manager reverts to past behavior and demands overtime and weekend working. Maybe the team start busting a gut to keep their commitments. Either way the team is heading for burn-out.

That’s the wall.

A true story about CMM

About a decade ago I worked for a major supplier of financial information in London. The company decided it wanted to improve its software development process and the answer to this problem was to adopt CMM - the Capability Maturity Model. Initially the become level 2 and then move on the level 3.

The little group I worked in interfaced to financial trading institutions in the City of London. My project in particular had to respond quickly to changes at the institutions.

The CMM roll-out was a akin to a neutron bomb in reverse. The people were left in place, but their processes and practices were replaced en-mass. Developers attended CMM training and instructed in how to develop software according to the new process.

The effect was devastating. One manager likened it to pouring quick setting concrete on the railway tracks of progress.

I went out and I bought myself a book on CMM so I could understand it. What I learned was that CMM was a way of measuring maturity. It was a meter, imaging it as a ruler. It was a way of assessing your organization. You could measure yourself as 1 CMM, or 2 CMM, up to 5 CMM. It was not a method of working.

The CMM body of knowledge also contained recommendations, “best practices” and examples of good practices but it was not itself a process. It was a meter for measuring your maturity.

From the book I learned that Watts Hymphrey advised companies to: focus on improving their processes first and let the CMM measures improve by themselves.

What I observed in practice was the destruction of working practices an processes. I never observed any attempt to involve those doing the work in process improvement. The consultants knew best. Opposition was to be overcome. Process improvement was something that was done to others.

This wasn’t the company for me and I left. The company was eventually certified as CMM Level 2 and was expected to move to level 3 in time. Time went by and I don’t think the company ever made level 3. I heard that internally CMM was recognised as a mistake and the work was quietly allowed to die. The company also announced major moves to offshore development.

Fast forward to 2009. I recently met someone who was an outside consultant to the company and advised them on CMM adoption. He told me that he, and the other consultants, had advised the company to build on the practices they had and improve them. They specifically advised them not to write new processes and impose them on people.

He also told me that when the company did its own retrospective on the CMM project it was seen as a mistake to have written and imposed new processes on developers. A far better way would have been to improve what was in place with the people who did the work.

So the moral of the story:
• There is a lot of good in CMM - and the more recent CMMI.
• CMM itself is not a process, it is a meter, it is a means of measuring yourself.
• The key is process improvement, focus on improving and the levels will come.
• Don’t replace, improve, evolve. Involve your own people.