Agile and the Demand Curve

In the last two posts I’ve taken two different views on what might happen to the Software Demand curve when we introduce Agile:

• The good Agile demand curve
• The bad Agile demand curve

When we discussed how Agile effects the software supply curve things were pretty clear cut. Agile, in the long term boosts supply, and may even flatten the curve - making it more responsive to price changes, i.e. more elastic. Unfortunately things are not so clear cut with the demand curve.

Good news: There is a story Agile people - OK me! - sometimes tell - about what “should” happen with demand in an Agile environment.

Bad news: The is another story about demand in an Agile environment, one that Classicist (i.e. believer in “Waterfall” style up front requirements and planning) fear. And I’ve seen that too. It ain’t pretty.

But, the key point is: Few of the Agile tools work directly on the demand curve. After all, Agile is about software and the demand for software is derived from something else so why would they?

Let me say that again in other words because it is important: Nobody wants software for the sake of software, the demand for software is derived from some other demand, e.g. a retailer wants to sell online, that means they need software.

The current Agile tool set (e.g. User Stories, Acceptance Criteria, Specification by Example, etc.) which touches on demand - call it need, want, requirements, specification, whatever you want - does not address the underlying demand. These tools operate with the demand as presented.

Sure there can be feedback from the tools and Agile deliveries but feedback is often absent. Even when feedback is present those responsible for the underlying demand may not want to hear - or act - on the feedback. And in an environment where success is measured by criteria like “On Budget, On Schedule, On Features” acting on feedback may be impossible.

Which brings us back to the 3 Styles of Agile: Iterative, Incremental and Evolutionary (part 1) and 3 Styles of Agile part 2. (At some point in the future I will reprise those three styles in the light of this analysis.)

One of the key factors in determining whether the feedback can affect underlying demand is whether those responsible (e.g. senior managers) who largely originate the underlying demand are open to feedback. Which brings us to another story I need to tell, another time...

Where does this leave us then?

If the current Agile toolset cannot address the underlying demand we need new tools. We might choose to add those tools to the thing we call Agile or we might choose to put them in another category.

Personally I believe many of those tools do exist but they exist outside the Agile space. They are fellow travellers if you will. For example: Beyond Budgeting is clearly one of those tools. However, many of those tools have their own names, they are not Agile, they are what they are (e.g. Beyond Budgetting). They fit very nicely with Agile but they are not.

However that is my personal view. I also think that Agile is a marketing bang-wagon and those tools will be conscripted to the Agile course and in many cases put under the Agile umbrella.

Either way the problem we need to recognise is: addressing the software demand curve Agile is necessary but not sufficient by itself.

The bad Agile demand curve

In explaining the Agile Demand curve I told a good news story in the previous entry. This time I want to tell a bad news story. This is every classical Project Managers’ fear about Agile, its story we don’t tell often but it can happen. Once I’ve told this story we can proceed to the proper analysis of the Agile Demand curve.

So a classical project manager looks at Agile and he sees his greatest fear: No Scope limit.

There are no signatures against exactly what is to be build, not even in pencil, let alone blood. There are some loose “Story things.” But he is told it is Agile and Agile works this way so against his better judgement he gets stuck in - after all, he needs to get Agile on his CV sometime.

And to some degree the classical Project Manager is right to have this fear.

After years of being pushed around by IT and IT’s sub-contractors the business hears of Agile and thing “Yes! - we are finally going to get what we want. No more messing about with long documents, no longer waiting for ever and no more bugs.”

These days when I go to talk to a company about “going Agile” the Agile publicity machine has beaten me to it. Unfortunately for some on the business side they believe that “going Agile” means they can have whatever they want whenever they want it. If the development team are Agile, so the logic goes, then they will deliver fast and will deliver exactly what is wanted even if the business keeps changing their mind.

You can hardly blame the business side from believing this, after all, it is how Agile presents itself. Unfortunately Agile as “all things to all people” and “I don’t need to change what I’m doing” is more prevalent than is healthy.

The next effect of this attitude is to boost the Demand curve significantly, and if it does affect elasticity it probably reduces it further.

The demand curve moves from D to D’. At every price point the business wants more. Of course the business doesn’t think of it that way. They are just asking for what they always wanted but now, because it is “Agile” they are not restricted.

Previously, the need to put every request in writing - voluminous writing - and argue it with various managers meant making the request required effort (a price) which they were not willing to pay. No with Agile all that is gone and demand is unbounded.

There is one “project” I know which is quite large, we are talking dozens of people all counted. Demand is rampant, the business keeps on thinking of new ideas, analysts keep finding more work in what has been asked for, developers find more work and the testers more (because initial quality is not what it could be.)

In short: I’ve seen rampant demand happen and I think Agile makes it more likely because it removes many of the restrictions previously used to control demand.

If the team don’t deliver on (often unreasonable) expectations “Agile” becomes another tool with which to beat the team.

It is worth noting here: that as demand is derived much of that demand always existed but was hidden. What is a more interesting question to ask is: which elements of this demand are worth doing? That question will need to wait.

We need to add supply to our diagram above. There are two scenarios we need to consider here and shows in this diagram:

Scenario 1: Demand moves to D’ but the developers are still operating on the original Supply curve S. In which case the amount and the price is going to rise in both dimensions. The next diagram shows this, the yellow area is the pre-Agile cost while green area shows the additional cost.

Scenario 2: Demand moves to D’ again but this time the team are operating on the Sa (Supply Agile) curve. Thus we get this diagram:

Things are more complex now: because supply is elastic and responds to price more software is produced than at the beginning but the unit price is lower. So the blue are represents increased cost but the purple area represents money saved.

The immediate question I bet you are asking is: have they saved more than they increased?

As it happens yes. On my graphing tool the purpose area takes up 45 grid squares and the blue 46. So they have saved, just.

But, those curves were placed by my for readability without any data for any actual team. The difference is small and demonstrates things could go either way.

A more important question which should be asked is: are the team delivering more benefit? More value?

When I started this blog entry my aim was to show that talking Agile could by itself increase demand. The problem I want to highlight is rampant demand, and this is not confined to Agile. Although I think Agile can make it worse.

What I have also shown is that if the development team really is Agile they can handle this but it isn’t clear.

The good Agile demand curve

I’ve been slow with this entry on Agile and the Demand Curve, apologies. Part of the reason for the delay is software demand is not simple.

In my last entry on software economics I discussed why the Software Demand curve for custom software is both high (i.e. people want a lot of software) and inelastic (i.e. increases in costs don’t reduce demand by much.) This time I want to look at how applying Agile changes the demand curve.

In order to explain what happens to the demand for software in an Agile environment we need to consider different possibilities. So lets start with a good news story - the bad new story and analysis will come in later blog posts.

This story is about how, using Agile software development, a team reduced the demand for software and shipped a product sooner. This is the story most Agile advocates will associate with and happily promote. This is the way Agile is supposed to work....

There was once a team who adopted Agile software development. At first they were able to demo their enhanced software to others in the business on a more regular schedule. They made a big thing about this and invited others to come and look. This generated feedback on the thing they were creating: some good, some not so good, some requests for more and some cancellations of things which had been asked for.

A little while later they stated making more regular releases. And because the software was “ready to release” more often the business could decide: “Do we want it now - with any pain that might bring? Or do we want to wait a little longer for more?” (A bird in the hand is worth two in the bush as they say.)

Simultaneously the team were adopting technical practices which improved the quality of their code - they also had the side effect of reducing the amount of code. Not only did this make the team more productive because they had less code to maintain and bugs to fix but it also meant the business had fewer reasons to complain about bugs.

(In my experience, for some teams when the conversations about “Have you fixed it yet?” and “Can we release it yet?” start to disappear it turns out business don’t know what to ask for. They’ve forgotten or never knew what they really wanted.)

The team also adopted User Stories at some point. As they got better at this technique they were challenged to think about “Who actually wants this story?”. With a little time stories advanced from “As a User” to something more specific.

And one day the “Product Owner” realised: they were not building one product but two.

The Product Backlog of work to do would be better throughout of as two backlogs for two products, each with its own user/customer base. User Stories had helped him see this.

When he restructured the backlog he found he had three piles. Product A, Product B and not justified. A chunk of the backlog, maybe as much as a third, didn’t fit in either product. They were nice ideas, things people had asked for, but when you came to analyse them they didn’t belong anywhere and justification was weak.

This diagram shows the Demand Curve graphically. The team start with demand D, Agile supply is Sa. After the restructuring to separate products appear with their own demand curves Da and Db. Each on of these can be reasoned about independently.

At any price point the combined demand of A plus B is less than D (the original total) because some work has gone away. Note: I have not suggested any changes to the slope of the curves, it is possible that different products will have different elastics but that is another discussion.

While the team had been battling lots of bugs and their output was a black hole to the rest of the organization, and while they thought they were building one big product this insight had been hidden from them.

That is the way Agile is supposed to work on the demand side. By being clear, by being delivery focused, by feeding back to users and customers people get what they want.

But, do you know what? All the members of this team are still employed and still work on software. In fact I think the team might have grown a bit. There is still work to do, there is still demand.

Agile reduced the demand in one part but there is still plenty that company wants to do, and much of that means software. Getting demand reduced in one place just leaves the team more time to do other work, other demand.

(OK, if I’m being honest while I have one team in mind while I was retelling this story I’ve simplified it (knocked a few rough edges off) and composited it a bit to include other examples. I don’t think that invalidates the story as this is how it is supposed to be.)

Honoured - I'm on a 100-top lists

Bit thank yo to the guys at ProgramCreek.com for including this blog on their list of “100 High-Quality Java Developers’ Blogs”.

Strictly speaking this is not a Java blog but most of the issues I talk about do effect Java developers.

It also seems that in listing this blog at number 55 (yeah, still room for improvement!) the ProgramCreek.com team have set up a little competition between me and my #NoProjects conspiracist Steve Smith who is just two places behind me :)

(Whoops, should have posted this when it was announced! - better late than never)

Software demand curve

Returning to my series of posts applying the tools of economics to software development - Supply & Demand in software development, Software supply over time and Software supply & demand - this time its Agile - it is time to turn our attention to the demand curve. First a reminder of how things start...

First I need to explain why I believe there is so much demand for software and why I believe the curve is inelastic, i.e. why a higher price doesn’t reduce demand very much. The reasoning here breaks into five groups: technology progress, separation of demand from cost, separation of benefit from delivery, lack of evaluation and the effect of “fixed” work - although the last in that list doesn’t always play a part.

It is important to remember in all this discussion that software is a derived demand. Nobody wants software for its own right, they want it to achieve some other aim.

Technology Progress

Moore’s Law implies that processor power doubles every two years (give or take a bit). Consequently the ability of computers to tackle ever more complex problems and fill more needs constantly increases. Supply creates demand, we want it because we can.

For example, I’m hoping Farther Christmas will bring me an activity tracker for Christmas, I didn’t know such devices existed until recently but I can see it solving one “problem” in my life. These devices didn’t exist until recently, I didn’t need one because they didn’t exist. The underlying problem existed but had been put in the “thats life” bin.

At a company level this means the benefits computers can bring to our companies are constantly increasing and there are new opportunities to save costs or generate revenue. And if the incumbent company doesn’t take these opportunities then Amazon, Google or some start-up will.

Moore’s Law is not alone, Metcalfe’s Law compounds the effect. The more devices we enable and the more devices we connect to the internet the greater the value and opportunities from connecting these and more devices.

Separation of Demand from Cost

Traditional development process - and even Agile processes - tend to separate the demand for software technology from the supply, consequently those requesting technology are isolated from the cost of that technology.

Many practices in software development make this problem worse. We send Business Analysts to talk to “users” about “What they Want” i.e. what is their demand. These BAs are sometime little more than order takers, waiters. People are encouraged to ask for more and more. Indeed, some of our practices incentivise people to ask for more and punish them for not asking for it.

Even if “users” don’t ask for it Analysts may be encouraged to include any need they can comprehend. I once had a BA tell me “If we miss a something in the requirements document it is seen as a black mark against the BA.”

Separation of Benefits from Delivery

To make things worse its not just demand which is separated from costs but so too are the benefits. Nobody wants IT for IT’s sake, they want it for some benefits. But achieving the benefits may well take time, it might be months or years after an IT system is delivered before the benefits are seen.

That is, if the benefits are ever seen, IT alone is not enough to bring about benefit. Users may need training, changes to processes may be needed, customer agreements may need to be changed and so on.

Although it is a few years old now Wired for Innovation contains a good, short, discussion of why IT benefits don’t appear in the way we would like them too (i.e. quickly and easily) and what can be done to help.

Lack of evaluation

Ideally companies would evaluate the benefits delivered by IT work but all too often this fails to happen. Instead companies may rely on the original claimed benefits. But this might the result of optimistic thinking itself: “38% of businesses openly admit benefits are overstates in business cases in order to obtain project funding” - from a Cranfield Business School study I reported in October 2010.

And when one manager exaggerates benefit to get his project funded it creates an incentive for the next manager to exaggerate the benefit of the next project and so on. Benefit inflation cascades through the organisation.

This isn’t helped by vendors who claim benefits for their product which might not apply to a particular business or in a particular context. If company A used product B to save $C millions then surely company X can use product B to save about $C millions, right?

In our personal lives we have become accustomed to technology getting cheaper and more powerful and we expect the same in business. But consumer technology is paid for by millions of people buying millions of products while business technology may have one customer. Consequently we don’t appreciate the costs involved in creating new software.

Goal Displacement: Floorded contracts, projects and governance

In an effort to control IT, to maximise the return organizations frequently fixate on controlling the costs. They demand things like price, time and scope are decided in advance and fixed. Sometimes this is done internally - projects are approved and must be finished according to some set budget or date. And sometimes it is done by finding a supplier who will agree to fixed price/time work. (I’ve written about this before in Dear Customer on this blog and on Agile Connection/TechWell.)

The problem is, when you put these arrangements in place the objective becomes: meeting the fixed criteria, not delivering value. It is what psychologists and management students call Goal Displacement.

Options to reduce scope, time and cost are lost in the process. The things which were meant to be maximums, ceilings, become minimums, floors. Opportunities to reduce costs are lost to more features and more software.

Net effect

Taken individually, together and in combination with other forces we end up with a demand curve which is very high and inelastic. Changing the price of the technology doesn’t reduce the demand by much, Moore and Metcalfs Law mean there are always new opportunities, failure to capture original benefits mean they are still there for the taking, and lack of evaluation means nobody is counting, boys will always make arguments for more toys.

In the next instalment of this series I intend to look at what Agile can do about this and whether Agile makes things better or worse.

Agile is like sex

I’ll get back to software development economics, there are more posts in the works, but right now I need to say something....

I have blogged before on the subject of “What is Agile” - I’ve even expanded on that blog in an unfinished piece of writing called “What is Agile? Perspectives on Agile” - but sometimes I think its just sex....

(OK, this blog just got filtered out of lots of feeds and stopped by lots of firewalls but lets continue.)

In the software development world there are, broadly speaking, two groups of people: those who create the software (coders, testers, etc.) and those who manage the process (project managers, development managers, etc.). When discussing "Agile" I find that both sides think the problem is with the other.

To put it another way, if I'm talking to developers they think its managers who are the block to adopting more Agile techniques and improvement. But when I'm talking to managers they say its the developers who resist Agile and other improvements.

Which always reminds me of the old Philip Crosby quote:

"Quality has much in common with sex. Everyone is for it. (Under certain conditions, of course.) Everyone feels they understand it. (Even though they wouldn't want to explain it.) Everyone thinks execution is only a matter of following natural inclinations. (After all, we do get along somehow.) And, of course, most people feel that all problems in these areas are caused by other people." Philip Crosby

Lets bring it up to date and make it Agile specific, substitute the work 'Agile' for 'Quality':

"Agile has much in common with sex. Everyone is for it. (Under certain conditions, of course.) Everyone feels they understand it. (Even though they wouldn't want to explain it.) Everyone thinks execution is only a matter of following natural inclinations. (After all, we do get along somehow.) And, of course, most people feel that all problems in these areas are caused by other people." Allan Kelly after Philip Crosby

You see? It fits.

Agile isn’t someone else’s thing, it doesn’t belong to another person: Its yours. You are as likely to be the block as the person you think is the block.

Software supply & demand - this time its Agile

Carrying on from my previous posts applying the economists tools to thinking about software development (Supply & Demand in software development and Software supply over time). In this post I want to see what happens when we apply Agile...

We start with our supply and demand curves as before:

First we need to define Agile, not a trivial matter. Lets simplify this by saying “Agile practices”, specifically:

• Short iterations and regular releases (or at least demos), e.g. every two weeks
• Planning meetings, planning poker estimation (or similar, or #NoEstimates), velocity (or commitment if you prefer)
• User Stories, Epics, task breakdown (if you do this sort of thing)
• Stand-up meetings
• Team retrospectives and team planning boards or electronic equivalent
• Test driven development and continuous integration
• Pair programming, code review and shared code-ownership
• Acceptance Test Driven Development (ATDD), Behaviour Driven Development (BDD), Specification by Example (SbE), and other automated testing mechanisms

The first thing to note about all these techniques - and indeed the majority of what we tend to call “Agile” - is that these techniques work on the supply side. They operate on the supply curve, the net effect is to increase supply.

Possibly, if used in the right fashion, BDD and SbE might effect the supply side. But since most of the efforts to use BDD I encounter are little more than super-charged TDD we’ll leave it there for now - see Dan North’s blog for a better discussion BDD v. TDD.

Projects - specifically projects but other work initiatives too - are often still judged on the basis of: on schedule, on feature and on budget. Agile in this fashion is what I have called Iterative Agile, or at best Incremental. The team is still seeking to deliver a fixed quantity of software. Thus the demand curve is unchanged. (We will return to the demand curve in another post and discuss how Agile techniques might affect it.)

While Agile tools promise increased performance this is only in the long run. In the short run we can expect teams to slow down as they adopt these tools. We can also expect costs to go up as companies to spend money to buy in this expertise.

Maybe they hire the likes of myself to supply training and consulting in adoption (yes! I’m for hire, see my company website Software Strategy). Even if they don’t pay the likes of me costs will go up because engineers need to learn these skills. Companies rationale for hiring me is that I will help reduce the period of time for which they are slower and reduce the risk. The deal is: hire me and I will help you get to increased productivity faster (and with less risk).

Putting this in terms of curves:

A team starting on Ss can expect to move left (1) to Sasr (agile short run) before moving to the right (2) - Salr (agile long run) - reduced supply before greater supply. In the first instance the price per unit increases while the quantity delivered falls but after the second move the price per unit falls and the team produces more software.

Leave consultants like me to one side for a moment and simplify it. Imagine a team which “just does it”. Assume for a moment that the team members stay the same and the amount they are paid each week stays the same. When the team start doing something new they slow down, they produce less software each week so the average price of per unit increases. As they get better the same team better their original performance, earn the same money, produce more software and as a result the average price per unit falls.

Hopefully that all makes sense and isn’t controversial.

But I think something else happens. This is a hypothesis, at the moment I can’t measure it so please tell me - in the comments on this blog - whether you agree or think I’m wrong.

Staying with the supply curve, consider again the tools we were looking at:

• Short iterations and regular releases make it cheaper to make small changes. Therefore the supply of the small quantities of can should take place at a lower cost. And since the delivery - or at least a demo - is earlier the whole “project” could be cancelled if things don’t work out. Thus the bottom of the curve moves left over time.
• Automated testing (TDD, ATDD, etc.) reduce the quantity of faulty software produced and increase the amount of useful software increased at any point thereby raising all points on the curve. With less “technical debt” teams can deliver more functionality more cheaply.
• The same techniques - TDD, ATDD, etc. - and shared code ownership also make it easier for new engineers to become productive because a) there are more code examples of how the system works and b) places a safety net under any work they do, allowing them to experiment and learn faster, and get their changes to production sooner.

The net consequence of these changes is to flatten the curve, not entirely but certainly reduce of an angle. In economics terms we increase the elasticity. Changes in price - adding or removing a developer - causes a bigger effect on the quantity supplied than before.

Supply moves from the initial Ss curve to the Savlr (Supply Agile Very Long Run). As the chosen name implies, this takes some time to occur.

Now there is a catch. Consider the total amount of software bought and the total price paid (price x quantity.) To do this look at the next two diagram.

The green area represents the total paid for the software. You can’t see it here but on my graphics tool I can see price is 10 high and quantity 7 wide, so 70 (whatever) units.

In this diagram, on the new supply curve, the blue area represents the total spend. Again you can’t see it but on OmniGraffle it is 5 high and slightly more than 9 wide, almost 45 units of total spend.

Now if you are producing software for your own company - either to resell (e.g. Microsoft) or to consume yourself (e.g. a bank) - this model is good news. You get more software for a lower overall spend.

But, if you are an outsourced provider of software selling custom software to a client this is bad news. You now deliver more software for less money, about 36% less in total. If profit is tied to revenue (time and materials) then this is bad.

No wonder outsourced providers can be reluctant to embrace Agile or to do it badly. If you are a consumer of outsourced software development services you might want to think about that.

Next time I want to turn my attention to the software demand curve.

(As a footnote, using the same analysis techniques it is possible to show that reducing supply of software - moving the supply curve left, either because the software ages or because Agile adoption is slowing things down - will result in a greater overall spend for less software. This might be good for outsourcer suppliers but isn’t so good for in-house development.)

Blog revisions: economics

Although I don’t think I’ve ever stated it I operate a “no revisions” policy to this blog. I don’t take things down, and I rarely modify past entries. OK, I might change it in the 10 minutes or so after I first post to fix typos and gaffes I see once I see it online but otherwise I avoid doing so.

But my recent posts on software economics, supply and demand, contained inaccuracies. Or rather the graphs I published didn’t follow economics convention. The logic was (mostly) right but the graphs were wrong. I rushed out one fix and breathed a sigh of relief.

Then last night on my way back from Riga I noticed another flaw. I won’t go into details but I’m now fixing/fixed it.

Several times in the past I’ve attempted to capture my thoughts on software economics but nothing I’ve ever written has seen the light of day. As I continue this analysis I’m finding more insights I hope to share. I’m also - possibly - making errors.

I hope other readers who think they know economics can help debug my posts. They will be better for it.

In the meantime, sorry.

Software supply over time

Having outlined the theory of supply and demand in the software market I want to spend this, and the next few posts examining what this analysis can teach us. I’m starting with the supply curve and will look at the demand curve later.

First I want to consider what happens to a software system over time if nothing else is done. (Consider this as a small detour before we consider Agile.)

Over time changes to software systems will add code, this means there is more code to maintain, more code for engineers to understand, more code for testers to test. Internally code quality tends to decay, some people call it software entropy.

Engineers leave the team and new engineers (might) arrive thereby loosing knowledge of how the system works. At least the knowledge is diluted. And if maintenance is outsourced or offshore such knowledge may be destroyed altogether. The new, cheaper, outsourced developers need to relearn.

Meanwhile the business environment seldom becomes any simpler. New business rules get added one place or another - increasing complexity. New systems need to be connected - either as data sources or destinations - which again increase complexity.

This all gives rise to the Big Ball of Mud pattern.

Finally technology changes. The operating system and databases which form the foundations of the system may be upgraded or replaced. In the worse cases the language the system is written in dies.

The net result of all of this is that the system becomes harder to maintain. Harder translates to more time and more time translates to more money. Thus the system becomes more expensive to maintain. Consequently the supply curve for the system moves to the left. For the same price we can now get less software.

Consider the graph below. We have the Supply of Software (Ss) and Demand for Software (Ds) curves as before. Now we add the Supply of Software over Time (Sst) curve. Over time, of nothing is done, the curve itself moves, all other things being equal we do not move to a new position on the curve. We get an entirely new curve.

Now it is possible to fight against this trend. And certainly there are plenty of tool vendors out there who will happily sell you a tools which they claim will move the curve in the opposite direction. (In economics terms we break the “all other things being equal” assumption.”)

But either way: unless you fight it things will get more expensive for the same quantity of software - or you get less software for the same money.

Supply & Demand in software development

(Whoops, I’m publishing a revision of this blog entry - something I don’t normally do. I realised after publication that I had mislabelled the graphs - they were correct but did not follow economists convention hence this revision. The core argument still stands although I have had to modify the reasoning slightly. Sorry.)

I’m proud to say I am, or at least was, a Software Engineer. I’m proud to say that I’m a third generation Engineer. My farther and grandfather got their hands dirty with the oily sort of engineering. But I’m also a frustrated Economist. Economics forms part of both of my degrees and on a couple of occasionsI’ve been temped to become a real Economist.

So I am always disappointed when someone talks about Software Economics. I have yet to find a serious study of software economics. Most of the studies, books and papers about “software economics” would be better called “software accounting.” They are about apportioning costs and adding up numbers. Economics is not Accountancy.

I have long believed software engineers and their managers would benefit from a better understanding of economics. In fact many of my clients will be used to hearing me refer to the supply side (development) and demand side (requirements, analysis, etc.).

I find it very useful to think of software development in supply and demand terms and use the tools of economists to understand what is happening. Usually thinking like this shows there supply and demand to be mismatched.

(This blog post that has been a long time coming. Several previous attempts have floundered because there is so much I want to say about software economics - and thinking in supply and demand terms. So while there might be little in this post which is earth shattering - especially to those of you who have studied economics - later posts will build on these ideas and analysis.)

Lets start with the fundamental problem which economic theory addresses: the allocation of scarce resources. After all we have scarce resources in software development so lets use an Economists tools: supply and demand.

Lets have a look at the supply curve:

The line marked Sn is a normal supply curve. When price is a bit greater than zero individuals will start supplying their services and companies will come into being to supply the market. As the price increases more individuals and companies will be attracted to the market and supply will increase, more resources are brought to play.

Note two things here:

1. Economists usually draw these curves for a product within a market. Here I am not examining the supply of multiple copies of a software product, rather I am examining the supply of software development capacity for a unique product. The marginal costs of supplying existing software are as close to zero as to be zero, the marginal costs of supplying new software products - i.e. the capacity to create a product - is significantly larger than zero and that is what we are examining here. In software terms we need to consider a software product, perhaps a single program, perhaps a “project” or a small system.
2. If you are not an economist don’t worry about the fact that these “curves” are straight lines, the basic analysis still holds, we hold actual curves for an advanced class.

The software supply curve - Ss - starts the same way. When price is zero nobody supplies - perhaps not entirely true if we consider open source but close enough. As price increases more resources are brought into play and supply increases.

Software development is not cheap. If you are buying software developer services they are rarely cheap. The only time you get “cheap” software is when you buy (some) off the shelf stuff. Consequently when the price is low there is no supply of custom software.

But - and this is the BIG BUT - supply raises very slowly. Two people might write twice as much code as one but when they work together the generalised form of Brook’s Law comes into play: adding people to a software development effort slows a team down. Therefore supply responds slowly to price increases - what economists call inelastic.

Now consider the demand for new software:

Here Dn represents demand for a normal product, when the price is high there is little demand for the product. As the price falls the demand increases because more people can afford it. (Remember an Economist’s definition of “Demand” is “A want with the ability to pay”. I may want a new iPad Air but if it is more than I can pay there is no demand - some might call this “latent demand.”)

For normal products, in time, the market will “clear” and balance will be reached when the curves meet:

At some price (Pn) the quantity supplied (Qn) will match that demanded. Now economists accept that this might not happen instantly - and there are various models for that - but in the scheme of things the time it takes for the market to settle is not a big issue. It will happen eventually.

Until that happens all sorts of problems will ensue, some people might make a lot of profit, some people might go without and some people might have to pay a non-financial price, e.g. queues will form. All good stuff we could consider but lets leave that for another day.

Returning to the demand for software. People and companies do not want software for its own sake, they want software because of what it allows them to do. What Economists call a “derived demand”.

The problem is, and the reason I’ve drawn the the Ds curve as highly inelastic is that we are faced with an ever increasing demand for software. Processor power doubles every 18 months to two years (Moore’s Law) which means the things our computers are capable of doing - with the right software - is forever expanding. When we get this right the benefits can be massive. Which increases the demand for software…

• How many readers wanted to read newspapers on their telephone three years ago?
• How many readers knew they needed to watch TV on an iPad five years ago?
• How many readers took digital pictures with their telephone and posted them online on a social network to instantly share them with friends and relatives seven years ago?

So lets look at the software supply and demand curves together:

For me, the problem in software economics:

• Demand is rampant but inelastic so as price rises it does not fall off very fast
• Supply is severely constrained and is very difficult to increase (particularly in the short run) so is also inelastic
• Consequently the market only clears at high prices and slowly: consequently markets take a long time to reach equilibrium, which in turn means we need to consider the tensions arising in a market which spends so much time out of equilibrium. Thus the time lags which normally only play a small part in classical economic analysis are more significant.

Each of these statements deserves further examination and explanation in their own right - future blogs, maybe.

More importantly, and also deserving of its own blog entry, is the question: what can be done to bring this market into equilibrium?

That is the question I find myself wrestling with on behalf of clients, and it is one I will return to.

Sprint, Iteration or Marathon?

A shot blog on terminology...

I have always found it hard to differentiate an Iteration from a Sprint, so much so that I commonly say they mean the same think. They are synonymous. I use the two interchangeably. The difference is historical, term “Iteration” originated in XP and “Sprint” in Scrum.

To me both mean: a short period of time in which work is done.

Occasionally people have attempted to differentiate the two terms but I’ve never been convinced.

At Agile on the Beach, in a private conversation, Rachel Davies suggested a difference which I think stands up:

“Sprint means a fixed period of time during which the team commit and work towards a fixed goal which does not change. Iteration means a fixed period during which the team undertake some work - no commitment, no fix work.”

Most of the time the difference probably does matter but I like Rachel’s differentiation. It ties Sprint quite closely to Hard Core Scrum.

I really should try to stick to Iteration.

Actually, I think the real Sprints occur when:

• A team starts a new piece or work and they have no idea what they are up against. They are sprinting to maintenance mode. Risk is highest when sprinting, until the team have their walking skeleton - a thin slice of functionality that does something tiny
• Or when a team are first trying a new way of working (probably Agile like). They have a short period of time where the rest of the organization doesn’t know what they are doing, or if they do will extend good will to let the team try. A honeymoon, if you like. At the end of that time the team need to be able to show something better. Otherwise the good will is gone and with nothing to show for it people feel let down Again risk is high.

Real software isn’t about sprinting, real software development is a Marathon. You need to pace yourself. The danger with constantly sprinting is that you burn out.

Real software development is a marathon run in laps. We call the laps iterations.

Beyond Projects, Beyond #NoProjects

My recent “#NoProjects - why projects don’t make sense” post stirred up a fair bit of discussion, both on the blog and especially on Twitter - check the #NoProjects hashtag. But, while that post railed against “projects” it didn’t say much about what we should do instead. Indeed as was pointed out it might be better to advocate and rally behind a positive alternative rather than negative idea.

So let me try and put in place the outline of how I see a #NoProjects world working.

(Strictly speaking it might be better to use the hashtag #BeyondProjects but a) #NoProjects has already built up momentum, b) #NoProjects actually describes what we are talking about quite well and c) #NoProjects has 4 fewer letters than #BeyondProjects.)

Stable teams: first off the teams that do the work need to be stable and they need to be kept together over the long run. Of course teams will sometimes loose people and they will sometimes expand. But over the long run we want team members who will live with their creation and grow with their creation.

Teams rooted in business: These stable teams needs to be based with their business users and customers. There should no divide between “the business” and “IT”. IT shouldn’t be hired guns who get called up when there is a problem.

Increasingly, as more and more everyday business depends on technology “the business” is a technology business. IT departments as we know them may cease to exist in time.

(Naturally there are some differences between Beyond Projects in a corporate IT environment and in a software vendor environment. I won’t go into the details here. But, on the whole it is corporate IT departments who have a project obsession and practice corporate psychopathy. Continuity, if not stability, is usual the norm for software vendors.)

Business as usual (BAU) is the source and destination: Everyday business, business as usual, is the source of requests for technology changes and the destination for such changes. Many of these changes will be small - incremental - but others will be big. The people who do the work will be involved with uncovering the need for change and in working with the change when it occurs.

The need for change management will decrease considerably because IT will no longer be inflicting change on unsuspecting users. Instead technologists will be bringing requested improvements to willing customers.

If you need change management then it too will be based in the business lines of work. So too will be testers and business analysts. In fact the role of business analyst will become far more important because they will be charged with work with real stakeholders and helping them find ways to improve the business processes, products and delivery. BAs will look more like internal consultants than the order takers they are sometime used as.

Continuous stream of value: the result of this model will be a continuous stream of value. Technologists will be delivering improvements to the systems regularly, every few weeks if not every day. The batch size for work request and delivery will be one.

Again, as more and more businesses become IT businesses - business which can only exist and compete through the use of IT and IS systems - the value delivered will be bottom line value allowing the business to compete and stay ahead of competitors.

Portfolio management: this is not to say that all lines of business - and technology teams - will exist for ever. Some lines will need be to closed. Some teams will need to survive with less technology. Conversely, work streams which are seeing growth and performing stronger than the wider company will benefit from more resources.

The activity of reviewing work streams, increasing capacity, reducing capacity, starting new initiatives and terminating existing ones needs to be pursued more rigorously than before.

True portfolio management needs to be practices on a regular basis - at least every three months. And it needs to look at value delivered, potential future value and costs. Portfolio management which looks at progress against plan, or measures success as “on time, budget and scope” has no place is the Beyond Projects world.

Like trains: trains run on schedules and so should software projects. When trains leave the station late and arrive late they are poor performers. Just the same goes for software work. We want work which begins on schedule and arrives on schedule.

When this happens co-ordination becomes simpler. The train schedules are known weeks and months in advance. Yes they change sometimes but not often. When you know the train you are going to get you arrange your journey around that.

When you have to take connecting trains you know how much time you need. You might allow a buffer or you might take a risk and run between platforms. How many of you readers have missed a train or a plane recently? Humans are very good at working to deadlines and very poor at estimate how long activities will take.

Co-ordinating multiple pieces of work between different teams becomes a process of putting the features/functionality you need from another team on their train. You inject work into the other team. And the receiving team will be able to tell you their schedule and whether the train you want to put a feature on is overloaded. They might even offer you a discount for going off-peak!

Project managers: obviously without projects there are no projects to manage. But does that mean the end for the people we call project managers?

I think not, there is still work to manage. I disagree with many in the Agile community - and particularly the Scrum community - because I don’t think all management work goes away. There might be less but I think those who argue “you don’t need managers” don’t really understand management work. (In truth most managers don’t either. Go and read Managing by Mintzberg then we can talk about what managers actually do.)

Many of the skills good project managers have will still be needed. They may take on another co-ordination role or they may become business-as-usual managers. Rather than having “Project Manager Call Centre Renewal Project” on their business card we might see “Call Centre Development Manager”.

Management roles will still exist but they will manage real work.

Goodbye matrix management: with projects gone and technologists based in the business much matrix management should disappear. Matrix management has never looked like a great idea, at best it is confusing at worst it is dysfunctional.

There will be more line management and there will be more need for business line managers to understand technology. Some of those surplus project managers will be well positioned to take up the new roles.

(Indeed I suspect that the proliferation of project managers in the last 20 years has much to do with the reduction of middle and junior managers that occurred in the 10 years before hand. First corporations cleared out their management ranks, then reinvented them under the cover of “projects.”)

There then are the basics of #NoProjects, Beyond Projects. I’m sure there is much more I could say and much more I will say in time.

If you have any opinions on this please add them below, I’d really like to hear what people think. And especially why you think I’m wrong, what have I missed? Go on, tell me!

11 Myths & 2 Truths of Agile - Infographic

Abi Crafter at Countersoft liked by “11 Myths and 2 Truths of Agile” blog entry so much he turned it into this Infographic. Thanks Abi!

(Some of you might have noticed that a reworked version of that blog appeared on Agile Connection as Top Twelve Myths of Agile Development.)



By the way, notice I’ve linked to Abi and his company Countersoft. Abi did something I like, Abi gave me something, I’m more than happy to give him a link for that. I don’t necessarily endorse his tool but I might well look at it.

I regularly decline - or plain ignore - requests to review particular products in this blog. Similarly I mark comments which don’t really say anything but include a link as spam. Unfortunately most such comments seem to come from Indian companies which reduces my opinion of the Indian software industry.

#NoProjects - why projects don't make sense

In the last few months Steve Smith, myself and others have been Tweeting a lot with the hash tag #NoProjects. We have independently come to believe Projects are a smell, they are bad for our industry (the technology industry). Steve set out his thinking and now it is my turn.

Right now this is little more than a bullet point list, I’m presenting to PROMS-G in February (“The End of Projects”) where I will present a more coherent argument. Until then here goes.
Lets start of by defining “A project”.

The Project Management Institute says:

“what is a project? It’s a temporary group activity designed to produce a unique product, service or result. A project is temporary in that it has a defined beginning and end in time, and therefore defined scope and resources. … The development of software for an improved business process, the construction of a building … — all are projects.
And all must be expertly managed to deliver the on-time, on-budget results, learning and integration that organizations need.”

PRINCE2 defines a project as

“A temporary organization that is needed to produce a unique and predefined outcome or result at a pre-specified time using predetermined resources.”

Actually, PRINCE2 has two definitions of a project:

“project management is the way of managing change. Everything from the Olympics to organising a wedding can be considered a project. It describes the activities that meet specific objectives and can be used to introduce or improve new or existing products and services.”

Bearing this in mind I suggest the project model does not fit software development:

• Projects have end dates - successful software doesn’t end, it not temporary. Finished software is dead software, software that isn’t used and doesn’t change
• Successful software is continually developed and improved, the team is not temporary, it continues to exist (although it might change composition and the imposition of the project may at times destaff it)
• Resources change: sometimes the people decide to leave of their own accord, sometimes the wider organizations decided to remove people or add others
• The project model implies we can define what needs doing before we begin and that it won’t change. Software development uncovers need as it proceeds.
• Project thinking implies that the problem can be defined independently before the solution: in software development the creating of a (partial) solution can lead to a redefinition of the problem being addressed.

Those points might be enough to persuade you the project model is not a good fit for software development. I go further, I think the project metaphor actively damages software development and destroys business value:

• Project success criteria are “On time, On budget, On scope” and not business value. These criteria become a distraction and a barrier to hide behind - a “social defence” to use the jargon. Work should be value seeking and judged by the value/capability/benefit/[whatever you want] rather than arbitrary criteria which seemed good one day in the past.
• Corporate Psychopathy: why disband a performing team just because a project has ended? Why start a project by pulling together a new team who have never played before?
• Destroying team destroys knowledge - knowledge has value, knowledge exists in heads not documents
• Project thinking induces short-termism - around quality, around productivity and around teams especially. Project thinking builds pre-fab houses when it should be building to last
• Large batch size: projects lump lots of work together and attempt to manage it all as one large unit. This is inefficient (software has dis-economies of scale), breaks flow and delays delivery which delays value realisation
• Project thinking makes BAU a dirty word; BAU is not a dirty work, BAU is where the action is: if it is worth doing it is worth doing more of, if it is not worth doing then close it down. Use portfolio management rather than pre-defined cut-offs. (BAU = Business As Usual, a common corporate term)
• Projects inflict change on people (employees): why not seek their involvement? They own the work, they are BAU, the work stream should be seeking to improve itself as part of its daily work
• Continuous improvement is continuous. Projects stop continuity, projects break flow and reduce effectiveness. Continuous improvement should be part of the fabric of everyday work, supported if need be by technologists.
• Project thinking create scheduling projects: “Project B after Project A has finished, but if A is late….” - define work streams and feed them work
• Projects confine change to projects and freeze change elsewhere
• Start-up costs: bureaucracy, administration, etc. of starting a project mean the project model is expensive and slow at taking action. Project setup costs detract from value and the time it takes to “launch” a project detract from responsiveness. On a small work effort the costs of creating a project may be disproportionally high which may detract from attractiveness of doing the work. Similarly the time it takes to get a project scoped, a team assembled, everything kicked-off, performing and delivering creates a barrier to seizing value.

I recognise that a lot of what is called “a project” isn’t, it doesn’t fit the definitions at above. That simply makes things worse. Calling something a project when it isn’t is at best sloppy thinking, at worst it imports the problems listed into work that would otherwise be free of them.

Then there is scaling: so much of the “scaling agile” debate revolves around projects. If instead you accept standing teams dedicated to improving business as usual many of the scaling issues simply go away.

Projects are accounting codes to bill work back to. Projects are for accountants. Leave them at that.

Ask not “When will it be done?” ask instead “When will it start delivering value?”

Pessimistic about Agile in retail banks

Almost exactly two years ago I wrote a strident blog entitled “Agile will never work in Investment banks”. Its a blog post that keeps getting rediscovered by those who agree or disagree with my position and just such event has occurred on Twitter with Kevin Burns @martinburnsuk and several others @kev_austin @sandromancuso @gordonmcmahon 

During the intervening two years I have had many conversations which support my point of view. Most of these conversations have been with people who have more, and more recent, experience in banks than I do. In particular I found Peter Pilgrim’s blog closely aligned with my own views.

Conversations with the likes of Andy and Chris (you know who you are!) and others have led me to believe that while retail banking might be more hopeful it isn’t a massively better there. In part that is due to the colonisation of retail banking by investment banking that has occurred over the last twenty years.

Lets be clear: although investment banking and retail banking share the word “banking” in their title they have nothing else in common. I’ll leave it to John Kay to say something about the why they are in conflict.

The move by many investment banks to occupy retail space has been purely opportunistic, they have sought cheap funds and to embed themselves in the economic fabric so they become “too big to fail” and “systematically important institutions”, thereby guaranteeing Government bail out, thereby reducing their funding costs and increasing banking bonuses. (See The Bankers New Clothes for a discussion of the many problems with current banks and solutions.)

If you wanted to summarise my argument in one line I’d say: Too big to fail means too big to manage.

Now there are two caveats on my original post which I want to make clear before expanding on the discussion:

1. Agile is interpreted by many in banking to mean Iterative Development - see my recent “3 Styles of Agile” post. This can work to some degree. Simply adding the technical practices, improving software craftsmanship, can improve IT. However if an Agile ever becomes more than - incremental or evolutionary - there then conflicts will be created within the management regime which may lead to its own destruction.
2. Incremental and Evolutionary Agile can exist in banking, and have existed in many banks I know of but only in the short to medium term. This kind of Agile requires a guardian angel. Again this will create conflicts, while the guardian angel is strong enough the team can be protected. However given time either the angel is successful and will presented with a better opportunity (probably at another bank in which case he might have the team follow him) or he is not successful and will be pushed to one side and his creation will be dismantled.

Turning specifically to retail banks, and thereby extending my argument. On the one hand I don’t believe retail banks suffer with quite the same rent-seeking culture found in investment banks so I’m more hopeful of them. But on the other hand retail banks have an additional set of problems:

Legacy retails banks are dependent on Cobol mainframe systems. This brings several problems. Firstly while you can do some Agile like things in a Cobol environment there are many things you can’t do.

Specifically TDD: as far as I can tell several attempts have been made to create a CobolUnit (this CobolUnit may be the same to different) and they are all pretty much dead. No action since 2009 or 2010 on those links.

We can’t expect the OpenSource community to come to the rescue here. Few people write Cobol at home and the generation of people who write Cobol are different to the OpenSourcers who created JUnit, nUnit, etc.

Unit IBM or MicroFocus get interested in a CobolUnit then I don’t expect much to happen - it seems IBM might be moving in this direction. But generally I don’t see the incentive for either unless they can see money.

Second these Cobol systems are ageing. They have patch upon patch applied to them and by some accounts are reaching the end of their lives. Much of the technology they are built with just isn’t as flexible or modifiable as the current generation. There is only so much you can do with a hierarchical IMS database or a mainframe batch processing system. (If you could do these things, and you could do them as easily as modern technology then we wouldn’t have needed to invent modern technology.)

Since retail bankers were early adopters of computers some of these systems are among the oldest systems in use. Many of them have been patched together through mergers and acquisitions making things even more difficult.

True I’d a great proponent of never rewrite and I think it would be a mistake for banks to rewrite these systems. Indeed, given the chronically poor management skills in these banks authorising a rewrite would be a waste of money. Which means you are damned if you do, damned if you don’t.

Third, compounding the first two: the people who wrote these systems are nearing retirement age. Not only do they understand the technology but they understand the business too. When they are gone these systems are lost.

Banks could have alleviated this problem by training replacements but they don’t generally do this. Actually they have made things worse in the last 20 years by outsourcing and offshoring work thereby discarding their own experience and acerbating the generation change. Some of the knowledge might now reside in the outsourced and offshore but getting this into the bank will prove difficult and expensive.

The solution for the banks is to buy off-the-shelf systems. But this is incredibly difficult so...

Fourth, buying an off the shelf product means not only process changes but potentially product changes. That lands the retail bank with change problems. They will need to customise off the shelf software and that isn’t cheap either.

Fifth, new “challenger” banks like Metro bank are already buying off the shelf. As the costs of maintaining legacy systems or migrating to new systems escalate these banks will have a greater and greater advantage over the legacy banks.

And its not just challenger banks: Paypal, Zopa, Funding Circle and a host of other online payment and loans providers are applying technology to reinvent banking. In doing so they are eating the retail banks market. (And the banks can’t respond because of the reasons listed here.) In other words: the banks are suffering from disruptive innovation

Sixth - something I just hinted at - the quality of IT management in banks is absolutely appalling. Again investment banks have made retail banks worse because the big money was in investment, anyone who was any good and wanted more money had an incentive to move from the retail to the investment side. Retail IT has been robbed of its best staff.

Management quality is not entirely down to investment banks. Banks are about money management after all. Promotion to the senior positions comes from the banking side, not the IT side. Almost by definition the senior managers in banks don’t understand IT. It also means that IT managers sooner or later face a ceiling on progression.

Many of these problems have been made worse because investment bankers controlled retail banks. Investment banking is essentially a rent extracting activity, and they have managed retail IT just like that. The existing IT has been milked and without adequate investment. Hence problems like well the documented RBS outages and Lloyds payment problems.

To make matters worse banks are where the money is so they have been preyed on by major systems companies and tool vendors. These companies sell technology fixes which may - or may no - address a problem but do little if anything to build capacity. Put it this way: banks are major buyers of automated testing tools which aren’t used.

You see my argument?

I’m sure Agile could help with some of these problems - not all but some - but the way I see it the problems facing the banks mean Agile will have a very hard time.

Finally, as others have pointed out many of my arguments can be applied to other types of large corporations. I agree, I confine my arguments to banks because I think the case is clearer there, banks are all very similar, because banks exhibit more systematic failures than most and banks are something we should be worried about - as customers, as tax payers, as citizens in economies that depend on banks.

3 Styles of agile (part 2)

In my last entry I set out what I call “3 Styles of Agile - Iterative, Incremental and Evolutionary.” In this entry I’d like to discuss the model as a whole.

Lest anyone start “my style is better than your style” I am at pains to point out that each style has merits. Regardless of whether any of us regards one style as superior, or one style as “wrong” the model describes what I find in companies and teams. I also believe you can use Scrum, XP or Kanban in any of the three styles.

That said, I confess, I see evolutionary as “superior” to iterative and I probably don’t do a very good job of hiding my preference. Rationally I must accept that each style has its own advantages and disadvantages depending on your context.

So how should one use the model? Or rather what does the model show?

Firstly (#1) the model is useful in naming the way a team is working and resolving conflicts. I frequently find that one group of people in the organization think Agile development means one style (e.g. Evolutionary) while another group thing it means another (e.g. Iterative). The result is a conflict with each group thinking the other is stupid and doesn’t understand.

This extends to much of the writing in books, blogs and elsewhere about agile. Sometimes the writer is implicitly assuming one style but their reader may be thinking another. In the extreme subscribers to one style will see the other as “Idealistic” or “Purist”, or contra wise: thinking someone “wants to do Waterfall” or “lacks flexibility.” One person’s “Pragmatic Agile” is anthers “Flawed Agile.”

Simply recognising the model and talking about it exposes assumptions and beliefs. It can also help to recognise that different teams within the same organization will work to different models.

The second (#2) use of the model is as a change model. Typically a team with a dysfunctional (or sub-functional) process and wanting one particular style, say, evolutionary, can enter at iterative and advance up in steps. It is fairly easy to see how a team working in iterative style could become incremental. (Of course a chaotic team finding itself in a controlling environment might make the opposite transition, from evolutionary to iterative.)

Thirdly (#3) - and already implied the last paragraph: organisations might rationally choose to adopt one style in preference to others. For example, for an organization which follows a strict budgeting process, operates in a slow paced market and has no intention of changing its management approach (e.g. a large insurance company) it is probably right to adopt an Iterative approach. Adopting evolutionary would just upset too much of the organization and possibly destabilise it.

One of the things I would like to understand - and something I should devote more time to is: understanding the forces which lead an organization to adopt one style over another. There are rational reasons for attempting to freeze out change (while achieving efficiency benefits) and there are rational reasons for embracing change (and sacrificing efficiency).

Now, while it sounds rational to say “A company can choose its Agile style” I don’t believe that is how it happens. Rather the style a company operates in has more to do with where they came from - what economists call path dependency.

Companies and teams seldom rationally choose the style they want. Rather their existing mental models and assumptions leads them to understand “agile” in one of the styles listed. As a result when they encounter engineers, consultants and books which assume a different style they a) ignore them, b) get confused or c) dismiss them as “idealistic” or “not in the real world.”

The net result is that it is more important than ever to name these things. Just naming them (#1) and recognising the styles for what they are is a great step forward. What you do next is a far deeper issue tied up with the wider organization.

3 Styles: Iterative, Incremental and Evolutionary Agile (part 1)

When I’m teaching training courses (as I was this week at Skills Matter) or advising clients on the requirements side of software development (which I’m doing a lot of just now) I talk about model I call “3 Styles of Agile”. Incredibly I’ve never blogged about this - although the model is hidden inside a couple of articles over the years.

So now the day has come… I don’t claim the “3 Styles model” is the way it should be, I only claim it is the way I find the world.

While “doing agile” on the code side of software development always comes back to the same things (stand-up meetings, test/behaviour driven development, code review/pair programming, stories, boards, etc.) the requirements side is very very variable. The advice that is given is very variable and the degree to which that advice is compatible with corporate structures and working is very variable.

However I find three reoccurring styles in which the requirements side operates and interfaces to development. I call these styles: Iterative, Incremental and Evolutionary, and I usually draw this diagram:


I say style because I’m looking for a neutral word. I think you can use Scrum, XP and Kanban (or any other method) in any of the three styles. That said, I believe Kanban is a better fit for evolutionary while Scrum/XP are a better fit for Iterative and Incremental.

I try not to be judgemental, I know a lot of agile folk will see Evolutionary as superior, they may even consider Evolutionary to be the only True Agile but actually I don’t think that is always the case. There are times when the other styles are “right.”

Let me describe the three styles:

Iterative

In this style the development team are doing lots of good stuff like: stand up meetings, planning meetings, short iterations or Kanban flow, test driven development, code review, refactoring, continuous integration and so on. I say they are doing it might be better to say “I hope they are doing” because quite often some bit or other is missing. Thats not important for this model. The key thing is the dev team are doing it!

In this model requirements are arrive in a requirements document on mass. In fact, the rest of the organization carries on as if nothing has changed, indeed this may be what the organization wants. In this model you hear people say things like “Agile is a delivery mechanism” and “Agile is for developers”.

The requirement document may even have been written by a consultant or analyst who is now gone. The document is “thrown over the fence” to another analyst or project manager who is expected to deliver everything (scope, features) within some fixed time frame for some budget. Delivery is most likely one “big bang” at the end of the project (when the team may be dissolved.)

In order to do this they use a bacon slicer. I’ve written about this before and called it Salami Agile. The requirements document exists and the job of the “Product Owner” is to slice off small pieces for the team to do every iteration.

The development team is insulated from the rest of the organization. There is probably still a change review board and any increase scope is seen as a problem.

I call this iterative because the team is iterating but thats about it. This is the natural style of large corporations, companies with annual budgets, senior managers who don’t understand IT and in particular banks.

Incremental

This style is mostly the same as Iterative, it looks similar to start with. The team are still (hopefully) doing good stuff and iterating. There is still a big requirements document, the organization still expects it all delivered and it is still being salami sliced.

However in this model the team are delivering the software to customers. At a very lest they are demonstrating the software and listening to feedback. More likely they are deploying the software and (potential) users can start using it today.

As a result the customer/users give feedback about what they want in the software. Sometimes this is extra features and functionality (scope creep!) and sometimes it is about removing things that were requested (scope retreat!). The “project” is still done in the traditional sense of everything in the document is “done” but now some things are crossed out rather than ticked. Plus some additional stuff might be done over and above the requirements document.

I call this incremental because the customers/users/stakeholders are seeing the thing grow in increments - and hopefully early value is being delivered.

I actually believe this is the most common style of software development - whether that work is called agile, waterfall or anything else. However in some environments this is seen as wrong, wrong because the upfront requirements are “wrong” or because multiple deliveries need to be made, or because the team are’t delivering everything they were originally asked to deliver.

Evolutionary

Here again the development team are iterating much as before. However this time there is no requirements document. Work has begun with just an idea. Ideally I would want to see a goal, an objective, an aim, which will guide work and help inform what should be done - and this goal should be stated in a single sentence, a paragraph at most. But sometimes even this is missing, for better or worse.

In this model the requirements guy and developers both start at the beginning. They brainstorm some ideas and select something to do. While Mr Requirements runs off talk to customers and stakeholders about what the problem is and what is needed the tech team (maybe just 1 person) get started on the best idea so far.

Sometime soon (2 weeks tops) they get back together. Mr Requirements talks about what he has found and the developers demonstrate what they have built. They talk some more and decide what to do next.

With that done the developers gets on with building and Mr Requirements gets on his bike again, he shows what has been built and talks to people - some people again and some new people. As soon as possible the team start to push software out to users and customers to use. This delivers value and provides feedback.

And so it goes. It finishes, if it finishes, when the goal is met to the organization decided to use its resources somewhere else.

Evolutionary style is most at home in Palo Alto, Mountain View and anywhere else that start-ups are the norm. Evolutionary is actually a lot more common than is recognised but it is called maintenance or “bug fixing” and seen as something that shouldn’t exist.

Having set out the three styles I’ll leave discussion of how to use the model and why you might use each style to another entry. If you want to know more about each model and how I see Agile as spectrum have a look my 2011 “The Agile Spectrum” from ACCU Overload or the recently revised (expanded but unfinished) version by the same title: “Agile Spectrum” (the 2013 version I suppose, online only).

Events! - Speaking and training

With the impromptu Quality mini-series out of the way normal blog-service will now be resumed!
First up a bit of self-publicity! I’ve got a busy few months coming up and I thought I’d post a list of the events and public training courses I’m giving…

• Skills Matter In the Brain, “Xanpan - not a methodology, a personal take on Agile”, London, 16 September 2013, this is a free evening (6.30pm event) - registration required at Skills Matter site - THATS THIS MONDAY
• Skills Matter, Essential Agile for Business Analysts, this is a 3-day course at Skills Matter in London, 16-18 September (booking essential - with Skills Matter). Its called “for Business Analysis” but anyone concerned with the requirements side will benefit. I’m also thinking of renaming it and tweaking to to “Agile for Business Analysts and Project Managers”
• IIBA Business Analysis Conference, 23-25 September, 2013, “Patterns and Pattern Thinking for Analysis and Innovation - when I’m not preaching about agile, or wishing I’m an Economist, I’m a patterns guy, its my dirty little secret
• Agile Cambridge conference: "Requirements: Whose job are they anyway?" - 25-27 September 2013
• Program Utvikling, Kanban Software Development 101, Oslo, 21-22 October 2013: A semi-new 2-day course, some tried and tester material bundled into a new format and focused on Kanban
• Agile Tour London, "Xanpan (What do you get if you cross XP with Kanban?)”, London mini-conference, 1 November 2013, a chance to hear me outline Xanpan (and save on reading the book)

Looking further ahead, I’m taking on a challenge by speaking to the BCS PROMS-G group (thats the project management group) on "The End Of Projects - and what happens next" (5 February 2014) free, register with the BCS.

Still banging on about Quality

The last couple of blog entires have been a public investigation - personal think - about software quality - see What is Software Quality? and Software quality? Thanks to all those who have provided feedback and comments, they’ve all contributed to my thought process.

I think I am moving towards some kind of conclusion. I’ll write up that conclusion for Xanpan and post it elsewhere too. Right now I’d like to add some more thoughts and hint at the conclusion.

I’m starting to think it is easier to talk about how to achieve high quality and about the benefits of quality than it is to actually define quality itself. Without actually defining what quality is or what the benefits of quality are then talk of how to achieve it is unlikely to be more than opinion.

I think that defining quality is one thing, and we also need to recognise that there may be multiple ways of achieving that end point. I might not agree with all those ways, I might even think that some are misguided but I will recognise that mine is not the only way.

So far my definition of “Software Quality” has focused on two aspects: defects and maintainability (and particularly extendability).

That is a small view, Tom Gilb and Jerry Weinberg’s suggestions are much wider. Tom would - I think - say my two aspects are but two “qualities” among many. For example: speed performance, ease of use and conformance to specification might all be part of quality.

I’m now thinking of quality like an onion. Defects and maintainability are at the core of the onion, and defects is probably the core.

You, your organization, your team, your customers might choose to added additional qualities to this definition which you are quite at liberty to do. For example, this might be your onion:

I believe this model is entirely compatible with Gilb’s concept of quality as qualities and Weinberg’s “Quality is meeting some person’s requirements.” (Thanks to Mark Nilsen for this).
Points here:

• Quality is multifaceted and finding a single definition to satisfy all situations is tough. (Even if we had such a definition it might be too abstract or academic to be meaningful in everyday work.)
• Each organization/team/project/product could benefit from defining what they take to be quality. (And I would suggest that few actually do this and that this resulting difference in individuals opinion on “what is quality” is the cause of much friction and mis-aligned aims.)

Quality is itself a statement of values - not necessarily financial value, its as much about personal values. That said financial value should neither be ignored or belittled - after all, most software is developed for commercial reasons, profit is not a dirty word.

Thus let me make another statement I believe is true:

Quality results in business benefit - money saved, money made, happier customers, business aims realised

Once we accept this the statements like “Quality is free”, “High quality save money” and “Quality sells” become axiomatic. The aim of “high quality” is to produce one of these business benefits.

Consequently statements like “We can’t afford this much quality” and beliefs like “Lower quality is acceptable to customers, is faster and saves money” are the result of a mismatch in different individuals understanding of what is quality.

For example: one person believes customers are happy with buggy software as long as it is available soon while another believes that customers want bug-free software later. When such mismatched occur using the word “Quality” is itself a problem because it means different things to the different parties.

That might all be too circular for some, sorry about that, I wish it wasn’t but I suspect that is the nature of the beast.

Now back to my definition and the inner layers of my onion. I think defects and maintainability are at the heart of the onion and need to be present in every definition of software quality because…

Defects: the research I have seen (Jones and elsewhere) and personal experience (as a developer,manager and customer) tells me that defects (bugs) are not a good thing - whether absolute defects or common defects. My experience and intuition tell me Jones is right when he says: “projects with low defect potentials and high defect remove efficiency also have the shortest schedules, lowest costs, and best customer satisfaction levels.”

Maintainability, changeability, and its half-brother extendability, is life. Software which can’t change can’t live.

At a basic level if you have maintainability you can fix defects and you can add any other quality you like. If your software is maintainable but not performant you can change it to have the performance you want. If your software is hard to use but maintainable you can change it to be easy to use.

If your software lacks maintainability, changeability, you will find it hard - expensive but perhaps not impossible - to make any of these changes. When software lacks maintainability it is not soft.

Lets agree to call software which is not (easily) changeable (maintainable) sclero-ware.

Definition: Sclero-ware - software which is not practically changeable.
(Where practically is context dependent.)

“But” you might ask, “are there not some examples of software which does not need to be maintained?”. “Surely,” you say, “there is some software which does its job and it is done. Defects might be core of the onion but maintainability is just another quality you might, or might not want.”

While this I can follow the logic here I am lacking an example. Do you know any? Indeed I suspect there are no such examples.

Because: successful software is used, and because it is used it needs to change.

Software needs to change because the world changes: Windows 8 replaced Windows 7 which replaced Vista which…. the Euro replaced the Deutschmark which replaced Reichsmark, the 747 replaced the 707 which replaced the Queen Mary and Queen Elizabeth I, telephones became mobile, telephones replaced iPods which replaced Discman which replaced the Walkman and along the way iPods replace Hifi systems, Banks replaced Building Societies, Debit cards displaced cash, ….

Must I continue? - the world changes.

Software which is not used does not need to change.

Software which is not used is the past. It is dead. Look here, SourceForge has plenty of dead software projects you can download, try one (here’s a page of examples.)

Successful software lives, it changes, it moves on.

So yes, software which did its job and is no longer used doesn’t need to change. You don’t need to change if you are dead.

Also, “maintenance” starts a lot sooner than is many people recognised. Maintenance starts the day after coding, not the day after the “project” ended. If you need to go back a fix a bug - at anytime! - you are maintaining. If your software lacks maintainability you are going to have a hard time getting any bugs out (and I suspect if it is not maintainable you probably have a lot of defects).

To summarise: I consider software quality to be a function of high maintainability and a low number of defects. I focus on these to attributes because I believe these are invariant. I am happy for add more qualities in a given context.

I accept there are different views on the best way of achieving these qualities. For example:

• The approach I was taught in the 1980s and 1990s involved detailing what was required, perhaps writing a logical specification, engaging in a design process intended to minimise defects and maximise maintainability, then coding and finally testing the software. Some people call this “Waterfall”, I prefer “Traditional”.
• The approach I advocate today involves setting a goal, identifying small pieces which can help build towards the goal, setting tests, engineering to those tests and repeating the process again and again and again. Some people call this “Agile”, I believe there are actually three styles better called “Iterative”, “Incremental” and “Evolutionary”.

And I believe that if you lack quality - you have a high number of defects and sclerotic code - sclero-ware - you will fail every time.