Agile and the Demand Curve

In the last two posts I’ve taken two different views on what might happen to the Software Demand curve when we introduce Agile:

• The good Agile demand curve
• The bad Agile demand curve

When we discussed how Agile effects the software supply curve things were pretty clear cut. Agile, in the long term boosts supply, and may even flatten the curve - making it more responsive to price changes, i.e. more elastic. Unfortunately things are not so clear cut with the demand curve.

Good news: There is a story Agile people - OK me! - sometimes tell - about what “should” happen with demand in an Agile environment.

Bad news: The is another story about demand in an Agile environment, one that Classicist (i.e. believer in “Waterfall” style up front requirements and planning) fear. And I’ve seen that too. It ain’t pretty.

But, the key point is: Few of the Agile tools work directly on the demand curve. After all, Agile is about software and the demand for software is derived from something else so why would they?

Let me say that again in other words because it is important: Nobody wants software for the sake of software, the demand for software is derived from some other demand, e.g. a retailer wants to sell online, that means they need software.

The current Agile tool set (e.g. User Stories, Acceptance Criteria, Specification by Example, etc.) which touches on demand - call it need, want, requirements, specification, whatever you want - does not address the underlying demand. These tools operate with the demand as presented.

Sure there can be feedback from the tools and Agile deliveries but feedback is often absent. Even when feedback is present those responsible for the underlying demand may not want to hear - or act - on the feedback. And in an environment where success is measured by criteria like “On Budget, On Schedule, On Features” acting on feedback may be impossible.

Which brings us back to the 3 Styles of Agile: Iterative, Incremental and Evolutionary (part 1) and 3 Styles of Agile part 2. (At some point in the future I will reprise those three styles in the light of this analysis.)

One of the key factors in determining whether the feedback can affect underlying demand is whether those responsible (e.g. senior managers) who largely originate the underlying demand are open to feedback. Which brings us to another story I need to tell, another time...

Where does this leave us then?

If the current Agile toolset cannot address the underlying demand we need new tools. We might choose to add those tools to the thing we call Agile or we might choose to put them in another category.

Personally I believe many of those tools do exist but they exist outside the Agile space. They are fellow travellers if you will. For example: Beyond Budgeting is clearly one of those tools. However, many of those tools have their own names, they are not Agile, they are what they are (e.g. Beyond Budgetting). They fit very nicely with Agile but they are not.

However that is my personal view. I also think that Agile is a marketing bang-wagon and those tools will be conscripted to the Agile course and in many cases put under the Agile umbrella.

Either way the problem we need to recognise is: addressing the software demand curve Agile is necessary but not sufficient by itself.

The bad Agile demand curve

In explaining the Agile Demand curve I told a good news story in the previous entry. This time I want to tell a bad news story. This is every classical Project Managers’ fear about Agile, its story we don’t tell often but it can happen. Once I’ve told this story we can proceed to the proper analysis of the Agile Demand curve.

So a classical project manager looks at Agile and he sees his greatest fear: No Scope limit.

There are no signatures against exactly what is to be build, not even in pencil, let alone blood. There are some loose “Story things.” But he is told it is Agile and Agile works this way so against his better judgement he gets stuck in - after all, he needs to get Agile on his CV sometime.

And to some degree the classical Project Manager is right to have this fear.

After years of being pushed around by IT and IT’s sub-contractors the business hears of Agile and thing “Yes! - we are finally going to get what we want. No more messing about with long documents, no longer waiting for ever and no more bugs.”

These days when I go to talk to a company about “going Agile” the Agile publicity machine has beaten me to it. Unfortunately for some on the business side they believe that “going Agile” means they can have whatever they want whenever they want it. If the development team are Agile, so the logic goes, then they will deliver fast and will deliver exactly what is wanted even if the business keeps changing their mind.

You can hardly blame the business side from believing this, after all, it is how Agile presents itself. Unfortunately Agile as “all things to all people” and “I don’t need to change what I’m doing” is more prevalent than is healthy.

The next effect of this attitude is to boost the Demand curve significantly, and if it does affect elasticity it probably reduces it further.

The demand curve moves from D to D’. At every price point the business wants more. Of course the business doesn’t think of it that way. They are just asking for what they always wanted but now, because it is “Agile” they are not restricted.

Previously, the need to put every request in writing - voluminous writing - and argue it with various managers meant making the request required effort (a price) which they were not willing to pay. No with Agile all that is gone and demand is unbounded.

There is one “project” I know which is quite large, we are talking dozens of people all counted. Demand is rampant, the business keeps on thinking of new ideas, analysts keep finding more work in what has been asked for, developers find more work and the testers more (because initial quality is not what it could be.)

In short: I’ve seen rampant demand happen and I think Agile makes it more likely because it removes many of the restrictions previously used to control demand.

If the team don’t deliver on (often unreasonable) expectations “Agile” becomes another tool with which to beat the team.

It is worth noting here: that as demand is derived much of that demand always existed but was hidden. What is a more interesting question to ask is: which elements of this demand are worth doing? That question will need to wait.

We need to add supply to our diagram above. There are two scenarios we need to consider here and shows in this diagram:

Scenario 1: Demand moves to D’ but the developers are still operating on the original Supply curve S. In which case the amount and the price is going to rise in both dimensions. The next diagram shows this, the yellow area is the pre-Agile cost while green area shows the additional cost.

Scenario 2: Demand moves to D’ again but this time the team are operating on the Sa (Supply Agile) curve. Thus we get this diagram:

Things are more complex now: because supply is elastic and responds to price more software is produced than at the beginning but the unit price is lower. So the blue are represents increased cost but the purple area represents money saved.

The immediate question I bet you are asking is: have they saved more than they increased?

As it happens yes. On my graphing tool the purpose area takes up 45 grid squares and the blue 46. So they have saved, just.

But, those curves were placed by my for readability without any data for any actual team. The difference is small and demonstrates things could go either way.

A more important question which should be asked is: are the team delivering more benefit? More value?

When I started this blog entry my aim was to show that talking Agile could by itself increase demand. The problem I want to highlight is rampant demand, and this is not confined to Agile. Although I think Agile can make it worse.

What I have also shown is that if the development team really is Agile they can handle this but it isn’t clear.

The good Agile demand curve

I’ve been slow with this entry on Agile and the Demand Curve, apologies. Part of the reason for the delay is software demand is not simple.

In my last entry on software economics I discussed why the Software Demand curve for custom software is both high (i.e. people want a lot of software) and inelastic (i.e. increases in costs don’t reduce demand by much.) This time I want to look at how applying Agile changes the demand curve.

In order to explain what happens to the demand for software in an Agile environment we need to consider different possibilities. So lets start with a good news story - the bad new story and analysis will come in later blog posts.

This story is about how, using Agile software development, a team reduced the demand for software and shipped a product sooner. This is the story most Agile advocates will associate with and happily promote. This is the way Agile is supposed to work....

There was once a team who adopted Agile software development. At first they were able to demo their enhanced software to others in the business on a more regular schedule. They made a big thing about this and invited others to come and look. This generated feedback on the thing they were creating: some good, some not so good, some requests for more and some cancellations of things which had been asked for.

A little while later they stated making more regular releases. And because the software was “ready to release” more often the business could decide: “Do we want it now - with any pain that might bring? Or do we want to wait a little longer for more?” (A bird in the hand is worth two in the bush as they say.)

Simultaneously the team were adopting technical practices which improved the quality of their code - they also had the side effect of reducing the amount of code. Not only did this make the team more productive because they had less code to maintain and bugs to fix but it also meant the business had fewer reasons to complain about bugs.

(In my experience, for some teams when the conversations about “Have you fixed it yet?” and “Can we release it yet?” start to disappear it turns out business don’t know what to ask for. They’ve forgotten or never knew what they really wanted.)

The team also adopted User Stories at some point. As they got better at this technique they were challenged to think about “Who actually wants this story?”. With a little time stories advanced from “As a User” to something more specific.

And one day the “Product Owner” realised: they were not building one product but two.

The Product Backlog of work to do would be better throughout of as two backlogs for two products, each with its own user/customer base. User Stories had helped him see this.

When he restructured the backlog he found he had three piles. Product A, Product B and not justified. A chunk of the backlog, maybe as much as a third, didn’t fit in either product. They were nice ideas, things people had asked for, but when you came to analyse them they didn’t belong anywhere and justification was weak.

This diagram shows the Demand Curve graphically. The team start with demand D, Agile supply is Sa. After the restructuring to separate products appear with their own demand curves Da and Db. Each on of these can be reasoned about independently.

At any price point the combined demand of A plus B is less than D (the original total) because some work has gone away. Note: I have not suggested any changes to the slope of the curves, it is possible that different products will have different elastics but that is another discussion.

While the team had been battling lots of bugs and their output was a black hole to the rest of the organization, and while they thought they were building one big product this insight had been hidden from them.

That is the way Agile is supposed to work on the demand side. By being clear, by being delivery focused, by feeding back to users and customers people get what they want.

But, do you know what? All the members of this team are still employed and still work on software. In fact I think the team might have grown a bit. There is still work to do, there is still demand.

Agile reduced the demand in one part but there is still plenty that company wants to do, and much of that means software. Getting demand reduced in one place just leaves the team more time to do other work, other demand.

(OK, if I’m being honest while I have one team in mind while I was retelling this story I’ve simplified it (knocked a few rough edges off) and composited it a bit to include other examples. I don’t think that invalidates the story as this is how it is supposed to be.)

Honoured - I'm on a 100-top lists

Bit thank yo to the guys at ProgramCreek.com for including this blog on their list of “100 High-Quality Java Developers’ Blogs”.

Strictly speaking this is not a Java blog but most of the issues I talk about do effect Java developers.

It also seems that in listing this blog at number 55 (yeah, still room for improvement!) the ProgramCreek.com team have set up a little competition between me and my #NoProjects conspiracist Steve Smith who is just two places behind me :)

(Whoops, should have posted this when it was announced! - better late than never)

Software demand curve

Returning to my series of posts applying the tools of economics to software development - Supply & Demand in software development, Software supply over time and Software supply & demand - this time its Agile - it is time to turn our attention to the demand curve. First a reminder of how things start...

First I need to explain why I believe there is so much demand for software and why I believe the curve is inelastic, i.e. why a higher price doesn’t reduce demand very much. The reasoning here breaks into five groups: technology progress, separation of demand from cost, separation of benefit from delivery, lack of evaluation and the effect of “fixed” work - although the last in that list doesn’t always play a part.

It is important to remember in all this discussion that software is a derived demand. Nobody wants software for its own right, they want it to achieve some other aim.

Technology Progress

Moore’s Law implies that processor power doubles every two years (give or take a bit). Consequently the ability of computers to tackle ever more complex problems and fill more needs constantly increases. Supply creates demand, we want it because we can.

For example, I’m hoping Farther Christmas will bring me an activity tracker for Christmas, I didn’t know such devices existed until recently but I can see it solving one “problem” in my life. These devices didn’t exist until recently, I didn’t need one because they didn’t exist. The underlying problem existed but had been put in the “thats life” bin.

At a company level this means the benefits computers can bring to our companies are constantly increasing and there are new opportunities to save costs or generate revenue. And if the incumbent company doesn’t take these opportunities then Amazon, Google or some start-up will.

Moore’s Law is not alone, Metcalfe’s Law compounds the effect. The more devices we enable and the more devices we connect to the internet the greater the value and opportunities from connecting these and more devices.

Separation of Demand from Cost

Traditional development process - and even Agile processes - tend to separate the demand for software technology from the supply, consequently those requesting technology are isolated from the cost of that technology.

Many practices in software development make this problem worse. We send Business Analysts to talk to “users” about “What they Want” i.e. what is their demand. These BAs are sometime little more than order takers, waiters. People are encouraged to ask for more and more. Indeed, some of our practices incentivise people to ask for more and punish them for not asking for it.

Even if “users” don’t ask for it Analysts may be encouraged to include any need they can comprehend. I once had a BA tell me “If we miss a something in the requirements document it is seen as a black mark against the BA.”

Separation of Benefits from Delivery

To make things worse its not just demand which is separated from costs but so too are the benefits. Nobody wants IT for IT’s sake, they want it for some benefits. But achieving the benefits may well take time, it might be months or years after an IT system is delivered before the benefits are seen.

That is, if the benefits are ever seen, IT alone is not enough to bring about benefit. Users may need training, changes to processes may be needed, customer agreements may need to be changed and so on.

Although it is a few years old now Wired for Innovation contains a good, short, discussion of why IT benefits don’t appear in the way we would like them too (i.e. quickly and easily) and what can be done to help.

Lack of evaluation

Ideally companies would evaluate the benefits delivered by IT work but all too often this fails to happen. Instead companies may rely on the original claimed benefits. But this might the result of optimistic thinking itself: “38% of businesses openly admit benefits are overstates in business cases in order to obtain project funding” - from a Cranfield Business School study I reported in October 2010.

And when one manager exaggerates benefit to get his project funded it creates an incentive for the next manager to exaggerate the benefit of the next project and so on. Benefit inflation cascades through the organisation.

This isn’t helped by vendors who claim benefits for their product which might not apply to a particular business or in a particular context. If company A used product B to save $C millions then surely company X can use product B to save about $C millions, right?

In our personal lives we have become accustomed to technology getting cheaper and more powerful and we expect the same in business. But consumer technology is paid for by millions of people buying millions of products while business technology may have one customer. Consequently we don’t appreciate the costs involved in creating new software.

Goal Displacement: Floorded contracts, projects and governance

In an effort to control IT, to maximise the return organizations frequently fixate on controlling the costs. They demand things like price, time and scope are decided in advance and fixed. Sometimes this is done internally - projects are approved and must be finished according to some set budget or date. And sometimes it is done by finding a supplier who will agree to fixed price/time work. (I’ve written about this before in Dear Customer on this blog and on Agile Connection/TechWell.)

The problem is, when you put these arrangements in place the objective becomes: meeting the fixed criteria, not delivering value. It is what psychologists and management students call Goal Displacement.

Options to reduce scope, time and cost are lost in the process. The things which were meant to be maximums, ceilings, become minimums, floors. Opportunities to reduce costs are lost to more features and more software.

Net effect

Taken individually, together and in combination with other forces we end up with a demand curve which is very high and inelastic. Changing the price of the technology doesn’t reduce the demand by much, Moore and Metcalfs Law mean there are always new opportunities, failure to capture original benefits mean they are still there for the taking, and lack of evaluation means nobody is counting, boys will always make arguments for more toys.

In the next instalment of this series I intend to look at what Agile can do about this and whether Agile makes things better or worse.