Agile is like sex

I’ll get back to software development economics, there are more posts in the works, but right now I need to say something....

I have blogged before on the subject of “What is Agile” - I’ve even expanded on that blog in an unfinished piece of writing called “What is Agile? Perspectives on Agile” - but sometimes I think its just sex....

(OK, this blog just got filtered out of lots of feeds and stopped by lots of firewalls but lets continue.)

In the software development world there are, broadly speaking, two groups of people: those who create the software (coders, testers, etc.) and those who manage the process (project managers, development managers, etc.). When discussing "Agile" I find that both sides think the problem is with the other.

To put it another way, if I'm talking to developers they think its managers who are the block to adopting more Agile techniques and improvement. But when I'm talking to managers they say its the developers who resist Agile and other improvements.

Which always reminds me of the old Philip Crosby quote:

"Quality has much in common with sex. Everyone is for it. (Under certain conditions, of course.) Everyone feels they understand it. (Even though they wouldn't want to explain it.) Everyone thinks execution is only a matter of following natural inclinations. (After all, we do get along somehow.) And, of course, most people feel that all problems in these areas are caused by other people." Philip Crosby

Lets bring it up to date and make it Agile specific, substitute the work 'Agile' for 'Quality':

"Agile has much in common with sex. Everyone is for it. (Under certain conditions, of course.) Everyone feels they understand it. (Even though they wouldn't want to explain it.) Everyone thinks execution is only a matter of following natural inclinations. (After all, we do get along somehow.) And, of course, most people feel that all problems in these areas are caused by other people." Allan Kelly after Philip Crosby

You see? It fits.

Agile isn’t someone else’s thing, it doesn’t belong to another person: Its yours. You are as likely to be the block as the person you think is the block.

Software supply & demand - this time its Agile

Carrying on from my previous posts applying the economists tools to thinking about software development (Supply & Demand in software development and Software supply over time). In this post I want to see what happens when we apply Agile...

We start with our supply and demand curves as before:

First we need to define Agile, not a trivial matter. Lets simplify this by saying “Agile practices”, specifically:

• Short iterations and regular releases (or at least demos), e.g. every two weeks
• Planning meetings, planning poker estimation (or similar, or #NoEstimates), velocity (or commitment if you prefer)
• User Stories, Epics, task breakdown (if you do this sort of thing)
• Stand-up meetings
• Team retrospectives and team planning boards or electronic equivalent
• Test driven development and continuous integration
• Pair programming, code review and shared code-ownership
• Acceptance Test Driven Development (ATDD), Behaviour Driven Development (BDD), Specification by Example (SbE), and other automated testing mechanisms

The first thing to note about all these techniques - and indeed the majority of what we tend to call “Agile” - is that these techniques work on the supply side. They operate on the supply curve, the net effect is to increase supply.

Possibly, if used in the right fashion, BDD and SbE might effect the supply side. But since most of the efforts to use BDD I encounter are little more than super-charged TDD we’ll leave it there for now - see Dan North’s blog for a better discussion BDD v. TDD.

Projects - specifically projects but other work initiatives too - are often still judged on the basis of: on schedule, on feature and on budget. Agile in this fashion is what I have called Iterative Agile, or at best Incremental. The team is still seeking to deliver a fixed quantity of software. Thus the demand curve is unchanged. (We will return to the demand curve in another post and discuss how Agile techniques might affect it.)

While Agile tools promise increased performance this is only in the long run. In the short run we can expect teams to slow down as they adopt these tools. We can also expect costs to go up as companies to spend money to buy in this expertise.

Maybe they hire the likes of myself to supply training and consulting in adoption (yes! I’m for hire, see my company website Software Strategy). Even if they don’t pay the likes of me costs will go up because engineers need to learn these skills. Companies rationale for hiring me is that I will help reduce the period of time for which they are slower and reduce the risk. The deal is: hire me and I will help you get to increased productivity faster (and with less risk).

Putting this in terms of curves:

A team starting on Ss can expect to move left (1) to Sasr (agile short run) before moving to the right (2) - Salr (agile long run) - reduced supply before greater supply. In the first instance the price per unit increases while the quantity delivered falls but after the second move the price per unit falls and the team produces more software.

Leave consultants like me to one side for a moment and simplify it. Imagine a team which “just does it”. Assume for a moment that the team members stay the same and the amount they are paid each week stays the same. When the team start doing something new they slow down, they produce less software each week so the average price of per unit increases. As they get better the same team better their original performance, earn the same money, produce more software and as a result the average price per unit falls.

Hopefully that all makes sense and isn’t controversial.

But I think something else happens. This is a hypothesis, at the moment I can’t measure it so please tell me - in the comments on this blog - whether you agree or think I’m wrong.

Staying with the supply curve, consider again the tools we were looking at:

• Short iterations and regular releases make it cheaper to make small changes. Therefore the supply of the small quantities of can should take place at a lower cost. And since the delivery - or at least a demo - is earlier the whole “project” could be cancelled if things don’t work out. Thus the bottom of the curve moves left over time.
• Automated testing (TDD, ATDD, etc.) reduce the quantity of faulty software produced and increase the amount of useful software increased at any point thereby raising all points on the curve. With less “technical debt” teams can deliver more functionality more cheaply.
• The same techniques - TDD, ATDD, etc. - and shared code ownership also make it easier for new engineers to become productive because a) there are more code examples of how the system works and b) places a safety net under any work they do, allowing them to experiment and learn faster, and get their changes to production sooner.

The net consequence of these changes is to flatten the curve, not entirely but certainly reduce of an angle. In economics terms we increase the elasticity. Changes in price - adding or removing a developer - causes a bigger effect on the quantity supplied than before.

Supply moves from the initial Ss curve to the Savlr (Supply Agile Very Long Run). As the chosen name implies, this takes some time to occur.

Now there is a catch. Consider the total amount of software bought and the total price paid (price x quantity.) To do this look at the next two diagram.

The green area represents the total paid for the software. You can’t see it here but on my graphics tool I can see price is 10 high and quantity 7 wide, so 70 (whatever) units.

In this diagram, on the new supply curve, the blue area represents the total spend. Again you can’t see it but on OmniGraffle it is 5 high and slightly more than 9 wide, almost 45 units of total spend.

Now if you are producing software for your own company - either to resell (e.g. Microsoft) or to consume yourself (e.g. a bank) - this model is good news. You get more software for a lower overall spend.

But, if you are an outsourced provider of software selling custom software to a client this is bad news. You now deliver more software for less money, about 36% less in total. If profit is tied to revenue (time and materials) then this is bad.

No wonder outsourced providers can be reluctant to embrace Agile or to do it badly. If you are a consumer of outsourced software development services you might want to think about that.

Next time I want to turn my attention to the software demand curve.

(As a footnote, using the same analysis techniques it is possible to show that reducing supply of software - moving the supply curve left, either because the software ages or because Agile adoption is slowing things down - will result in a greater overall spend for less software. This might be good for outsourcer suppliers but isn’t so good for in-house development.)

Blog revisions: economics

Although I don’t think I’ve ever stated it I operate a “no revisions” policy to this blog. I don’t take things down, and I rarely modify past entries. OK, I might change it in the 10 minutes or so after I first post to fix typos and gaffes I see once I see it online but otherwise I avoid doing so.

But my recent posts on software economics, supply and demand, contained inaccuracies. Or rather the graphs I published didn’t follow economics convention. The logic was (mostly) right but the graphs were wrong. I rushed out one fix and breathed a sigh of relief.

Then last night on my way back from Riga I noticed another flaw. I won’t go into details but I’m now fixing/fixed it.

Several times in the past I’ve attempted to capture my thoughts on software economics but nothing I’ve ever written has seen the light of day. As I continue this analysis I’m finding more insights I hope to share. I’m also - possibly - making errors.

I hope other readers who think they know economics can help debug my posts. They will be better for it.

In the meantime, sorry.

Software supply over time

Having outlined the theory of supply and demand in the software market I want to spend this, and the next few posts examining what this analysis can teach us. I’m starting with the supply curve and will look at the demand curve later.

First I want to consider what happens to a software system over time if nothing else is done. (Consider this as a small detour before we consider Agile.)

Over time changes to software systems will add code, this means there is more code to maintain, more code for engineers to understand, more code for testers to test. Internally code quality tends to decay, some people call it software entropy.

Engineers leave the team and new engineers (might) arrive thereby loosing knowledge of how the system works. At least the knowledge is diluted. And if maintenance is outsourced or offshore such knowledge may be destroyed altogether. The new, cheaper, outsourced developers need to relearn.

Meanwhile the business environment seldom becomes any simpler. New business rules get added one place or another - increasing complexity. New systems need to be connected - either as data sources or destinations - which again increase complexity.

This all gives rise to the Big Ball of Mud pattern.

Finally technology changes. The operating system and databases which form the foundations of the system may be upgraded or replaced. In the worse cases the language the system is written in dies.

The net result of all of this is that the system becomes harder to maintain. Harder translates to more time and more time translates to more money. Thus the system becomes more expensive to maintain. Consequently the supply curve for the system moves to the left. For the same price we can now get less software.

Consider the graph below. We have the Supply of Software (Ss) and Demand for Software (Ds) curves as before. Now we add the Supply of Software over Time (Sst) curve. Over time, of nothing is done, the curve itself moves, all other things being equal we do not move to a new position on the curve. We get an entirely new curve.

Now it is possible to fight against this trend. And certainly there are plenty of tool vendors out there who will happily sell you a tools which they claim will move the curve in the opposite direction. (In economics terms we break the “all other things being equal” assumption.”)

But either way: unless you fight it things will get more expensive for the same quantity of software - or you get less software for the same money.

Supply & Demand in software development

(Whoops, I’m publishing a revision of this blog entry - something I don’t normally do. I realised after publication that I had mislabelled the graphs - they were correct but did not follow economists convention hence this revision. The core argument still stands although I have had to modify the reasoning slightly. Sorry.)

I’m proud to say I am, or at least was, a Software Engineer. I’m proud to say that I’m a third generation Engineer. My farther and grandfather got their hands dirty with the oily sort of engineering. But I’m also a frustrated Economist. Economics forms part of both of my degrees and on a couple of occasionsI’ve been temped to become a real Economist.

So I am always disappointed when someone talks about Software Economics. I have yet to find a serious study of software economics. Most of the studies, books and papers about “software economics” would be better called “software accounting.” They are about apportioning costs and adding up numbers. Economics is not Accountancy.

I have long believed software engineers and their managers would benefit from a better understanding of economics. In fact many of my clients will be used to hearing me refer to the supply side (development) and demand side (requirements, analysis, etc.).

I find it very useful to think of software development in supply and demand terms and use the tools of economists to understand what is happening. Usually thinking like this shows there supply and demand to be mismatched.

(This blog post that has been a long time coming. Several previous attempts have floundered because there is so much I want to say about software economics - and thinking in supply and demand terms. So while there might be little in this post which is earth shattering - especially to those of you who have studied economics - later posts will build on these ideas and analysis.)

Lets start with the fundamental problem which economic theory addresses: the allocation of scarce resources. After all we have scarce resources in software development so lets use an Economists tools: supply and demand.

Lets have a look at the supply curve:

The line marked Sn is a normal supply curve. When price is a bit greater than zero individuals will start supplying their services and companies will come into being to supply the market. As the price increases more individuals and companies will be attracted to the market and supply will increase, more resources are brought to play.

Note two things here:

1. Economists usually draw these curves for a product within a market. Here I am not examining the supply of multiple copies of a software product, rather I am examining the supply of software development capacity for a unique product. The marginal costs of supplying existing software are as close to zero as to be zero, the marginal costs of supplying new software products - i.e. the capacity to create a product - is significantly larger than zero and that is what we are examining here. In software terms we need to consider a software product, perhaps a single program, perhaps a “project” or a small system.
2. If you are not an economist don’t worry about the fact that these “curves” are straight lines, the basic analysis still holds, we hold actual curves for an advanced class.

The software supply curve - Ss - starts the same way. When price is zero nobody supplies - perhaps not entirely true if we consider open source but close enough. As price increases more resources are brought into play and supply increases.

Software development is not cheap. If you are buying software developer services they are rarely cheap. The only time you get “cheap” software is when you buy (some) off the shelf stuff. Consequently when the price is low there is no supply of custom software.

But - and this is the BIG BUT - supply raises very slowly. Two people might write twice as much code as one but when they work together the generalised form of Brook’s Law comes into play: adding people to a software development effort slows a team down. Therefore supply responds slowly to price increases - what economists call inelastic.

Now consider the demand for new software:

Here Dn represents demand for a normal product, when the price is high there is little demand for the product. As the price falls the demand increases because more people can afford it. (Remember an Economist’s definition of “Demand” is “A want with the ability to pay”. I may want a new iPad Air but if it is more than I can pay there is no demand - some might call this “latent demand.”)

For normal products, in time, the market will “clear” and balance will be reached when the curves meet:

At some price (Pn) the quantity supplied (Qn) will match that demanded. Now economists accept that this might not happen instantly - and there are various models for that - but in the scheme of things the time it takes for the market to settle is not a big issue. It will happen eventually.

Until that happens all sorts of problems will ensue, some people might make a lot of profit, some people might go without and some people might have to pay a non-financial price, e.g. queues will form. All good stuff we could consider but lets leave that for another day.

Returning to the demand for software. People and companies do not want software for its own sake, they want software because of what it allows them to do. What Economists call a “derived demand”.

The problem is, and the reason I’ve drawn the the Ds curve as highly inelastic is that we are faced with an ever increasing demand for software. Processor power doubles every 18 months to two years (Moore’s Law) which means the things our computers are capable of doing - with the right software - is forever expanding. When we get this right the benefits can be massive. Which increases the demand for software…

• How many readers wanted to read newspapers on their telephone three years ago?
• How many readers knew they needed to watch TV on an iPad five years ago?
• How many readers took digital pictures with their telephone and posted them online on a social network to instantly share them with friends and relatives seven years ago?

So lets look at the software supply and demand curves together:

For me, the problem in software economics:

• Demand is rampant but inelastic so as price rises it does not fall off very fast
• Supply is severely constrained and is very difficult to increase (particularly in the short run) so is also inelastic
• Consequently the market only clears at high prices and slowly: consequently markets take a long time to reach equilibrium, which in turn means we need to consider the tensions arising in a market which spends so much time out of equilibrium. Thus the time lags which normally only play a small part in classical economic analysis are more significant.

Each of these statements deserves further examination and explanation in their own right - future blogs, maybe.

More importantly, and also deserving of its own blog entry, is the question: what can be done to bring this market into equilibrium?

That is the question I find myself wrestling with on behalf of clients, and it is one I will return to.

Sprint, Iteration or Marathon?

A shot blog on terminology...

I have always found it hard to differentiate an Iteration from a Sprint, so much so that I commonly say they mean the same think. They are synonymous. I use the two interchangeably. The difference is historical, term “Iteration” originated in XP and “Sprint” in Scrum.

To me both mean: a short period of time in which work is done.

Occasionally people have attempted to differentiate the two terms but I’ve never been convinced.

At Agile on the Beach, in a private conversation, Rachel Davies suggested a difference which I think stands up:

“Sprint means a fixed period of time during which the team commit and work towards a fixed goal which does not change. Iteration means a fixed period during which the team undertake some work - no commitment, no fix work.”

Most of the time the difference probably does matter but I like Rachel’s differentiation. It ties Sprint quite closely to Hard Core Scrum.

I really should try to stick to Iteration.

Actually, I think the real Sprints occur when:

• A team starts a new piece or work and they have no idea what they are up against. They are sprinting to maintenance mode. Risk is highest when sprinting, until the team have their walking skeleton - a thin slice of functionality that does something tiny
• Or when a team are first trying a new way of working (probably Agile like). They have a short period of time where the rest of the organization doesn’t know what they are doing, or if they do will extend good will to let the team try. A honeymoon, if you like. At the end of that time the team need to be able to show something better. Otherwise the good will is gone and with nothing to show for it people feel let down Again risk is high.

Real software isn’t about sprinting, real software development is a Marathon. You need to pace yourself. The danger with constantly sprinting is that you burn out.

Real software development is a marathon run in laps. We call the laps iterations.