Estimates or #NoEstimates? that is the question

To estimate or not to estimate, to join the #NoEstimates bang-wagon or not, that is the question.

Maybe it is a navel gazing exercise for agile-folk but it does seem to be the reoccurring theme. And I can’t get over this feeling that some of my peers think I’m a bit stupid for continuing to support estimates.

Complicating matters I’m finding my own work and research is starting to be cited in support of #NoEstimate - Dear Customer (PDF version), my publicity for Vierdort’s Law, Notes on Estimation and More notes on estimation. Add my own #NoProjects / #BeyondProjects logic isn’t far removed from the whole estimates discussion.

At Lean Agile Scotland a few weeks ago Seb Rose and I were discussing the subject, in the end Seb said something to the effect:

“How can you continue to believe in estimation when your own arguments show it is pointless?” (I’m sure Seb will correct me if my memory is fault.)

My reply to Seb was something along the lines:

I continue to believe that estimation can be both useful and accurate, however I increasingly believe the conditions under which this holds are beyond most organizations.

To which Seb challenged me to list those conditions. Well here is that list.

I’ve blogged about this before (well, I’ve mentioned it in lots of blogs, see this one Conclusions and Hypothesis about estimates) and I’ve devoted a large section of the Xanpan book to talking about I see estimates working but I think its worth revisiting the subject.

Before continuing I should say: I’m talking about Effort Estimates specifically. There is another discussion which needs to be had around business value/benefit estimation.

Here is, a probably incomplete, list of conditions I think are required in order for effort estimates to be accurate:

1. The people and teams who will undertake the work need to do the estimates
2. Estimates go off if not used: estimates only remain valid for a short period of time (days), the longer the elapsed time between making the estimate and doing the work the less accurate they will prove
3. Estimates will only be accurate if the teams are stable
4. Estimates much be calibrated against past performance, i.e. data is needed
5. Together #3 and #4 imply that only teams which have been working in this fashion for a while can produce accurate estimates
6. Teams must have a record of delivering and must be largely able to undertake the work needed, i.e. there are few dependencies on other teams and few “call outs” to elsewhere in the organization
7. Estimates must be used as is, they cannot be adjusted
8. Estimates cannot be used as targets (Goodharts Law will cut if they are)
9. Estimates made in units of time (hours, days, etc.) are not reliable
10. The tracking and measurement process must measure all work done, not just “project” work
11. Financial bonus should not be tied to estimates or work
12. People outside the team should not coerce the team in any way

There are probably some other conditions which need to be on this list but I haven’t realised. I’m happy to have additional suggestions.

Perhaps this list is already so long enough as to be unachievable for most teams. Perhaps meeting this conditions are unachievable for many, even most, organizations. In which case the #NoEstimate’ers are right.

So… I believe estimation can be useful, I also believe it can be accurate but I believe there are a lot of factors that can cause effort estimates to go wrong. In fact, I know one team, possibly two, who claim their estimates and planning processes is very accurate. Perhaps I cling to my belief in estimates because I know these teams.

When estimates do work I don’t believe they can work far into the future. They are primarily a tool for teams to decide how much work to take on for the next couple of weeks. I don’t expect estimates further out will prove reliable. Estimates for 2 to 12 weeks have some value but beyond the 3 month mark I don’t believe they will prove accurate. So my advice: don’t estimate anything that isn’t likely to happen in the next 3 months, and don’t plan any work based on estimates which extend more than 3 months into the future.

Which means: that even if you accept my argument that estimates work they may not tell you what you want to know, they may not have much value to you under these conditions.

And to further complicate matters I suspect that for mature teams estimation becomes pointless too.

As implied by the list above I would not expect a team new to this (agile) way of working to produce reliable estimates. With experience, and the conditions above, I think they can. One of the ways I think it works is by helping teams break down work into small pieces which flow. As a team get better I would expect the effort estimation to exhibit a very tight distribution. When this happens then simply counting the number of card (tasks, stories, whatever the thing you are estimating is) will have about the same information value for a fraction of the cost.

(For example, suppose a team normally do 45 points of work per iteration, if the teams average size estimate is 5 with a standard deviation of 0.5 then they would be expected to accept 9 pieces of work per iteration. If these statistics are stable then estimation works. But at this point simply taking in 9 pieces of work would also prove a reliable guide.)

So:

1. Effort estimation doesn’t work for immature teams because they don’t exhibit the conditions above
2. Effort estimation does work for mature teams but
3. Effort estimation is pointless for very mature teams

Even given all this I think estimation is a worthwhile activity for teams of type 1 and 2 because it has other benefits.

One benefit is that is promotes discussion - or at least it should. Another is that it forms part of a design activity that helps teams make pieces of work smaller.

But there is another reason I want teams to do it: Credibility.

Estimation is so enshrined in the way many businesses work that teams and those trying to introduce change/agile risk undermining their own credibility if they remove estimation early. And I don’t just mean credibility with “the business” I think many developers also expect estimation and if asked to adopt a process without it will be skeptical.

So its just possible that estimation as we knowing - planning poker, velocity and such - is a placebo. It doesn’t actually help many teams but it helps people feel they are doing something right. In time they may find the placebo actually works or they may find they don’t need it.

Another reason why I like developers to think about “how long will the take” is that I believe it helps them set their own deadline. It helps them focus their own work.

Thus I keep advocating estimates because I think they are useful to the team, the fact that you might be able to tell when something might be “done” is a side effect. Since I find long range estimates questionable I advocate a cheap approach which might be usefulness or might just be a placebo.

However, I do believe, that given the right conditions teams can estimate accurately, and can deliver to those estimates. Increasing I believe very few organizations can provide those conditions to their teams.