Monday, August 30, 2010

Study on benefits of TDD

OK, this isn’t news, this study came out a couple of years ago and was covered by many people then. But, I find myself regularly referring to it trying to find the link. So I’m going to blog about it then I’ll always be able to find the link.

The study is by Nagappan, Maximilien, Bhat and Williams and is entitled: Realizing quality improvement through test driven

development: results and experiences of four industrial

teams and is freely downloadable from Microsoft.

This is the key findings are summarised in this table:

Team /

Metric

IBM drivers

Microsoft Windows

MS MSN

MS Visual Studio

Defect density of comparable team not using TDD

W

X

Y

Z

Defect density of team using TDD

0.61W

0.38X

0.24Y

0.09Z

Increase in time taken because of TDD

15-20%

25-25%

15%

25-20%

The way to read this is: the researches looked at two Microsoft MSN teams, one team did not use TDD and had a defect density of Y. The second MSN team had a defect density less than a quarter of Y but took 15% longer.

To my mind that proves that which was to be proven, i.e. TDD reduced bugs. But, I’m also aware that other writers have disputed this and I’ve heard of studies which disprove it. (Anyone got a link? Thanks)

Most people who I’ve met, and who have practices, or understand TDD agree it is effective. However there are those who don’t believe it. It reminds me of the episode of Black Adder where Rowan Atkinson’s Black Adder hires a ship Captained by Tom Baker. When there is a problem it plays out like this:

Black Adder: “Someone in the crew will know... you do have a crew don’t you?”

Captain: “Arh, opinion is divided on the subject... all the other captains say you need a crew, and I say You Don’t”

At the end of the day Confirmation Bias will probably decide which set of results you choose to believe.

5 comments:

  1. Why was TDD less relatively effective for IBM drivers than for MS Visual Studio? Is it because of different domains? Or different quality of programmers? Or what?

    Is the increased time worth the lower number of bugs(It definitely is for Visual Studio, but one could argue either way for IBM)?

    Are there better ways to spend those extra hours? For example code review?

    All those need to be answered before something conclusive can be said...

    ReplyDelete
  2. Pavel - I've supplied the link, now read the paper. You might have some of them answered.

    I think the only conclusive thing I can say is: confirmation bais exists.

    ReplyDelete
  3. Hi Allan, this is a very interesing results. Always wanted to see some research work on TDD in big organisations.

    Thanks for sharing this,

    ReplyDelete
  4. Link no longer works - current link (as of 25 Feb 2013) is http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf

    ReplyDelete
  5. The link might be dead in the meantime?

    ReplyDelete