OK, this isn’t news, this study came out a couple of years ago and was covered by many people then. But, I find myself regularly referring to it trying to find the link. So I’m going to blog about it then I’ll always be able to find the link.
The study is by Nagappan, Maximilien, Bhat and Williams and is entitled: Realizing quality improvement through test driven
development: results and experiences of four industrial
teams and is freely downloadable from Microsoft.
This is the key findings are summarised in this table:
Team / Metric | IBM drivers | Microsoft Windows | MS MSN | MS Visual Studio |
Defect density of comparable team not using TDD | W | X | Y | Z |
Defect density of team using TDD | 0.61W | 0.38X | 0.24Y | 0.09Z |
Increase in time taken because of TDD | 15-20% | 25-25% | 15% | 25-20% |
The way to read this is: the researches looked at two Microsoft MSN teams, one team did not use TDD and had a defect density of Y. The second MSN team had a defect density less than a quarter of Y but took 15% longer.
To my mind that proves that which was to be proven, i.e. TDD reduced bugs. But, I’m also aware that other writers have disputed this and I’ve heard of studies which disprove it. (Anyone got a link? Thanks)
Most people who I’ve met, and who have practices, or understand TDD agree it is effective. However there are those who don’t believe it. It reminds me of the episode of Black Adder where Rowan Atkinson’s Black Adder hires a ship Captained by Tom Baker. When there is a problem it plays out like this:
Black Adder: “Someone in the crew will know... you do have a crew don’t you?”
Captain: “Arh, opinion is divided on the subject... all the other captains say you need a crew, and I say You Don’t”
At the end of the day Confirmation Bias will probably decide which set of results you choose to believe.
Why was TDD less relatively effective for IBM drivers than for MS Visual Studio? Is it because of different domains? Or different quality of programmers? Or what?
ReplyDeleteIs the increased time worth the lower number of bugs(It definitely is for Visual Studio, but one could argue either way for IBM)?
Are there better ways to spend those extra hours? For example code review?
All those need to be answered before something conclusive can be said...
Pavel - I've supplied the link, now read the paper. You might have some of them answered.
ReplyDeleteI think the only conclusive thing I can say is: confirmation bais exists.
Hi Allan, this is a very interesing results. Always wanted to see some research work on TDD in big organisations.
ReplyDeleteThanks for sharing this,
Link no longer works - current link (as of 25 Feb 2013) is http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
ReplyDeleteThe link might be dead in the meantime?
ReplyDelete