In my Agile Business Conference present (“How much quality can we afford?”) I talked about the Reuse Myth, this is something always touch on when I deliver a training course but I’ve never taken time to write it down. Until now.
Lets take as our starting point Kevlin Henney’s observation that “there is no such thing as reusable code, only code that is reused.” Kevlin (given the opportunity!) goes on to examine what constitutes “reuse” over simple “use.” A good discussion itself but right now the I want to suggest that an awful lot of code which is “designed for reuse” is never actually re-used.
In effect that design effort is over engineering, waste in other words. One of the reasons developers want to “design for reuse” is not so much because the code will be reused but rather because they desire a set of properties (modularity, high cohesion, low coupling, etc.) which are desirable engineering properties but sound a bit abstract.
In other words, striving for “re-usability” is a developers way of striving for well engineered code. Unfortunately in striving for re-usability we loose focus which brings us to the second consideration.... cost of re-usability.
In Mythical Man Month (1974) Fred Brooks suggests that re-usable code costs three times as much to develop as single use code. I haven’t seen any better estimates so I tend to go with this one. (If anyone has any better estimates please send them over.)
Think about this. This means that you have to use your “reusable” code three times before you break even. And it means you only see a profit (saving) on the fourth reuse.
How much code which is built for reuse is reused four times?
I would suggest the answer to this question is: very little.
Consequently development teams which write a lot of reusable code are costing their organisations a lot of time and money. Waste.
Reusable code is not the solution to any of our IT problems. It is a supply side only solution and not a very good one at that. While it may reduce the amount of code that is written it reduces it by artificially constructing supply rather.
Thus, reuse, as a solution to software supply problems, is a myth.
This leave two questions we need to answer.
First: how are we to get those desirably engineering properties if we can’t/don’t push reuse?
The good news here is that Test Driven Development, i.e. writing code that can be automatically tested with other code we write, also has the effect of promoting modularity, high cohesion, low coupling, etc. Thus the answer to this question is: Make your code testable.
This approach will retain focus and deliver worthwhile benefits.
The second question: How should we manage reuse? After all, there are some genuine situations were reuse is the right answer.
Here there are two answers, a micro and a macro answer.
The micro answer, when your working in a development team is: look for emergent reuse.
Don’t plan for reuse but look for opportunities to reuse something that has gone before. When you see the opportunity take the previous work and enhance it just enough to reuse it in your case. This way you only pay the price for what you actually need when you need it.
If you later find the same code is useful again then repeat the process. Improve it just enough to use it a third time.
Remember: you have the tests from the earlier answer to make it safe to do this. Without the tests things get difficult.
Now you’ve reused your code three times, you’ve pay the price a bit every time, and you have the tests to show it still works. By now the code is going to be getting pretty close to generically reusable.
Still, maybe you go round this look a third or fourth time. It doesn’t matter is making one piece of code reusable costs more in the long run than it would have done if you did it the first time because... You haven’t spent money making a lot more code reusable that never needed it.
Telling the future is hard. It is difficult to know whether code will be reused or not, so default to Not and save the money. Occasionally you will be wrong and it will cost more, but overall you will be right and you will save money.
The macro answer: if this code really is widely reusable by many people then, go into business. Market this as a library, a plug in, an application what ever. Let the market decide the economics of reusability.
Great post! I've been tempted by the sirens of reuse myself and once burned, believe I have learned.
ReplyDeleteA good article about studies into the cost of reuse can be found at http://www.informationweek.com/708/08iuhid.htm. In it, the author quotes Norman Kerth (the same as the author of "Retrospectives", I think) as saying: ". Kerth found that it takes at least three times the effort to build a reusable part that it does to build a part for one-time use". This article is newer than "Mythical Man Month," so maybe Kerth has some research backing up his assertion...
A question: You talk about reuse of code. What about reuse of services? Does how you package the reuse change the equation?
Nice article Allan.
ReplyDeleteI thought about one thing while I was reading it though... Isn't being able to change or add functionality to an existing code base one kind of re-use? If you had to rewrite everything for every new release of your software, that would be pretty painful. So you reuse the old codebase, right?
Do you have any thoughts on that?
Andre, you should be able to extended code, I don't think that is really reuse - have a read of Kevlin Henney's article. Making it "extendable" on day-1 might be over engineering but you should be think about this. As the late John Vlissides once said "Extendability is a (perhaps the) hallmark of good code."
ReplyDeleteDon't frameworks qualify as re-usable code? They're written with the express purpose of being the foundation of many projects, and to keep the programmer from having to re-invent the wheel over and over, and over...
ReplyDeleteYes, frameworks are genuine reuse but they are the exception.
ReplyDeleteHow many of the frameworks ever written are actually successful?
"Its is a framework" is just another cover for spending more on code than is necessary.
Post would be clearer to understand if you spelt the words correctly:
ReplyDelete"How much code which is build for reuse if reused four times? "
This is an interesting post and I would like to share my opinion. I agree with Mr. Kelly as much as I disagree. Maybe we just use different names or have different experience.
ReplyDeleteFirst of all, this article state that writing reusable code requires 3 times more time. And as a solution, it suggests TDD. Just to make it clear, I'm not against TDD.
But what is the cost of test-driven development? Because it also requires more effort which is often not paid until the code is refactored or complex bug have to be fixed.
The second point I would like to state is the cost of writing code. This is something hard to measure at all, but "Peopleware: productive projects and teams" have some good observations on this topic. There is the point about quality of the code, or about writing of reusable code: it is done to reach the developer's satisfaction from his work.
Writing mediocre code (and mediocre is defined by the developers themselves) leads to reduced productivity (because of missing work satisfaction), which increases the cost of the code (and this is one of the reasons measuring code cost is something so difficult). Both "reusable code" and TDD are proved to cut-off cost in the long run (on support, bug fixing, requirement changes, etc.). And there is no single reason TDD paradigm to be put against "writing reusable code" paradigm, because they share common characteristics, as pointed by the article.
Generally I accept the idea to improve the code as much as you need for the current task, which is the way to develop software using some of the agile practices like continuous refactoring and evolving code design. But the message in the article is not that clear, and it is focused on the cost. But in my opinion it's not the cost you'll save from the code writing, if you go with this suggestion; it's the cost you'll save from changing the 'reusable code' because the product requirements have changed at 180 degree. In today's world, it's just happens everyday, and the result is not that you are not going to reuse your well-designed code - there is a chance you won't use it at all.
Oh please, you're confusing adoption with the cost of reuse. As well you're ignoring the current research which shows that library code is high quality and very expensive to write.
ReplyDeleteThis is pretty typical of agile folks, just give a touchy feely anecdote and assume it is way things are.
Anonymous, shame you didn't leave an address, we could continue this conversation.
ReplyDeleteFirstly you state "Current research" - do you have some references? I'd like to read up on this, happy to be proved wrong :)
Secondly, you write: "library code is high quality and very expensive to write" which I think actually proves my point. If the (reusable, I assume) high-quality library is expensive to write then why pay the cost?
I'm not saying never write for reuse, but I am saying very strongly: it is wrong to start off with that assumption.
In my experience - and yes it is experience, I'm proud of that - an awful lot of code is written for "reuse" and never reused. How many string classes have you seen? Date and time? etc. etc.
Maxim, thanks for your comments, sorry this reply is a little out of order.
ReplyDeleteYou ask: "what is the cost of test-driven development? Because it also requires more effort which is often not paid until the code is refactored or complex bug have to be fixed"
At first sight TDD may appear to be more work but this is not really the case. Would you advocate releasing untested code? I'm pretty sure you wouldn't, very few people would, but in some cases this happens. Not necessarily by design but in practice.
So the first thing is to compare like with like: I do not believe TDD is more expensive than traditional, i.e. manual, unit testing - assuming you actually do do manual testing. I accept that it is more expensive than releasing untested code but only in the short term. Anything other than the next day the bugs come back and bite you.
I suggest you have a look at my blog entry form a few weeks ago Study on benefits of TDD http://allankelly.blogspot.com/2010/08/study-on-benefits-of-tdd.html.
Your second point: I think we are in agreement here. I'm not advocating TDD as a replacement for reuse. I'm only observing that they both lead to similar beneficial characteristics, which are characteristics which make engineers feel good about their work.
BTW Seeing the "green bar" of the nUnit test framework always make me feel good.
On your final point I also agree. It is not just about cost, neither reuse or TDD rely on cost as the only argument but it is an argument in both cases. What I am arguing in the post is that, in the case of reuse the argument does not add up the way people think.
Doug McIlroy is quoted in Eric Raymond's The Art Of Unix Programming (http://catb.org/~esr/writings/taoup/html/ch01s06.html)
ReplyDelete(i) Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features.
Now, this would blow a hole through reuse, and would also blow a hole through TDD, and even through automated testing. (why do you need to run test to see if there have been changes in functionality when the program was modified if you do no modify the program?)
Some of the other things he said:
(ii) Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
(iii) Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them.
(iv) Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them.
ESR wrote: He later summarized it this way (quoted in A Quarter Century of Unix [Salus]):
This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
Before people get all in-my-face about this, note that this Doub McIlroy invented the Unix pipe, so isn't a lightweight, and probably knows a thing or two about programming.
Just resist the siren call to cut corners and hack something in that you know is wrong. Those are what accumulate and eventually bite you when you do try to reuse some code. Don't over-engineer systems, but try to gaze at least a few months into the future, and resist explicit tight coupling and other rush jobs that can't possibly be re-used.
ReplyDeleteManagers and engineers think about code quality very differently. I know from experience that short-sighted engineering inevitably leads to an accumulation of cruft so great that it prevents not only future re-use, but also prevents just getting things done in that codebase. Every few years the organization will just throw out the whole thing and start over, or continue on with a codebase that nobody can work in and watch the deadlines get missed.
So be short-sighted if you want, just don't complain to me when your project is failing and you can't do anything about it because your hands are tied due to all the short-sighted decisions leading to your predicament.
Nice post Allan!
ReplyDeleteThere is a common source of cost, in terms of reuse which is often neglected, which is the communication cost involved in knowing that a piece of usable code is already there. This type of cost, expands with the organization size. Le larger, it is the harder it will be to have everybody on board aware of the fact that there is already some piece of code that soves my problem.
If we take communication cost into account, it might turn out that the DRY principle, it's not an axiom, but more a balance of forces. Which definitely inclines towards avoiding duplication in the small scale and a little more of laissez faire in the large scale.
Also, the more a piece of our code becomes reused, it starts becoming more difficult to change, even if it's perfectly designed. A change must eventually be verified against a larger number of test suites, or need some more sophisticated work to avoid touching a young but heavily reused API.
In my comment, at the bottom, it should be Doug, not Doub... Do'h.
ReplyDeleteThanks for fixing...
I'm afraid that while this may be more efficient, single-use means that more knowledgeable programmers have less to offer more than just regular cubicle drones. It also makes development less enjoyable and should be a drag on morale. I worry about the "eliminating the free soda" savings type effect of some of these suggestions.
ReplyDeleteAllan,
ReplyDeleteIn your post about TDD there is even a number that indicates how "slower" is TDD against non-TDD - 15%. Of course, this can be compared to 300% in the "reuse" case, but here the measurement is different - it is cost vs defects. So let us try to measure: no code reuse (1x), code reuse (3x), and no code reuse, but TDD (?x)
In order to write in TDD manner, we have to educate our team, and to use right tools. But this is a personal investment, and also it is one-time investment, so we won't take it into account (after all, it's the same for the code reuse, but code reuse is widely accepted and we assume our developers already have the mind-set and skill-set required, it is possible to hire such developers for TDD as well).
If we do not plan reuse and we don't need test, we can write coupled code just to prove that it works. We can decouple the code as soon as we need to reuse it.
If we are going to reuse code, we need loosely coupled code. Also we have to think more about design, what responsibility have each class, and where exactly to put our method. What exact parameters it might need in the future. And in the process we are going to 3x.
Well, for TDD, we need to write the tests, but in order to do it we also need loosely coupled code. Or if we have no loosely coupling, we have to provide a mock object. As system is getting more complex, it's becoming harder to make independent testes, so you need to constantly simplified it (which is good, this lead to continuous refactoring to keep design clean).
So my point is, that "code for reuse" may be 3x time slower than "code for single use", but it's not 3x time slower than "code in TDD manner". TDD may be more efficient in the result you get in the end, it may be or may not be cheaper in terms of time needed to write it, but for sure it provides better results for the same (or lower) cost.
I don't argue that TDD is good thing, but it's not a panacea, and it's not something you can use instead of all other tools and concepts. It something you need to add to your toolset and use appropriately.
By the way, one of the common myths about agile I'm facing, is that it abandons code design/architecture. But it's just rethinks and redefines them. (The last comment is for the readers, not for the author)
@maxim Interesting!
ReplyDelete