If any of you have heard me speak in a training session or conference you’ll know I am found of quoting Philip Crosby: “Quality is free!”. Crosby was talking from a background in missile production but the message was picked up by the car industry and silicon chip industry (“The Anderson Bombshell” in 1980 explained how Japanese RAM manufacturers were cheaper than Americans and better quality). I am quite fond of applying the argument to software.
I like to cite Capers Jones: “projects with low defect potentials and high defect remove efficiency also have the shortest schedules, lowest costs, and best customer satisfaction levels.” (Capers Jones, Applied Software Measurement, 2008)
He’s not alone in this, Tom Gilb says: “The reduction in defects … saves ‘rework’, which otherwise is about half of all effort in software projects.” (Tom Gilb, Competitive Engineering, 2005).
Jon Jagger pulled me up on some sloppy discussion of software quality, defects and rework in Xanpan. So I’m trying to come up with a (concise) definition of what I mean by software quality and it turns out that this more difficult than you might think.
To my disappointment Jones doesn’t give a definition.
Gilb offers “Defect: a failure to observe a formal, written, required rule. It is not a personal opinion or personal taste. It is a failure to observe a group norm, or required best practice.”
That sounds good until you take the statement to pieces:
- What if rules are informal? Well I suppose we can allow informal, tacit, rules, because they are “group norms”.
- “written” assumes there is something written which isn’t an assumption I can accept. Jones points out that documentation is the second most expensive activity after fixing defects, so I’d hate to eliminate defects at the expense of increase writing costs.
- “personal opinion or taste” seems fair enough but putting this into practice can be incredibly difficult. I know plenty of times when I would call a defect a personal taste but the person raising the issue wouldn’t
- “group norm” is particularly difficult when you are developing products which will change group norms
- And “best practice” …. who says it is best practice? who says it can’t be bettered?
I like Gilb’s definition but I don’t think is enough. Crucially even in saying “not a personal opinion” it does nothing to avoid the “one man’s bug is another man’s feature” problem.
What can we say about software quality and defects?
- Software quality is inversely proportion to the number of defects in the system: high quality implies few defects and vice versa
- Defects have undesirable consequences
- Defects incur costs, in all likelihood financial costs but there are others, time in particular. Even if defects are not fixed they will incur costs, e.g. over payments from a financial system or people ringing the helpline to report a spelling mistake
- Removing defects requires rework and rework costs time and money
This is probably the start of a longer list, what I am describing are the attributes - or qualities - I attribute to “high quality software”.
The list is also self fulfilling: everything I have said so far implies that low quality, lots of defects, will increase costs, so the quotes from Crosby, Jones and Gilb all become self fulfilling. Perhaps this isn’t a problem, perhaps the quality attributes we want from our software is that costs are kept down.
But there is another quality I would like from high quality software which is insidious. High quality software should be changeable - actually all software is changeable (its soft!) but some code is easier to change than other code.
High quality software as easy to change
Lets leave to one side a definition of easy, I agree it should be quantified but not right now.
What do I mean when I say “change” ? I think the spirit is captured by an old John Vlissidees quote I’ve long been fond of:
"A hallmark - if not the hallmark - of good object oriented design is that you can modify and extend a system by adding code rather than hacking it.... In short, change is additive, not invasive. Additive change is potentially easier, more localized, less error-prone, and ultimately more maintainable than invasive change." John Vlissides, The C++ Report, February 1998
I’m prepared to generalise this to all software, not just OO software. I might even go as far as focusing on the “rather than hacking it” - although one then needs to define “hacking”. Good software needs to allow for change rather than having change forced into it.
Actually this quote also provides the attributes we need to define “Easy to change”
- Change is localized
- Change is less error-prone - perhaps better stated as “change does not inject defect” (Somewhere in Jones writing he suggests 7% of defect fixes inject new defects so high quality software would have a bad fix injection rate less than 7%)
- Change is more maintainable, i.e. changing software does not detract from the changeability of the software
If have these attributes (qualities if you prefer) then software quality is high and as a result change is cheap(er). The relationship between quality and costs appears again.
But the way I describe the quality-cost link is the reverse of the way many people perceive it: the stereotypical Project Manager views quality as an attribute that can be reduced in order to accelerate development and reduce costs. I have to say I have difficulty in actually understanding this point of view but perhaps its because of the way I am defining quality.
Apart from that there is another danger of approaching: Over engineering.
Given all we have said so far you could make an argument for spending a lot of time designing your software to exhibit all these attributes. You could seek to build, design, software which would not require the design itself to be revisited. That after all is rework isn’t it?
Now I’ve long believed there is rework and there is rework:
- Rework to fix bugs, defects, is bad and wasteful because you shouldn’t have put the bug in there in the first place.
- Rework to change software for new requirements, even if that means reworking (refactoring) the design is good, or at least acceptable, because you couldn’t know about this up front therefore any effort to cater for this requirement might be misplaced and could actually end up complicating the design. In other words it is self defeating.
As I see it there is a question of knowledge here: you need to engineer within your knowledge, if you know, or could easily find out, some piece of information which would cause you to work differently then you should. But if there is information you don’t know, and would be time consuming/costly, or even impossible, to find out then it is acceptable to defer knowing and accept rework will be required later.
So the question starts to become one of knowledge acquisition. One way of acquiring more knowledge is through feedback, when feedback is rapid, timely and cheap to get we can rapidly expand the knowledge we are working with.
High quality software should be as free as possible from deficiencies given the current knowledge of what is required, but open to change when new knowledge becomes available which necessitates a change.
It’s tempting to write this as:
Quality(T) = Changeability(T) / Known defects(T)
Where T represents some point in time, as time progresses onwards changeability may well decrease which known defects may go up or down.
Notice I’ve said: “Known defects” not “Known changes”. For a piece of living, successful, software there will be a list of changes people would like made to the software. The existence of this list actually demonstrates another attribute of quality software: people use it and value changes. (Low quality software on the other hand may be so buggy that people avoid using it and thus don’t request changes.)
Excluding non-defect changes like this does leave open the problem of whether a defect report (a bug) is actually a defect report or a request for change. In some organisations such debates are heated but usually they are pointless. Sometimes they are really a Cap Ex v. Op Ex discussion, sometimes they are a “Who will do it?” or a “When will it be done?” discussion, sometimes they are a “Who will pay?” discussion. All these, and more, problems get in the way of this measurement.
While I would like to throw the door open and say: “Its all work to be done, one backlog” to do so would be to blow the equation and argument out of the water because, as I just said, high quality software may well have a longer list of change requests than low quality software.
So now we have to consider the argument about internal v. external quality…. but this blog entry is all ready too long.
Does any of this make sense?
Does any of this help?
Am I any closer to defining software quality?
Perhaps but I don’t think I’ve answered my own question yet!
Writing this entry has helped me. I think I’ve found a possible definition of quality, although I still need a definition of defect. I think we need to consider the attributes of both quality and defects. I think there is a temporal issue here related to knowledge (but I don’t know how to model or define that.) I’m even more confused then ever about the relationship between cost and quality because it appear circular.
Anyone got any better ideas? - or just comments?
