A few months ago Markus Gartner introduced me to the Testing Triangle, or Testing Pyramid. It looks like this:
If you Google you will find a few slightly different version and some go by the name of Testing Pyramid.
Now a question: where did this come from? Who should I credit with the original? Markus thinks it mike be Mike Cohn but he’s not sure.
This triangle is actually pretty similar to a diagram I’ve been drawing for a while when I do Agile training:
But it occurs to me the triangle should be pushed to the side, and when you do that you can add some axis which add more information:
At the base the, Unit Tests, there are lots and lots of tests and they typically execute in milliseconds. I typically hear people say there are between two and four times as much test code (for unit tests) as production code.
As you rise up there are fewer tests, tests take longer to execute and therefore tend to be run less often. Also as you rise up it becomes more difficult to automate the tests, you can automate but it requires more effort and more co-ordination. Therefore manual testing continues to exist.
True, manual testing can never be eliminated entirely for various reasons (e.g. exploratory testing) but you can get very high levels of automated acceptance or system tests.
Cost is proportional to time - and manual testing is an order of magnitude more expensive than automated tests over anything other than the very short run - and therefore as tests take longer to run costs go up.
Now a word on UAT or Beta testing.
As far as I am concerned User Acceptance Testing is the same as proper Beta Testing. Both mean: showing a potentially finished product to real life users and getting their response.
The difference is: UAT tends to happen in corporate environments where users work for the company and will need to use the software. Beta testing tends to happen in software vendor environments and it means showing the software, even giving it to, potential users outside the company.
Thus: UAT and Beta testing can only truly be performed by USERS (yes I am shouting). If you have professional testers performing it then it is in effect a form of System Testing.
This also means UAT/Beta cannot be automated because it is about getting real life users to user the software and get their feedback. If users delegate the task to a machine then it is some other form of testing.
And having users play with software means they are not doing their actual job so UAT is very expensive: expensive because it is manual and expensive because something else is not being done. Given this cost it is sensible to minimise UAT whenever possible.
In my experience most UAT phases (although not beta phases) are in effect an additional round of System Test and are frequently performed by Professional Testers. The fact that these professional testers are doing UAT is the give away. Professional Testers are not Users, they are Professional Testers.
(The other give away is that Professional Testers doing UAT are usually paid for by some group other than the IT department, i.e. IT test is not trusted, perhaps for good reason.)
More than once I have seen System Test / Acceptance Test cycles which are either completely absent or very poorly done. This necessities the need for a second round. Which is called UAT (possibly to hide the actual problem?)
I also see situations were Unit Test is poorly done or completely omitted.
If the low levels of this triangle are done well - and quality built in - then UAT should be reduced to a) real users, b) as near as possible a formality.
UAT is a very expensive way to find bugs, it also shows that something was missing in the development process. Either in coding or before that in understanding the users. It also, perhaps more worryingly, shows a failing in the earlier test steps.