Friday, May 24, 2013

Testing triangles, pyramids and circles, and UAT

A few months ago Markus Gartner introduced me to the Testing Triangle, or Testing Pyramid. It looks like this:

If you Google you will find a few slightly different version and some go by the name of Testing Pyramid.

Now a question: where did this come from? Who should I credit with the original? Markus thinks it mike be Mike Cohn but he’s not sure.

This triangle is actually pretty similar to a diagram I’ve been drawing for a while when I do Agile training:

But it occurs to me the triangle should be pushed to the side, and when you do that you can add some axis which add more information:

At the base the, Unit Tests, there are lots and lots of tests and they typically execute in milliseconds. I typically hear people say there are between two and four times as much test code (for unit tests) as production code.

As you rise up there are fewer tests, tests take longer to execute and therefore tend to be run less often. Also as you rise up it becomes more difficult to automate the tests, you can automate but it requires more effort and more co-ordination. Therefore manual testing continues to exist.

True, manual testing can never be eliminated entirely for various reasons (e.g. exploratory testing) but you can get very high levels of automated acceptance or system tests.

Cost is proportional to time - and manual testing is an order of magnitude more expensive than automated tests over anything other than the very short run - and therefore as tests take longer to run costs go up.

Now a word on UAT or Beta testing.

As far as I am concerned User Acceptance Testing is the same as proper Beta Testing. Both mean: showing a potentially finished product to real life users and getting their response.

The difference is: UAT tends to happen in corporate environments where users work for the company and will need to use the software. Beta testing tends to happen in software vendor environments and it means showing the software, even giving it to, potential users outside the company.

Thus: UAT and Beta testing can only truly be performed by USERS (yes I am shouting). If you have professional testers performing it then it is in effect a form of System Testing.

This also means UAT/Beta cannot be automated because it is about getting real life users to user the software and get their feedback. If users delegate the task to a machine then it is some other form of testing.

And having users play with software means they are not doing their actual job so UAT is very expensive: expensive because it is manual and expensive because something else is not being done. Given this cost it is sensible to minimise UAT whenever possible.

In my experience most UAT phases (although not beta phases) are in effect an additional round of System Test and are frequently performed by Professional Testers. The fact that these professional testers are doing UAT is the give away. Professional Testers are not Users, they are Professional Testers.

(The other give away is that Professional Testers doing UAT are usually paid for by some group other than the IT department, i.e. IT test is not trusted, perhaps for good reason.)

More than once I have seen System Test / Acceptance Test cycles which are either completely absent or very poorly done. This necessities the need for a second round. Which is called UAT (possibly to hide the actual problem?)

I also see situations were Unit Test is poorly done or completely omitted.

If the low levels of this triangle are done well - and quality built in - then UAT should be reduced to a) real users, b) as near as possible a formality.

UAT is a very expensive way to find bugs, it also shows that something was missing in the development process. Either in coding or before that in understanding the users. It also, perhaps more worryingly, shows a failing in the earlier test steps.

Tuesday, May 07, 2013

Dialogue Sheets - update & new planning sheet

Last month InfoQ carried an update on the use of retrospective dialogue sheets. The use of these sheets continues to grow and I continue to receive good feedback.

If you’ve tried the sheets and haven’t sent me some feedback than please e-mail me and let me know about your experiences.

And for those of you who’ve not tried a dialogue sheet retrospective, whats stopping you? The Dialogue Sheet PDFs are free to download. For those who don’t have a large A1 Printer/Plotter there is a print on demand service for all the dialogue sheets.

I have also added a new sheet, this one is designed for Iteration Planning Meetings. I think one of the things teams new to Agile struggle with is the initial planing meetings. So I’ve created a new A1 sheet to guide teams through the planning meeting. This one has a bit more of a board game feel because there are activities to do and steps to repeat.

One thing I realised when creating this sheet is: planning meetings are not simple, and there are lots of variations in them. So I set about writing a Guide to Iteration Planning meetings. This turned out to be a little longer than I expected but this just goes to show what is involved. If you intend to use the sheet then I recommend you read the guide too.

Like the other sheets the planning sheet is free to download although I ask people to register - the guide is free without registration. Again, the printed version is available from the print on demand service too.

When I do training course I always give teams one or two retrospective dialogue sheets for them to use for their first retrospectives. I’m hoping that the planning sheet will fill a similar role at the start of the iteration.

I should say, I’m not completely sure this iteration planning sheet qualifies as a dialogue sheet in the strictest sense because it is much more guided and less about discussion. That said, I don’t think there are any hard and fast rules on what is, and what is not, a true dialogue sheet.