Manual testing sinful?

One of the asides I made in “Programmers without TDD will be unemployable” which caused a bit of outrage in the testing community was my comment “Manual testing is a sin.” While I have been unfair to many testers, and somewhat simplistic, I still stand by the statement. Let me expand on why I still stand by the comment and why I am wrong.

It is all a question of context. Look at the full line the quote appeared in:

“Unit testing will be overwhelmingly automated. Manual testing is a sin. Manual unit testing doubly so.”

In the context of Unit Testing, as one who has undertaken formal and informal manual unit testing I believe unit testing can and should be fully automated. Informal unit testing is too ad hoc, prone to error, time consuming and difficult to replicate. Formal manual unit testing suffers all these problems and is hideously expensive.

In the context of unit testing I stand by “Manual testing is a sin.” I will also go further.

The vast majority of testing I see performed by professional testers - i.e. not unit testing - is manual, the vast majority of this testing could be automated. If you automate the testing the initial costs might go up - I say might because a good natural language test script is expensive to write itself - but the cost of executing the test will fall. More importantly the time it takes to run the test will fall dramatically.

The combination of low cost and fast turn execution means the tests become a very very effective feedback loop.

Failure to seize this amazing opportunity is something I consider sinful. It may be expensive but it is necessary. Even seizing 20% of this testing would be massively beneficial.

What I do not consider sinful is Exploratory Testing which almost by definition cannot be automated and therefore needs to be a manual process. Exploratory testing is about looking for the unusual, the unexpected, the thing you didn’t think of before, the thing you only think of because you now see it.

Automated exploratory testing might even be a contradiction in terms therefore manual exploratory testing is not sinful.

But, and this is a pretty big BUT: if the initial quality is low (e.g. unit testing is absent or poor) and the testing (e.g. system, integration, acceptance testing) that comes before exploratory testing is ineffective then exploratory testing will be effective at finding defects (because the earlier rounds didn’t find them) but it will be ineffective as exploratory testing.

The time and effort spent on "exploratory testing" is really being spent doing system testing. Real exploratory testing itself will not be possible when there are many defects because the exploration will continually be interrupted. 

Unfortunately the subtlety of this logic is lost on most organizations who just label everything that happens after writing actual production code “testing” and is made worse when testing is separated from the developers and, more importantly, those who understand what the system should do (the business, the BAs, the product managers, the users, etc.)

Worse still, the easy availability of cheap offshore testing capability means that rather than address the root cause of defects (low initial quality, system ineffective testing, misunderstanding of exploratory testing) means the whole thing can be hidden away cheaply.

There are probably some other examples of testing which cannot be automated and are not sinful. But I repeatedly find individuals and organizations who too ready to jump to “it cannot be automated” after very limited, or no, effort is spent on trying to automate it.

Similarly the fact that a professional tester cannot automate the testing of a piece of software system does not mean it cannot be automated. Automated testing - at least the setup - involves programming skills which many testers simply do not have, indeed it is wrong to assume they would have them.

Finally, there are a class of systems which as far as I know defy automated testing. Or rather defy automation at the moment. They defy automation because the creators of these systems have little or no interest in allowing automation. This set includes: SAP, Microsoft Sharepoint, most CRM systems such as Microsoft Dynamics, Murex (although I believe efforts are afoot here) and some other large packages.

I find it incredible that companies like SAP, Oracle and Microsoft can sell systems which cannot be effectively tested. But actually this is part of the sales-pitch. Because these systems are marketed and sold as “no programming required” to offer automated test tools would expose the lie. If you are involved with selecting one of these systems ask: how will we test it?

The inability to effectively test these systems is their Achilles heal, it is one of the reasons I don’t like these systems and it makes me wonder about whether organizations are right to use these systems.