What Software Test Automation Can Do But Humans Testers Can’t

I was reviewing the Apache JMeter book by PACKT publishing (nice book, will publish the full review soon) and came across a common misunderstanding in software testers about test automation.

On the other hand, despite of the appeals of test automation, we need to bear in mind that test automation may just be suitable for only parts of the software testing process. Automated testing IS NOT a total replacement for manual testing. Certain aspects of testing an application would rely more on the human tester than on test automation. The ultimate testers still are the human testers themselves; where applicable, test automation only complements manual testing. Test automation may not test any better than the human tester, but if implemented wisely, can certainly help the tester test faster. Since certain testing of the application can be automated, the tester can spend more quality time on more important and critical aspects of the testing. Ultimately, the tester can test better and more effectively.

Source: Apache JMeter (e-book) Page 8

The key underlying assumption is that test automation can automate only a subset of testing done by human testers. In reality there are tests which cannot be done by human testers, except through automation. I am not just talking about load testing and stress testing which cannot be conceivably done by human testers for a reasonable web application unless you have a really really large test team. I am also talking about subtle load related problems which occur only with a particular load pattern or even only on live site.

While working at Extensity (now part of GEAC) we uncovered several subtle but serious server side bugs much before QA or even our customers had any inkling about it. We did it by analysing the architecture and then coming up with test automation plans which exposed the defects.

One of them was a deadlock issue which may occur only once in few months on live site, later confirmed on client site. Such subtle defects can almost never be exposed by human testers but only through targeted test automation of system.

In multi-threaded server side programming test automation is a must for detecting subtle deadlock and other issues.

The author of the book previously commented:

Can we do without automation? Yes, of course—if time is abundant and your client (or boss) is NOT on your tail for the application’s next release.

That is not correct as I explained above. There are many software defects which can be discovered only through software test automation. The bottomline is that software test automation not only complements manual testing but also supplements manual testing. Any good software QA team cannot be reasonably sure of product quality without extensive test automation in place.