Can automated testing really replace all manual testing? Software development strategies such as Agile methods are highly committed with the idea of devoting a large part of an organization’s testing resources, if not 100%, to automated testing. Jonathan Kohl reflects on this “over-automation” debate in his blog entry about the risks of Reckless Test Automation.
This controversy isn’t new news - for years the software industry has been trying to understand what constitutes Agile methods such as Extreme Programming (XP). The idea that all tests should be automated continues to saturate the industry. Yes, there are some clear benefits of test automation. Tests can be run faster, they’re consistent, there is less overhead, the quality assurance cycle is presumed shorter. Kohl does point out that test automation will truly deliver, however some amount of manual ad-hoc testing is the supplement needed to fill in the holes.
I agree with Kohl’s outlook - although I don’t necessarily agree that every member of the Agile community is enthusiastically supporting the 100% automation methodology. Automated unit tests and functional tests have great value and are a cornerstone of many agile development methods but this is just a small part of the overall testing effort. In instances where you’re only going to run the test once or twice tests are better suited for manual execution.
In the end, it really comes down to the appropriate choice of tools, techniques and approaches for the test at hand. The moral of Kohl’s story is that sometimes there is just no substitute for simple human involvement with a product. Every software development shop needs to find a good balance between both automated and manual testing, if you go too far in either direction, the results will be reckless.
April 26th, 2006 at 9:56 am
Hey, cool post. Sounds like we’re on the same page: finding a good balance in testing is key.
We are also on the same page with this statement: “I don’t necessarily agree that every member of the Agile community is enthusiastically supporting the 100% automation methodology” — I never said that every member of the Agile community supports this, just some louder ones.
The “100%” thing can cause problems because experienced people look for more of a balance, but decision makers take it as an absolute. Some companies I’ve worked in got obsessed with this number and discouraged /any/ manual testing. “This is XP after all.” I’d prefer hearing about more of a balanced testing approach more often, so thanks for this.
Nice post.
May 25th, 2006 at 3:21 pm
[…] My last two blog entries, Effectively Exploring Exploratory Testing and Reckless Test Automation covered two Agile techniques and the controversies surrounding their acceptance in the industry. I am continuing my discussion of practices championed by Agile approaches by further exploring Pair Programming (PP), the extreme of Extreme Programming (XP). […]
October 31st, 2006 at 9:39 am
[…] It’s important to note that static analysis alone is not a silver bullet solution. It is, however, a great addition to the QA toolbox and an effective complement to managing the quality of software. Keep in mind, (as I’ve said before), automated and manual code analysis methods are most effective when used in conjunction with each other - there is no replacement for a little human intellect. […]