My first thought was here is another bumber sticker statement from Michael. Usually I just read these statements and wonder how he can compress his knowledge and experience to so short statements. And does he expect every reader fully understand them?
Well, I read it and went outside to remove the snow from my car. During this activity my thoughts fell back on Michaels statement and I tried to follow the track of thoughts.
“Testing cannot confirm that something WILL work or even that it DOES work; only that it CAN work”
In my company we use different test environments for different test phases (yes, we have a lot of “Factory” testing going on). We use a pre-production environment for our user acceptance test before deploying a release to production.
And almost every release deployed to production creates failure scenarios that the previous test phases did not find. Why?
One reason is that the test environments never will have the exact same configuration as production – mainly due to costs of licenses and hardware not to mention maintenance costs of test environments. But also configuration and interfaces can be significantly different.
Test data is another reason. It can be very difficult to create every bit of test data to match production – same goes for test configurations for that matter.
Testing is also about using your time on specific prioritized areas. As Michael often points out: “What is the problem?”. Maybe this focus on specific problems can take some of our focus away from secondary and low priority areas? And this will later create new problems.
Testers are very skilled (hopefully) and they can be very creative and eager to learn and play with the software under test. Yet sometimes obvious problem slips through to production. Can a tester become “blind” to problems during testing and not see what 1000 users later discover 10 minutes after release have gone in production?
Michael statement contains a lot of truth (if I understand it correctly); When we perform test, it is done under certain conditions with certain test configurations with certain tester skills etc. and under these circumstances we, the testers, can conclude that the software CAN work.
I am not saying that every tester should have a tattoo with “Testing cannot confirm that something WILL work or even that it DOES work; only that it CAN work”
– but maybe every responsible CEO should have this Tattoo?