Quality vs. Automated Software Testing

By | 9 March, 2010

Inspection is not enough

Deming is world famous, especially in the Agile/Lean world. During his life Deming taught top management how to improve design (and thus service), product quality, testing and sales. His name is synonymous to quality improvement and his philosophies are still seen as visionary.

One of  Deming’s principles is: “Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.” Sounds good, right? In stead of testing every last inch of your product, you increase it’s intrinsic quality which is more sustainable, has a direct benefit for the customer and eventually lowers cost.

Now how does that relate to software development? In software development we do quite a bit of inspection, not only by peer reviews but mostly by testing. We test documents, units of code, modules of code, systems, user interfaces, services etc. We test functionality, speed, error handling, ease of use, fit for purpose etc. etc. My interpretation of what Deming would say to us: focus less on creating more tests, focus more on product and process quality.

Let’s start out with the obvious, tests are useful. Even when you’re 100% Deming compliant™ you’ll still have tests, especially automated tests. But I’m not arguing to stop testing, the point I’m making is that tests lure you into a false sense of security and are not a guarantee for improving quality. A symptom of that is in software development QA people (often called testers) seem mainly concerned with ‘checking’ and spend very little time on ‘mistake proofing’.

Mistake proofing

John Hunter says tests are a form of prevention. They are a form of Poke Yoke, Poke Yoke means “mistake-proofing”. His reasoning is: with a test in place, you can’t introduce a bug in the software. But this is not true, the test detects the bug, but doesn’t prevent the bug from entering the code. A better example of Poke Yoke in software development is using high level language constructs, these prevent mistakes that are easily made with low level languages. The difference is that the mistake-proofing is done at the level of human actions where the mistakes occur and prevents the mistake being made. A test doesn’t prevent a mistake being made, it just signals that the product contains (at least) one bug. So even if your test servers as a Poke Yoke, it’s only detecting, not preventing.

Prevent Construction Errors

Tests and other forms of inspection have one fundamental limitation. I’ll try to clarify this with an analogy:

Suppose you’re a gardener who has to keep a big lawn. The worst enemy of the gardener is the mole and its molehills. Now suppose you’ve discovered a molehill. You don’t want those on your lawn. So you can remove it and place a heavy object (a slab of stone) where the hole was dug. You’ve made sure the mole can’t create a molehill at that particular place, but you haven’t prevented the next molehill from appearing.

After talking to your software tester friend, he helps you to find a way to strategically put the slabs in place before the mole arrives, preventing a lot of the molehills from appearing. You’ll soon end up with a garden covered in stone slabs and it keeps your lawn fairly green. However, the obvious question is, why aren’t you trying to get rid of the mole?

In software development, I’ve seen too few team getting rid of the mole. Too few discussions on the real cause of bugs. I would expect a Deming compliant team™ asking: “What preceded writing that particular bit of code that led to the bug? What would make it impossible to create this bug again? How can we increase quality across the board to prevent these kind of bugs, not just this single one?”

Prevention vs. Inspection

Donald Reinertsen says “Inspection makes economic sense if it is more cost-effective than prevention.”  Of course, this is a truism and automated tests are a relatively cheap way to do inspection. When inspection is your major instrument for assuring and improving quality, there’s a risk of test-addiction. Suppose you have lots of tests, but in production the customers finds a bug. What do you do? You create a test for it and that might not be the best way to handle the bug. If the actual cause is low software quality due to overly complex design or sloppy coding, it’s not enough to create a test.The opposite could even be true: the number of tests per quantity of software (LoC) should decrease over time because quality increases to such a level tests are no longer needed in large numbers.

Decrease the necessity of testing

In software development we need to fight the mole more often than patching the molehill. Do a root cause analysis with the team to uncover the cause of the bug and change your process, change they way you do design, introduce patterns, use pair programming, take a training course or do whatever prevents the mistake from occurring again, than will not only prevent that particular bug from re-occuring, it will prevent a whole slew of bugs because overall quality is higher. Your customer will thank you for it.

3 thoughts on “Quality vs. Automated Software Testing

  1. John Hunter

    Good discussion. The way I look at it is a bit differently. Lets look at a typical poka-yoke example. A USB connector must be put in the right way up – for the connection to work properly and the communication to occur as intended. So to mistake proof the process the connector won’t allow the USB device to be put in upside down – the hardware connection designed to not allow that type of connection.

    Using a deployment process that prevents code from being submitted that has an error follows a nearly identical process. It blocks the mistake from being made. I guess you can say that the connection doesn’t mistake proof the prevent me from trying to put it in the wrong way. But I think the general notion of mistake proofing is that the process blocks an error from being made. It seems to me a process that blocks code with a bug from being deployed with an error is the basically the same as a USB connection that will not accept the device being put in upside down.

    I completely agree you want to focus on improving the process. The mistake-proofing that process both improves it (many poka-yoke solution make it easier to use) and prevents an error in case you still try something wrong. So I would see the automated tests as a way to serve as a backstop, in case the process improvement you made to the software development process failed in some form. Then the automated testing required to deploy would prevent the introduction of that error to the production environment.

  2. Machiel Groeneveld Post author

    @John Hunter
    Thanks for your reply. Thinking about your view on Poke Yoke, I can now see a unit test as a custom fit Poke Yoke for a small piece of code.

  3. Pingback: How small batches improve product development | Manufacturology

Leave a Reply

Your email address will not be published. Required fields are marked *