Principles of Software Quality Management

Poor Man's Approach

To start the conversation, let's first examine a wide spread idea of quality management: Quality is a compromise between Scope, Resources and Schedule (The Iron Triangle of Projects); and, Bugs must be prioritized based on Impact and Likelihood. Based on a metaphor it shows why the above idea is a very limited and an impractical approach to software quality management, and it allows us to immediately make two conclusions:

software development is a society and QA is the hospital in that society; people that get ill are sent to the hospital to get cured.

1. Quality is non-negotiable: letting people die is always unacceptable;
2. Prioritizing won't improve anything: it won't reduce the number of people getting ill.

Principles of Software Quality Management

Quality of software is measured through bugs. There is no known or set formula to quantify the software quality, but what is measured is number of bugs and this is the basis to imply the quality. Thus first we need to know what the definition of a bug is. According to Wikipedia: "A software bug is an error, flaw or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways." We will formulate it for this article's purpose as: any operation that software makes that has an unexpected result.

Testing is Sampling

The product is realized by software, and we define the set 'All' as all the operations that this software can perform. This set is off course a super set of the set of operations that is actually needed for the product. Therefore we define the set 'Supported' as the set of all supported operations by the product. QA is testing operations, but it is practically impossible to test all the supported operations of the product. The set of operations that is tested by QA is named 'Tested'.

A set contains of operations, and each operation could potentially have an unexpected result. Thus bugs will show up in one or more sets. The implication is that when you fix all bugs in the 'Tested' set, many more bugs (relative to the amount found in the 'Tested' set) likely will come up in production. It is more important to understand the quality as relation to other releases than an absolute number. Furthermore, fixing many bugs, and only releasing after all have been fixed, does not imply no production bugs.

Phase in Lifecycle Matters

For quality purposes, software development can be simplified into: change and usage stages. During the change stage, some of the functionality get altered, added or deleted, leading to code changes. First this "new" functionality is tested. After the functionality has been completed, a regression test is done to ensure the code changes don't break the product somewhere else. Finally the software is taken into production where end users make use of its value.

Feature Development Phase

After all code changes have been made, there is a specific set of operations affected, we shown this set as "Change". The next step is that functional testing is done. This will test only the set of operations that are both in the "Change" and "Tested" set. The amount of bugs found is correlated with the number of lines of the code change, and a specific technical quality factor. When bugs get classified, only bugs where is no workaround for, should be fixed. Other bugs can be fixed if times permits: there hasn't been any previous expectations on how it should work.

Regression Phase

When the functionality has been finished a regression cycle/testing is started. The amount of bugs found is normally correlated to the size of the change, the number of tests, and a technical quality factor. All bugs that are found, independent if there is a workaround or not - even if it is just cosmetic -, should be fixed. Since the bug was not related to the intended change - read feature (otherwise it was captured during Feature Testing) -, it should not have changed. Because testing is sampling, it is an indicator that code has been changed that shouldn't have changed, and it could have more significant impact on Supported operations that are not part of the Tested set. Also, customers have already got an expectation of the outcome of the operation, and thus this will directly impact the quality perceived by the customers.

Production Phase

Customers always use the software in their own way. That means it might not exactly match the specified product. Since only a part of the specified product is tested, bugs will be found independent of how much Feature and Regression Testing is done. Also, different customers might have different expectations of the same operation. This leads per definition to a bug, and can't be fixed; the only thing that can be done is to adjust the expectation of one or more customers, but that is not fixing their bug... Therefor the amount of bugs found in production is correlated to the number of customers and some usability quality factor. This also leads to the observation that:

if you want a system with no bugs, then you should ensure there are no users/customers...

Quality Factor

To improve your quality, there are a few steps to take. First option is to improve/tune your sample set (test cases). Second, improving the technical quality factor. And/or you could improve your usability quality factor. Usability could be increased by improving user documentation, training, and product user experience. And better requirements and architecture, code and design reviews, could help improve your technical quality factor.

Conclusion

Not all bugs should be treated equally, especially because the phase of discovery determines greatly how it should be handled. Furthermore, quality is not measured as an absolute number, but everyone - including customers - do see and experience quality. Therefore it is still important to actually manage your software quality even if there are no set metrics.

It is best to come up with your own strategy on software quality management, but a good start is this diagram below.

This understanding should give you control to reason when your quality needs to improve: there are too many bugs. For example, if there are "too many" production bugs, you might consider a better or broader 'Tested' set. When you have too many regression bugs, likely the code review process needs improvement. And for feature development bugs, design reviews should help immediately to improve your quality.