Question #4: Is there a process being followed to evaluate each defect and prioritize its resolution?

You're really asking: Is the organization tackling the most severe problems first and agreeing on the contents and timing of the next release?

You're trying to determine: If the organization is making good decisions about where to apply its assets.

Interpreting the response: Defects vary in severity. For example, a defect in the cosmetics of a screen form is less severe than a defect that stops the software cold. A defect that impacts many users is more severe than one that impacts few users. The order in which the development team resolves defects should be in line with their severity.

Trouble occurs when the development and test teams do not communicate about which defects to remedy and in which order. To ensure improvement of the software and for the test phase to move toward completion, the development and test teams must collaborate.

A related symptom to check: The number of highest-severity defects does not diminish over time; friction exists between development and test organizations.

Question #5: Does the organization collect testing metrics at regular intervals? The total number of test cases? The number that passed and failed? The number of defects - by degree of severity - in the process of being fixed?

You're really asking: Can the organization quantify the state of testing?

You're trying to determine: Can the organization measure progress?

Interpreting the response: Metrics enable informed testing decisions. If metrics are not recorded and published on a regular basis, progress will remain uncertain.

Metrics relating to test cases and defects must be captured, published and tracked. With these metrics you can determine whether defects are climbing, cresting or diminishing, and whether the most severe defects are being attacked first. You will see trends and be able to make corrections.

A related symptom to check: There are differing opinions about the state of testing, open defects and trends.

Because software testing ultimately exposes human failure, it's difficult to know whether the process is achieving its goal of creating the best software. People don't like to admit mistakes. They can go to extraordinary lengths to hide mistakes or take unilateral steps to try to remedy problems before others can discover them. "Busy-ness" is no guarantee of progress - indeed, it may indicate the worst kind of testing failure. CIOs can provide a critically important perspective on the process to get testing back on track and keep it there.

Paul Garbaczeski has held a variety of systems development, management and business positions at major enterprises over the past 30 years