Minimum Acceptable Code Coverage

This paper discusses how to decide what percentage of code coverage you need.

By Steve Cornett. Copyright © Bullseye Testing Technology 2006-2007. All rights reserved. Redistribution in whole or in part is prohibited without permission.

warning

Do not copy any part of this document without permission.

Summary

Code coverage of 70-80% is a reasonable goal for system test of most projects with most coverage metrics. Use a higher goal for projects specifically organized for high testability or that have high failure costs. Minimum code coverage for unit testing can be 10-20% higher than for system testing.

Introduction

Empirical studies of real projects found that increasing code coverage above 70-80% is time consuming and therefore leads to a relatively slower bug detection rate. Where to set your goal depends on the risk assessment and economics of the project. Consider the following factors.

Many projects set no particular minimum percentage required code coverage. Instead they use code coverage analysis only to save time. Measuring code coverage can quickly find those areas overlooked during test planning.

Defer choosing a code coverage goal until you have some measurements in hand. Before measurements are available, testers often overestimate their code coverage by 20-30%.

Full Coverage Generally Impractical

Although 100% code coverage may appear like a best possible effort, even 100% code coverage is estimated to only expose about half the faults in a system. Low code coverage indicates inadequate testing, but high code coverage guarantees nothing.

In a large system, achieving 100% code coverage is generally not cost effective. Some reasons are listed below.

Generally, the tester should stop increasing code coverage when the tests become contrived. When you focus more and more on making the coverage numbers better, your motivation shifts away from finding bugs.

Unit, Integration and System Testing

You can attain higher code coverage during unit testing than in integration testing or system testing. During unit testing, the tester has more facilities available, such as a debugger to manipulate data and conditional compilation to simulate error conditions.

Likewise, higher code coverage is possible during integration testing than in system testing. During integration testing, the test harness often provides more precise control and capability than the system user interface.

Therefore it makes sense to set progressively lower goals for unit testing, integration testing, and system testing. For example, 90% during unit testing, 80% during integration testing, and 70% during system testing.

Coverage Metrics

The information in this paper applies to code coverage metrics that consider control structures independently. Specifically, these are:

Although some of these metrics are less sensitive to control flow than others, they all correlate statistically at a large scale.

Formal Standards

DO-178B

The aviation standard DO-178B requires 100% code coverage for safety critical systems. This standard specifies progressively more sensitive code coverage metrics for more critical systems.

Effect of System Failure Level Example Required Code Coverage
Catastrophic A Crash 100% modified condition/decision coverage and 100% statement coverage
Hazardous B Passenger fatality 100% decision coverage and 100% statement coverage
Major C Passenger injury 100% statement coverage
Minor D Flight plan change No code coverage requirement
No effect E Entertainment system failure No code coverage requirement

These requirements consider neither the probability of a failure nor the cost of performing test cases.

ANSI/IEEE 1008-1987

The IEEE Standard for Software Unit Testing section 3.1.2 specifies 100% statement coverage as a completeness requirement. Section A9 recommends 100% branch coverage for code that is critical or has inadequate requirements specification. Although this document is now quite old, it was reaffirmed in 2002.

FDA requirements for Medical Devices

The United States Federal Drug Administration FDA document General Principles of Software Validation recommends structural testing but does not specify any particular code coverage requirement. The document says only:

The amount of structural coverage should be commensurate with the level of risk posed by the software.

References

Efficient use of code coverage in large-scale software development, Yong Woo Kim, 2003

Code coverage, what does it mean in terms of quality? Williams et al, 2001

An Empirical Study of the Branch Coverage of Different Fault Classes, Melissa Cline and Linda Werner, 1994

Coverage measurement experience during function test, Paul Piwowarski et al, 1993

How Do You Know When You Are Done Testing?, Richard Bender, 2000

Lessons learned on the road to 100% code coverage, Tony Obermeit, 2006

How to Misuse Code Coverage, Brian Marick, 1997

Comparing the Effectiveness of Software Testing Strategies, Basili and Selby, 1987

DO-178B, Software Considerations in Airborne Systems and Equipment Certification, 1992