Abstract

In this discussion paper, I examine the layman and technical definitions of the word "bug" and related terms. I later posit more precise operational definitions, and present the scenarios where bugs appear (or not).

Created: November 30, 2002
Updated: December 25, 2002

Introduction

We, humans, are notoriously imprecise. Do you always agree with what someone else calls an anomaly, a bug, a defect, a deficiency, an error, an exception, a fault, a problem, or an issue? When was the last time you matter-of-factly said:

• That's not a bug.
• That's a "feature".
• That'll be an enhancement (or change request).
• We need to reset the customer's expectations.

That a reported problem may meet with any (or all) of these responses, leads us to consider the need for more concise operational definitions.

Literature Review

The literature is rife with inconsistent usage of terminology. At one extreme, McConnell [69-72] makes no distinction between errors and defects in the examples he cites. At the polar opposite, Humphrey [311-313] elaborately states a bug is a defect but not all defects are bugs, and all defects result from errors but not all errors produce defects. Even the software measurements collected by pseudo-authoritative organizations reflect a lack of consensus: NASA and DoD refer to "defects", the Software Engineering Laboratory refers to "errors", and the Army refers to "faults" and "anomalies". [Christensen and Thayer, 478-483]

Turning to standards and more authoritative sources, let us examine the prevailing definitions: (Source: "Randy Rice's Software Testing Glossary", except where noted otherwise):

anomaly

(IEEE) anything observed in the documentation or operation of software that deviates from expectations based on previously verified software products or reference documents.

bug

(1) things the software does that it is not supposed to do, [or] something the software doesn't do that it is supposed to. [Telles and Hsieh, 5];

(2) a fault in a program which causes the program to perform in an unintended or unanticipated manner.

crash

(IEEE) the sudden and complete failure of a computer system or component.

defect

(1) (IEEE) a product anomaly;

(2) synonymous with fault, both imply a quality problem discovered after the software has been released to end-users (or to another activity in the software process); [Pressman, 203]

(3) fault; [SWEBOK, Ch.5, p.6]

(4) error, fault, or failure; [SWEBOK, Ch.11, p.9]

(5) noncomformance to requirements.

error

(1) (ISO) a discrepancy between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition;

(2) a quality problem discovered by software engineers (or others) before the software is released to the end-user (or to another activity in the software process). [Pressman, 203]

exception

(IEEE) an event that causes suspension of normal program execution; types include addressing exception, data exception, operation exception, overflow exception, protection exception, underflow exception.

failure

(IEEE) the inability of a system or component to perform its required functions within specified performance requirements.

fault

an incorrect step, process, or data definition in a computer program which causes the program to perform in an unintended or unanticipated manner.

mishap

(DOD) an unplanned event or series of events resulting in death, injury, occupational illness, or damage to or loss of data and equipment or property, or damage to the environment.

mistake

(IEEE) a human action that produces an incorrect result, and the subject of the quality improvement process.

requirement

(1) (IEEE) a condition or capability needed by a user to solve a problem or achieve an objective;

(2) a condition or capability that must be met or possessed by a system or system component to satisfy a contract, standard. specification, or other formally imposed documents;

(3) a documented representation of a condition or capability as in (1) or (2).

risk

(IEEE) a measure of the probability and severity of undesired effects; often taken as the simple product of probability and consequence.

side effect

an unintended alteration of a program's behavior caused by a change in one part of the program, without taking into account the effect the change has on another part of the program.

specification

(IEEE) a document that specifies, in a complete, precise, verifiable manner, the requirements, design, behavior, or other characteristics of a system or component, and often, the procedures for determining whether these provisions have been satisfied.

Analysis

First, we state the following indisputable facts: (see tangent discussion in Scenarios)

• There is the user/customer's intention or expectation.
• There is a specification (or documented requirement).
• There is a software implementation, provided by the developer/vendor.

The general, imprecise definition of "bug" captures this relationship:

bug

any deviation between the customer's expectation, the specification, and/or implementation.

Next, we examine the typical use of "error" vs. "defect" in these common messages (with these Google-based rankings):

• An error has occurred (~75,000 pages, vs. 68 for defect)
• Unknown error (~57,700 pages, vs. 478 for defect)
• Unrecoverable error (~26,900 pages, vs. 3 for defect)

Clearly, based on dominant usage, "error" and its ilk (i.e., mishap, crash, failure, exception, and anomaly) can be considered observed events, i.e., manifestastions that occur at some time, often with observable symptoms or effects. We'll choose to differentiate "error" from "defect" in this manner, as the latter is rarely used in the context of an event (e.g., "a defect has occurred").

Thus, we define "error" as follows:

error

any observed event (incident) or detected deviation between the specification and the implementation.

Pressman's use of temporal constraints (i.e., before vs. after) to differentiate between "error" and "defect" is obscure, but not without merit or justification. Constrained by a qualifier, the "Zero-Defects" concept (as advocated by Crosby) may seem less daunting to the organization considering a quality improvement program. Also, in some circles (or cultures), "defect" carries harsher overtones than "error".

And so, we state our definition of "defect" as follows:

defect

a deviation between the specification and the implementation, detected after release to the customer (or the next activity in the software process).

The operational definitions established herein are neither earth-shattering nor revolutionary. We simply refine and tighten the definitions, hopefully resulting in more consistent usage.

Scenarios

Considering expectation, specification, and implementation, we arrive at these five scenarios:

1. Expectation = Specification = Implementation. No bugs. Many happy campers.
2. Expectation = Specification != Implementation. Error or defect to be corrected by developer/vendor.
3. Expectation = Implementation != Specification. Customer is happy with code-to-intent. Vendor has a requirements bug, to be addressed in the "as-built" product specification.
4. Expectation != Specification = Implementation. Customer is unhappy with code-to-specifications. Customer has a requirements bug, to be addressed by a change request or reset expectations.
5. Expectation != Specification != Implementation. Likely the result of ambiguous or undocumented requirement(s). General disagreement, unhappiness, and finger pointing.

Some closing comments:

• Scenario 2 implies employing software development and QA practices for error preventation and detection, such as prebugging, inspections, and testing.
• Scenario 4 may explain the increasing interest in methodologies relating to "Requirements Engineering" (or "Requirements Management") and "Domain Analysis". Unfortunately, this interest would be customer-driven (given the aforementioned scenario which favors the vendor/developer) and in itself, may present little quantitative (risk or cost reduction) incentive for the vendor/developer to add process overhead.
• Scenario 5 is somewhat rare (i.e., breakdown in communications and understanding).

References

Standards

• IEEE Standard 610.12-1990. IEEE Standard Glossary for Software Engineering Terminology.
• IEEE Trial Draft SWEBOK Version 1.00. Software Engineering Body of Knowledge.

Books

• Christensen, Mark J., and Richard H. Thayer. The Project Manager's Guide to Software Engineering's Best Practices. Los Alamitos: IEEE, 2001.
• Crosby, Philip B. Quality Is Free. New York: Penguin, 1980.
• Humphrey, Watts S. Managing the Software Process. Reading: Addison-Wesley, 1989.
• McConnell, Steve. Rapid Development. Redmond: Microsoft, 1996.
• Pressman, Roger S. Software Engineering: A Practitioner's Approach. 5th ed. New York: McGraw-Hill, 2001.
• Telles, Matt and Yuan Hsieh. The Science of Debugging. Scottsdale: Coriolis, 2001.

Copyright

Copyright © 2002 Anthon Pang.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".