Volume 6, Issue 1

Reliability Edge Home

The Cost Benefits of Predicting Software Reliability

Guest Submission

Ann Marie Neufelder

Software reliability growth models have been used since the 1960s to project software failure rate and reliability given defect data observed during testing [Ref. 1]. The primary disadvantage of these growth models is that they cannot be used until system level software testing commences. By this phase it is generally too late to improve the reliability of the software via any other means than adding additional testing resources and/or delaying the release schedule. These models are, however, still valuable as they are particularly useful for proactively planning warranty costs and maintenance staffing.

Since 1987, models that predict software reliability at the beginning of the project have been available in the public domain [Ref. 2]. These models facilitate advanced planning of all resources required to reach a specific reliability objective. They are also useful for identifying key strengths and key areas of improvement that correspond to reduced failure rate and higher reliability.

The primary question to be answered by this article is "How cost effective are these reliability models?" The author has analyzed the actual defects delivered by 32 software organizations. Of these, 10 organizations were predicting software reliability early in the life cycle, 13 were measuring it during testing and 9 weren't measuring it at all. The prediction and estimation models used by the 32 organizations in this study are summarized in Tables 1 and 2.

Table 1: Software Reliability Prediction Models
Table 1: Software Reliability Prediction Models

Table 2: Software Reliability Growth Models
Table 2: Software Reliability Growth Models*

In the study, "cost benefit" is measured by these three things:

  • a reduction in escaped defects
  • a reduction in late software deliveries (missed market window)
  • a reduction in how much the market window is missed by

These performance metrics were chosen because fixing defects and missing a market/development window are the primary root causes for software development cost overruns.

Table 3 presents the results of the study. In this table, the actual normalized defect density is a normalized metric of escaped defects and is in terms of defects per normalized effective size of software. The probability of a late delivery is simply the percentage of time that the organization delivers a software product late. Magnitude of late delivery measures how late the deliveries are.

Table 3: Performance Measures vs. Software Reliability Metrics
Table 3: Performance Measures vs. Software Reliability Metrics

As you can see from Table 3, the group that does predictions early in the life cycle has significantly fewer escaped defects and less likelihood of delivering software late than either of the other groups. When this group is late delivering software, it is by a much smaller margin than the others. So, even if they are late, they may still be able to hit the market window. The organizations that use reliability growth models versus no models at all have fewer escaped defects, deliveries that are late less often and by a smaller margin of error. So, measuring late is better than not at all.

Now let's see how much it costs on average to use the reliability models -- see Table 4. Please note that a variety of tools/documents/templates exist in every price range. Generally, the more expensive tools require less effort to do the modeling. In addition, all models require that effective size be predicted in a parallel effort. Size prediction is a necessary part of software management, which is not included in the time estimate for using the models. The primary cost of using the growth models is in developing and maintaining an interface between the defect tracking system and the tool/template containing the reliability growth model.

Table 4: The Cost of Using the Software Reliability Models
Table 4: The Cost of Using the Software Reliability Models

On average, a single escaped software defect can take from a week to several weeks of effort to correct when including the time to isolate, repair, checkout, retest, reconfigure and redistribute. So, a typical organization needs only to reduce its escaped defects by 1 to compensate for the cost of using the prediction models or the simple reliability growth models.

1. Neufelder, Ann Marie, "System Software Reliability Assurance Notebook," P. Lakey Boeing Corporation, 1997. This was produced for Rome Laboratory.

2. Rome Laboratory, "Methodology for Software Reliability Estimation and Assessment," Technical Report RLTR- 92-52, Vol. 1 and 2, 1992. This was one of the first publicly available multi-parameter models for predicting software reliability without the need for testing defect data.

3. Neufelder, Ann Marie, "The Naked Truth About Software Engineering," January 2004.

End Article


About the Author
Ann Marie Neufelder has been in software engineering and software reliability for more than 20 years. She has invented several methods for predicting software defects. Ann Marie has published numerous articles on the subject which can be found at http://www.softrel.com/publicat.htm. Ann Marie is a 1983 graduate of Georgia Tech and resides in Portland, Oregon with her family. She can be contacted via e-mail at amneufelder@softrel.com.

On the Web: http://www.SoftRel.com


ReliaSoft.com Footer

Copyright © HBM Prenscia Inc. All Rights Reserved.
Privacy Statement | Terms of Use | Site Map | Contact | About Us

Like ReliaSoft on Facebook  Follow ReliaSoft on Twitter  Connect with ReliaSoft on LinkedIn  Follow ReliaSoft on Google+  Watch ReliaSoft videos on YouTube