Firmware defect rate

Question

What frequency of defects can I expect in a C ++ codebase written for the embedded processor (DSP), given that there were no unit tests, no code reviews, no static code analysis, and that compiling the project generates about 1500 warnings. Is 5 defects / 100 lines of code a reasonable estimate?

Answer 1

Despite my skepticism of the applicability of any assessment in this case, I found some statistics that may be relevant.

In this article, the author cites figures from a “large body of empirical research” published in “Software Evaluations, Benchmarks, and Best Practices” (Jones, 2000) . At SIE CMM Level 1 , which sounds like the level of this code, you can expect a defect rate of 0.75 per function point . I will leave it to you to determine how function points and LOCs can relate to your code - you will probably need a metrics tool to perform this analysis.

Steve McConnell at Code Complete cites a study of 11 projects developed by the same team, 5 with no code reviews, 6 with code reviews. The defect coefficient for the unvalued code was 4.5 per 100 LOC, and for the considered one - 0.82. Therefore, on this basis, your assessment seems fair in the absence of any other information. However, I have to take on the level of professionalism among this team (simply because they felt the need to conduct research), and that they would at least follow the warnings; your defect rate can be much higher.

A warning point is that some of them are benign, and some of them are bugs (i.e. lead to undesirable behavior of the software), if you ignore them on the assumption that they are all benign, you introduce errors. In addition, some of them will become service errors if other conditions change, but if you have already decided to accept the warning, you do not have protection against introducing such errors.

Answer 2

Your question: "Is 5 defects / 100 lines of code a reasonable estimate?" This question is extremely difficult to answer, and it greatly depends on the complexity of the code and the code.

You also mentioned in the commentary “show control, that there are probably a lot of errors in the code base” - this is wonderful, though directly.

In order to open imaginative eyes for control, I would suggest at least a 3-way approach:

• accept special warnings about the compiler and show how some of them can cause undefined / catastrophic behavior. Not all warnings will be as weighty. For example, if you have someone using an uninitialized pointer, this is pure gold. If you have someone stuffing an unsigned 16-bit value into an 8-bit unsigned value, and you can show that the 16-bit value will always be <= 255, this will not help make your case so strong.
• run the static analysis tool. PC-Lint (or Flexelint) is cheap and provides a good "hit on the dollar." This will almost certainly catch the compiler, and it can also work through translation units (combine everything together, even with two or more passes) and find more subtle errors. Again, use some of them as indications.
• run a tool that will give other indicators of code complexity, another source of errors. I would recommend Medium Squared M-Meters (RSMs) , which will provide you with more information and indicators (including code complexity) than you might expect. When you tell management that a score of more than 50 is “generally unverifiable,” and you have a score of 200 in one routine that should open some eyes.

Another point: I need clean compilations in my groups and pure Lint output. This can usually be achieved solely by writing good code, but occasionally compiler / ling warnings need to be set up to reassure the tool for things that are not problems (use wisely).

But the important point I want to do is the following: be very careful when entering and fixing compiler and lint warnings . This is a great goal, but you can also inadvertently break working code and / or uncover undefined behavior that accidentally triggered in broken code. Yes this is true. So walk carefully.

Finally, if you already have a reliable test suite, this will help you determine if you accidentally break something during refactoring.

Good luck

Answer 3

It also depends on who wrote the code (level of experience) and how big the code base is.

I would consider all warnings as errors.

How many errors do you get when you run the static code analysis tool?

EDIT

Run cccc and check the mccabe cyclic complexity. He has to say how complicated the code is.

http://sourceforge.net/projects/cccc/

Launch other static analysis tools.

Answer 4

Look at the quality of the code. This will quickly give you an indication of the number of problems lurking in the source. If the source is ugly and takes a long time to realize that there will be many errors in the code.

Well-structured, consistent-style code that is easy to understand will contain fewer problems. The code shows how much effort and thought went into it.

My guess is, if the source contains many warnings, there will be many errors in the code.

Answer 5

If you want to get an estimate of the number of defects, the usual way of statistical estimation is a subset of data. I would choose three medium-sized routines at random and carefully check them for errors (exclude compiler warnings, run the static analysis tool, etc.). If you find three errors in 100 common lines of code selected at random, it seems reasonable that a similar error density is in the rest of the code.

The issue of introducing new errors is an important issue, but you do not need to check the modified code on the production branch to run this test. I would like to offer a thorough set of unit tests before modifying any routines and cleaning up all the code, followed by a thorough system test before releasing new code for production.

Answer 6

If you want to demonstrate the benefits of unit tests, code reviews, static analysis tools, I suggest conducting an experimental study.

Perform some unit tests, review the code, and run the static analysis tools on a piece of code. Show management how many errors you find using these methods. I hope the results speak for themselves.

Answer 7

The following article provides some numbers based on real-world projects that have been subjected to static analysis: http://www.stsc.hill.af.mil/crosstalk/2003/11/0311German.html

Of course, the criteria by which the anomaly is calculated can significantly affect the results, which leads to a large change in the figures shown in table 1. In this table, the number of anomalies per thousand lines of code for C ranges from 500 (!) To 10 (automatically generated) .