What is statistical debugging?

Question

What is statistical debugging? I have not yet found a clear and concise explanation, but the term certainly sounds impressive.

Is this just a research topic, or is it used somewhere for actual development? In other words: Would it help me find bugs in my program?

Answer 1

I have created statistical debugging as well as various wonderful employees over the years. I would like to notice your question a few months ago! But if you're still interested, perhaps this late answer will be better than nothing.

At a very high level, statistical debugging is the idea of ​​using statistical models of program success / failure to track errors. These statistical models reveal the relationship between specific program behaviors and possible success or failure of a run. For example, suppose you notice that the program has some branch that sometimes goes to the left, sometimes correctly. And you also notice that runs where the branch goes to the left are good, but working where the branch goes to the right is 75% more likely to fail. So, there is a statistical correlation, which may be worth a closer look. Statistical debugging formalizes and automates this process of finding software (incorrect) behaviors that correlate with failure, thereby directing developers to the root causes of errors.

Returning to the original question:

Is this just a research topic or is it used somewhere for actual development?

This is mainly a research topic, but it is in the “real” world in two ways:

• publicly deploying a collaborative error isolation project hunts for bugs in various open source programs running Fedora Linux. You can download ready-made packages , and every time you use them, you feed us data to help us find errors.

• Microsoft has released Holmes, the introduction of statistical debugging for .NET . It is well integrated in Visual Studio and should be a very simple way to use statistical debugging to help you find your own errors in your own code. I worked closely with Microsoft Research in Holmes, and these are good smart people who know how to produce high-quality tools.

One caveat to keep in mind: for statistical debugging, you need sufficient baseline data to build good statistical models. In a public CBI deployment, this raw data comes from real users. With Holmes, I think Microsoft assumes that the raw data will come from internal automatic unit tests and manual tests. What will not work is code without any runs, or only with errors, but without successful counterexamples. Statistical debugging works on the contrast between good and bad runs, so you need to feed them both. If you want to use tools to find errors without runs, you will need some kind of static analysis. I also do research, but this is not statistical debugging. :-)

Hope this helps and not too long. I am happy to answer any further questions. Happy hunt for bugs!

Answer 2

that when you submit software that says "well, it probably works ..."; -)

EDIT: This is a research topic where machine learning and statistical clustering are used to try to find patterns in programs that are good predictors of errors, to determine where more errors can be hidden.

Answer 3

This sounds like a statistical sample. When you buy a product, there is a good chance that not every product coming out of the "assembly line" has been tested for quality.

Statistical sampling requires checking a certain percentage of products to ensure that they are all without problems. This minimizes the risk of some problems getting through them, and is absolutely essential when the testing process is destructive - if you conduct destructive testing at 100% of your production line, which does not leave much to spread :-)

To be honest, if you do not check every execution path and all possible input values, you already do this in your testing. The amount of effort required to test everything on any, but the most simplified systems is not worth it. The extra cost will make your product uncompetitive.

Note that statistical sampling doesn’t just include testing every 100th block. There are ways to target the sample to improve the likelihood of trap problems. For example, if historical data indicates that most errors are entered at a particular phase, indicate this step. If one of your developers is more problematic than others, better check its material.

From what I see from a quick look at some research articles, statistical debugging is only targeting areas based on a past history of problems.

I know that we are already doing this for our software. Since any corrections that are fixed must pass single and system tests that replicate the problem (and our TDD says that these tests must be written before you try to fix the error), these tests are automatically added to the regression test suite so that they areas that cause more problems are naturally more likely to be checked in the future.