What type of smoothing to use?

Question

Not sure if this may or may not be valid here on SO, but I was hoping someone could advise using the correct algorithm.

I have the following RAW data.

In the image you see "steps". Essentially, I want to get these steps, but then get a moving average of all the data between them. In the following image you can see the moving average:

However, you will notice that with "steps" the moving average reduces the gradient, where I want to keep a high vertical gradient.

Is there any smoothing method that takes into account the large vertical “offset” but smooths other data?

Answer 1

Yes, I should have done something similar with images from a spaceship.

Simple Technique # 1: Uses a medium filter with a modest width - say about 5 samples or 7. This provides an output value that is the median of the corresponding input value, and several of its closest neighbors on either side. He will get rid of these peaks and do a good job, keeping the edges of the steps.

A media filter is provided in all the number set tools that I know, such as Matlab, Python / Numpy, IDL, etc. and libraries for compiled languages ​​like C ++, Java (although specific names don't come to mind right now ...)

Technique No. 2 may not be exactly the same: Use the Savicki-Naked smoothing filter. This works by efficiently deriving a least-square polynomial for the data on each output sample using the corresponding input sample and a neighborhood of points (which is very similar to an average filter). SG smoothing is known to be good enough to maintain peaks and sharp transitions.

The SG filter is usually provided by most signal processing packages and the number of crunches, but may not be as general as the median filter.

Technique number 3, the largest work and requiring the most experience and judgment. . Go ahead and use a smoother shifting field, Gaussian, any - but then create a conclusion that mixes between the original and the smoothed data. A mix controlled by the new data sequence that you create ranges from completely original (mix at 0% of smoothed) to completely smoothed (100%).

To control mixing, start with an edge detector to detect jumps. You can first transfer median data filtering to get rid of spikes. Then expand (expansion in the jargon of image processing) or smoothly and renormalize the output of the edge detector, and turn it over so that it gives 0.0 when jumping and near them, and 1.0 everywhere. You may need a smooth transition connecting them. It’s an art to get this right, which depends on how the data will be used - for me, these are usually images that people should view. An automatic integrated control system may work better if configured differently.

The main advantage of this method is that you can connect any smoothing filter that you like. It will have no effect when the value of the mix control is zero. The main disadvantage is that the jumps, a small neighborhood determined by the output of the controlled edge, will contain noise.

Answer 2

I recommend that you first identify the steps and then smooth each step individually.

You know how to do anti-aliasing, and edge / step detection is pretty simple (see here , for example). A typical edge detection scheme is to smooth your data and then multiply / collapse / cross match with some filter (for example, an array [-1,1], which will show you where the steps are). In a mathematical context, this can be seen as a study of the derivative of your plot to find the inflection points (for some filters).

An alternative "hacker" solution would be to make a moving average, but exclude emissions from smoothing. You can decide what the outlier is by using some threshold t. In other words, for each point p with value v, take x points surrounding it and find the subset of those points that are between v - t and v + t, and take the average value of these points as the new value of p.