Detection of a fiduciary marker in the presence of camera shake

Question

I am trying to make my detection of a OpenCV-based fiduciary marker more reliable when the user moves the camera (phone) violently. Markers are an ArTag style with a Hamming code embedded in a black border. Borders are detected using the image threshold, then search for ATVs based on the found contours, and then check the internal parts of the quads.

In general, marker decoding is reasonably stable if a black frame is recognized. I tried the most obvious thing, which reduces the image twice, and also performs quad detection at these levels. This helps to focus the camera on extreme close-up markers, as well as with very low levels of image blur, but it doesn’t help in the general case of motion blur.

Are there available studies on ways to improve detection reliability? Ideas that I'm interested in include:

• Can you do some optical flow tracking to “guess” the position of the marker in the next frame, and then some kind of angular detection in the area of ​​these guesses, and not consider the rectangle search as a full-frame threshold?
• On computers, can I get blur coefficients (perhaps by registering in the last video frames where the marker was found) and deblock the image before processing?
• On smartphones, can I use a gyroscope and / or accelerometers to get attenuation coefficients and pre-process the image? (I guess not, simply because if that were the case, the market would be flooded with shake correction applications).

Links to bad ideas will also be appreciated if this saves me from trying.

Answer 1

• Yes, you can use the optical stream to evaluate where the marker is and to localize your search, but this is just relocalization, your tracking will break for blurry frames.
• I don’t know enough about the exception, except that it is very computationally intensive, so it can be difficult in real time
• You can use sensors to guess the kind of blur you're encountering, but I would suggest that deblurring is too computational for real-time mobile devices.

Then some other approaches:

There are some really smart things here: http://www.robots.ox.ac.uk/~gk/publications/KleinDrummond2004IVC.pdf , where they do edge detection (which you could use to search for a border marker, even if you're looking ATVs right now), simulating camera movements from sensors and using these values ​​to estimate how the edge in the blurring direction should appear taking into account the frame rate and searching for it. Very elegant.

Similarly here http://www.eecis.udel.edu/~jye/lab_research/11/BLUT_iccv_11.pdf they just pre-blur the tracking targets and try to match the blurry targets that are suitable given the blur direction, They use Gaussian filters to model Blurs that are symmetrical, so you need half as much pre-blurred targets as you might expect.

If you try to implement any of them, I would be very interested to hear how you are doing!

Answer 2

From some related work (trying to use sensors / gyroscopes to predict the likely location of functions from one frame to another in a video), I would say that 3 is likely to be difficult, if not impossible. I think that at best you can get an idea of ​​the approximate direction and angle of movement, which can help you blur the model using the approaches dabhaid refers to, but I think that you are unlikely to get enough accuracy to be much more useful. .