Strange result with SURF comparison

Question

I am trying to implement a traffic sign recognizer using the OpenCV and SURF method. My problem is that I get random results (sometimes very accurate, sometimes obviously wrong), and I can’t understand why. This is how I implemented the comparison:

• First I find outlines in my image.
• Then on each circuit I use SURF to find out if the road sign is inside and which road sign

Path detection works fine: with gaussain and canny edge blur, I manage to find a path like this:

Then I extract the image corresponding to this contour and compare this image with the image of the road sign template, for example:

cvExtractSURF returns 189 descriptors for the outline image. Then I use the naiveNearestNeighbor method to find out the similarities between my outline image and each template image.

Here are my results:

6/189 for the first template (which I expect to find)

92/189 for the second template (which, obviously, is very different from the outline image)

I really don't understand these results ...

Here is a list of the steps I follow:

• Rotate a grayscale outline image
• Rotate the grayscale pattern image
• Equals the histogram of the outline image (cvEqualizeHist)
• Resize the template image to fit the outline image.
• Blur Image Template (cvSmooth)
• Edge Blur (cvSmooth)
• Make cvExtractSURF on the template image
• Make cvExtractSURF on the outline image
• For each descriptor o, the outline image I make naiveNearestNeighbor
• I keep the number of "good" points

To assess the similarity between the two images, I use the ratio:

number of goog points / total number of descriptors

PS: For information, I follow this guide: http://www.emgu.com/wiki/index.php/Traffic_Sign_Detection_in_CSharp

And used the find_obj OpenCV sample to adapt it to C.

Answer 1

SURF descriptors are great for comparing heavily textured images ... I think textures in road signs are not enough for them.

When extracting the descriptors, the first "selection points" are located in your case, for example, in the corners of rectangular labels on both signs (rectangle and letter P), then local properties are collected for them. For example, what the angle of the rectangle looks like, from a close-up, the shades of gray are blurry.

Then these descriptors are compared with the angle of the rectangle from the letter P. They are not all different ... (since we do not take into account the form information). Perhaps the corners of the letter P are slightly closer to the corners of the no-entry sign. Random.

Of course, all this is just an assumption ... the only way to find out is to carefully debug it. Try displaying images with small circles where the descriptors were found (the size of the circle may depend on the scale at which the point was found). Or put both images in one IplImage and draw lines between the corresponding descriptors. Something like that:

http://www.flickr.com/photos/ 22191989@N00 / 268039276

As for how to fix it ... how about using the same inside shape matching method that you use to detect the outer edges of the road sign? (For example, you can search for P-shaped objects after finding a character.)

Answer 2

To assess the similarity between the two images, I use the ratio: number of goog points / total number of descriptors

I believe this is a bad metric in which you need to use a metric based on descriptor vectors, and you should use spatial information between points.

This is because functions like SIFT only correspond to “the same points” but not to similar points, perhaps you can customize it by changing the matching criteria. Since opencv has the closest point in the matching criteria (by descriptor) and check if there is another descriptor with about 0.6 similarities.

matching descriptors consists of two steps. The first step follows David Law’s simple but powerful matching algorithm. More precisely, to see if the descriptor A in the left image coincides with some descriptor in the desired image or not, we first calculate the Euclidean distance d (A, A ') between the descriptor A in the left image and all descriptors A' in the correct image. If the closest distance, say d (A, A1 '), is less than k times the second nearest distance, say d (A, A2'), then A and A1 'are considered consistent. Put k = 0.6

maybe you can change k, but I think it gives more false positives.