Image comparison for vector images (based on edge detection)?

Question

I have been dealing with messages and examples over the past two days, and all the fragments that I tested and thoroughly tested turned out to be completely useless, at least for my purposes.

What I want to do is compare a black vector symbol photographed on a wall or a sheet of paper (a quality similar to poorly scanned images that can be said) and compare this with an electronic version of the same or similar symbol (which would be stored locally and compared with photograph). Please take a look at the two attached images, the first blank (reference image) is the version of the symbol database, and the second is the shitty picture that I made on a piece of paper, which I then photographed using my iPad.

I wanted the procedure to be executed as follows:

• Two images are downloaded and then cropped using a modified version of this cropping algorithm, which I found here: Cropping images using PIL . I found that the “threshold” value of 50 and the “evidence” value of 20 (parameters in a related script) give good results for these images
• Then the images will be resized by one size and compared

Now for comparison, I tried a ton of different approaches proposed, but so far the results are terrible. I can get better comparison results with a random image than with a verified one. I tried comparing the RMS difference based on real images, their edges (created using the filter function with ImageFilter.CONTOUR or ImageFilter. FIND_EDGES), Pixel-Based Comparison , but so far nothing I have found on the Internet (despite to my ongoing Google search), or here at StackOverflow, gave me decent results.

I believe that the problem lies in the noisy background of the test image, but I could not prove it. Does anyone know if there is a way to get a vector outline from the edges on these images and compare them not only as images, but also as vector vectors? Despite my crappy drawing, I find the two images to be pretty similar, and it should be possible to get a good comparison from it.

Answer 1

To get the best answer, you need to better limit the scope of your application. Here is something that can help you. I believe that your “shitty picture” input is always similar to the one you provided in the sense that it has strong edges and the color present on it doesn't matter. To solve (or, better, come closer to a solution) your problem in a simple way, you need to describe both images in terms of descriptors of scale invariants.

I take it upon myself: binarize both images, count the number of connected components (CC) inside both, drop the CCs of irrelevant size (too far from the median, average, stddev related, etc., you decide). You might want to supplement the second step to better distinguish your image from other inputs, that is, the more powerful you want your approach to be, the more discriminant descriptors you will need. At some point, you may also consider using SVM or other machine learning methods.

So, the binarization step: perform a morphological gradient and reset the weak gradient. It is very simple if the entries are similar to what was published. Here's what I get with a threshold with an intensity of 60 (I also assume that your input is in the range [0, 255]):

I quickly experimented with thresholds of up to 90, and they all worked for these images. Crop them easily, and you can also fill the background and object:

Now you can extract the connected components in white and analyze them. In this case, the easiest way to count them. For these inputs, we get 12 in the “perfect” image and 14 in the “bad” one. But, in the "bad", we have 2 components of size 1 (in each of them there is only one pixel), which are trivially eliminated. There are many other ways to compare connected components, but I hope this can get you started. If you need code for these tasks, I can enable it.

Answer 2

I'm not sure how to do this with PIL in particular, but I can point you to some good working examples to help you learn on your own (this is not a trivial task for image processing!).

A good working example is DeTeXify , a program that matches a character drawn with a mouse, with a large library of known characters (in this case, characters that can be created in a LaTeX typing program). Source code available for front and back.

Another example is a ShapeCatcher , which:

... uses the so-called "form contexts" to find similarities between the two forms. Contexts of form, a reliable mathematical way of describing the concept of similarity between figures, is a function descriptor first proposed by Serge Belonge and Jitendra Malik.

A freely available research document for form contexts can be found on the Berkeley website.

Answer 3

I think that if you visualize lines as edges on the graph, and intersections as nodes, even if one of them turns out to be awkward, the computer should be able to see that they are one and the same sign. Just play with the levels to get white and black, and then try to analyze adjacent black spots.

Answer 4

There is a method called SIFT . OpenCV uses it, and there is an implementation in Python OpenCV. An implementation of SURF in OpenCV, various questions and examples can be found, and it works well. This is an example.

(sending a response as additional help)

Answer 5

I know that they already answered it, but maybe someone will find it anyway.

Unlike the accepted answer, I would not consider the gradient to perform binarization, but instead would look at Otsu Thresholding . This should work well if all of your images contain only very dark and very bright areas, as it searches for two peaks in the histogram of the images (one for all pixels of light, one for all dark pixels), and then threshold values ​​for the value in between two peaks.