Compare SIFT / SURF images in .yml OpenCV files?

Question

I am a java android developer, I know little about C / C ++ or Matlab functions. The simple thing I do in my code is to simply create sift / Surf image data and save the data in .yml files.

Here is the code how I create sift

vector < KeyPoint > keypoints; Mat descriptors; // Create a SIFT keypoint detector. SiftFeatureDetector detector; detector.detect(image, keypoints); LOGI("Detected %d keypoints\n", (int) keypoints.size()); // Compute feature description. detector.compute(image, keypoints, descriptors); 

Saving the resulting descriptor in a (.yml) file and then comparing these yml files with FlannBasedMatcher OpenCV

here is my code

descriptors1 and descriptors2 are two mat objects created from .yml files.

 FlannBasedMatcher matcher; vector<double> ee; vector < DMatch > good_matches; double good_matches_sum = 0.0; vector < vector<DMatch> > matches; matcher.knnMatch(descriptors1, descriptors2, matches, 2); for (int i = 0; i < matches.size(); i++) { if (matches[i][0].distance < 0.8 * matches[i][1].distance) { good_matches.push_back(matches[i][0]); good_matches_sum += matches[i][0].distance; } } LOGI("good_matches_sum %ds\n", good_matches_sum); LOGI("the distance k %fs\n", good_matches_sum); double score = (double) good_matches_sum / (double) good_matches.size(); LOGI("score %fk %ds\n", score, good_matches.size()); 

The problem is in the above code - I get different results every time.

here are my two images like

 //first time run for two images good_matches_sum 1006632960 the distance k 3054.279755 scores 254.523313 k 12 s //Second time run for same two images good_matches_sum -402653184 the distance k 2835.513489 score score scores 257.773954 k 11 s //Third time run for same two images good_matches_sum -1946157056 the distance k 2794.588959 score score scores 254.053542 k 11 s 

I assume that the image is more subtle or dissimilar, based on a positive result and a negative one. But with such a different result every time, I can’t say that the image is similar or not.

Please help me. I don't know about Opencv and c, so if anyone has an idea, please suggest a fixed code. thanks.

Answer 1

Basically, SIFT / SURF cannot determine if two images are similar or not. He can only tell you which key point in one image corresponds to that which corresponds to the point of another image.

Fortunately, the two functions also provide you with additional information, such as the "distance" from the match and the total number of matches in the image.

My idea of ​​how similar both images are:

 1. Use findHomography() to get the homography of two images 2. Check whether the homography is valid (such as 4 of the points form a rectangle). 3. Count the number of "good" matches (match.distance < SOME_DISTANCE) 4. Use some math to generate a score 5. If the previous homography is valid, increase the score 

Here is a piece of code that I once used, but it has some drawbacks (for some specific types of matches, the estimate is unfounded, but this situation is rare).

Please note: MY_****_DISTANCE depends on whether you use SIFT or SURF, and the parameters of the SIFT / SURF functions that you give.

 //===========SCORE ============ vector<double> matchDistance; double avg = 0; int avgCount = 0; int goodCount = 0 ; for( unsigned i = 0; i < matches.size(); i++ ) { double dist = matches[i].distance; //This is a "average match" if( dist < MY_AVG_DISTANCE && dist > MY_LEAST_DISTANCE ) { avg += dist; //Count the average of distance of "average matches" avgCount++; } //This is a good match, and number of good match have a great impact on the result //Good matches are also average matches, that is, {GOOD_MATCH} is a subset of {AVERAGE_MATCH} if(dist < MY_GOOD_DISTANCE && dist > MY_LEAST_DISTANCE ){ goodCount++; //Count the number of "good matches" } } if(avgCount > 6){ avg /= avgCount; //Gives the average value if(goodCount < 12){ //If there are too few good matches, make a punishment avg = avg + (12-goodCount) * 4; } }else{ avg = MY_MAX_DISTANCE; } avg = avg > MY_AVG_DISTANCE ? MY_AVG_DISTANCE : avg; avg = avg < MY_MIN_DISTANCE ? MY_MIN_DISTANCE : avg; double score_avg = (MY_AVG_DISTANCE - avg) / ( MY_AVG_DISTANCE - MY_MIN_DISTANCE ) * 100; if(homography_valid){ //If the previous homography is valid, make a reward. score_avg += 40; score_avg = score_avg > 100 ? 100 : score_avg; }else{ score_avg -= 5; //Or, make a little punishment score_avg = score_avg < 0 ? 0 : score_avg; } return score_avg 

Update

My _ **** _ DISTANCE

MY_****_DISTANCE is what I defined myself, sorry for not explaining it before. This is the value that I use, but you can change it to better match your code.

 #define MY_MIN_DISTANCE 200 #define MY_GOOD_DISTANCE 310 #define MY_AVG_DISTANCE 350 #define MY_MAX_DISTANCE 500 #define MY_MIN_HESSIAN 2000 #define MY_LEAST_DISTANCE 100 

When starting SIFT with SiftFeatureDetector detector(400, 3, 0.04, 10.0, 1.6); parameters SiftFeatureDetector detector(400, 3, 0.04, 10.0, 1.6); the result of some "wrong" match compared to some "right" match:

 Wrong Right 307.993 330.124 470.419 307.374 219.775 371.026 294.389 400.696 355.321 259.239 331.926 189.042 222.317 457.089 320.718 379.061 423.09 201.95 371.098 200.646 362.427 343.229 441.167 359.32 253.253 382.15 367.191 215.678 405.19 358.686 390.251 343.798 341.905 238.002 341.073 226.519 363.775 262.5 340.742 174.877 320.214 415.802 249.405 195.261 347.357 328.76 343.839 116.331 351.058 383.93 286.224 111.472 352.976 138.701 298.409 238.044 385.34 223.716 264.571 331.115 333.339 208.103 329.588 128.168 372.971 267.83 331.804 222.578 301.935 232.459 351.504 342.524 300.762 379.87 346.872 390.031 374.281 308.198 304.746 401.452 307.184 193.298 229.943 98.0714 286.163 133.978 363.634 171.415 361.656 111.077 357.108 134.186 289.712 123.199 371.496 339.944 318.708 192.164 360.547 425.937 331.225 336.535 297.688 309.419 351.898 162.296 408.206 311.055 309.023 457.352 281.375 337.529 362.266 407.757 229.295 388.567 317.005 161.118 386.907 108.936 363.942 215.311 374.832 343.376 311.264 184.318 364.745 188.963 466.795 308.48 381.667 318.828 381.826 119.591 377.338 105.527 377.333 199.206 279.228 369.394 295.078 387.979 267.408 196.942 386.063 307.815 372.14 184.83 294.927 417.138 348.458 97.8621 234.199 144.094 172.377 131.412 250.503 227.139 233.32 116.258 205.331 354.505 95.0368 108.434 116.46 138.246 406.135 308.2 92.817 194.838 312.103 323.163 312.946 377.798 359.393 396.191 320.272 375.025 309.383 280.826 278.456 

Obviously, the average value of the “incorrect” match is greater than the average value of the “correct” match. But if you look at a lot of photos, you will get some exceptions (for example, an incorrect match with fewer matches, but a smaller average distance), some of which I did not understand how to fix. Thus, based on the source data of a comparable distance, you should also make a rather "personal" account.

findHomography

Here is a demo fragment that you may need.

 std::vector<Point2f> obj_corners(4); obj_corners[0] = Point(0,0); obj_corners[1] = Point( img_object.cols, 0 ); obj_corners[2] = Point( img_object.cols, img_object.rows ); obj_corners[3] = Point( 0, img_object.rows ); std::vector<Point2f> scene_corners(4); Mat H = findHomography( obj, scene, RANSAC ); perspectiveTransform( obj_corners, scene_corners, H); 

And you can find detailed information in this lesson.

Update # 2

Check_Valid_Homography

It depends on how you define "valid." Look at the official example, the image on the left (image "object") is part of the image on the right (image "scene"). Then apply the homography matrix to the vertex vectors of the image “object”, finally, we get a green rectangle on the right, which marks where the “object” image is located.

As I, on the basis of the textbook, my definition of "valid": 1. Four points form a quadrangle 2. None of the sides of the side of the quadrangle should be too small 3. None of the corners in the quadrangles should be too small. 4. The quadrangle should not be up and down.

And, performing some basic math operations, here is one part of the verification process.

 bool RectChecker::check_order(const vector<Point2f> &corners, int height, int width){ if(corners[0].y + 5 >= corners[3].y || corners[1].y + 5 >= corners[2].y){ //Point 0 should be above Point 3, so do Point1 and Point2 return false; } if(corners[0].x + 5 >= corners[1].x || corners[3].x + 5 >= corners[2].x){ //Point 0 should be on the left of Point 1, so do Point3 and Point2 return false; } int cnt = 0; for(int i = 0 ; i < corners.size() ; i++){ if(corners[i].x < -30 || corners[i].x > width + 30){ cnt++; continue; } if(corners[i].y < -20 || corners[i].y > height + 20){ //This point is outside the image cnt++; } } if(cnt == 4){ //All points are outside the image return false; } return true; } 

Once again, you can create your own verification process, and as for verification, it depends a lot on your requirements. But, in my opinion, this part is the simplest, while the most important part is the scoring process.

Answer 2

The problem is in the above code - I get different results every time

This is because the FLANN match finds the closest nearest neighbors and is based on initialization using random numbers. To guarantee the same answer in every run, either start the random number generator (for example, calling srand(123456); ), or use the BruteForceMatcher , which is slower, but will always get the most accurate result.