Using OpenCV to create three-dimensional points (assuming a frontal parallel configuration)

Question

I'm currently trying to create 3D points defined by a pair of stereo images in OpenCV. This has been done quite a bit as far as I can search.

I know that the external stereo settings that I'm going to suggest are in the frontal parallel configuration (indeed, this is not so bad!). I know the focal length, the base level, and I'm going to consider the highlight the center of the image (I know, I know ...).

I compute a psuedo-decent mismatch map using StereoSGBM, and a manual Q-matrix encoding after an O'Reilly Learning OpenCV book that points out:

Q = [ 1 0 0 -c_x 0 1 0 -c_y 0 0 0 f 0 0 -1/T_x (c_x - c_x')/T_x ] 

I will take that (c_x, c_y) is the main point (which I indicated in the image coordinates), f is the focal length (which I described in mm), and T_x is the translation between two cameras or the base line (which I also described in mm).

 int type = CV_STEREO_BM_BASIC; double rescx = 0.25, rescy = 0.25; Mat disparity, vdisparity, depthMap; Mat frame1 = imread( "C:\\Users\\Administrator\\Desktop\\Flow\\IMG137.jpg", CV_LOAD_IMAGE_GRAYSCALE ); Mat frame1L = frame1( Range( 0, frame1.rows ), Range( 0, frame1.cols/2 )); Mat frame1R = frame1( Range( 0, frame1.rows ), Range( frame1.cols/2, frame1.cols )); resize( frame1L, frame1L, Size(), rescx, rescy ); resize( frame1R, frame1R, Size(), rescx, rescy ); int preFilterSize = 9, preFilterCap = 32, disparityRange = 4; int minDisparity = 2, textureThreshold = 12, uniquenessRatio = 3; int windowSize = 21, smoothP1 = 0, smoothP2 = 0, dispMaxDiff = 32; int speckleRange = 0, speckleWindowSize = 0; bool dynamicP = false; StereoSGBM stereo( minDisparity*-16, disparityRange*16, windowSize, smoothP1, smoothP2, dispMaxDiff, preFilterCap, uniquenessRatio, speckleRange*16, speckleWindowSize, dynamicP ); stereo( frame1L, frame1R, disparity ); double m1[3][3] = { { 46, 0, frame1L.cols/2 }, { 0, 46, frame1L.rows/2 }, { 0, 0, 1 } }; double t1[3] = { 65, 0, 0 }; double q[4][4] = {{ 1, 0, 0, -frame1L.cols/2.0 }, { 0, 1, 0, -frame1L.rows/2.0 }, { 0, 0, 0, 46 }, { 0, 0, -1.0/65, 0 }}; Mat cm1( 3, 3, CV_64F, m1), cm2( 3, 3, CV_64F, m1), T( 3, 1, CV_64F, t1 ); Mat R1, R2, P1, P2; Mat Q( 4, 4, CV_64F, q ); //stereoRectify( cm1, Mat::zeros( 5, 1, CV_64F ), cm2, Mat::zeros( 5, 1, CV_64F ), frame1L.size(), Mat::eye( 3, 3, CV_64F ), T, R1, R2, P1, P2, Q ); normalize( disparity, vdisparity, 0, 256, NORM_MINMAX ); //convertScaleAbs( disparity, disparity, 1/16.0 ); reprojectImageTo3D( disparity, depthMap, Q, true ); imshow( "Disparity", vdisparity ); imshow( "3D", depthMap ); 

So, I submit the resulting map of inconsistencies with StereoSGBM and this Q-matrix to get the 3D points that I write to the layer file.

But the result is this: http://i.stack.imgur.com/7eH9V.png

It’s interesting to see, but not what I need :( I read on the Internet that it gets better results after dividing the nonconformity map by 16, and indeed, it looked a little better (it actually looks like there was a camera that took the picture!) .

This is my discrepancy map if you are interested: http://i.stack.imgur.com/lNPkO.png

I understand that without cavitation, this is unlikely to look like the best 3D projection, but I was expecting something ... better.

Any suggestions?

Answer 1

According to the assumption of parallels, the relationship between the difference and the 3D depth is: d = f*T/Z , where d is the mismatch, f is the focal length, T is the baseline, and Z is the 3D depth. If you consider the center of the image as the base point, the 3D coordinate system will be solved. Then for a pixel (px,py) its three-dimensional coordinate (X, Y, Z) is equal to:

X = (px-cx)*Z/f, Y = (py- cy)*Z/f, Z = f*T/d ,

where cx, cy is the pixel coordinate of the center of the image.

Your mismatch image seems pretty good, and it can generate reasonable 3D clouds.

Simple differences browser on github .