Opencv - Depth map from an uncalibrated stereo system

Question

I am trying to get a depth map in a non-calibrated way. I can get the fundamental matrix through various corresponding points from the SIFT method and cv2.findFundamentalMat . Then using cv2.stereoRectifyUncalibrated I can get the rectification matrix. Finally, I can use cv2.warpPerspective to fix and calculate the mismatch, but this does not lead to a good depth map. The values ​​are very high, so I wonder if I need to use warpPerspective or if I need to calculate the rotation matrix from the homography matrix obtained with stereoRectifyUncalibrated .

So I'm not sure about the projective matrix with the case of the homography matrix obtained with stereoRectifyUncalibrated for straightening.

Code Part:

 #Obtainment of the correspondent point with SIFT sift = cv2.SIFT() ###find the keypoints and descriptors with SIFT kp1, des1 = sift.detectAndCompute(dst1,None) kp2, des2 = sift.detectAndCompute(dst2,None) ###FLANN parameters FLANN_INDEX_KDTREE = 0 index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5) search_params = dict(checks=50) flann = cv2.FlannBasedMatcher(index_params,search_params) matches = flann.knnMatch(des1,des2,k=2) good = [] pts1 = [] pts2 = [] ###ratio test as per Lowe paper for i,(m,n) in enumerate(matches): if m.distance < 0.8*n.distance: good.append(m) pts2.append(kp2[m.trainIdx].pt) pts1.append(kp1[m.queryIdx].pt) pts1 = np.array(pts1) pts2 = np.array(pts2) #Computation of the fundamental matrix F,mask= cv2.findFundamentalMat(pts1,pts2,cv2.FM_LMEDS) # Obtainment of the rectification matrix and use of the warpPerspective to transform them... pts1 = pts1[:,:][mask.ravel()==1] pts2 = pts2[:,:][mask.ravel()==1] pts1 = np.int32(pts1) pts2 = np.int32(pts2) p1fNew = pts1.reshape((pts1.shape[0] * 2, 1)) p2fNew = pts2.reshape((pts2.shape[0] * 2, 1)) retBool ,rectmat1, rectmat2 = cv2.stereoRectifyUncalibrated(p1fNew,p2fNew,F,(2048,2048)) dst11 = cv2.warpPerspective(dst1,rectmat1,(2048,2048)) dst22 = cv2.warpPerspective(dst2,rectmat2,(2048,2048)) #calculation of the disparity stereo = cv2.StereoBM(cv2.STEREO_BM_BASIC_PRESET,ndisparities=16*10, SADWindowSize=9) disp = stereo.compute(dst22.astype(uint8), dst11.astype(uint8)).astype(np.float32) plt.imshow(disp);plt.colorbar();plt.clim(0,400)#;plt.show() plt.savefig("0gauche.png") #plot depth by using disparity focal length 'C1[0,0]' from stereo calibration and 'T[0]' the distance between cameras plt.imshow(C1[0,0]*T[0]/(disp),cmap='hot');plt.clim(-0,500);plt.colorbar();plt.show() 

Here are the corrected images with the non- warpPerspective method (and warpPerspective ):

Here are the corrected snapshots with the calibrated method:

I don’t know how much the difference is so important between the two types of images ... and for a calibrated method it does not look aligned ... strange Map of non-calibrated method mismatch:

And the depth map is calculated using: C1[0,0]*T[0]/(disp) with T from stereoCalibrate but the values ​​are very high ...

-------- EDIT LATER ------------

I tried to "mount" the recovery matrix ( [Devernay97] , [Garcia01] ) using the homography matrix obtained with "stereoRectifyUncalibrated", but the result is not very good ... Am I used correctly?

 Y=np.arange(0,2048) X=np.arange(0,2048) (XX_field,YY_field)=np.meshgrid(X,Y) #I mount the X, Y and disparity in a same 3D array stock = np.concatenate((np.expand_dims(XX_field,2),np.expand_dims(YY_field,2)),axis=2) XY_disp = np.concatenate((stock,np.expand_dims(disp,2)),axis=2) XY_disp_reshape = XY_disp.reshape(XY_disp.shape[0]*XY_disp.shape[1],3) Ts = np.hstack((np.zeros((3,3)),T_0)) #i use only the translations obtained with the rectified calibration...Is it correct? # I etablish the projective matrix with the homography matrix P11 = np.dot(rectmat1,C1) P1 = np.vstack((np.hstack((P11,np.zeros((3,1)))),np.zeros((1,4)))) P1[3,3] = 1 # P1 = np.dot(C1,np.hstack((np.identity(3),np.zeros((3,1))))) P22 = np.dot(np.dot(rectmat2,C2),Ts) P2 = np.vstack((P22,np.zeros((1,4)))) P2[3,3] = 1 lambda_t = cv2.norm(P1[0,:].T)/cv2.norm(P2[0,:].T) #I define the reconstruction matrix Q = np.zeros((4,4)) Q[0,:] = P1[0,:].T Q[1,:] = P1[1,:].T Q[2,:] = lambda_t*P2[1,:].T - P1[1,:].T Q[3,:] = P1[2,:].T #I do the calculation to get my 3D coordinates test = [] for i in range(0,XY_disp_reshape.shape[0]): a = np.dot(inv(Q),np.expand_dims(np.concatenate((XY_disp_reshape[i,:],np.ones((1))),axis=0),axis=1)) test.append(a) test = np.asarray(test) XYZ = test[:,:,0].reshape(XY_disp.shape[0],XY_disp.shape[1],4)