How do I apply camera view conversion calculated using EPnP to a VTK camera?

Question

For my augmented reality project, I have a 3D model viewed using a VTK camera and a real model object viewed using a real camera.

I used EPnP to evaluate the external matrix of a real camera (this camera was already calibrated before starting work, so I know the internal parameters), providing 3D points from VTK and corresponding 2D points from a real camera image and internal parameters of a real camera for the algorithm to work EPnP.

After that, I got a rotation and translation matrix with elements → R1, R2, R3, ....., R9 and t1, t2 and t3.

So my external matrix of a real camera looks like this (let me call it extrinsicReal)

R1 R2 R3 T1 R4 R5 R6 T2 R7 R8 R9 T3 0 0 0 1 

After that, I evaluate the external matrix of my VTK camera using the following code:

 vtkSmartPointer<vtkMatrix4x4> extrinsicVTK = vtkSmartPointer<vtkMatrix4x4>::New(); extrinsicVTK->DeepCopy(renderer->GetActiveCamera()->GetViewTransformMatrix()); 

To connect the 3D model of the VTK camera to a real camera, the VTK camera must be set to the same position as the real camera position, and the focal length of the VTK camera must be the same as the real camera. Another important step is to apply the same external matrix of the real camera to the VTK camera. How to do it?

What I did was that I took the inverse of extrinsicReal and multiplied it by extrinsicVTK to get a new 4 * 4 matrix (let it name newMatrix). I applied this matrix to convert a VTK camera.

 vtkSmartPointer<vtkMatrix4x4> newMatrix = vtkSmartPointer<vtkMatrix4x4>::New(); vtkMatrix4x4::Multiply4x4(extrinsicRealInvert,extrinsicVTK,newMatrix); vtkSmartPointer<vtkTransform> transform = vtkSmartPointer<vtkTransform>::New(); transform->SetMatrix(NewM); transform->Update(); renderer->GetActiveCamera()->ApplyTransform(transform); 

I am not sure if this is the correct method. But I checked the real camera position (which I got after EPnP) and the VTK camera position (after applying the conversion above), and they are both exactly the same. In addition, the orientation of the real camera and the projection direction of the VTK camera are also the same.

The problem is that even after the above parameters correspond to both the VTK and the real camera, the 3D VTK model does not seem to fit perfectly with the real video camera. Can someone help me step by step to debug the problem?

Answer 1

Yes, it’s more difficult when applying these parameters to a vtk camera. Here's how I did it (just excerpts from important snippets of code, all the code would be too large to be inserted here and would be useless to you anyway). Other points to consider:

• I draw an endoscope image as a background texture in my vtkRenderWindow.
• I use a combination of VTK, ITK (vnl), OpenCV functions, but they must be interchangeable (for example, cvRound can also be replaced with vtkMath :: Round (), etc.).

First of all, I use the active camera from my vtkRenderer:

 d->m_Renderer->GetActiveCamera() 

The next step is to continuously update the active camera by applying your conversion. Depending on whether your rendering window is changed or not, you must initialize or also continuously update two additional parameters: 1. ViewAngle, 2. WindowCenter (It is extremely important that vtk is not documented. In the end you should apply your main point here, which you’ve discovered by calibration, or you’ll have surfaces treated with an offset. It took me 3 months to find this two-line solution).

Calculation of the viewing angle:

  double focalLengthY = _CameraIntrinsics->GetFocalLengthY(); if( _WindowSize.height != _ImageSize.height ) { double factor = static_cast<double>(_WindowSize.height)/static_cast<double>(_ImageSize.height); focalLengthY = _CameraIntrinsics->GetFocalLengthY() * factor; } _ViewAngle = 2 * atan( ( _WindowSize.height / 2 ) / focalLengthY ) * 180 / vnl_math::pi; 

Apply viewing angle:

 d->m_Renderer->GetActiveCamera()->SetViewAngle(viewAngle); 

Calculation of WindowCenter:

  double px = 0; double width = 0; double py = 0; double height = 0; if( _ImageSize.width != _WindowSize.width || _ImageSize.height != _WindowSize.height ) { double factor = static_cast<double>(_WindowSize.height)/static_cast<double>(_ImageSize.height); px = factor * _CameraIntrinsics->GetPrincipalPointX(); width = _WindowSize.width; int expectedWindowSize = cvRound(factor * static_cast<double>(_ImageSize.width)); if( expectedWindowSize != _WindowSize.width ) { int diffX = (_WindowSize.width - expectedWindowSize) / 2; px = px + diffX; } py = factor * _CameraIntrinsics->GetPrincipalPointY(); height = _WindowSize.height; } else { px = _CameraIntrinsics->GetPrincipalPointX(); width = _ImageSize.width; py = _CameraIntrinsics->GetPrincipalPointY(); height = _ImageSize.height; } double cx = width - px; double cy = py; _WindowCenter.x = cx / ( ( width-1)/2 ) - 1 ; _WindowCenter.y = cy / ( ( height-1)/2 ) - 1; 

Setting the center of windows:

  d->m_Renderer->GetActiveCamera()->SetWindowCenter(_WindowCenter.x, _WindowCenter.y); 

Apply external matrix to camera :

 // create a scaling matrix (THE CLASS TRANSFORM IS A WRAPPER FOR A 4x4 Matrix, methods should be self-documenting) d->m_ScaledTransform = Transform::New(); d->m_ScaleMat.set_identity(); d->m_ScaleMat(1,1) = -d->m_ScaleMat(1,1); d->m_ScaleMat(2,2) = -d->m_ScaleMat(2,2); // scale the matrix appropriately (m_VnlMat is a VNL 4x4 Matrix) d->m_VnlMat = d->m_CameraExtrinsicMatrix->GetMatrix(); d->m_VnlMat = d->m_ScaleMat * d->m_VnlMat; d->m_ScaledTransform->SetMatrix( d->m_VnlMat ); d->m_VnlRotation = d->m_ScaledTransform->GetVnlRotationMatrix(); d->m_VnlRotation.normalize_rows(); d->m_VnlInverseRotation = vnl_matrix_inverse<mitk::ScalarType>( d->m_VnlRotation ); // rotate translation vector by inverse rotation P = P' d->m_VnlTranslation = d->m_ScaledTransform->GetVnlTranslation(); d->m_VnlTranslation = d->m_VnlInverseRotation * d->m_VnlTranslation; d->m_VnlTranslation *= -1; // save -P' // from here proceed as normal // focalPoint = P-viewPlaneNormal, viewPlaneNormal is rotation[2] d->m_ViewPlaneNormal[0] = d->m_VnlRotation(2,0); d->m_ViewPlaneNormal[1] = d->m_VnlRotation(2,1); d->m_ViewPlaneNormal[2] = d->m_VnlRotation(2,2); d->m_vtkCamera->SetPosition(d->m_VnlTranslation[0], d->m_VnlTranslation[1], d->m_VnlTranslation[2]); d->m_vtkCamera->SetFocalPoint( d->m_VnlTranslation[0] - d->m_ViewPlaneNormal[0], d->m_VnlTranslation[1] - d->m_ViewPlaneNormal[1], d->m_VnlTranslation[2] - d->m_ViewPlaneNormal[2] ); d->m_vtkCamera->SetViewUp( d->m_VnlRotation(1,0), d->m_VnlRotation(1,1), d->m_VnlRotation(1,2) ); 

Finally, make the clipping range reset :

 d->m_Renderer->ResetCameraClippingRange(); 

Hope this helps. I don’t have time to explain the details. Especially the last code (applying appearance to the camera) has some consequences that are related to the orientation of the coordinate system. But it worked for me.

Best michael