Kinnect RGB Capture with Openni and Display with OpenCV

Question

I need to capture an RGB color image from a Kinnect camera, but I want to show it in OpenCV, as this is only part of a larger program. I know that OpenCV is compatible with OpenNI, if you set the flag, but although I tried, CMake could not find the path to OpenNI2, so I could not create OpenCV with OpenNI. In any case, I think it’s good to know how to manually convert OpenNI frameworks to openCV frameworks, so I decided to follow this path.

To capture a color frame in OpenNI, I tried the following:

openni::Device device; openni::VideoStream color; openni::VideoFrameRef colorFrame; rc = openni::OpenNI::initialize(); rc = device.open(openni::ANY_DEVICE); rc = color.create(device, openni::SENSOR_COLOR); rc = color.start(); color.readFrame(&colorFrame); const openni::RGB888Pixel* imageBuffer = (const openni::RGB888Pixel*)colorFrame.getData(); 

But now I do not understand how to do the conversion in cv :: Mat.

Has anyone anyone managed to do this?

Answer 1

So, first you have to separate the initialization from the loop to read frames this way.

Initialization

 openni::Device device; openni::VideoStream color; openni::VideoFrameRef colorFrame; rc = openni::OpenNI::initialize(); rc = device.open(openni::ANY_DEVICE); rc = color.create(device, openni::SENSOR_COLOR); rc = color.start(); Mat frame; 

Now comes the main loop for reading frames. You have done almost everything, it remains only to copy the buffer to the openCV layout.

Loop for reading frames

 while (true) { color.readFrame(&colorFrame); const openni::RGB888Pixel* imageBuffer = (const openni::RGB888Pixel*)colorFrame.getData(); frame.create(colorFrame.getHeight(), colorFrame.getWidth(), CV_8UC3); memcpy( frame.data, imageBuffer, 3*colorFrame.getHeight()*colorFrame.getWidth()*sizeof(uint8_t) ); cv::cvtColor(frame,frame,CV_BGR2RGB); //this will put colors right } 

Answer 2

Following @Jav_Rock's answer above, for IR and Depth the solution is similar, except for the following (instead of SENSOR_COLOR , CV_8UC3 and RGB888Pixel respectively):

Depth

• Sensor Type: SENSOR_DEPTH
• OpenCV Type: CV_16UC1
• VideoFrameRef Data: DepthPixel

Please note that you probably want to set the PixelMode for VideoFormat VideoStream to PIXEL_FORMAT_DEPTH_100_UM , otherwise the image of your depth will look very black.

IR

• Sensor Type: SENSOR_IR
• OpenCV Type: CV_16UC1
• VideoFrameRef Data: Grayscale16Pixel

Finally, note that neither depth nor IR needs a call to cv::cvtColor .