Get a frame from the video with libvlc smem and convert it to opencv Mat. (C ++)

Question

[UPDATED PARTIAL ANSWER]
Here is my code:

void cbVideoPrerender(void *p_video_data, uint8_t **pp_pixel_buffer, int size) { // Locking imageMutex.lock(); videoBuffer = (uint8_t *)malloc(size); *pp_pixel_buffer = videoBuffer; } void cbVideoPostrender(void *p_video_data, uint8_t *p_pixel_buffer , int width, int height, int pixel_pitch, int size, int64_t pts) { // Unlocking imageMutex.unlock(); Mat img = Mat(Size(width,height), CV_8UC3, p_pixel_buffer); //cvtColor(img,img,CV_RGB2BGR); } int main(int argc, char ** argv) { libvlc_instance_t * inst; char smem_options[1000]; sprintf(smem_options , "#transcode{vcodec=RV24}:smem{" "video-prerender-callback=%lld," "video-postrender-callback=%lld," "video-data=%lld," "no-time-sync}," , (long long int)(intptr_t)(void*)&cbVideoPrerender , (long long int)(intptr_t)(void*)&cbVideoPostrender //This would normally be useful data, 100 is just test data , (long long int)200 //Test data ); const char * const vlc_args[] = { "-I", "dummy", // Don't use any interface "--ignore-config", // Don't use VLC config "--extraintf=logger", // Log anything "--verbose=1", // Be verbose "--sout", smem_options // Stream to memory }; // We launch VLC inst = libvlc_new(sizeof(vlc_args) / sizeof(vlc_args[0]), vlc_args); ... return 0; } 

UPDATED QUESTION
I have verified that my two callback functions execute correctly.
_What data does RV32 accurately output? Does this match CV_8U3C (unsigned 8bit int 3-channel channel required here? _Do I need to add a step to my Mat class? (Step is the number of bytes that each row of the matrix occupies)
UPDATED2
I changed RV32 to RV24, which makes more sense. I add cvtColor because the Mat matrix seems to need a BGR pixel, not RGB, but the image does not display correctly.
_ Is there a vcodec that will give me the YUV format as output so that I can check the pixel data before trying to output opencv :: Mat img?
[CHANGE IMG IMAGE] (changing vlc type to CV_8UC4 on four channels (I don’t know why), we can almost see the frame, but in really low quality, why is this so?
[DECISION]
I found out that the images at the beginning of my video were of poor quality, so my Mat imshow () showed me such an ugly thing that the code above should work now (apparently there is no need for cvtColor)

Answer 1

Firstly, a quick warning: starting with VLC2.2 (the current version of git, which will be released soon), the size parameter is size_t. There is no API for smem (but hopefully this will change), which sucks, so it will quietly break your application.

Then, a quick comment about the "data" parameter: it should contain what you need to process. This is a pointer to a structure, an instance of a class, you name it. I strongly doubt that you would work on a 32-bit machine for a long time, since you would force 64 bits into something that can contain only 32. What you have to do is declare the structure and save what you need in it . Here is a good example:

 struct MyParamStruct { YourMutexType imageMutex; // Here mutex is not a global variable anymore int otherParam; // You can use this to store the value 200 that you were passing before }; //... // Init the struct somewhere MyParamStruct* param = new MyStructParam; param->otherParam = 200; //... sprintf(smem_options , "#transcode{vcodec=h264}:smem{" "video-prerender-callback=%lld," "video-postrender-callback=%lld," "video-data=%lld," "no-time-sync}," , (long long int)(intptr_t)(void*)&cbVideoPrerender , (long long int)(intptr_t)(void*)&cbVideoPostrender //This would normally be useful data, 100 is just test data , (long long int)(intptr_t)(void*)param ); 

Regarding the use of a mutex, this looks good to me. It actually seems that you have no problem with concurrency here, since you are simultaneously allocating a new buffer for each frame. If you used a pre-allocated buffer each time, you will need to consider locking when exiting the postrender function.

In fact, I'm not even sure what exactly the void p_video_data pointer is.

It depends on your image format. For H264, this will depend on the pixel format that will be output by the decoder. Since you are requesting H264 output, it is likely that you will get a planar pixel format, although the exact type will depend on your H264 profile.

If you expect rawdata to appear (it looks like CV_8UC3 seems to refer to a 3-channel raw image after a quick look at google), I would recommend you switch to RV32: #transcode{vcodec=RV32}

What you need to pass to the transcoding module is your fourcc output , VLC will handle the input for you :)

Update

I have no idea if the Mat class is in the ownership of your pointer, but you can also check this.

Update 2

To answer your further question about what RV32 is:

 /* 24 bits RGB */ #define VLC_CODEC_RGB24 VLC_FOURCC('R','V','2','4') /* 24 bits RGB padded to 32 bits */ #define VLC_CODEC_RGB32 VLC_FOURCC('R','V','3','2') /* 32 bits RGBA */ #define VLC_CODEC_RGBA VLC_FOURCC('R','G','B','A') 

If you expect only 3 bytes, then you should probably try RV24! I probably suggested that from the start, since 8CU3 definitely offers only 3 bytes ...