The fastest way to decode and display many H264 videos simultaneously C #

Question

As you can guess from the title of the question, we must simultaneously decode and display several (for example, eight) videos in H.264 format (and keep them in sync all the time, but this other question at another time). Videos usually have a value of 25 FPS with 640x480 resolution.

I am going to provide a little background before I get to the bottom of the problem.

The function should be baked into a fairly large C # 3.5 application (WinForms). Video captures will occupy rectangles in the application - managed code should be able to specify where each video will be drawn, as well as size.

We get H264 packets in C # and run them in our own H264 decoder to get YUV12 image data.

An early attempt was to convert YUV12 images to RGB24 and BitBlt's of them in HWND, transferred to native code from C #. While functional, all BitBlt'ing was supposed to happen in the user interface stream, which made him afraid when displaying more than a few videos (on a 2.6 GHz dual-core processor).

The current attempt will launch a single-threaded processor core at startup, and the download will balance the decoding / display of the video across these streams. The performance of this is legibility (I believe the task manager is much more interesting than the displayed videos). UI-wise, it leaves much to be desired.

Millisecond, we started drawing on HWND, created in a user interface thread (for example, a panel docked in a WinForms control) from a thread other than the UI, we began to get all kinds of funky behaviors due to error-free security from WinForms. This led us to create HWND in our own code and attract those for which C # provided rectangles onto which they should be drawn in screen coordinates.

G! CanOfWorms.Open ().

Problem. When a C # application receives focus, it jumps to the top of the Z-Order and hides the video window. Decision. Put the video window Always On Top.

Problem. When the user switches to another application, the video windows are still on top. Decision. Define the activation and deactivation of the C # application and accordingly show / hide the video windows.

Problem: the user says: "I want my videos to play on one monitor when I edit a Word document in another!" Solution: tell the user that he will shut up, and that Word sucks in any case.

Problem: I get fired.

etc .. and others.

I assume the problem is that we created HWNDs for a thread other than the UI, and want to "simulate" those that are embedded in a C # application.

Any thoughts / suggestions? Am I completely having lunch here?

I am more than open to use a completely different approach, if one exists (for this project, a lot of training is required - winning the lottery will have a greater probability than me, choosing the best approach at every step along the way).

Answer 1

Forget BitBlt-ing and do the following:

• for each window you want your video to play, create one DirectShow graph and attach a graph renderer to this window.
• the samplegrabber filter is placed in front of the visualizer on the chart. This will allow you to have a callback in which you can fill the buffer.
• instead of blitting, decode to the buffer provided by samplegrabber.

In addition, I assume that you will be able to put raw YUV12 in the buffer since VMRenderer can display them directly.

Use the DirectShowNet library.

EDIT:

And yes, BTW, if the videos are on the same “canvas”, you can use the same technique with the visualizer and create only one large window, and then shift the decoded video angles “manually” and put them in the framebuffer buffer.

GIVE ANOTHER EDITING:

BitBlts are ALWAYS serialized, i.e. they cannot work in parallel.

Answer 2

Millisecond, we started drawing on HWND, created in a user interface thread (for example, a panel docked in a WinForms control) from a thread other than the UI, we began to get all kinds of funky behaviors due to error-free security from WinForms. This led us to create HWNDs in our own code and attract those for which C # provided rectangles onto which they should be drawn in screen coordinates.

What is far-fetched behavior? If you mean flickering or pulling pulls, have you tried to lock () a panel or some other class to synchronize streams / drawings? Again: What is the exact problem when sending data to a decoder, receiving an image, converting it, and then drawing with the OnPaint handler. (Configure another thread that will run at 25 frames per second, call panel 1.Invalidate () then)

I assume the problem is that we created HWNDs for a thread other than the UI, and want to "simulate" those that are embedded in a C # application.

Do not do this. Try drawing the resulting data in your C # application. In general, I would not recommend mixing native code and C #. The one exception to this is the native code h264 decoder.

Use your streams to decode the video packets (as you already did), then one stream, which loops and calls Invalidate (as indicated above). Then you have an OnPaint handler for each panel in which you display the video. In this handler, you will get the latest video image and draw it (e.Graphics).

I hope this helps, but additional information about the problem will also be required ...

Answer 3

I like the DirectShow answer posted earlier, but I wanted to include an additional option that might be easier for you to implement based on this excerpt from your question:

While functional, all BitBlt'ing was supposed to happen in the user interface stream, which made him afraid when displaying more than a few videos

My idea is to start with this code and use Async CTP for Visual Studio 2010, which is currently available and includes a go-live license. From there, it should be relatively easy to modify this existing code to be more responsive: just add expectation and async keywords in several places, and the rest of the code should be pretty much unchanged.