Implement Media Foundation Transform (MFT) with DirectX Video Acceleration (DXVA)

Question

The goal is to create a custom MFT for video processing and synchronization with an external application. Details are not important. What I would like to do as a first step is to start the MFT using DXVA or DXVA-HD video processing. I could not do that.

Here's what I did: I started building the topology with the original input node (my webcam), MFT (example MFT_Grayscale) and EVR. I have included this in a small application. The topology worked, and I saw a monochrome stream from the camera. Now I want to change the example code MF_Grayscale so that it supports DXVA video processing and can use the hardware acceleration provided by the VideoProcessBlt method. Microsoft documentation provides bits and pieces of information, but I could not get the MFT to start.

What i have done so far:

• In the GetAttributes method GetAttributes I specify that this MFT is equal to MF_SA_D3D_AWARE .
• In the ProcessMessage method, I process the MFT_MESSAGE_SET_D3D_MANAGER message to get the device descriptor, IDirect3DDeviceManager9 and IDirectXVideoProcessorService .
• In the SetInputType method SetInputType I use the methods described here https://msdn.microsoft.com/en-us/library/windows/desktop/ms694235(v=vs.85).aspx to get the structure DXVA2_VideoDesc and follow this code https: // msdn .microsoft.com / en-us / library / windows / desktop / cc307964 (v = vs. 85) .aspx to create a video processing device. In addition, I create surfaces using IDirectXVideoProcessorService->CreateSurface
• In the GetOutputStreamInfo method, the GetOutputStreamInfo variable looks like this:

pStreamInfo->dwFlags = MFT_OUTPUT_STREAM_PROVIDES_SAMPLES | MFT_OUTPUT_STREAM_WHOLE_SAMPLES | MFT_OUTPUT_STREAM_SINGLE_SAMPLE_PER_BUFFER | MFT_OUTPUT_STREAM_FIXED_SAMPLE_SIZE;

Everything seems to be in order while here. Now my questions (sorry I can't be more specific):

• Do I need to adapt the GetOutputAvailableType/SetOutputType ?

• In the ProcessInput method ProcessInput I get IMFSample and retrieve IMFMediaBuffer . The buffer does not control IDirect3DSurface9 according to my function calls. Do I need to store buffer data on a Direct3D surface?

• In the ProcessOutput method, to make a starting point, I want to forward the incoming frame to the output. VideoProcessBlt should make a 1: 1 flare from input to output. The documentation states:

  Get an accessible surface that is currently not in use.

How to determine if a surface is used?

• How can I output the surface? Should I use MFCreateVideoSampleFromSurface or MFCreateDXSurfaceBuffer ?

• Unfortunately, I really lost and could not achieve any results using the documentation.

Now the situation is that I do not see any video output (the window has a default background color for the window), and the webcam stops to capture frames after the first frame (the LED turns off). In addition, nothing happens - the application just continues to work, not showing anything).

I hope someone can help me. I would also appreciate if someone could direct me to the sample code for the MFT using DXVA or DXVA-HD video processing. I could not find anything ...

thanks

Answer 1

Do I need to adapt the GetOutputAvailableType / SetOutputType methods?

Yes. You need to provide attributes, as usual, without special requirements.

Do I need to rewrite buffer data to a Direct3D surface?

Yes. In doing so, you need to take care of IDirect3DDeviceManager9 and LockDevice, because EVR can use the surface at the same time.

How to determine if a surface is used?

You must take care, especially if the surface is free to use. Your MFT should implement the IMFAsyncCallback interface. After using MFCreateVideoSampleFromSurface, you request the IMFTrackedSample interface and call SetAllocator. The Invoke method will tell you when the surface is free.

How can I bring out the surface? Should I use MFCreateVideoSampleFromSurface or MFCreateDXSurfaceBuffer?

Since you are using IDirectXVideoProcessorService-> CreateSurface, MFCreateVideoSampleFromSurface is the right place. You output the surface through IMFSample.

Check out this project:

Mfnode

In MFTDxva2Decoder and MFSkDxva2Renderer you will find some dxva2 processing.

Answer 2

The easiest way to get Media Foundation samples is to download and install the "Microsoft Windows SDK for Windows 7 and .NET Framework 4" from http://www.microsoft.com/en-us/download/details.aspx?id=8279 . Install the samples, and then find them in the "v7.1 / Samples / Multimedia / MediaFoundation" section. Samples of the Media Foundation are allegedly located on the Microsoft Code Gallery, but I could not find them there. Samples are no longer included in Windows SDK releases.

Samples that are directly relevant to your question are probably decoders, DXVA_HD, DXVA2_VideoProc, EVRPresenter, MPEG1Source and topoedit.

I can no longer help, as I am currently struggling with some of the same problems.