How to capture frames from a video stream in modern Windows 8 applications?

Question

I am trying to extract images from mp4 video stream. After you look, it seems like this is the right way to do it using Media Foundations in C ++ and open the frame / read material from it.

There is very little documentation and samples, but after some digging, it seems that some people have succeeded in this by reading frames in the texture and copying the contents of this texture to a readable texture (I’m not even sure that I am using the right conditions). Trying to find what I found gives me errors, and I'm probably wrong.

Here is a short piece of code where I am trying to do this (the project itself is attached at the bottom).

ComPtr<ID3D11Texture2D> spTextureDst; MEDIA::ThrowIfFailed( m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst)) ); auto rcNormalized = MFVideoNormalizedRect(); rcNormalized.left = 0; rcNormalized.right = 1; rcNormalized.top = 0; rcNormalized.bottom = 1; MEDIA::ThrowIfFailed( m_spMediaEngine->TransferVideoFrame(m_spRenderTexture.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor) ); //copy the render target texture to the readable texture. m_spDX11DeviceContext->CopySubresourceRegion(m_spCopyTexture.Get(),0,0,0,0,m_spRenderTexture.Get(),0,NULL); m_spDX11DeviceContext->Flush(); //Map the readable texture; D3D11_MAPPED_SUBRESOURCE mapped = {0}; m_spDX11DeviceContext->Map(m_spCopyTexture.Get(),0,D3D11_MAP_READ,0,&mapped); void* buffer = ::CoTaskMemAlloc(600 * 400 * 3); memcpy(buffer, mapped.pData,600 * 400 * 3); //unmap so we can copy during next update. m_spDX11DeviceContext->Unmap(m_spCopyTexture.Get(),0); // and the present it to the screen MEDIA::ThrowIfFailed( m_spDX11SwapChain->Present(1, 0) ); } 

The error I get is:

First opportunity exception in 0x76814B32 in App1.exe: Microsoft C ++ exception: platform :: InvalidArgumentException ^ in memory location 0x07AFF60C. HRESULT: 0x80070057

I’m not quite sure how to pursue him further, since, as I said, there are very few documents about this.

Here's the sample I'm working on. This question is specific to WinRT (Windows 8 applications).

Answer 1

UPDATE SUCCESS! see edit below

---

Some partial success, but perhaps enough to answer your question. Read please.

On my system, debugging an exception showed that when I tried to call TransferVideoFrame() the OnTimer() function OnTimer() . The error he gave was an InvalidArgumentException .

So, a bit of Googling led to my first discovery - obviously, a bug in the NVIDIA drivers - which means that video playback seems to have 11 and 10 levels of features.

So, my first change was in the CreateDX11Device() function as follows:

 static const D3D_FEATURE_LEVEL levels[] = { /* D3D_FEATURE_LEVEL_11_1, D3D_FEATURE_LEVEL_11_0, D3D_FEATURE_LEVEL_10_1, D3D_FEATURE_LEVEL_10_0, */ D3D_FEATURE_LEVEL_9_3, D3D_FEATURE_LEVEL_9_2, D3D_FEATURE_LEVEL_9_1 }; 

Now TransferVideoFrame() still fails, but instead indicates E_FAIL (like HRESULT).

More googling led to my second discovery -

What an example was shown using TransferVideoFrame() without using CreateTexture2D() to pre-create the texture. I see that you already have code in OnTimer() similar to this but not used, so I think you found the same link.

In any case, I used this code to get the video frame:

 ComPtr <ID3D11Texture2D> spTextureDst; m_spDX11SwapChain->GetBuffer (0, IID_PPV_ARGS (&spTextureDst)); m_spMediaEngine->TransferVideoFrame (spTextureDst.Get (), nullptr, &m_rcTarget, &m_bkgColor); 

After that, I see that TransferVideoFrame() succeeds (good!), But the Map() call on your copied texture - m_spCopyTexture - fails because this texture was not created with read access to the CPU.

So, I just used your read / write m_spRenderTexture as the copy target instead, because it has the correct flags, and because of the previous change, I no longer used it.

  //copy the render target texture to the readable texture. m_spDX11DeviceContext->CopySubresourceRegion(m_spRenderTexture.Get(),0,0,0,0,spTextureDst.Get(),0,NULL); m_spDX11DeviceContext->Flush(); //Map the readable texture; D3D11_MAPPED_SUBRESOURCE mapped = {0}; HRESULT hr = m_spDX11DeviceContext->Map(m_spRenderTexture.Get(),0,D3D11_MAP_READ,0,&mapped); void* buffer = ::CoTaskMemAlloc(176 * 144 * 3); memcpy(buffer, mapped.pData,176 * 144 * 3); //unmap so we can copy during next update. m_spDX11DeviceContext->Unmap(m_spRenderTexture.Get(),0); 

Now, on my system, the OnTimer() function does not work. Video fragments are displayed in the texture, and pixel data is successfully copied to the memory buffer.

Before looking to see if there are any additional issues, this may be the right time to see if you can make the same progress as I still have. If you comment on this answer with additional information, I will edit the answer to add additional help, if possible.

EDIT

Changes to the texture description in FramePlayer::CreateBackBuffers()

  //make first texture cpu readable D3D11_TEXTURE2D_DESC texDesc = {0}; texDesc.Width = 176; texDesc.Height = 144; texDesc.MipLevels = 1; texDesc.ArraySize = 1; texDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM; texDesc.SampleDesc.Count = 1; texDesc.SampleDesc.Quality = 0; texDesc.Usage = D3D11_USAGE_STAGING; texDesc.BindFlags = 0; texDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE; texDesc.MiscFlags = 0; MEDIA::ThrowIfFailed(m_spDX11Device->CreateTexture2D(&texDesc,NULL,&m_spRenderTexture)); 

Note also that there is a memory leak that needs to be cleaned up someday (I'm sure you know) - the memory allocated on the next line is never freed:

 void* buffer = ::CoTaskMemAlloc(176 * 144 * 3); // sizes changed for my test 

SUCCESS

I managed to save a separate frame, but now without using copy texture.

Firstly, I downloaded the latest DirectXTex library , which provides DX11 texture helper functions, for example, to extract an image from a texture and save it to a file. The instructions for adding the DirectXTex library to your solution as an existing project must be carefully followed, taking into account the changes necessary for the Windows 8 Store applications.

After the above library is enabled, FramePlayer.cpp and built, add the following #include to FramePlayer.cpp

 #include "..\DirectXTex\DirectXTex.h" // nb - use the relative path you copied to #include <wincodec.h> 

Finally, the central section of code in FramePlayer::OnTimer() should look like the following. You will see that I just save one file name every time, so this will require changes, for example. frame number for name

 // new frame available at the media engine so get it ComPtr<ID3D11Texture2D> spTextureDst; MEDIA::ThrowIfFailed(m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst))); auto rcNormalized = MFVideoNormalizedRect(); rcNormalized.left = 0; rcNormalized.right = 1; rcNormalized.top = 0; rcNormalized.bottom = 1; MEDIA::ThrowIfFailed(m_spMediaEngine->TransferVideoFrame(spTextureDst.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor)); // capture an image from the DX11 texture DirectX::ScratchImage pImage; HRESULT hr = DirectX::CaptureTexture(m_spDX11Device.Get(), m_spDX11DeviceContext.Get(), spTextureDst.Get(), pImage); if (SUCCEEDED(hr)) { // get the image object from the wrapper const DirectX::Image *pRealImage = pImage.GetImage(0, 0, 0); // set some place to save the image frame StorageFolder ^dataFolder = ApplicationData::Current->LocalFolder; Platform::String ^szPath = dataFolder->Path + "\\frame.png"; // save the image to file hr = DirectX::SaveToWICFile(*pRealImage, DirectX::WIC_FLAGS_NONE, GUID_ContainerFormatPng, szPath->Data()); } // and the present it to the screen MEDIA::ThrowIfFailed(m_spDX11SwapChain->Present(1, 0)); 

I don’t have time right now to take this further, but I am very pleased with what I have achieved so far :-))

Can you take a fresh look and update your results in the comments?

Answer 2

See the sample video thumbnails and source reading documentation.

You can find sample code under the SDK Root\Samples\multimedia\mediafoundation\VideoThumbnail

Answer 3

I think OpenCV can help you. OpenCV offers api for capturing frames from cameras or video files. You can download it here http://opencv.org/downloads.html . Below is a demo that I wrote using "OpenCV 2.3.1".

 #include "opencv.hpp" using namespace cv; int main() { VideoCapture cap("demo.avi"); // open a video to capture if (!cap.isOpened()) // check if succeeded return -1; Mat frame; namedWindow("Demo", CV_WINDOW_NORMAL); // Loop to capture frame and show on the window while (1) { cap >> frame; if (frame.empty()) break; imshow("Demo", frame); if (waitKey(33) >= 0) // pause 33ms every frame break; } return 0; }