Play custom avi data stream using QtMultimedia

Question

I need to play a custom AVI file containing a classic video stream, an audio stream , as well as a user data stream .

A user stream contains data that is rendered using some custom widgets; these widgets only need to have each user frame written to the buffer at the right time.

Our application is based on Qt and already uses QMediaPlayer / QVideoWidget to play traditional videos, but an additional user stream complicates the work because AFAIK QMediaPlayer only plays video / audio and ignores everything else.

I would like to avoid reusing all of qt-multimedia , but I'm not sure how to do my best from the available Qt classes.

---

My ideas so far:

• Create your own media player that demultiplexes and decodes video using ffmpeg , implements synchronization, uses QAudioOutput to play audio, creates a QVideoFrame stream to play on video and write user data to some buffer for visualization.

  Problem . To avoid writing code for scaling / converting video frames, I would like to reuse QVideoWidget , but it seems to work only with the “real” QMediaPlayer .

• Demux input file and QMediaPlayer channel with AV streams. Demux type ffmpeg (possibly leaving decoding to the Qt backend), ask QIODevice to extract only the video / audio streams from the input file, and the other to receive the data stream. Play video / audio using QMediaPlayer .

    +-------+ | QFile | +---^---+ | inherits | +--------------------+ | MyAviDemuxer | | | | holds a queue of | | demuxed packets | +--------------------+ | | readDataPacket readVideoPacket | | +-------v--------+ +--------v-----------+ +-----------+ | MyCustomReader | | MyVideoAudioStream +--inherits--> QIODevice | +----------------+ +--------+-----------+ +-----------+ | setMedia | +-------v-------+ | QMediaPlayer | +---------------+ 

  Problem : synchronize data stream synchronization with QMediaPlayer , process headers and metadata correctly.

---

I am a bit prone to option 1, just because it gives me more control, but I wonder if I missed a simpler solution (even for Windows).

Answer 1

I understand that you have a customized class structure, but maybe you can use some tips from a newbie encoder. I think you should use some more basic existing data types along with your custom classes.

Solution for: time synchronization of data flow with QMediaPlayer:
Try using some timer threads (a combination of Thread and a timer ). Create one that uses any stream index from MyVideoAudioStream (using time as a variable in the index) and "Mycustomreader" (using an array of packets over time as a variable in the index) as the body. Add some logic to the body that cycles through the position (@param: time) in QMediaPlayer. From this, you can analyze the execution code of both at the same time. As time increases, the position in QMediaPlayer and the index of your stream will increase.

If you do not have a pointer or position in a user thread, I highly recommend that you create one.

Answer 2

It seems that Qt actually already to some extent supports the concept of data streams - http://doc.qt.io/qt-5/qmediastreamscontrol.html#details shows that this is among the selected types of streams for controlling qmediastreams.

Other documents, including http://doc.qt.io/qt-5/qmediaserviceproviderplugin.html , suggest that you can create a QMediaServiceProviderPlugin that implements the QMediaServiceProviderPlugin video and audio interfaces (possibly by subclassing an existing media service provider), also create your own subclass of the QMediaControl interface to create a control to process your raw data.

We hope that the implementation in this way will allow you to use the existing capabilities for splitting threads, processing headers and similar functions.

Unfortunately, the construction features of QMediaService are apparently "outside the scope of this documentation and support in the appropriate mailing lists or IRC channels." ( http://doc.qt.io/qt-5/qmediaservice.html#details ). The source ( http://code.qt.io/cgit/qt/qtmultimedia.git/tree/src/multimedia ) could use this, however, in addition to perhaps the source in http://code.qt.io /cgit/qt/qtmultimedia.git/tree/src/plugins , which includes directshow / gstreamer / coreaudio plugins.

Anyway, I would try to subclass and reimplement as little as possible