Defragment the NAL H264 stream (originally 1722 avb packets)

Question

The task at hand:

Capture 1722 avb video packets passing through the Ethernet port and play them as live video in Android. Video packets consist of a H.264 NAL stream.

What is already available:

The code for reading data from the Ethernet port and capturing packets is ready. In short, I have payload data with me.

What I'm looking for:

• C code that can parse these NAL H264 packets.
• Define start, intermediate, and end frames from a continuous stream of payloads.
• Combine all the H264 NALs associated with it to create a video frame.

I think the above process is called de-fragmentation. After defragmentation, I will send this video frame to watch videos in android and show them on the screen.

Any useful resources would be really appreciated.

Answer 1

First, I assume that your analysis will be carried out on NAL units. The following table shows an incomplete list of NAL unit types. In the main implementations of H264 codecs, you can only find NAL unit types 1, 5, 6, 7, and 8. You can find other NAL units very rarely.

Access Block Separator:

Your problem is easier to solve if the stream has a NAL block number 9, i.e. access block separator. All NAL units, between 2 NAL units of the access division unit, belong to one video frame. Since this type of NAL block is optional, most encoders usually skip embedding this NAL block. So. it is very likely that you cannot find this NAL block in your stream.

NAL units - 6 and 7:

These 3 NAL units are not directly involved in defragmentation, but they are necessary for the decoding operation. In most cases, these 2 types arrive only once in a sequence, that is, at the beginning of a video sequence.

NAL Units - 1 and 5:

These are NAL units that are critical to de-fragmentation. For a given video frame, all NAL units must have the same NAL unit, i.e. 1 or 5. These NALs carry frame slices. I assume that fragments arrive in order, since ASO support (random order of fragments) is an extremely rare function for searching encoders. The first fragment of the frame contains a flag indicating that this is the beginning of the video frame.

The above image is formed by combining two partial tables (only applicable here) from the H264 standard.

Once you decode the NAL header (1-byte information), you will find out if it has a NAL type of 1 or 5 (slice NAL block). After the NAL is found as a slice unit, analyze the stream for the "first_mb_in_slice" character (this information comes immediately after 1 byte of NAL header information). If this flag is set, then this is the first fragment of the video frame. The following NAL units will have this flag as zero until the last fragment of the current video frame. If the flag of the NAL fragment "first_mb_in_slice" is detected, this means that this new fragment belongs to the next video frame and is the beginning of the next video card.

I hope these details help in solving your problem.