First, it’s important to understand that there is no single standard H.264 bit-bit format. The specification document contains an Appendix, in particular Appendix B, which describes one possible format, but this is not an actual requirement. The standard defines how video is encoded in separate packets. How these packets are stored and transmitted remains open to the integrator.
1. Appendix B
Network Abstraction Level Units
Packets are called units of the network abstraction layer. Often abbreviated NALU (or sometimes simply NAL) each packet can be processed separately and processed separately. The first byte of each NALU contains the NALU type, in particular bits 3 through 7. (bit 0 is always off, and bits 1-2 indicate whether the NALU refers to another NALU).
There are 19 different types of NALUs, divided into two categories: VCL and non-VCL:
- VCL or Video Coding Layer packages contain actual visual information.
- Non-VCLs contain metadata that may or may not be required for video decoding.
One NALU, or even VCL NALU, is not the same as a frame. A frame can be “sliced” into several NALU units. Just like you can chop pizza. Then one or more fragments are actually grouped into access units (AU), which contain one frame. Slicing is really insignificant, so it is often not used.
Below is a table of all defined NALUs.
0 Unspecified non-VCL 1 Coded slice of a non-IDR picture VCL 2 Coded slice data partition A VCL 3 Coded slice data partition B VCL 4 Coded slice data partition C VCL 5 Coded slice of an IDR picture VCL 6 Supplemental enhancement information (SEI) non-VCL 7 Sequence parameter set non-VCL 8 Picture parameter set non-VCL 9 Access unit delimiter non-VCL 10 End of sequence non-VCL 11 End of stream non-VCL 12 Filler data non-VCL 13 Sequence parameter set extension non-VCL 14 Prefix NAL unit non-VCL 15 Subset sequence parameter set non-VCL 16 Depth parameter set non-VCL 17..18 Reserved non-VCL 19 Coded slice of an auxiliary coded picture without partitioning non-VCL 20 Coded slice extension non-VCL 21 Coded slice extension for depth view components non-VCL 22..23 Reserved non-VCL 24..31 Unspecified non-VCL
There are several types of NALUs in which knowledge may be useful later.
- Sequence Parameter Set (SPS). This non-VCL NLU contains the information necessary to configure the decoder, such as profile, level, resolution, frame rate.
- Image Parameter Set (PPS). Like SPS, this non-VCL contains information on entropy coding mode, slice groups, motion prediction filters, and release filters.
- Instant Decoder Update (IDR). This VCL NALU is a standalone image slice. That is, the IDR can be decoded and displayed without reference to other SALs and PPS NALUs.
- Access Block Separator (AUD). AUD is an optional NALU that can be used to delimit frames in an elementary stream. This is not required (unless otherwise indicated by the container / protocol, for example TS) and is often not included to save space, but it may be useful to find the beginning of the frame without a complete analysis of each NALU.
NALU Start Codes
A NALU does not contain its size. Therefore, simply concatenating NALU to create a stream will not work, because you won’t know where to stop, and then it starts.
The specification of Appendix B permits this by requiring that "Initial Codes" precede each NALU. The start code is 2 or 3 0x00 bytes with byte 0x01 . e.g. 0x000001 or 0x00000001 .
The 4-byte variation is useful for transmission over a serial connection, since trivially bytes align the stream, looking for 31 zero bits, followed by one. If the next bit is 0 (since each NALU starts with 0 bits), this is the beginning of the NALU. The 4-byte variation is usually used only for signaling random access points in the stream, such as SPS PPS AUD and IDR. Where, as a variation, 3 bytes are used everywhere to save space.
Warning Management Byte
Running codes works because the four byte sequences 0x000000 , 0x000001 , 0x000002 and 0x000003 are illegal in a NALU other than RBSP. Therefore, when creating a NALU, care is taken for these values, which otherwise could be confused with the startup code. This is achieved by inserting the 0x03 emulation warning byte, so that 0x000001 becomes 0x00000301 .
When decoding, it is important to look for and ignore emulation prevention bytes. Since emulation prevention bytes can occur almost anywhere in the NALU, it is often more convenient in the documentation to assume that they are already deleted. The byte-free representation of emulation warnings is called the serial byte sequence payload (RBSP).
Example
Take a look at the full example.
0x0000 | 00 00 00 01 67 64 00 0A AC 72 84 44 26 84 00 00 0x0010 | 03 00 04 00 00 03 00 CA 3C 48 96 11 80 00 00 00 0x0020 | 01 68 E8 43 8F 13 21 30 00 00 01 65 88 81 00 05 0x0030 | 4E 7F 87 DF 61 A5 8B 95 EE A4 E9 38 B7 6A 30 6A 0x0040 | 71 B9 55 60 0B 76 2E B5 0E E4 80 59 27 B8 67 A9 0x0050 | 63 37 5E 82 20 55 FB E4 6A E9 37 35 72 E2 22 91 0x0060 | 9E 4D FF 60 86 CE 7E 42 B7 95 CE 2A E1 26 BE 87 0x0070 | 73 84 26 BA 16 36 F4 E6 9F 17 DA D8 64 75 54 B1 0x0080 | F3 45 0C 0B 3C 74 B3 9D BC EB 53 73 87 C3 0E 62 0x0090 | 47 48 62 CA 59 EB 86 3F 3A FA 86 B5 BF A8 6D 06 0x00A0 | 16 50 82 C4 CE 62 9E 4E E6 4C C7 30 3E DE A1 0B 0x00B0 | D8 83 0B B6 B8 28 BC A9 EB 77 43 FC 7A 17 94 85 0x00C0 | 21 CA 37 6B 30 95 B5 46 77 30 60 B7 12 D6 8C C5 0x00D0 | 54 85 29 D8 69 A9 6F 12 4E 71 DF E3 E2 B1 6B 6B 0x00E0 | BF 9F FB 2E 57 30 A9 69 76 C4 46 A2 DF FA 91 D9 0x00F0 | 50 74 55 1D 49 04 5A 1C D6 86 68 7C B6 61 48 6C 0x0100 | 96 E6 12 4C 27 AD BA C7 51 99 8E D0 F0 ED 8E F6 0x0110 | 65 79 79 A6 12 A1 95 DB C8 AE E3 B6 35 E6 8D BC 0x0120 | 48 A3 7F AF 4A 28 8A 53 E2 7E 68 08 9F 67 77 98 0x0130 | 52 DB 50 84 D6 5E 25 E1 4A 99 58 34 C7 11 D6 43 0x0140 | FF C4 FD 9A 44 16 D1 B2 FB 02 DB A1 89 69 34 C2 0x0150 | 32 55 98 F9 9B B2 31 3F 49 59 0C 06 8C DB A5 B2 0x0160 | 9D 7E 12 2F D0 87 94 44 E4 0A 76 EF 99 2D 91 18 0x0170 | 39 50 3B 29 3B F5 2C 97 73 48 91 83 B0 A6 F3 4B 0x0180 | 70 2F 1C 8F 3B 78 23 C6 AA 86 46 43 1D D7 2A 23 0x0190 | 5E 2C D9 48 0A F5 F5 2C D1 FB 3F F0 4B 78 37 E9 0x01A0 | 45 DD 72 CF 80 35 C3 95 07 F3 D9 06 E5 4A 58 76 0x01B0 | 03 6C 81 20 62 45 65 44 73 BC FE C1 9F 31 E5 DB 0x01C0 | 89 5C 6B 79 D8 68 90 D7 26 A8 A1 88 86 81 DC 9A 0x01D0 | 4F 40 A5 23 C7 DE BE 6F 76 AB 79 16 51 21 67 83 0x01E0 | 2E F3 D6 27 1A 42 C2 94 D1 5D 6C DB 4A 7A E2 CB 0x01F0 | 0B B0 68 0B BE 19 59 00 50 FC C0 BD 9D F5 F5 F8 0x0200 | A8 17 19 D6 B3 E9 74 BA 50 E5 2C 45 7B F9 93 EA 0x0210 | 5A F9 A9 30 B1 6F 5B 36 24 1E 8D 55 57 F4 CC 67 0x0220 | B2 65 6A A9 36 26 D0 06 B8 E2 E3 73 8B D1 C0 1C 0x0230 | 52 15 CA B5 AC 60 3E 36 42 F1 2C BD 99 77 AB A8 0x0240 | A9 A4 8E 9C 8B 84 DE 73 F0 91 29 97 AE DB AF D6 0x0250 | F8 5E 9B 86 B3 B3 03 B3 AC 75 6F A6 11 69 2F 3D 0x0260 | 3A CE FA 53 86 60 95 6C BB C5 4E F3
This is a complete speaker containing 3 NALUs. As you can see, we start with a startup code followed by SPS (SPS starts at 67). Inside SPS you will see two bytes of warning emulation. Without these bytes, an illegal sequence of 0x000000 will occur at these positions. You will then see the start code followed by PPS (PPS starts at 68) and one final start code, followed by the IDR snippet. This is a complete H.264 stream. If you enter these values in a hex editor and save the file with the extension .264 , you can convert it to this image:
Appendix B is commonly used in live and streaming formats such as traffic streams, broadcasts, and DVDs. In these formats, SPS and PPS are often repeated periodically, usually before each IDR, thus creating an arbitrary access point for the decoder. This allows you to connect to an existing stream.
2. AVCC
Another common way to store an H.264 stream is the AVCC format. In this format, each NALU is preceded by its length (in large end format). This method is easier to parse, but you lose the function of aligning bytes in application B. Just to complicate things, the length can be encoded using 1, 2, or 4 bytes. This value is stored in the header object. This header is often called "extradata" or "header". Its main format is as follows:
bits 8 version ( always 0x01 ) 8 avc profile ( sps[0][1] ) 8 avc compatibility ( sps[0][2] ) 8 avc level ( sps[0][3] ) 6 reserved ( all bits on ) 2 NALULengthSizeMinusOne 3 reserved ( all bits on ) 5 number of SPS NALUs (usually 1) repeated once per SPS: 16 SPS size variable SPS NALU data 8 number of PPS NALUs (usually 1) repeated once per PPS 16 PPS size variable PPS NALU data
Using the same example above, AVCC extradata would look like this:
0x0000 | 01 64 00 0A FF E1 00 19 67 64 00 0A AC 72 84 44 0x0010 | 26 84 00 00 03 00 04 00 00 03 00 CA 3C 48 96 11 0x0020 | 80 01 00 07 68 E8 43 8F 13 21 30
You will notice that SPS and PPS are now stored out of range. That is, separately from the elementary stream data. The storage and transfer of this data is the job of the file container and is beyond the scope of this document. Note that although we do not use start codes, emulation prevention bytes are still inserted.
In addition, there is a new variable called NALULengthSizeMinusOne . This variable with named names tells us how many bytes to use to store the length of each NALU. So, if NALULengthSizeMinusOne set to 0, then each NALU is preceded by one byte indicating its length. Using one byte to store the size, the maximum NALU size is 255 bytes. This is obviously quite a bit. The path is too small for the entire keyframe. Using 2 bytes gives us 64k per NALU. This will work in our example, but still remains pretty low. 3 bytes would be ideal, but for some reason are not supported everywhere. So 4 bytes are by far the most common, and this is what we used here:
0x0000 | 00 00 02 41 65 88 81 00 05 4E 7F 87 DF 61 A5 8B 0x0010 | 95 EE A4 E9 38 B7 6A 30 6A 71 B9 55 60 0B 76 2E 0x0020 | B5 0E E4 80 59 27 B8 67 A9 63 37 5E 82 20 55 FB 0x0030 | E4 6A E9 37 35 72 E2 22 91 9E 4D FF 60 86 CE 7E 0x0040 | 42 B7 95 CE 2A E1 26 BE 87 73 84 26 BA 16 36 F4 0x0050 | E6 9F 17 DA D8 64 75 54 B1 F3 45 0C 0B 3C 74 B3 0x0060 | 9D BC EB 53 73 87 C3 0E 62 47 48 62 CA 59 EB 86 0x0070 | 3F 3A FA 86 B5 BF A8 6D 06 16 50 82 C4 CE 62 9E 0x0080 | 4E E6 4C C7 30 3E DE A1 0B D8 83 0B B6 B8 28 BC 0x0090 | A9 EB 77 43 FC 7A 17 94 85 21 CA 37 6B 30 95 B5 0x00A0 | 46 77 30 60 B7 12 D6 8C C5 54 85 29 D8 69 A9 6F 0x00B0 | 12 4E 71 DF E3 E2 B1 6B 6B BF 9F FB 2E 57 30 A9 0x00C0 | 69 76 C4 46 A2 DF FA 91 D9 50 74 55 1D 49 04 5A 0x00D0 | 1C D6 86 68 7C B6 61 48 6C 96 E6 12 4C 27 AD BA 0x00E0 | C7 51 99 8E D0 F0 ED 8E F6 65 79 79 A6 12 A1 95 0x00F0 | DB C8 AE E3 B6 35 E6 8D BC 48 A3 7F AF 4A 28 8A 0x0100 | 53 E2 7E 68 08 9F 67 77 98 52 DB 50 84 D6 5E 25 0x0110 | E1 4A 99 58 34 C7 11 D6 43 FF C4 FD 9A 44 16 D1 0x0120 | B2 FB 02 DB A1 89 69 34 C2 32 55 98 F9 9B B2 31 0x0130 | 3F 49 59 0C 06 8C DB A5 B2 9D 7E 12 2F D0 87 94 0x0140 | 44 E4 0A 76 EF 99 2D 91 18 39 50 3B 29 3B F5 2C 0x0150 | 97 73 48 91 83 B0 A6 F3 4B 70 2F 1C 8F 3B 78 23 0x0160 | C6 AA 86 46 43 1D D7 2A 23 5E 2C D9 48 0A F5 F5 0x0170 | 2C D1 FB 3F F0 4B 78 37 E9 45 DD 72 CF 80 35 C3 0x0180 | 95 07 F3 D9 06 E5 4A 58 76 03 6C 81 20 62 45 65 0x0190 | 44 73 BC FE C1 9F 31 E5 DB 89 5C 6B 79 D8 68 90 0x01A0 | D7 26 A8 A1 88 86 81 DC 9A 4F 40 A5 23 C7 DE BE 0x01B0 | 6F 76 AB 79 16 51 21 67 83 2E F3 D6 27 1A 42 C2 0x01C0 | 94 D1 5D 6C DB 4A 7A E2 CB 0B B0 68 0B BE 19 59 0x01D0 | 00 50 FC C0 BD 9D F5 F5 F8 A8 17 19 D6 B3 E9 74 0x01E0 | BA 50 E5 2C 45 7B F9 93 EA 5A F9 A9 30 B1 6F 5B 0x01F0 | 36 24 1E 8D 55 57 F4 CC 67 B2 65 6A A9 36 26 D0 0x0200 | 06 B8 E2 E3 73 8B D1 C0 1C 52 15 CA B5 AC 60 3E 0x0210 | 36 42 F1 2C BD 99 77 AB A8 A9 A4 8E 9C 8B 84 DE 0x0220 | 73 F0 91 29 97 AE DB AF D6 F8 5E 9B 86 B3 B3 03 0x0230 | B3 AC 75 6F A6 11 69 2F 3D 3A CE FA 53 86 60 95 0x0240 | 6C BB C5 4E F3
The advantage of this format is the ability to configure the decoder at the beginning and jump into the middle of the stream. This is a common use case when the media is available on random access media such as a hard disk, and is therefore used in conventional container formats such as MP4 and MKV.