Multithreading compression

Question

This is my understanding of Brotli stores block information in the header of the metablock with only the final uncompressed block size and no information about the compression length ( 9.2 ). I assume that you will need to create a wrapper in order to use it with multiple threads, or perhaps with something similar to Mark Adler pigz .

Will the same streaming principles apply to Brotli as with gzip, or are there any predictable problems to know when it comes to multi-threaded implementations?

Answer 1

You can use brotli format as for this purpose. I made them add the ability to place metadata in empty metablocks (where “empty” means that the meta-block produces zero uncompressed data). You can put markers in metadata to help find metablocks. A nested empty meta block also starts the next meta block at the byte boundary.

Each meta block can be independent of other meta blocks. If the flow is designed in this way, then there is no problem combining them during compression or separate decompression. Areas of potential dependency are a circular buffer of the four most recently used distances and backlinks preceding the start of the current metablock. For parallel use, the meta-block can and should be designed in such a way that it does not depend on the last four distances, without referring to the ring buffer, until it is filled with distances from the current metablock. In addition, distances that reach the current metablock will not be allowed (which does not include static links). Finally, you must add a blank meta-block or metadata to bring the sequence to the byte boundary for easy concatenation.

By the way, it looks like you're referring to an older version of the draft. Below is a link to the current version .