Geek Answers Handbook

Compression library for C / C ++ capable of handling more than 32-bit elements in an array

I have a problem in that I need to compress about 6 GB of std::vector() (1.5 billion float in it), and so far I have used lz4, but it only processes int count char s. Since I have 6 billion characters in my vector, this will require 33bit to represent, and compression with LZ4 does not work as I need.

From what I saw in zlib libraries, it accepts int as well as input for the length of the compressed data.

Do I need to segment my data, or is there an environment capable of dealing with more than 32 bits from char s, or am I missing something?

+4
source share
2 answers

Use zlib and pass the array as multiple fragments. The DEFLATE algorithm used by zlib has a window size of about 32 KB, and it already buffers the compressed data, so transferring data in the form of several fragments does not affect the compression efficiency.

+2
source

Take a look at the XZ ; it seems to handle really large sizes. The CLI executable itself is a thin wrapper over the library, so this should match your score.

OTOH, the stream of binary floats should not compress well ...

0
source

More articles:


All Articles