Deflater.deflate and small output buffers

Question

I see a strange situation with small output buffers with Java 8u45 and the java.util.Deflater.deflate(byte[] b, int off, int len, int flush) method java.util.Deflater.deflate(byte[] b, int off, int len, int flush) when used with small output buffers.

(I am working on some low-level networking code related to the expanding WebSocket permessage-deflate extension, so I need small buffers for me)

Code example:

 package deflate; import java.nio.charset.StandardCharsets; import java.util.zip.Deflater; public class DeflaterSmallBufferBug { public static void main(String[] args) { boolean nowrap = true; Deflater deflater = new Deflater(Deflater.DEFAULT_COMPRESSION,nowrap); byte[] input = "Hello".getBytes(StandardCharsets.UTF_8); System.out.printf("input is %,d bytes - %s%n",input.length,getHex(input,0,input.length)); deflater.setInput(input); byte[] output = new byte[input.length]; // break out of infinite loop seen with bug int maxloops = 10; // Compress the data while (maxloops-- > 0) { int compressed = deflater.deflate(output,0,output.length,Deflater.SYNC_FLUSH); System.out.printf("compressed %,d bytes - %s%n",compressed,getHex(output,0,compressed)); if (compressed < output.length) { System.out.printf("Compress success"); return; } } System.out.printf("Exited compress (maxloops left %d)%n",maxloops); } private static String getHex(byte[] buf, int offset, int len) { StringBuilder hex = new StringBuilder(); hex.append('['); for (int i = offset; i < (offset + len); i++) { if (i > offset) { hex.append(' '); } hex.append(String.format("%02X",buf[i])); } hex.append(']'); return hex.toString(); } } 

In the above case, I am trying to create compressed bytes for input "Hello" using an output buffer of 5 bytes long.

I would suggest the following resulting bytes:

 buffer 1 [ F2 48 CD C9 C9 ] buffer 2 [ 07 00 00 00 FF ] buffer 3 [ FF ] 

What does it mean

 [ F2 48 CD C9 C9 07 00 ] <-- the compressed data [ 00 00 FF FF ] <-- the deflate tail bytes 

However, when Deflater.deflate() used with a small buffer, this normal loop continues indefinitely with 5 bytes of compressed data (it seems to only occur with buffers of 5 bytes or lower).

Result for running the above demo ...

 input is 5 bytes - [48 65 6C 6C 6F] compressed 5 bytes - [F2 48 CD C9 C9] compressed 5 bytes - [07 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] Exited compress (maxloops left -1) 

If you do I / O more than 5 bytes, the problem seems to go away. (Just enter the "Hellox" input line to check it out for yourself)

6 byte buffer creation results (input as "Hellox" )

 input is 6 bytes - [48 65 6C 6C 6F 78] compressed 6 bytes - [F2 48 CD C9 C9 AF] compressed 6 bytes - [00 00 00 00 FF FF] compressed 5 bytes - [00 00 00 FF FF] Compress success 

Even these results are a little strange for me, as it seems that there are 2 tail sequence deflates.

So, I think my last question is am I missing something about using Deflater , which makes something strange for me, or does this indicate a possible error in the implementation of the Deflater JVM?

Update: August 7, 2015

This discovery was accepted as bugs.java.com/JDK-8133170

Answer 1

This is the "zlib" function registered in zlib.h:

In the case of Z_FULL_FLUSH or Z_SYNC_FLUSH, make sure avail_out is greater than six to avoid repeated flush markers due to avail_out == 0 when returning.

What happens is that every deflate() call with Z_SYNC_FLUSH inserts a five-byte flush token. Since you do not provide enough space for the output of the token, you call again to get more output, but ask it to insert another clear token at the same time.

What you have to do is call deflate() with Z_SYNC_FLUSH once, and then get all available output with additional deflate() calls, if necessary, that use Z_NO_FLUSH (or NO_FLUSH in Java).