Reaching client load in client

Question

I have a jersey client that needs to upload a file large enough to require a progress bar.
The problem is that for a download that takes several minutes, I see that the transferred bytes go 100% as soon as the application starts . Then it takes a few minutes to print the Finish line.
It is as if bytes were sent to the buffer, and I read the transfer rate to the buffer, not the actual download speed. This makes the progress indicator useless.

This is the simplest code:

ClientConfig config = new DefaultClientConfig(); Client client = Client.create(config); WebResource resource = client.resource("www.myrestserver.com/uploads"); WebResource.Builder builder = resource.type(MediaType.MULTIPART_FORM_DATA_TYPE); FormDataMultiPart multiPart = new FormDataMultiPart(); FileDataBodyPart fdbp = new FileDataBodyPart("data.zip", new File("data.zip")); BodyPart bp = multiPart.bodyPart(fdbp); String response = builder.post(String.class, multiPart); 

To get the execution status, I added a ContainerListener filter, obviously, before calling builder.post:

 final ContainerListener containerListener = new ContainerListener() { @Override public void onSent(long delta, long bytes) { System.out.println(delta + " : " + long); } @Override public void onFinish() { super.onFinish(); System.out.println("on finish"); } }; OnStartConnectionListener connectionListenerFactory = new OnStartConnectionListener() { @Override public ContainerListener onStart(ClientRequest cr) { return containerListener; } }; resource.addFilter(new ConnectionListenerFilter(connectionListenerFactory)); 

Answer 1

should be enough to provide you with your own MessageBodyWriter for java.io.File that fires some events or notifies some listeners about the progress of the changes

 @Provider() @Produces(MediaType.APPLICATION_OCTET_STREAM) public class MyFileProvider implements MessageBodyWriter<File> { public boolean isWriteable(Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) { return File.class.isAssignableFrom(type); } public void writeTo(File t, Class<?> type, Type genericType, Annotation annotations[], MediaType mediaType, MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream) throws IOException { InputStream in = new FileInputStream(t); try { int read; final byte[] data = new byte[ReaderWriter.BUFFER_SIZE]; while ((read = in.read(data)) != -1) { entityStream.write(data, 0, read); // fire some event as progress changes } } finally { in.close(); } } @Override public long getSize(File t, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) { return t.length(); } } 

and make your client application just use this new provider:

 ClientConfig config = new DefaultClientConfig(); config.getClasses().add(MyFileProvider.class); 

or

 ClientConfig config = new DefaultClientConfig(); MyFileProvider myProvider = new MyFileProvider (); cc.getSingletons().add(myProvider); 

You will also need to enable some algorithm to recognize which file is being transmitted when receiving progress events.

Edited by:

I just discovered that by default, HTTPUrlConnection uses buffering. And to disable buffering, you can do several things:

• httpUrlConnection.setChunkedStreamingMode (chunklength) - disables buffering and uses encoded transmission encoding to send a request
• httpUrlConnection.setFixedLengthStreamingMode (contentLength) - disables buffering, but has some restrictions for streaming: the exact number of bytes must be sent

So, I suggest that the final solution to your problem use the first option and look like this:

 ClientConfig config = new DefaultClientConfig(); config.getClasses().add(MyFileProvider.class); URLConnectionClientHandler clientHandler = new URLConnectionClientHandler(new HttpURLConnectionFactory() { @Override public HttpURLConnection getHttpURLConnection(URL url) throws IOException { HttpURLConnection connection = (HttpURLConnection) url.openConnection(); connection.setChunkedStreamingMode(1024); return connection; } }); Client client = new Client(clientHandler, config); 

Answer 2

In Jersey 2.X, I used WriterInterceptor to wrap the output stream with a subclass of Apache Commons IO CountingOutputStream, which tracks the record and notifies the download progress code (not shown).

 public class UploadMonitorInterceptor implements WriterInterceptor { @Override public void aroundWriteTo(WriterInterceptorContext context) throws IOException, WebApplicationException { // the original outputstream jersey writes with final OutputStream os = context.getOutputStream(); // you can use Jersey target/builder properties or // special headers to set identifiers of the source of the stream // and other info needed for progress monitoring String id = (String) context.getProperty("id"); long fileSize = (long) context.getProperty("fileSize"); // subclass of counting stream which will notify my progress // indicators. context.setOutputStream(new MyCountingOutputStream(os, id, fileSize)); // proceed with any other interceptors context.proceed(); } } 

Then I registered this interceptor with the client or for specific purposes for which you want to use the interceptor.

Answer 3

I have successfully used David's answer. However, I would like to expand it:

The following implementation aroundWriteTo my WriterInterceptor shows how a panel (or similar) can also be passed to CountingOutputStream :

 @Override public void aroundWriteTo(WriterInterceptorContext context) throws IOException, WebApplicationException { final OutputStream outputStream = context.getOutputStream(); long fileSize = (long) context.getProperty(FILE_SIZE_PROPERTY_NAME); context.setOutputStream(new ProgressFileUploadStream(outputStream, fileSize, (progressPanel) context .getProperty(PROGRESS_PANEL_PROPERTY_NAME))); context.proceed(); } 

afterWrite in CountingOutputStream can then set the progress:

 @Override protected void afterWrite(int n) { double percent = ((double) getByteCount() / fileSize); progressPanel.setValue((int) (percent * 100)); } 

Properties can be set for the Invocation.Builder object:

 Invocation.Builder invocationBuilder = webTarget.request(); invocationBuilder.property( UploadMonitorInterceptor.FILE_SIZE_PROPERTY_NAME, newFile.length()); invocationBuilder.property( UploadMonitorInterceptor.PROGRESS_PANEL_PROPERTY_NAME, progressPanel); 

Perhaps the most important addition to David's answer and the reason I decided to post my own is the following code:

 client.property(ClientProperties.CHUNKED_ENCODING_SIZE, 1024); client.property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED"); 

client object is javax.ws.rs.client.Client .

You must also disable buffering with the WriterInterceptor approach. The code above is an easy way to do this with Jersey 2.x