How to get the progress status of a file uploaded to Amazon S3 using Java

Question

I upload multiple files to Amazon S3 using Java.

I am using the following code:

MultipartHttpServletRequest multipartRequest = (MultipartHttpServletRequest) request; MultiValueMap < String, MultipartFile > map = multipartRequest.getMultiFileMap(); try { if (map != null) { for (String filename: map.keySet()) { List < MultipartFile > fileList = map.get(filename); incrPercentge = 100 / fileList.size(); request.getSession().setAttribute("incrPercentge", incrPercentge); for (MultipartFile mpf: fileList) { /* * custom input stream wrap to original input stream to get * the progress */ ProgressInputStream inputStream = new ProgressInputStream("test", mpf.getInputStream(), mpf.getBytes().length); ObjectMetadata metadata = new ObjectMetadata(); metadata.setContentType(mpf.getContentType()); String key = Util.getLoginUserName() + "/" + mpf.getOriginalFilename(); PutObjectRequest putObjectRequest = new PutObjectRequest( Constants.S3_BUCKET_NAME, key, inputStream, metadata).withStorageClass(StorageClass.ReducedRedundancy); PutObjectResult response = s3Client.putObject(putObjectRequest); } } } } catch(Exception e) { e.printStackTrace(); } 

I need to create my own input stream to get the byte number consumed by Amazon S3. I got this idea from the question here: Upload a file or InputStream to S3 with a progress callback

My code for the ProgressInputStream class is as follows:

 package com.spectralnetworks.net.util; import java.io.IOException; import java.io.InputStream; import org.apache.commons.vfs.FileContent; import org.apache.commons.vfs.FileSystemException; public class ProgressInputStream extends InputStream { private final long size; private long progress, lastUpdate = 0; private final InputStream inputStream; private final String name; private boolean closed = false; public ProgressInputStream(String name, InputStream inputStream, long size) { this.size = size; this.inputStream = inputStream; this.name = name; } public ProgressInputStream(String name, FileContent content) throws FileSystemException { this.size = content.getSize(); this.name = name; this.inputStream = content.getInputStream(); } @Override public void close() throws IOException { super.close(); if (closed) throw new IOException("already closed"); closed = true; } @Override public int read() throws IOException { int count = inputStream.read(); if (count > 0) progress += count; lastUpdate = maybeUpdateDisplay(name, progress, lastUpdate, size); return count; }@Override public int read(byte[] b, int off, int len) throws IOException { int count = inputStream.read(b, off, len); if (count > 0) progress += count; lastUpdate = maybeUpdateDisplay(name, progress, lastUpdate, size); return count; } /** * This is on reserach to show a progress bar * @param name * @param progress * @param lastUpdate * @param size * @return */ static long maybeUpdateDisplay(String name, long progress, long lastUpdate, long size) { /* if (Config.isInUnitTests()) return lastUpdate; if (size < B_IN_MB/10) return lastUpdate; if (progress - lastUpdate > 1024 * 10) { lastUpdate = progress; int hashes = (int) (((double)progress / (double)size) * 40); if (hashes > 40) hashes = 40; String bar = StringUtils.repeat("#", hashes); bar = StringUtils.rightPad(bar, 40); System.out.format("%s [%s] %.2fMB/%.2fMB\r", name, bar, progress / B_IN_MB, size / B_IN_MB); System.out.flush(); }*/ System.out.println("name " + name + " progress " + progress + " lastUpdate " + lastUpdate + " " + "sie " + size); return lastUpdate; } } 

But this does not work properly. It prints right up to the file size as follows:

 name test progress 4096 lastUpdate 0 sie 30489 name test progress 8192 lastUpdate 0 sie 30489 name test progress 12288 lastUpdate 0 sie 30489 name test progress 16384 lastUpdate 0 sie 30489 name test progress 20480 lastUpdate 0 sie 30489 name test progress 24576 lastUpdate 0 sie 30489 name test progress 28672 lastUpdate 0 sie 30489 name test progress 30489 lastUpdate 0 sie 30489 name test progress 30489 lastUpdate 0 sie 30489 

And the actual load takes longer (more than 10 times after printing lines).

What should I do to get true download status?

Answer 1

I got an answer to my questions, the best way to get real progress status using the code below

 ObjectMetadata metadata = new ObjectMetadata(); metadata.setContentType(mpf.getContentType()); String key = Util.getLoginUserName() + "/" + mpf.getOriginalFilename(); metadata.setContentLength(mpf.getSize()); PutObjectRequest putObjectRequest = new PutObjectRequest( Constants.S3_BUCKET_NAME, key, mpf.getInputStream(), metadata) .withStorageClass(StorageClass.ReducedRedundancy); putObjectRequest.setProgressListener(new ProgressListener() { @Override public void progressChanged(ProgressEvent progressEvent) { System.out.println(progressEvent .getBytesTransfered() + ">> Number of byte transfered " + new Date()); progressEvent.getBytesTransfered(); double totalByteRead = request .getSession().getAttribute( Constants.TOTAL_BYTE_READ) != null ? (Double) request .getSession().getAttribute(Constants.TOTAL_BYTE_READ) : 0; totalByteRead += progressEvent.getBytesTransfered(); request.getSession().setAttribute(Constants.TOTAL_BYTE_READ, totalByteRead); System.out.println("total Byte read "+ totalByteRead); request.getSession().setAttribute(Constants.TOTAL_PROGRESS, (totalByteRead/size)*100); System.out.println("percentage completed >>>"+ (totalByteRead/size)*100); if (progressEvent.getEventCode() == ProgressEvent.COMPLETED_EVENT_CODE) { System.out.println("completed ******"); } } }); s3Client.putObject(putObjectRequest); 

The problem with my previous code was that I did not set the length of the content in the metadata, so I did not get the true progress status. The next line is a copy from the PutObjectRequest class API

Creates a new PutObjectRequest to load the data stream into the specified bucket and key. After creating a request, users can optionally specify object metadata or a canned ACL.

The content length for the data stream must be specified in the object metadata parameter; Amazon S3 requires it to be transferred before data is loaded. Failure to specify the length of the content will buffer the entire contents of the input stream locally in memory so that the length of the content can be calculated, which can lead to negative performance problems.

Answer 2

I am going to assume that you are using the AWS SDK for Java.

Your code works as it should: It shows that reading is invoked with reading 4K every time. Your idea (updated in the post) is also true: the AWS SDK provides the ProgressListener as a way to inform the application about the download progress.

The "problem" is the implementation of the AWS SDK, it buffers more than a file size of 30 KB (I'm going to read it 64 KB), so you do not receive any progress reports.

Try downloading a larger file (say 1M), and you will see that both methods give better results, because today's network speeds reporting progress in a 30K file are not even worth it.

If you need better control, you can implement the download yourself using the S3 REST interface (which ultimately uses the AWS Java SDK) is not very difficult, but it's a bit of work. If you want to go this route, I recommend finding an example to calculate the session authorization token instead of doing it yourself (sorry my search foo is not strong enough to link to the actual code sample right now.) However, as soon as you go over all these problems , you will find that you really want to have a 64K buffer in the socket stream to ensure maximum throughput on a fast network (which probably leads to the Java SDS SDK behaving the same way as it does).