How to set the contents of InputStream Length

Question

I upload files to an Amazon S3 bucket. Files are loading, but I get the following warning.

WARNING: no content length specified for stream data. The contents of the stream will be buffered in memory and may lead to memory errors.

So I added the following line to my code

metaData.setContentLength(IOUtils.toByteArray(input).length); 

but then I got the following message. I don’t even know if this is a warning or what.

Reading data has a different length than expected: dataLength = 0; expectedLength = 111992; includeSkipped = false; in.getClass () = class sun.net.httpserver.FixedLengthInputStream; markedSupported = false; marked = 0; resetSinceLastMarked = false; markCount = 0; resetCount = 0

How can I set contentLength for InputSteam metadata? Any help would be greatly appreciated.

Answer 1

When you read data using IOUtils.toByteArray , it consumes an InputStream. When the AWS API tries to read it, it is zero length.

Read the contents into an array of bytes and provide the InputStream package that this array applies to the API:

 byte[] bytes = IOUtils.toByteArray(input); metaData.setContentLength(bytes.length); ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes); PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, key, byteArrayInputStream, metadata); client.putObject(putObjectRequest); 

Answer 2

Please note that with ByteBuffer you simply do manually what the AWS SDK has already done automatically for you! It still buffers the entire stream to memory, and is as good as the original solution that throws a warning from the SDK.

You can get rid of the memory problem only if you have another way to find out the length of the stream, for example, when you create a stream from a file:

 void uploadFile(String bucketName, File file) { try (final InputStream stream = new FileInputStream(file)) { ObjectMetadata metadata = new ObjectMetadata(); metadata.setContentLength(file.length()); s3client.putObject( new PutObjectRequest(bucketName, file.getName(), stream, metadata) ); } } 

Answer 3

Last news! AWS SDK 2.0 has built-in support for downloading files:

  s3client.putObject( (builder) -> builder.bucket(myBucket).key(file.getName()), RequestBody.fromFile(file) ); 

There are also RequestBody methods for receiving strings or buffers that automatically and efficiently set the Content-Length. Only when you have another type of InputStream, you still need to specify the length yourself - however, this case should be more rare now with all the other options available.