Amazon S3 response flow after response has been deleted

Question

I use the Amazon SDK, and I have a method that returns a Stream for an object stored in the Amazon S3 service.

It contains something like this:

var request = new GetObjectRequest().WithBucketName(bucketName).WithKey(keyName); using (var response = client.GetObject(request)) { return response.ResponseStream; } 

Obviously, in this case, the stream is not read from the calling method, because the request object was deleted, and when this is done, it closes the stream.

I do not want to upload the file to a MemoryStream or FileStream.

If I do not use the use clause, the garbage collector will delete the request object at some point, so I cannot just not use it.

I ask, is there a way to wrap or copy a stream to another stream and then return it without downloading the file?

I am using .NET 3.5.

Edit: The method inherits from the abstract class, and the caller method does not know that it works with Amazon. Therefore, it must return the stream.

Answer 1

You cannot work with the stream after it has been placed, but you can delay the deletion of the response object until the response stream is used. There are three options I can offer.

• Return reply. One option is to return the response object to the caller. The caller can access the contained response stream and then delete the response upon completion. This is the simplest change, but also requires the caller to change too.

• Wrap the thread. Instead of directly responding to the response flow, create a new object that extends the flow and wraps the response flow, and also refers to the response itself. Then, when your shell is located, you can internally delete the response object. This only requires changing your called code, while the return type is just Stream .

• Callback. Do not return anything from the method and use the callback mechanism instead. Take Action<Stream> as a parameter and call this callback with the stream. This way the caller code is called, and you still control the deletion of the response and stream. This is the safest mechanism, since you do not rely on the caller to dispose of anything, but require most changes.

Answer 2

There is an OpenStream method in the TransferUtility class that returns a stream object.

public Stream OpenStream (String bucketName, String key)

I looked at the AWSSDK source code and found in the SDK that OpenStream simply returns a response stream directly. (It does not use the expression "using" for the response object.)

Answer 3

If your function returns a response object (without using the using statement) and the caller assigns it to a variable, there will still be a reference to the response object. Thus, it will not be a suitable garbage collection.

Answer 4

Had the same problem (I thought). But did you try not to use "use". This will not use the stream and send it to the caller, who will be responsible for its use. So simple.

  var request = new GetObjectRequest { BucketName = containerName, Key = blobName }; GetObjectResponse response = null; try { response = client.GetObject(request)); } catch (AmazonS3Exception ex) { if ((ex.ErrorCode == "NoSuchBucket") || (ex.ErrorCode == "AccessDenied") || (ex.ErrorCode == "InvalidBucketName") || (ex.ErrorCode == "NoSuchKey")) { return null; } throw; } return response.ResponseStream; 

Answer 5

For completeness, and since I went with the @Samuel option # 2 (Wrap the stream) that @spakinz commented on, he also included here my implementation of what I called AmazonS3Stream

 public class AmazonS3Stream : Stream { private Stream stream; private GetObjectResponse response; public AmazonS3Stream(GetObjectResponse response) { this.stream = response.ResponseStream; this.response = response; } // The whole purpose of this class protected override void Dispose(bool disposing) { // base.Dispose(disposing); // Do we really need this line? Probably not since I tested it and I can see that the stream is disposed when Response.Dispose is called by itself. And that makes sense because we know that this.stream is pointing to response.ResponseStream (that what we declared in the constructor: this.stream = response.ResponseStream; ) So, what do we expect from response.Dispose() ? Obviously the first thing it would do is call ResponseStream.Dispose() response.Dispose(); } public override long Position { get { return stream.Position; } set { stream.Position = Position; } } public override long Length { get { return stream.Length; } } public override bool CanRead { get { return stream.CanRead; } } public override bool CanSeek { get { return stream.CanSeek; } } public override bool CanWrite { get { return stream.CanWrite; } } public override void Flush() { stream.Flush(); } public override void Write(byte[] buffer, int offset, int count) { stream.Write(buffer, offset, count); } public override void SetLength(long value) { stream.SetLength(value); } public override long Seek(long offset, SeekOrigin origin) { return stream.Seek(offset, origin); } public override int Read(byte[] buffer, int offset, int count) { return stream.Read(buffer, offset, count); } } 

Answer 6

To add sample code to @Jun Y's answer. This is what I did, and it was the cleanest solution to date.

 using (var client = new AmazonS3Client(Amazon.RegionEndpoint.USEast2)) { var transferUtility = new TransferUtility(client); return await transferUtility.OpenStreamAsync(S3BucketName, key); }