S3.getObject (). createReadStream (): How to catch an error?

Question

I am trying to write a program to get a zip file from s3, unzip it, and then upload to S3. But I found two exceptions that I cannot catch.

1. StreamContentLengthMismatch: Stream content length mismatch. Received 980323883 of 5770104761 bytes. StreamContentLengthMismatch: Stream content length mismatch. Received 980323883 of 5770104761 bytes. This happens irregularly.

2. NoSuchKey: The specified key does not exist. This happens when I enter the wrong key.

When these two exceptions occur, this program crashes.

I would like to catch and handle these two exceptions correctly.

I want to prevent a crash.

  const unzipUpload = () => { return new Promise((resolve, reject) => { let rStream = s3.getObject({Bucket: 'bucket', Key: 'hoge/hoge.zip'}) .createReadStream() .pipe(unzip.Parse()) .on('entry', function (entry) { if(entry.path.match(/__MACOSX/) == null){ // pause if(currentFileCount - uploadedFileCount > 10) rStream.pause() currentFileCount += 1 var fileName = entry.path; let up = entry.pipe(uploadFromStream(s3,fileName)) up.on('uploaded', e => { uploadedFileCount += 1 console.log(currentFileCount, uploadedFileCount) //resume if(currentFileCount - uploadedFileCount <= 10) rStream.resume() if(uploadedFileCount === allFileCount) resolve() entry.autodrain() }).on('error', e => { reject() }) } }).on('error', e => { console.log("unzip error") reject() }).on('finish', e => { allFileCount = currentFileCount }) rStream.on('error', e=> { console.log(e) reject(e) }) }) } function uploadFromStream(s3,fileName) { var pass = new stream.PassThrough(); var params = {Bucket: "bucket", Key: "hoge/unzip/" + fileName, Body: pass}; let request = s3.upload(params, function(err, data) { if(err) pass.emit('error') if(!err) pass.emit('uploaded') }) request.on('httpUploadProgress', progress => { console.log(progress) }) return pass } 

This is the library that I use when unpacking. https://github.com/mhr3/unzip-stream

Help me!

Answer 1

If you want to catch the NoSuchKey error NoSuchKey by createReadStream , you have 2 options:

• Check if the key exists before reading it.
• Catch a bug from the stream

First

 s3.getObjectMetadata(key) .promise() .then(() => { // This will not throw error anymore s3.getObject().createReadStream(); }) .catch(error => { if (error.statusCode === 404) { // Catching NoSuchKey } }); 

The only case where you do not catch the error if the file was deleted within a second between the parsing response from getObjectMetadata and running createReadStream

Second :

 s3.getObject().createReadStream().on('error', error => { // Catching NoSuchKey & StreamContentLengthMismatch }); 

This is a more general approach and will catch all other errors, such as network issues.

Answer 2

You can listen to events (for example, errors, data, finish) in the stream that you get back. Event Details

 function getObjectStream (filePath) { return s3.getObject({ Bucket: bucket, Key: filePath }).createReadStream() } let readStream = getObjectStream('/path/to/file.zip') readStream.on('error', function (error) { // Handle your error here. }) 

Tested for "No Key" error.

 it('should not be able to get stream of unavailable object', function (done) { let filePath = 'file_not_available.zip' let readStream = s3.getObjectStream(filePath) readStream.on('error', function (error) { expect(error instanceof Error).to.equal(true) expect(error.message).to.equal('The specified key does not exist.') done() }) }) 

Tested for success.

 it('should be able to get stream of available object', function (done) { let filePath = 'test.zip' let receivedBytes = 0 let readStream = s3.getObjectStream(filePath) readStream.on('error', function (error) { expect(error).to.equal(undefined) }) readStream.on('data', function (data) { receivedBytes += data.length }) readStream.on('finish', function () { expect(receivedBytes).to.equal(3774) done() }) }) 

Answer 3

You need to listen to a previously emitted error. Your error handler searches for errors during unpacking.

A simplified version of your script.

 s3.getObject(params) .createReadStream() .on('error', (e) => { // handle aws s3 error from createReadStream }) .pipe(unzip) .on('data', (data) => { // retrieve data }) .on('end', () => { // stream has ended }) .on('error', (e) => { // handle error from unzip }); 

Thus, you do not need to make an additional AWS call to find out if it exists, if it exists.