Why doesn't the Google Storage API work with the Amazon SDK v3?

Question

I feel rather strange posting this here, but since SO is the only official Google API support channel, I think I need to ask it here.

If you have an API, use of a tool, or other software related to developing software for questions, searching for and publishing questions about stack overflows using the official Google Cloud Storage tag.

Ok, so here. I spent most of two days trying to get Google Storage to work on the Amazon PHP SDK v3 (latest version). I can’t use an older version of the SDK because I try to stick with the Laravel 5.1 file system without having to write a completely new driver for Google Storage. I believe this is in the spirit of what Google advertises for Google Storage:

https://cloud.google.com/storage/docs/migrating

In a simple migration from Amazon S3 to Google Cloud Storage, you can use existing tools and libraries to create authenticated REST requests on Amazon S3, as well as send authenticated requests to Google Cloud Storage. The changes that need to be made to existing tools and libraries are described in this section.

To configure for simple migration, follow these steps:

Set your default Google project.

Get the developer key.

In existing tools or libraries, make the following changes: Modify the request to use the Google Cloud Storage request endpoint. Replace Amazon Web Services Access (AWS) and private key with the corresponding Google Cloud Storage Access Key and Private Key (collectively called your Google Developer Key).

What is it! At this point, you can start using Google Cloud Storage using existing tools and libraries to send messages with a hash key (HMAC).

What a step! Let's try using compatibility credentials that work using gsutil.

$client = new S3Client([ 'credentials' => [ 'key' => 'GOOGxxxxxxxxxxxxxxx', 'secret' => 'ZfcOTxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', ], 'region' => 'US', 'version' => 'latest', 'endpoint' => 'https://storage.googleapis.com', ]); try { $result = $client->putObject(array( 'Bucket' => 'devtest', 'Key' => 'test', 'Body' => 'Hello world' )); echo $result['ObjectURL']; } catch (\Aws\S3\Exception\S3Exception $e) { // The AWS error code (eg, ) echo $e->getAwsErrorCode() . "\n"; // The bucket couldn't be created echo $e->getMessage() . "\n"; } 

Does not work. You get the "Invalid authentication header." Let's take a look at this heading.

AWS4-HMAC-SHA256 Credential = GOOGGUxxxxxxxxxxx / 20150611 / USA / s3 / aws4_request, SignedHeaders = host; x-AMZ-content-sha256; x-AMZ-date, Signature = 9c7de4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

At this point, I created a SO post, and someone suggested adding "signature" => "v2".

Invalid Google Storage authorization header using Amazon S3 PHP SDK v3

Try the following:

 $client = new S3Client([ 'credentials' => [ 'key' => 'GOOGxxxxxxxxxxxxxxx', 'secret' => 'ZfcOTxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', ], 'region' => 'US', 'version' => 'latest', 'endpoint' => 'https://storage.googleapis.com', 'signature' => 'v2', ]); 

Bad luck. The same mistake. The authorization header has not changed. Take a look at the S3Client code and see how the “signature” is used:

 public function createPresignedRequest(CommandInterface $command, $expires) { /** @var \Aws\Signature\SignatureInterface $signer */ $signer = call_user_func( $this->getSignatureProvider(), $this->getConfig('signature_version'), $this->getApi()->getSigningName(), $this->getRegion() ); return $signer->presign( \Aws\serialize($command), $this->getCredentials()->wait(), $expires ); } 

This is not true. So, now we deviate from the official S3 documentation, because they say the same thing:

http://docs.aws.amazon.com/aws-sdk-php/guide/latest/configuration.html

This is not a 'signature', this is a 'signature_version'. Let this change to v2.

 $client = new S3Client([ 'credentials' => [ 'key' => 'GOOGxxxxxxxxxxxxxxx', 'secret' => 'ZfcOTxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', ], 'region' => 'US', 'version' => 'latest', 'endpoint' => 'https://storage.googleapis.com', 'signature_version' => 'v2', ]); 

At least this time we get another error!

UnresolvedSignatureException in SignatureProvider.php 61 line: impossible to enable signature for v2 / s3 / US. Valid signatures include v4 and anonymous.

So, after a two-day training, it seems that this is impossible, at least not with the ease with which Google wants you to believe in your feed. I can't get this to work at all, so I hope someone here can shed some light on this. Either I missed something important, or Google mistakenly advertises that Google Storage works using the Amazon S3 SDK and is wasting developers time. I think that perhaps we need to manually grab the authorization header, but this is beyond my experience. Any help would be greatly appreciated.

Answer 1

So, I'm going to publish later than never, and say that I am using laravel 5.1, you need to make a choice whether you want to go with AWS or GCS during setup, since you cannot install two versions of aws-php client (which limits your choice of flysystem to v2 or v3). GCS requires v2 while the s3 implementation in laravel requires v3. I spent hours trying to get v3 before working with GCS, but the authentication headers are different from each other, so I was not worried.

You need to provide your own provider, but it is not so difficult to configure. Just create GcsAppsServiceProvider.php in application / providers

 <?php namespace App\Providers; use Illuminate\Support\ServiceProvider; use Storage; use Aws\S3\S3Client; use League\Flysystem\AwsS3v2\AwsS3Adapter; use League\Flysystem\Filesystem; class GcsAppsServiceProvider extends ServiceProvider { public function register() { } public function boot() { Storage::extend('gcs', function( $app, $config ) { $client = S3Client::factory([ 'key' => $config['key'], 'secret' => $config['secret'], 'base_url' => $config['base_url'], ]); return new Filesystem(new AwsS3Adapter($client, $config['bucket'])); }); } } 

Essentially in your configuration, you simply duplicate the s3 configuration in filesystem.php and change the region -> base_url. You can also change the default and cloud in gcs to support Google Cloud Storage.

 's3' => [ 'driver' => 's3', 'key' => env('AWS_KEY', 'your-key'), 'secret' => env('AWS_SECRET', 'your-secret'), 'region' => env('AWS_REGION','your-region'), 'bucket' => env('AWS_BUCKET','your-bucket'), ], 'gcs' => [ 'driver' => 'gcs', 'key' => env('GCS_KEY', 'your-key'), 'secret' => env('GCS_SECRET', 'your-secret'), 'base_url' => env('GCS_BASE_URL','your-base-url'), 'bucket' => env('GCS_BUCKET','your-bucket'), ], 

Last but not least, you also need to add the provider to the list of providers in app.php

 'providers' => [ ... App\Providers\GcsAppsServiceProvider::class, ], 

The AWS S3 adapter also mentions that GCS supports it in its documentation .