Deleting cloud storage objects in batch mode

Question

What is the best way to delete many cloudstorage objects? I have a bucket that contains ~ 500K objects, and I would like to delete them all.

Should I make 1 api request for each object that I want to delete, or is there some kind of batch method? I am currently using gsutil to remove one at a time.

Answer 1

You need to make 1 api request for each object. The easiest way to accomplish this is with gsutil :

$ gsutil -m rm gs://bucket_with_many_objects/** 

The -m option enables multithreading, which will delete many objects in parallel.

Answer 2

Note that with gsutil, the wildcard "*" will only match top-level objects (up to the next "/" in the path name). If you want to delete all objects that you can use:

 $ gsutil -m rm -R gs://bucket_with_many_objects 

or

 $ gsutil -m rm gs://bucket_with_many_objects/** 

Mike Schwartz, Google Cloud Storage Team

Answer 3

I had a similar problem with a bucket containing more than 800,000 objects, the gsutil -m rm gs: // bucket-name method works, but takes a lot of time since it is essentially still deleting each object one at a time.

After we contacted the Cloud Storage team at Google, they pointed me towards the bucket life cycle rules, although they are not instantaneous, they allow me to delete objects more efficiently.

I wrote a blog post about removing full buckets using this method.