TL;DR - The delete is the hard part. You can use S3 batch operations with an inventory list to do the copy very quickly. Alternatively, you can setup replication and "touch" each file using a self-copy CLI command, once the replication policy is in place. For the deletion, you're sort of stuck with lifecycle policies, which would take a day or so to clear out the old bucket, but could be supplemented with manual interaction, I'd imagine. There probably needs to be a better mechanic for completely wiping buckets. Last I checked there was not.
Yep. Had to delete a bucket with 100 million or so objects in it a while ago, with each having multiple versions. So it could have been a billion objects. It was something I needed to run every now and then (process to clear the production bucket while getting ready for the full cutover) and have it done in a few hours rather than wait for lifecycle policies to kick in.
S3 Batch Operations can also invoke a Lambda function per object, so it's straightforward enough to create a function that calls DeleteObject() and let Lambda scale out to silly levels.
Only, that's not actually deletion, nor would it have helped their business requirement:
> It is a 3rd party application that puts data into that origin bucket. They needed the bucket to be empty before the new version gets activated. And they wouldn't use another bucket. Something out of our control
- do this on a few threads per server to avoid being throttled
- which entails keeping track of which servers you're using (there are hundreds of servers available but dns only exposes a couple per second)
..and that assumes you've already got a list of keys - if you need to also enumerate the bucket you'll have to employ the same strategy and also partition your queries somehow to get multiple non-overlapping continuation-tokens for your concurrent calls to /?list. you can also request a bucket inventory but if you're in a hurry it's faster to do it yourself (for a 25TB bucket the turnaround on the inventory request is probably 2+ days)
