The source bucket had a ton of small files and most of the time was spent making network requests (O(<number of files>)). The suggested solution was to do a batch operation (open a ticket for a batch operation). Would rclone have helped in this scenario?
Yep, as mentioned in the other comment, with configurable `--transfers` & `--checkers`, the only bottleneck is the amount of CPU you have available and can allocate.
> the only bottleneck is the amount of CPU you have available and can allocate.
In this case S3 will be the bottleneck. 25 TB will have around 700M files (assuming it's as dense a linux installation). S3 can do 5000 operations per second. It'll take at least 40 hrs (assuming everything works at peak speed during the whole process).
