Not sure if this has been looked into already, but the speed of transfers seems to be vastly improved by threads/processes running in parallel.
My experience with is is using dreamhost, it's possible the official s3 is different.
I noticed a 5x improvement in speed in calling boto-rsync on each of 5 large directories, rather than on the single parent directory. I assume an internal system for using multiple upload threads/queues would see similar improvements.
Not sure if this has been looked into already, but the speed of transfers seems to be vastly improved by threads/processes running in parallel.
My experience with is is using dreamhost, it's possible the official s3 is different.
I noticed a 5x improvement in speed in calling boto-rsync on each of 5 large directories, rather than on the single parent directory. I assume an internal system for using multiple upload threads/queues would see similar improvements.