Right now, storelet uploads files to S3 as a single file. That means that there's a maximum limit of 5GB, and if there's a failure, it all needs to start again.
Perhaps implement multi-part upload to get around these issues? Preferably it should be sensible enough to use the best one for the job (multi-part over a certain size?), but be overridable.
An idea:
b.save_to_s3("bucket", "<access key>", "<secret key>", multi_part=MULTI_INTELLIGENT)
b.save_to_s3("bucket", "<access key>", "<secret key>", multi_part=MULTI_FORCE_ON)
b.save_to_s3("bucket", "<access key>", "<secret key>", multi_part=MULTI_FORCE_OFF)
Probably needs a little more thought as to organisation.
Right now, storelet uploads files to S3 as a single file. That means that there's a maximum limit of 5GB, and if there's a failure, it all needs to start again.
Perhaps implement multi-part upload to get around these issues? Preferably it should be sensible enough to use the best one for the job (multi-part over a certain size?), but be overridable.
An idea:
Probably needs a little more thought as to organisation.