Attila Megyeri
2015-01-20 17:57:54 UTC
Hi,
I have a few questions related to the s3 sync command:
- What is the approximate number of requests (LIST, PUT, COPY...) a sync operation would require, in case if the local directory to be synced would consist of approx. 40.000 files? Would it mean 40.000 operations? Or just 1 request? Perhaps as many requests as many subdirectories exist? Doesn't the md5 calculation use too much s3 requests?
- If I am not mistaken, encryption is not available when sync operation is used - and the reason for this is that the md5 would mismatch. Couldn't we have one of the following approaches:
o Ignore md5 check, and just sync the files if: they are changed (based on date/time, newly created or deleted)?
o Encrypt the local file temporarily just to have the right md5 value? I know that this is computation intensive, but that would be a logical tradeoff for someone needing this feature.
- Can you recommend any other approach to sync a large number of files to s3 in encrypted format? I would execute the operation once a week to sync the changes.
Thanks,
Attila
I have a few questions related to the s3 sync command:
- What is the approximate number of requests (LIST, PUT, COPY...) a sync operation would require, in case if the local directory to be synced would consist of approx. 40.000 files? Would it mean 40.000 operations? Or just 1 request? Perhaps as many requests as many subdirectories exist? Doesn't the md5 calculation use too much s3 requests?
- If I am not mistaken, encryption is not available when sync operation is used - and the reason for this is that the md5 would mismatch. Couldn't we have one of the following approaches:
o Ignore md5 check, and just sync the files if: they are changed (based on date/time, newly created or deleted)?
o Encrypt the local file temporarily just to have the right md5 value? I know that this is computation intensive, but that would be a logical tradeoff for someone needing this feature.
- Can you recommend any other approach to sync a large number of files to s3 in encrypted format? I would execute the operation once a week to sync the changes.
Thanks,
Attila