WagnerOne
2014-05-18 22:29:38 UTC
Hi,
During some of my initial S3 uploads using s3cmd, I didn't realize S3 behaved differently in terms of the hash it stores for multipart uploads compared to normal uploads.
I since bumped my s3cfg setting for multipart threshold well beyond any of my file sizes so multipart uploading doesn't engage.
However, with all the files that did go up with multipart enabled, s3cmd skips these files for md5 comparison on subsequent syncs - "INFO: disabled md5 check for filename"
What is being used to compare these files since s3cmd can't use the md5/Etag value? File size only?
I'm attempting to determine the best method to get usable md5/Etag values in this series of multipart uploaded files, but haven't figured out a way yet other than to delete them all and re-upload them with multipart effectively disabled.
I guess you used to be able to cp/mv the object over top of itself to fix this, but that isn't allowed anymore:
"ERROR: S3 error: 400 (InvalidRequest): This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes."
Thanks,
Mike
During some of my initial S3 uploads using s3cmd, I didn't realize S3 behaved differently in terms of the hash it stores for multipart uploads compared to normal uploads.
I since bumped my s3cfg setting for multipart threshold well beyond any of my file sizes so multipart uploading doesn't engage.
However, with all the files that did go up with multipart enabled, s3cmd skips these files for md5 comparison on subsequent syncs - "INFO: disabled md5 check for filename"
What is being used to compare these files since s3cmd can't use the md5/Etag value? File size only?
I'm attempting to determine the best method to get usable md5/Etag values in this series of multipart uploaded files, but haven't figured out a way yet other than to delete them all and re-upload them with multipart effectively disabled.
I guess you used to be able to cp/mv the object over top of itself to fix this, but that isn't allowed anymore:
"ERROR: S3 error: 400 (InvalidRequest): This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes."
Thanks,
Mike