[S3tools-general] Patch for using Xattr for checksums

[Jay McCanta]

I'm unsure of the proper protocol for proposing patches for the group. My apologies if this isn't the correct way.

We keep the md5 checksum of files in an extended attribute. When running s3cmd with --check-md5, this patch allow one to use the --xattr option and specify the name of the extended attribute to use for local md5sums. If the attribute is not there, a regular check is done, so no changes to existing procedures would be necessary. For us, the drastically improves check time since we are not re-reading every local file recalculating every md5sum.

Also there was a typo in s3cmd for the 'help' field of --acl-revoke option definition. It's corrected in this patch.

diff -ru s3cmd-1.5.0-alpha3/s3cmd s3cmd-1.5.0-alpha3.jtm/s3cmd
--- s3cmd-1.5.0-alpha3/s3cmd 2013-03-10 17:06:33.000000000 -0700
+++ s3cmd-1.5.0-alpha3.jtm/s3cmd 2013-05-12 15:20:44.013584123 -0700
@@ -32,6 +32,13 @@
from logging import debug, info, warning, error
from distutils.spawn import find_executable

+try:
+ hasXattr = False
+ import xattr
+ hasXattr = True
+except:
+ pass
+
def output(message):
sys.stdout.write(message + "\n")
sys.stdout.flush()
@@ -1141,7 +1148,7 @@

def _invalidate_on_cf(destination_base_uri):
cf = CloudFront(cfg)
- default_index_file = None
+ efault_index_file = None
if cfg.invalidate_default_index_on_cf or cfg.invalidate_default_index_root_on_cf:
info_response = s3.website_info(destination_base_uri, cfg.bucket_location)
if info_response:
@@ -1507,9 +1514,9 @@
ret_enc = gpg_encrypt(filename)
ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2], False)
hash = [
- Utils.hash_file_md5(filename),
- Utils.hash_file_md5(ret_enc[1]),
- Utils.hash_file_md5(ret_dec[1]),
+ Utils.hash_file_md5(filename, options.xattr),
+ Utils.hash_file_md5(ret_enc[1], options.xattr),
+ Utils.hash_file_md5(ret_dec[1], options.xattr),
]
os.unlink(filename)
os.unlink(ret_enc[1])
@@ -1762,11 +1769,13 @@
optparser.add_option( "--skip-existing", dest="skip_existing", action="store_true", help="Skip over files that exist at the destination (only for [get] and [sync] commands).")
optparser.add_option("-r", "--recursive", dest="recursive", action="store_true", help="Recursive upload, download or removal.")
optparser.add_option( "--check-md5", dest="check_md5", action="store_true", help="Check MD5 sums when comparing files for [sync]. (default)")
+ if hasXattr:
+ optparser.add_option ("--xattr", dest="xattr", action="store", help="If possible, use extended file attribute named (default:%default) instead of calculating it [sync].")
optparser.add_option( "--no-check-md5", dest="check_md5", action="store_false", help="Do not check MD5 sums when comparing files for [sync]. Only size will be compared. May significantly speed up transfer but may also miss some changed files.")
optparser.add_option("-P", "--acl-public", dest="acl_public", action="store_true", help="Store objects with ACL allowing read for anyone.")
optparser.add_option( "--acl-private", dest="acl_public", action="store_false", help="Store objects with default ACL allowing access for you only.")
optparser.add_option( "--acl-grant", dest="acl_grants", type="s3acl", action="append", metavar="PERMISSION:EMAIL or USER_CANONICAL_ID", help="Grant stated permission to a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all")
- optparser.add_option( "--acl-revoke", dest="acl_revokes", type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", help="Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, wr ite_acp, full_control, all")
+ optparser.add_option( "--acl-revoke", dest="acl_revokes", type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", help="Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all")

optparser.add_option( "--delete-removed", dest="delete_removed", action="store_true", help="Delete remote objects with no corresponding local file [sync]")
optparser.add_option( "--no-delete-removed", dest="delete_removed", action="store_false", help="Don't delete remote objects.")
Only in s3cmd-1.5.0-alpha3.jtm: s3cmd.orig


Jay McCanta | Senior Unix Administrator | F5 Networks, Inc.

[j***@enlyton.com]

Is there a typo (accidental one-char deletion) in the code block below?

J. Merrill

-----Original Message-----
From: "Jay McCanta" <***@F5.com>
Sent: Sunday, May 12, 2013 7:36pm
To: "s3tools-***@lists.sourceforge.net" <s3tools-***@lists.sourceforge.net>
Subject: [S3tools-general] Patch for using Xattr for checksums
[snip]
cf = CloudFront(cfg)


- default_index_file = None
+ efault_index_file = None
[snip]

[Jay McCanta]

My cat helped with that first version (rule #1: blame the firewall, rule #2: blame the cat. It was incomplete and included a typo. The reply to that has the correct stuff.

Jay McCanta
F5 Networks, Inc.

From: ***@enlyton.com [mailto:***@enlyton.com]
Sent: Tuesday, May 14, 2013 1:50 PM
To: s3tools-***@lists.sourceforge.net
Subject: Re: [S3tools-general] Patch for using Xattr for checksums


Is there a typo (accidental one-char deletion) in the code block below?



J. Merrill



-----Original Message-----

From: "Jay McCanta" <***@F5.com<mailto:***@F5.com>>
Sent: Sunday, May 12, 2013 7:36pm
To: "s3tools-***@lists.sourceforge.net<mailto:s3tools-***@lists.sourceforge.net>" <s3tools-***@lists.sourceforge.net<mailto:s3tools-***@lists.sourceforge.net>>
Subject: [S3tools-general] Patch for using Xattr for checksums
[snip]

cf = CloudFront(cfg)
- default_index_file = None
+ efault_index_file = None
[snip]

[Jay McCanta]

My apologies for cutting/pasting/posting without proofing. 

This patch allows s3cmd to utilize md5sums stored in the extended attributes of files.  If for any reason, that doesn’t work, the usual md5sum mechanisms are used.  This option only has impact if checkmd5 is enabled.
The last patch block corrects a typo in the s3cmd ‘--acl-revoke’ option help text.

diff -ru s3cmd-1.5.0-alpha3.orig/S3/Utils.py s3cmd-1.5.0-alpha3/S3/Utils.py
--- s3cmd-1.5.0-alpha3.orig/S3/Utils.py 2013-03-10 15:35:09.000000000 -0700
+++ s3cmd-1.5.0-alpha3/S3/Utils.py      2013-05-13 14:13:22.083721474 -0700
@@ -35,6 +35,12 @@
     import elementtree.ElementTree as ET
from xml.parsers.expat import ExpatError

+try:
+    import xattr
+    hasXattr = True
+except:
+    hasXattr = False
+
__all__ = []
def parseNodes(nodes):
     ## WARNING: Ignores text nodes from mixed xml/text.
@@ -226,7 +232,14 @@
     return mktmpsomething(prefix, randchars, createfunc)
__all__.append("mktmpfile")

-def hash_file_md5(filename):
+def hash_file_md5(filename, xattrName=None):
+    if hasXattr and xattrName is not None:
+        try:
+            md5sum = xattr.get(filename, xattrName)
+            return md5sum
+        except:
+            pass
+
     h = md5()
     f = open(filename, "rb")
     while True:
diff -ru s3cmd-1.5.0-alpha3.orig/s3cmd s3cmd-1.5.0-alpha3/s3cmd
--- s3cmd-1.5.0-alpha3.orig/s3cmd       2013-03-10 17:06:33.000000000 -0700
+++ s3cmd-1.5.0-alpha3/s3cmd    2013-05-13 14:10:33.927609005 -0700
@@ -32,6 +32,12 @@
from logging import debug, info, warning, error
from distutils.spawn import find_executable

+try:
+    import xattr
+    hasXattr = True
+except:
+    hasXattr = False
+
def output(message):
     sys.stdout.write(message + "\n")
     sys.stdout.flush()
@@ -1507,9 +1513,9 @@
                         ret_enc = gpg_encrypt(filename)
                         ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2], False)
                         hash = [
-                            Utils.hash_file_md5(filename),
-                            Utils.hash_file_md5(ret_enc[1]),
-                            Utils.hash_file_md5(ret_dec[1]),
+                            Utils.hash_file_md5(filename, options.xattr),
+                            Utils.hash_file_md5(ret_enc[1], options.xattr),
+                            Utils.hash_file_md5(ret_dec[1], options.xattr),
                         ]
                         os.unlink(filename)
                         os.unlink(ret_enc[1])
@@ -1762,11 +1768,13 @@
     optparser.add_option(      "--skip-existing", dest="skip_existing", action="store_true", help="Skip over files that exist at the destination (only for [get] and [sync] commands).")
     optparser.add_option("-r", "--recursive", dest="recursive", action="store_true", help="Recursive upload, download or removal.")
     optparser.add_option(      "--check-md5", dest="check_md5", action="store_true", help="Check MD5 sums when comparing files for [sync]. (default)")
+    if hasXattr:
+        optparser.add_option  ("--xattr", dest="xattr", action="store", help="If possible, use extended file attribute named (default:%default) instead of calculating it [sync].")
     optparser.add_option(      "--no-check-md5", dest="check_md5", action="store_false", help="Do not check MD5 sums when comparing files for [sync]. Only size will be compared. May significantly speed up transfer but may also miss some changed files.")
     optparser.add_option("-P", "--acl-public", dest="acl_public", action="store_true", help="Store objects with ACL allowing read for anyone.")
     optparser.add_option(      "--acl-private", dest="acl_public", action="store_false", help="Store objects with default ACL allowing access for you only.")
     optparser.add_option(      "--acl-grant", dest="acl_grants", type="s3acl", action="append", metavar="PERMISSION:EMAIL or USER_CANONICAL_ID", help="Grant stated permission to a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all")
-    optparser.add_option(      "--acl-revoke", dest="acl_revokes", type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", help="Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, wr     ite_acp, full_control, all")
+    optparser.add_option(      "--acl-revoke", dest="acl_revokes", type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", help="Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all")

     optparser.add_option(      "--delete-removed", dest="delete_removed", action="store_true", help="Delete remote objects with no corresponding local file [sync]")
     optparser.add_option(      "--no-delete-removed", dest="delete_removed", action="store_false", help="Don't delete remote

[Matt Domsch]

Jay - thanks for the patch. I've made some stylistic changes, as well as
simplifies testing for xattr
module import success, passes the result into the Config object,
and ensures the correct namespace is used. Also added a test to the
test
suite, and the entry on the manpage.

I gave you credit in the changelog of course.

I've pushed this to my xattr branch, and into my merge branch as well, and
have filed a pull request for same. I've tested this with the test suite
and with manual tests to be sure it works.
https://github.com/mdomsch/s3cmd/tree/merge

Thanks much for the idea!

Post by Jay McCanta
My apologies for cutting/pasting/posting without proofing.
This patch allows s3cmd to utilize md5sums stored in the extended
attributes of files. If for any reason, that doesn’t work, the usual
md5sum mechanisms are used. This option only has impact if check md5 is
enabled.
The last patch block corrects a typo in the s3cmd ‘--acl-revoke’ option
help text.
diff -ru s3cmd-1.5.0-alpha3.orig/S3/Utils.py s3cmd-1.5.0-alpha3/S3/Utils.py
--- s3cmd-1.5.0-alpha3.orig/S3/Utils.py 2013-03-10 15:35:09.000000000 -0700
+++ s3cmd-1.5.0-alpha3/S3/Utils.py 2013-05-13 14:13:22.083721474 -0700
@@ -35,6 +35,12 @@
import elementtree.ElementTree as ET
from xml.parsers.expat import ExpatError
+ import xattr
+ hasXattr = True
+ hasXattr = False
+
__all__ = []
## WARNING: Ignores text nodes from mixed xml/text.
@@ -226,7 +232,14 @@
return mktmpsomething(prefix, randchars, createfunc)
__all__.append("mktmpfile")
+ md5sum = xattr.get(filename, xattrName)
+ return md5sum
+ pass
+
h = md5()
f = open(filename, "rb")
diff -ru s3cmd-1.5.0-alpha3.orig/s3cmd s3cmd-1.5.0-alpha3/s3cmd
--- s3cmd-1.5.0-alpha3.orig/s3cmd 2013-03-10 17:06:33.000000000 -0700
+++ s3cmd-1.5.0-alpha3/s3cmd 2013-05-13 14:10:33.927609005 -0700
@@ -32,6 +32,12 @@
from logging import debug, info, warning, error
from distutils.spawn import find_executable
+ import xattr
+ hasXattr = True
+ hasXattr = False
+
sys.stdout.write(message + "\n")
sys.stdout.flush()
@@ -1507,9 +1513,9 @@
ret_enc = gpg_encrypt(filename)
ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2], False)
hash = [
- Utils.hash_file_md5(filename),
- Utils.hash_file_md5(ret_enc[1]),
- Utils.hash_file_md5(ret_dec[1]),
+ Utils.hash_file_md5(filename, options.xattr),
+ Utils.hash_file_md5(ret_enc[1],
options.xattr),
+ Utils.hash_file_md5(ret_dec[1],
options.xattr),
]
os.unlink(filename)
os.unlink(ret_enc[1])
@@ -1762,11 +1768,13 @@
optparser.add_option( "--skip-existing", dest="skip_existing",
action="store_true", help="Skip over files that exist at the destination
(only for [get] and [sync] commands).")
optparser.add_option("-r", "--recursive", dest="recursive",
action="store_true", help="Recursive upload, download or removal.")
optparser.add_option( "--check-md5", dest="check_md5",
action="store_true", help="Check MD5 sums when comparing files for [sync].
(default)")
+ optparser.add_option ("--xattr", dest="xattr", action="store",
help="If possible, use extended file attribute named (default:%default)
instead of calculating it [sync].")
optparser.add_option( "--no-check-md5", dest="check_md5",
action="store_false", help="Do not check MD5 sums when comparing files for
[sync]. Only size will be compared. May significantly speed up transfer but
may also miss some changed files.")
optparser.add_option("-P", "--acl-public", dest="acl_public",
action="store_true", help="Store objects with ACL allowing read for
anyone.")
optparser.add_option( "--acl-private", dest="acl_public",
action="store_false", help="Store objects with default ACL allowing access
for you only.")
optparser.add_option( "--acl-grant", dest="acl_grants",
type="s3acl", action="append", metavar="PERMISSION:EMAIL or
USER_CANONICAL_ID", help="Grant stated permission to a given amazon user.
Permission is one of: read, write, read_acp, write_acp, full_control, all")
- optparser.add_option( "--acl-revoke", dest="acl_revokes",
type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID",
help="Revoke stated permission for a given amazon user. Permission is one
of: read, write, read_acp, wr ite_acp, full_control, all")
+ optparser.add_option( "--acl-revoke", dest="acl_revokes",
type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID",
help="Revoke stated permission for a given amazon user. Permission is one
of: read, write, read_acp, write_acp, full_control, all")
optparser.add_option( "--delete-removed", dest="delete_removed",
action="store_true", help="Delete remote objects with no corresponding
local file [sync]")
optparser.add_option( "--no-delete-removed",
dest="delete_removed", action="store_false", help="Don't delete remote
objects.")
------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d
_______________________________________________
S3tools-general mailing list
https://lists.sourceforge.net/lists/listinfo/s3tools-general

[Matt Domsch]

In talking with Michal last night, he wants a different approach to the
xattr work, based on using xattrs as alternate storage for what's currently
stored externally in HashCache pickles, but offering the same kind of
protection (e.g. detection of file changes which cause the md5 to be
recalculated). If someone wants to take a stab at that approach, great.
Until then, the xattr patches from this thread have been reverted on the
upstream master branch. Use the --cache-file=FILE option for similar
functionality.

Thanks,
Matt

Post by Matt Domsch
Jay - thanks for the patch. I've made some stylistic changes, as well as
simplifies testing for xattr
module import success, passes the result into the Config object,
and ensures the correct namespace is used. Also added a test to the
test
suite, and the entry on the manpage.
I gave you credit in the changelog of course.
I've pushed this to my xattr branch, and into my merge branch as well, and
have filed a pull request for same. I've tested this with the test suite
and with manual tests to be sure it works.
https://github.com/mdomsch/s3cmd/tree/merge
Thanks much for the idea!

Post by Jay McCanta
My apologies for cutting/pasting/posting without proofing.
This patch allows s3cmd to utilize md5sums stored in the extended
attributes of files. If for any reason, that doesn’t work, the usual
md5sum mechanisms are used. This option only has impact if check md5 is
enabled.
The last patch block corrects a typo in the s3cmd ‘--acl-revoke’ option
help text.
diff -ru s3cmd-1.5.0-alpha3.orig/S3/Utils.py
s3cmd-1.5.0-alpha3/S3/Utils.py
--- s3cmd-1.5.0-alpha3.orig/S3/Utils.py 2013-03-10 15:35:09.000000000 -0700
+++ s3cmd-1.5.0-alpha3/S3/Utils.py 2013-05-13 14:13:22.083721474 -0700
@@ -35,6 +35,12 @@
import elementtree.ElementTree as ET
from xml.parsers.expat import ExpatError
+ import xattr
+ hasXattr = True
+ hasXattr = False
+
__all__ = []
## WARNING: Ignores text nodes from mixed xml/text.
@@ -226,7 +232,14 @@
return mktmpsomething(prefix, randchars, createfunc)
__all__.append("mktmpfile")
+ md5sum = xattr.get(filename, xattrName)
+ return md5sum
+ pass
+
h = md5()
f = open(filename, "rb")
diff -ru s3cmd-1.5.0-alpha3.orig/s3cmd s3cmd-1.5.0-alpha3/s3cmd
--- s3cmd-1.5.0-alpha3.orig/s3cmd 2013-03-10 17:06:33.000000000 -0700
+++ s3cmd-1.5.0-alpha3/s3cmd 2013-05-13 14:10:33.927609005 -0700
@@ -32,6 +32,12 @@
from logging import debug, info, warning, error
from distutils.spawn import find_executable
+ import xattr
+ hasXattr = True
+ hasXattr = False
+
sys.stdout.write(message + "\n")
sys.stdout.flush()
@@ -1507,9 +1513,9 @@
ret_enc = gpg_encrypt(filename)
ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2], False)
hash = [
- Utils.hash_file_md5(filename),
- Utils.hash_file_md5(ret_enc[1]),
- Utils.hash_file_md5(ret_dec[1]),
+ Utils.hash_file_md5(filename, options.xattr),
+ Utils.hash_file_md5(ret_enc[1],
options.xattr),
+ Utils.hash_file_md5(ret_dec[1],
options.xattr),
]
os.unlink(filename)
os.unlink(ret_enc[1])
@@ -1762,11 +1768,13 @@
optparser.add_option( "--skip-existing", dest="skip_existing",
action="store_true", help="Skip over files that exist at the destination
(only for [get] and [sync] commands).")
optparser.add_option("-r", "--recursive", dest="recursive",
action="store_true", help="Recursive upload, download or removal.")
optparser.add_option( "--check-md5", dest="check_md5",
action="store_true", help="Check MD5 sums when comparing files for [sync].
(default)")
+ optparser.add_option ("--xattr", dest="xattr", action="store",
help="If possible, use extended file attribute named (default:%default)
instead of calculating it [sync].")
optparser.add_option( "--no-check-md5", dest="check_md5",
action="store_false", help="Do not check MD5 sums when comparing files for
[sync]. Only size will be compared. May significantly speed up transfer but
may also miss some changed files.")
optparser.add_option("-P", "--acl-public", dest="acl_public",
action="store_true", help="Store objects with ACL allowing read for
anyone.")
optparser.add_option( "--acl-private", dest="acl_public",
action="store_false", help="Store objects with default ACL allowing access
for you only.")
optparser.add_option( "--acl-grant", dest="acl_grants",
type="s3acl", action="append", metavar="PERMISSION:EMAIL or
USER_CANONICAL_ID", help="Grant stated permission to a given amazon user.
Permission is one of: read, write, read_acp, write_acp, full_control, all")
- optparser.add_option( "--acl-revoke", dest="acl_revokes",
type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID",
help="Revoke stated permission for a given amazon user. Permission is one
of: read, write, read_acp, wr ite_acp, full_control, all")
+ optparser.add_option( "--acl-revoke", dest="acl_revokes",
type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID",
help="Revoke stated permission for a given amazon user. Permission is one
of: read, write, read_acp, write_acp, full_control, all")
optparser.add_option( "--delete-removed",
dest="delete_removed", action="store_true", help="Delete remote objects
with no corresponding local file [sync]")
optparser.add_option( "--no-delete-removed",
dest="delete_removed", action="store_false", help="Don't delete remote
objects.")
------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d
_______________________________________________
S3tools-general mailing list
https://lists.sourceforge.net/lists/listinfo/s3tools-general

[Matt Domsch]

I’m unsure of the proper protocol for proposing patches for the
group. My apologies if this isn’t the correct way.

Thanks for the patch Jay.

Patches by email is fine, though pull requests filed in github to pull
from a specific bug or feature branch in your own github-forked
repository also works quite well, and makes it harder to forget over
time as the pull request remains open until merged or closed
manually. :-)

We keep the md5 checksum of files in an extended attribute. When
running s3cmd with --check-md5, this patch allow one to use the
--xattr option and specify the name of the extended attribute to use
for local md5sums. If the attribute is not there, a regular check
is done, so no changes to existing procedures would be necessary.
For us, the drastically improves check time since we are not
re-reading every local file recalculating every md5sum.

Cool idea. What are you using to put such md5sums into the extended
attribute already?

Also there was a typo in s3cmd for the ‘help’ field of --acl-revoke
option definition. It’s corrected in this patch.

Ideally that would be split into a second patch.

diff -ru s3cmd-1.5.0-alpha3/s3cmd s3cmd-1.5.0-alpha3.jtm/s3cmd
--- s3cmd-1.5.0-alpha3/s3cmd 2013-03-10 17:06:33.000000000 -0700
+++ s3cmd-1.5.0-alpha3.jtm/s3cmd 2013-05-12 15:20:44.013584123 -0700
@@ -32,6 +32,13 @@
from logging import debug, info, warning, error
from distutils.spawn import find_executable
+ hasXattr = False
+ import xattr
+ hasXattr = True
+ pass
+

This can be shortened to:

try: import xattr
except: pass
...
if 'xattr' in sys.modules.keys():
optparser.add_option ("--xattr", dest="xattr", action="store", help="If possible, use extended file attribute named (default:%default) instead of calculating it [sync].")

This should be default=False, action="store_true". That way we know
it exists later.

sys.stdout.write(message + "\n")
sys.stdout.flush()
@@ -1141,7 +1148,7 @@
cf = CloudFront(cfg)
- default_index_file = None
+ efault_index_file = None

keystroke typo?

info_response = s3.website_info(destination_base_uri, cfg.bucket_location)
@@ -1507,9 +1514,9 @@
ret_enc = gpg_encrypt(filename)
ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2], False)
hash = [
- Utils.hash_file_md5(filename),
- Utils.hash_file_md5(ret_enc[1]),
- Utils.hash_file_md5(ret_dec[1]),
+ Utils.hash_file_md5(filename, options.xattr),
+ Utils.hash_file_md5(ret_enc[1], options.xattr),
+ Utils.hash_file_md5(ret_dec[1], options.xattr),

What's the corresponding change to S3/Utils.py please?

Given that this should also be a Config option, not just a command
line option, please stick it into the Config object, to be retrieved
in Utils.py.

Thanks,
Matt

[Matt Domsch]

Please ignore. It seems a mail server at work had a historical hiccup and
is resending mail sent in May.

Thanks,
Matt

Post by Matt Domsch

I’m unsure of the proper protocol for proposing patches for the
group. My apologies if this isn’t the correct way.

Thanks for the patch Jay.
Patches by email is fine, though pull requests filed in github to pull
from a specific bug or feature branch in your own github-forked
repository also works quite well, and makes it harder to forget over
time as the pull request remains open until merged or closed
manually. :-)

We keep the md5 checksum of files in an extended attribute. When
running s3cmd with --check-md5, this patch allow one to use the
--xattr option and specify the name of the extended attribute to use
for local md5sums. If the attribute is not there, a regular check
is done, so no changes to existing procedures would be necessary.
For us, the drastically improves check time since we are not
re-reading every local file recalculating every md5sum.

Cool idea. What are you using to put such md5sums into the extended
attribute already?

Also there was a typo in s3cmd for the ‘help’ field of --acl-revoke
option definition. It’s corrected in this patch.

Ideally that would be split into a second patch.

diff -ru s3cmd-1.5.0-alpha3/s3cmd s3cmd-1.5.0-alpha3.jtm/s3cmd
--- s3cmd-1.5.0-alpha3/s3cmd 2013-03-10 17:06:33.000000000 -0700
+++ s3cmd-1.5.0-alpha3.jtm/s3cmd 2013-05-12 15:20:44.013584123

-0700

@@ -32,6 +32,13 @@
from logging import debug, info, warning, error
from distutils.spawn import find_executable
+ hasXattr = False
+ import xattr
+ hasXattr = True
+ pass
+

try: import xattr
except: pass
...
optparser.add_option ("--xattr", dest="xattr", action="store",
help="If possible, use extended file attribute named (default:%default)
instead of calculating it [sync].")
This should be default=False, action="store_true". That way we know
it exists later.

sys.stdout.write(message + "\n")
sys.stdout.flush()
@@ -1141,7 +1148,7 @@
cf = CloudFront(cfg)
- default_index_file = None
+ efault_index_file = None

keystroke typo?

if cfg.invalidate_default_index_on_cf or
info_response = s3.website_info(destination_base_uri,

cfg.bucket_location)

@@ -1507,9 +1514,9 @@
ret_enc = gpg_encrypt(filename)
ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2],

False)

hash = [
- Utils.hash_file_md5(filename),
- Utils.hash_file_md5(ret_enc[1]),
- Utils.hash_file_md5(ret_dec[1]),
+ Utils.hash_file_md5(filename,

options.xattr),

+ Utils.hash_file_md5(ret_enc[1],

options.xattr),

+ Utils.hash_file_md5(ret_dec[1],

options.xattr),
What's the corresponding change to S3/Utils.py please?
Given that this should also be a Config option, not just a command
line option, please stick it into the Config object, to be retrieved
in Utils.py.
Thanks,
Matt
--
Matt Domsch
Technology Strategist
Dell | Office of the CTO
------------------------------------------------------------------------------
Sponsored by Intel(R) XDK
Develop, test and display web and hybrid apps with a single code base.
Download it for free now!
http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
_______________________________________________
S3tools-general mailing list
https://lists.sourceforge.net/lists/listinfo/s3tools-general