[S3tools-general] s3cmd put: repeats first upload

Discussion:

Russell Gadd

2015-04-06 09:40:09 UTC

I was testing s3cmd and it appears to repeat the first upload of put
command. There appears to be a delay between the confirmations on the
terminal so I expect it is actually uploading twice rather than just a
reporting error

This was the terminal output (Linux Mint 17, s3cmd version 1.5.2):

/data/WorkInProgress/testfiles $ s3cmd put * s3://mybucket-test2/
testfile1.pdf -> s3://mybucket-test2/testfile1.pdf [1 of 2]
153171 of 153171 100% in 1s 134.03 kB/s done
testfile1.pdf -> s3://mybucket-test2/testfile1.pdf [1 of 2]
153171 of 153171 100% in 1s 109.50 kB/s done
testfile2.xls -> s3://mybucket-test2/testfile2.xls [2 of 2]
62464 of 62464 100% in 0s 120.24 kB/s done
/data/WorkInProgress/testfiles $

Is there an explanation for this behaviour or is this a bug?

Russell

Matt Domsch

2015-04-06 11:43:19 UTC

Permalink

Is this amazon S3 or a clone? If a clone, this is expected as it falls
back from signature v4 to signature v2. Use --signature-v2 to avoid it.

Post by Russell Gadd
I was testing s3cmd and it appears to repeat the first upload of put
command. There appears to be a delay between the confirmations on the
terminal so I expect it is actually uploading twice rather than just a
reporting error
/data/WorkInProgress/testfiles $ s3cmd put * s3://mybucket-test2/
testfile1.pdf -> s3://mybucket-test2/testfile1.pdf [1 of 2]
153171 of 153171 100% in 1s 134.03 kB/s done
testfile1.pdf -> s3://mybucket-test2/testfile1.pdf [1 of 2]
153171 of 153171 100% in 1s 109.50 kB/s done
testfile2.xls -> s3://mybucket-test2/testfile2.xls [2 of 2]
62464 of 62464 100% in 0s 120.24 kB/s done
/data/WorkInProgress/testfiles $
Is there an explanation for this behaviour or is this a bug?
Russell
------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live
exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual-
event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
S3tools-general mailing list
https://lists.sourceforge.net/lists/listinfo/s3tools-general

Russell Gadd

2015-04-06 19:07:42 UTC

Permalink

It's Amazon s3.

Regards
Russell

Post by Matt Domsch
Is this amazon S3 or a clone? If a clone, this is expected as it falls
back from signature v4 to signature v2. Use --signature-v2 to avoid it.

------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live
exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual-
event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
S3tools-general mailing list
https://lists.sourceforge.net/lists/listinfo/s3tools-general

Jeremy Wadsack

2015-04-07 16:52:32 UTC

Permalink

I just upgraded locally (Mac) from 1.5.0-rc1 to 1.5.2 and see a double
output on get command using Amazon S3:

$ s3cmd get s3://bucket-name/2K12-02-u_ex140131.log
s3://bucket-name/2K12-02-u_ex140131.log -> ./2K12-02-u_ex140131.log [1 of
1]
s3://bucket-name/2K12-02-u_ex140131.log -> ./2K12-02-u_ex140131.log [1 of
1]
345829 of 345829 100% in 1s 336.48 kB/s done

Jeremy Wadsack

Post by Russell Gadd
It's Amazon s3.
Regards
Russell

Post by Matt Domsch
Is this amazon S3 or a clone? If a clone, this is expected as it falls
back from signature v4 to signature v2. Use --signature-v2 to avoid it.

Matt Domsch

2015-04-07 17:14:54 UTC

Permalink

Jeremy and/or Russell, can you run with --debug and see where the second
upload is coming from please?

Thanks,
Matt

Post by Jeremy Wadsack
I just upgraded locally (Mac) from 1.5.0-rc1 to 1.5.2 and see a double
$ s3cmd get s3://bucket-name/2K12-02-u_ex140131.log
s3://bucket-name/2K12-02-u_ex140131.log -> ./2K12-02-u_ex140131.log [1 of
1]
s3://bucket-name/2K12-02-u_ex140131.log -> ./2K12-02-u_ex140131.log [1 of
1]
345829 of 345829 100% in 1s 336.48 kB/s done
Jeremy Wadsack

Post by Russell Gadd
It's Amazon s3.
Regards
Russell

Post by Matt Domsch
Is this amazon S3 or a clone? If a clone, this is expected as it falls
back from signature v4 to signature v2. Use --signature-v2 to avoid it.

Jeremy Wadsack

2015-04-07 17:25:45 UTC

Permalink

It looks like the first request is returning 400 and then it falls back to
v2:

DEBUG: format_uri(): /2K12-02-u_ex140131.log
DEBUG: Response: {'status': 400, 'headers': {'x-amz-id-2':
'iX...J1ec/vu4VU=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked',
'connection': 'close', 'x-amz-request-id': 'FB...17', 'date': 'Tue, 07 Apr
2015 17:18:33 GMT', 'content-type': 'application/xml'}, 'reason': 'Bad
Request'}
DEBUG: Falling back to signature v2
DEBUG: Using signature v2
...
DEBUG: format_uri(): /2K12-02-u_ex140131.log
DEBUG: Response: {'status': 200, 'headers': {'content-length': '345829',
'x-amz-id-2': 'sb...3m8=', 'accept-ranges': 'bytes', 'x-amz-expiration':
'expiry-date="Wed, 22 Apr 2015 00:00:00 GMT", rule-id="Delete all files 14
days old or older"', 'last-modified': 'Tue, 07 Apr 2015 10:04:09 GMT',
'x-amz-request-id': 'D7...7E', 'etag': '"12...d53"', 'date': 'Tue, 07 Apr
2015 17:18:34 GMT', 'server': 'AmazonS3', 'content-type':
'application/octet-stream'}, 'reason': 'OK'}

I redacted a bunch of stuff but let me know what else you need to figure
this out.

Jeremy Wadsack

Post by Matt Domsch
Jeremy and/or Russell, can you run with --debug and see where the second
upload is coming from please?
Thanks,
Matt

Post by Jeremy Wadsack
I just upgraded locally (Mac) from 1.5.0-rc1 to 1.5.2 and see a double
$ s3cmd get s3://bucket-name/2K12-02-u_ex140131.log
s3://bucket-name/2K12-02-u_ex140131.log -> ./2K12-02-u_ex140131.log [1
of 1]
s3://bucket-name/2K12-02-u_ex140131.log -> ./2K12-02-u_ex140131.log [1
of 1]
345829 of 345829 100% in 1s 336.48 kB/s done
Jeremy Wadsack

Post by Russell Gadd
It's Amazon s3.
Regards
Russell

Post by Matt Domsch
Is this amazon S3 or a clone? If a clone, this is expected as it falls
back from signature v4 to signature v2. Use --signature-v2 to avoid it.

Matt Domsch

2015-04-07 17:32:49 UTC

Permalink

Bugger. That's the v4-signature-failure that several people have reported
but I haven't been able to reproduce or explain myself. :-( Can you post
the V4 signature bits (redacting your access key) which may help figure out
the failure?

Thanks,
Matt

Post by Jeremy Wadsack
It looks like the first request is returning 400 and then it falls back to
DEBUG: format_uri(): /2K12-02-u_ex140131.log
'iX...J1ec/vu4VU=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked',
'connection': 'close', 'x-amz-request-id': 'FB...17', 'date': 'Tue, 07 Apr
2015 17:18:33 GMT', 'content-type': 'application/xml'}, 'reason': 'Bad
Request'}
DEBUG: Falling back to signature v2
DEBUG: Using signature v2
...
DEBUG: format_uri(): /2K12-02-u_ex140131.log
DEBUG: Response: {'status': 200, 'headers': {'content-length': '345829',
'expiry-date="Wed, 22 Apr 2015 00:00:00 GMT", rule-id="Delete all files 14
days old or older"', 'last-modified': 'Tue, 07 Apr 2015 10:04:09 GMT',
'x-amz-request-id': 'D7...7E', 'etag': '"12...d53"', 'date': 'Tue, 07 Apr
'application/octet-stream'}, 'reason': 'OK'}
I redacted a bunch of stuff but let me know what else you need to figure
this out.
Jeremy Wadsack

Post by Matt Domsch
Jeremy and/or Russell, can you run with --debug and see where the second
upload is coming from please?
Thanks,
Matt

Post by Jeremy Wadsack
I just upgraded locally (Mac) from 1.5.0-rc1 to 1.5.2 and see a double
$ s3cmd get s3://bucket-name/2K12-02-u_ex140131.log
s3://bucket-name/2K12-02-u_ex140131.log -> ./2K12-02-u_ex140131.log [1
of 1]
s3://bucket-name/2K12-02-u_ex140131.log -> ./2K12-02-u_ex140131.log [1
of 1]
345829 of 345829 100% in 1s 336.48 kB/s done
Jeremy Wadsack

Post by Russell Gadd
It's Amazon s3.
Regards
Russell

Post by Matt Domsch
Is this amazon S3 or a clone? If a clone, this is expected as it
falls back from signature v4 to signature v2. Use --signature-v2 to avoid
it.

Jeremy Wadsack

2015-04-07 18:05:41 UTC

Permalink

Is this what you need?

DEBUG: signature-v4 headers: {'x-amz-content-sha256':
'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
'Authorization': 'AWS4-HMAC-SHA256
Credential=AK...A/20150407/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=128c7ce28096663f1be7d7bddaddba2ec784d3ab7ca6acadc6a333d01d16d64d',
'x-amz-date': '20150407T171824Z'}
INFO: Receiving file './NTMAITW2K12-02-u_ex140131.log', please wait...
DEBUG: get_hostname(bucket-name): bucket-name.s3.amazonaws.com
DEBUG: ConnMan.get(): creating new connection:
https://bucket-name.s3.amazonaws.com
DEBUG: Using ca_certs_file None
DEBUG: non-proxied HTTPSConnection(bucket-name.s3.amazonaws.com)
DEBUG: format_uri(): /NTMAITW2K12-02-u_ex140131.log
DEBUG: Response: {'status': 400, 'headers': {'x-amz-id-2':
'4N2pio5Ue9/6d7UPKxfZ/fGyft6rOe0IgBSMerKlCRLp+GOHGRtfalHGAF2I9iMEns5rkEWAMas=',
'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection':
'close', 'x-amz-request-id': 'DBD11555E34A9490', 'date': 'Tue, 07 Apr 2015
17:18:23 GMT', 'content-type': 'application/xml'}, 'reason': 'Bad Request'}
DEBUG: Falling back to signature v2
DEBUG: Using signature v2
DEBUG: SignHeaders:
'GET\n\n\n\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:Tue,
07 Apr 2015 17:18:25 +0000\n/bucket-name/NTMAITW2K12-02-u_ex140131.log'

Jeremy Wadsack

Post by Matt Domsch
Bugger. That's the v4-signature-failure that several people have reported
but I haven't been able to reproduce or explain myself. :-( Can you post
the V4 signature bits (redacting your access key) which may help figure out
the failure?
Thanks,
Matt

Post by Jeremy Wadsack
It looks like the first request is returning 400 and then it falls back
DEBUG: format_uri(): /2K12-02-u_ex140131.log
'iX...J1ec/vu4VU=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked',
'connection': 'close', 'x-amz-request-id': 'FB...17', 'date': 'Tue, 07 Apr
2015 17:18:33 GMT', 'content-type': 'application/xml'}, 'reason': 'Bad
Request'}
DEBUG: Falling back to signature v2
DEBUG: Using signature v2
...
DEBUG: format_uri(): /2K12-02-u_ex140131.log
DEBUG: Response: {'status': 200, 'headers': {'content-length': '345829',
'expiry-date="Wed, 22 Apr 2015 00:00:00 GMT", rule-id="Delete all files 14
days old or older"', 'last-modified': 'Tue, 07 Apr 2015 10:04:09 GMT',
'x-amz-request-id': 'D7...7E', 'etag': '"12...d53"', 'date': 'Tue, 07 Apr
'application/octet-stream'}, 'reason': 'OK'}
I redacted a bunch of stuff but let me know what else you need to figure
this out.
Jeremy Wadsack

Post by Matt Domsch
Jeremy and/or Russell, can you run with --debug and see where the second
upload is coming from please?
Thanks,
Matt
On Tue, Apr 7, 2015 at 11:52 AM, Jeremy Wadsack <

Post by Jeremy Wadsack
I just upgraded locally (Mac) from 1.5.0-rc1 to 1.5.2 and see a double
$ s3cmd get s3://bucket-name/2K12-02-u_ex140131.log
s3://bucket-name/2K12-02-u_ex140131.log -> ./2K12-02-u_ex140131.log [1
of 1]
s3://bucket-name/2K12-02-u_ex140131.log -> ./2K12-02-u_ex140131.log [1
of 1]
345829 of 345829 100% in 1s 336.48 kB/s done
Jeremy Wadsack

Post by Russell Gadd
It's Amazon s3.
Regards
Russell

Post by Matt Domsch
Is this amazon S3 or a clone? If a clone, this is expected as it
falls back from signature v4 to signature v2. Use --signature-v2 to avoid
it.