[S3tools-general] Doesn't seem to multipart upload

[Bob Smith]

Hello,

I believe I have a problem with multipart uploads. I'm
currently uploading a set of archive files that are all 150MB each. The
command I use is:

# s3cmd put <filename> s3://<bucket> --mime-type=application/x-dar 2>&1 >> backup.log

The
average upload speed that s3cmd is reporting is 105kB/s. Does that seem
slow? I believe there is only one single upload, as when I run netstat
-nputw, only one socket is listed for python:-

tcp 0 61736 192.168.1.2:51874 178.236.4.23:443 ESTABLISHED 7886/python

Would this list many connections if doing a multipart upload? If I check the multiparts using s3cmd, I get:-

***@nas:~# s3cmd multipart s3://<bucket>
s3://<bucket>/
Initiated Path Id
2014-06-26T18:09:42.000Z
s3://<bucket>/backup_2014-06-24_full.116.dar
SAuMBSqDY.7xXZ3bOVNzpiqcmbqPbgVhzeMx4hnCMNE_LKCqvPN5_Z8MlSxcReNBRARy9.r6iH6vBK9cyF1KsQ--

Can anyone tell whether I am getting multiparts uploads or not, and if not, why that might be?

My version of s3cmd is 1.5.0-beta1, with python version 2.7.3.

Thanks for any help, and thanks to all the developers for a great tool.

Regards
Rob Smith

[Matt Domsch]

You are getting multipart uploads, but not parallel uploads. The file is
getting chunked into 15MB pieces, then each piece is uploaded individually,
which is what allows you to upload large files in the first place. s3cmd
doesn't presently do parallel uploads, and certainly won't in the 1.5.0
release - there's a lot of work that would have to happen to enable such.

Post by Bob Smith
Hello,
I believe I have a problem with multipart uploads. I'm currently uploading
# s3cmd put <filename> s3://<bucket> --mime-type=application/x-dar 2>&1 >> backup.log
The average upload speed that s3cmd is reporting is 105kB/s. Does that
seem slow? I believe there is only one single upload, as when I run netstat
-nputw, only one socket is listed for python:-
tcp 0 61736 192.168.1.2:51874 178.236.4.23:443
ESTABLISHED 7886/python
Would this list many connections if doing a multipart upload? If I check
the multiparts using s3cmd, I get:-
s3://<bucket>/
Initiated Path Id
2014-06-26T18:09:42.000Z
s3://<bucket>/backup_2014-06-24_full.116.dar
SAuMBSqDY.7xXZ3bOVNzpiqcmbqPbgVhzeMx4hnCMNE_LKCqvPN5_Z8MlSxcReNBRARy9.r6iH6vBK9cyF1KsQ--
Can anyone tell whether I am getting multiparts uploads or not, and if
not, why that might be?
My version of s3cmd is 1.5.0-beta1, with python version 2.7.3.
Thanks for any help, and thanks to all the developers for a great tool.
Regards
Rob Smith
------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
S3tools-general mailing list
https://lists.sourceforge.net/lists/listinfo/s3tools-general

[Rob Smith]

Ah, I see. I misunderstood multipart. Thank you for clarifying.

I have a further question regarding the speed - I noticed that my .s3cfg file has bucket_location set to US, when actually its in Ireland. Will that negatively impact the upload speed? Is the file being routed via the US to Ireland?! (I'm in the UK)

I just wonder whether ~100kB/s is typical for S3 uploads?

Thanks again for your response

Rob

--- Original Message ---

From: "Matt Domsch" <***@domsch.com>
Sent: 26 June 2014 21:03
To: "s3tools-general" <s3tools-***@lists.sourceforge.net>
Subject: Re: [S3tools-general] Doesn't seem to multipart upload

You are getting multipart uploads, but not parallel uploads. The file is
getting chunked into 15MB pieces, then each piece is uploaded individually,
which is what allows you to upload large files in the first place. s3cmd
doesn't presently do parallel uploads, and certainly won't in the 1.5.0
release - there's a lot of work that would have to happen to enable such.

Post by Bob Smith
Hello,
I believe I have a problem with multipart uploads. I'm currently uploading
# s3cmd put <filename> s3://<bucket> --mime-type=application/x-dar 2>&1 >>
backup.log
The average upload speed that s3cmd is reporting is 105kB/s. Does that
seem slow? I believe there is only one single upload, as when I run netstat
-nputw, only one socket is listed for python:-
tcp 0 61736 192.168.1.2:51874 178.236.4.23:443
ESTABLISHED 7886/python
Would this list many connections if doing a multipart upload? If I check
the multiparts using s3cmd, I get:-
s3://<bucket>/
Initiated Path Id
2014-06-26T18:09:42.000Z
s3://<bucket>/backup_2014-06-24_full.116.dar
SAuMBSqDY.7xXZ3bOVNzpiqcmbqPbgVhzeMx4hnCMNE_LKCqvPN5_Z8MlSxcReNBRARy9.r6iH6vBK9cyF1KsQ--
Can anyone tell whether I am getting multiparts uploads or not, and if
not, why that might be?
My version of s3cmd is 1.5.0-beta1, with python version 2.7.3.
Thanks for any help, and thanks to all the developers for a great tool.
Regards
Rob Smith
------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
S3tools-general mailing list
https://lists.sourceforge.net/lists/listinfo/s3tools-general

[Matt Domsch]

bucket_location is just a default. Wherever the bucket is (in your case,
Ireland) is where the data will flow to directly; not through the US.

Post by Rob Smith
Ah, I see. I misunderstood multipart. Thank you for clarifying.
I have a further question regarding the speed - I noticed that my .s3cfg
file has bucket_location set to US, when actually its in Ireland. Will that
negatively impact the upload speed? Is the file being routed via the US to
Ireland?! (I'm in the UK)
I just wonder whether ~100kB/s is typical for S3 uploads?
Thanks again for your response
Rob
--- Original Message ---
Sent: 26 June 2014 21:03
Subject: Re: [S3tools-general] Doesn't seem to multipart upload
You are getting multipart uploads, but not parallel uploads. The file
is getting chunked into 15MB pieces, then each piece is uploaded
individually, which is what allows you to upload large files in the first
place. s3cmd doesn't presently do parallel uploads, and certainly won't in
the 1.5.0 release - there's a lot of work that would have to happen to
enable such.
Hello,
I believe I have a problem with multipart uploads. I'm currently uploading
# s3cmd put <filename> s3://<bucket> --mime-type=application/x-dar 2>&1 >>
backup.log
The average upload speed that s3cmd is reporting is 105kB/s. Does that
seem slow? I believe there is only one single upload, as when I run netstat
-nputw, only one socket is listed for python:-
tcp 0 61736 192.168.1.2:51874 178.236.4.23:443
ESTABLISHED 7886/python
Would this list many connections if doing a multipart upload? If I check
the multiparts using s3cmd, I get:-
s3://<bucket>/
Initiated Path Id
2014-06-26T18:09:42.000Z
s3://<bucket>/backup_2014-06-24_full.116.dar
SAuMBSqDY.7xXZ3bOVNzpiqcmbqPbgVhzeMx4hnCMNE_LKCqvPN5_Z8MlSxcReNBRARy9.r6iH6vBK9cyF1KsQ--
Can anyone tell whether I am getting multiparts uploads or not, and if
not, why that might be?
My version of s3cmd is 1.5.0-beta1, with python version 2.7.3.
Thanks for any help, and thanks to all the developers for a great tool.
Regards
Rob Smith
------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
S3tools-general mailing list
https://lists.sourceforge.net/lists/listinfo/s3tools-general
------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
S3tools-general mailing list
https://lists.sourceforge.net/lists/listinfo/s3tools-general