This is interesting. We've seen the same error arise from an S3 library. We
used to see it irregularly but lately the number of tines we get it has
increased.
My research suggested it was related to file size and specifically to not
using multipart chunks. We upload about 60 files a day using s3cmd, usually
in the 10MiB to 100MiB range and I haven't seen that error out in a while.
We upload thousands of small files (~10KB) with our library and have not
seen issues with that. Where we so see the failure is in very large files
in the 100MiB range.
We're addressing this by setting the chunk size to 100 MiB. We just made
this change Monday so I'm still waiting to see if it really helps.
"Socket reset by peer" is pretty generic so these could be unrelated issues
(especially as I'm seeing it in a different library). I'll keep your bucket
changes in mind.
Have you seen an increase in these errors lately? Could this be a change at
AWS?
Jeremy Wadsack
We are seeing (and have seen online references to) random connection reset
errors during put operations. Following online hints, we have seen these
errors fixed by using
- the --region=<region> command line argument (odd, since it is not
noted to effect the put command)
- setting the bucket region in the config file via both
- the hostbase=s3-<region>.amazonaws.com and
- the host_bucket = %(bucket)s.s3-<region>.amazonaws.com
configuration setting (its in unclear which is having the effect)
Neither of these works 100% of the time; so far we hope the combination
will prove to be robust.
File size seems unrelated to the error - we have successful 500MiB uploads
followed by 3 MiB file upload failures.
When we get the errors, they last for some indeterminate period (portions
of an hour up to several hours), long enough to be certain the above
settings resolve the issue (testing with and without the settings show
success and failure during an extended, hour or so long, testing period.)
Again, none of the settings works 100% of the time - oddly sometimes one
setting works and the other does not, and vice versa.
This seems like something that
- Should be addressed internally by the s3cmd
- you could perhaps determine the bucket region and explicitly use
that region
- or use any other method the team determines will make the command
robust
- Should be posted in a FAQ (with clear instructions for what work
around(s) help) in the interim
Thanks -
Stephen
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140
_______________________________________________
S3tools-general mailing list
https://lists.sourceforge.net/lists/listinfo/s3tools-general