Jamie Jackson
2015-04-10 22:08:28 UTC
Hi Folks,
I've got a weird use-case where my source is an SFTP folder on a
third-party server. I've got the remote filesystem mounted, and I've got
the sync set up, but it's pretty slow (and will get slower as the
third-party adds files). (This slowness isn't surprising.)
I'm afraid you're going to tell me this is just as hard as I think it is,
but maybe you'll surprise me: Is there a way I can calculate md5sums on the
remote server (since I've got SSH access, and md5sum is available) and then
use that information? I'm thinking that's going to be orders of magnitude
faster than calculating them over the wire. I know there's the --cache-file
option, but I don't have much clue what goes into generating that file, nor
exactly what it tells s3cmd.
Let me know what you think.
Thanks,
Jamie
I've got a weird use-case where my source is an SFTP folder on a
third-party server. I've got the remote filesystem mounted, and I've got
the sync set up, but it's pretty slow (and will get slower as the
third-party adds files). (This slowness isn't surprising.)
I'm afraid you're going to tell me this is just as hard as I think it is,
but maybe you'll surprise me: Is there a way I can calculate md5sums on the
remote server (since I've got SSH access, and md5sum is available) and then
use that information? I'm thinking that's going to be orders of magnitude
faster than calculating them over the wire. I know there's the --cache-file
option, but I don't have much clue what goes into generating that file, nor
exactly what it tells s3cmd.
Let me know what you think.
Thanks,
Jamie