[S3tools-general] Error on large (> 1 million files) file trees

[Whit Blauvelt]

Hi,

I'm seeing repeated errors like this:

INFO: [1478000/1480985]
INFO: [1479000/1480985]
INFO: [1480000/1480985]
INFO: Retrieving list of remote files for s3://xyzlan.backup1/data10/LMN_datastore/ ...
INFO: Forwarding request to us-west-2
WARNING: Retrying failed request: /?marker=data10/LMN_datastore/MANAGER/archive/2013/2013_02_28/12072112_20130228_ABH_149959.ARK&prefix=data10/LMN_datastore/ ('')
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?marker=data10/LMN_datastore/MANAGER/archive/2013/2013_04_05/81617342_20130405_ACH_163219.ARK&prefix=data10/LMN_datastore/ ('')
WARNING: Waiting 3 sec...

Directories off the same system that are half that number of files work fine.

I increased the RAM on the VM running this from 8 to 12 G, but no change.
Indeed it had been using the 8G an a bit of swap on these. But it's just at
about 8 G now, so 4 G still free, and the same error.

With 8 G this was freezing the VM running it eventually. With 12 G, hasn't
happended. But once it starts doing this in a directory it keeps doing it
... no apparent recovery.

Whit