I need to periodically traverse a site to update the server cache. It is a database driven site with very many (thousands of) pages. I use wget to mirror the site locally on the same server, so I use this command:
wget --mirror localhost
After some time it stops suddenly with this message:
HTTP request sent, awaiting response... Terminated
It happens always, but not on the exact same url. And I have tried on another server as well with the same result.
The --debug option does not provide any helpful information, nor does Apache's log file.
What could be the cause of this problem? I suspect some buffer running out of memory etc. Or a stack overflow.
Alternatively, are there other command line tools that can do the same?
It's Wget 1.11.4 on Debian Lenny.
Martin
No comments:
Post a Comment