Thank you for the advice. I was running it on my home machine with 1 GB of RAM.
At work, one of my machines has 4 GB. It looks like I'm running version 1.3 in
all locations, so I'll try upgrading it.
And the funny thing is I am excluding large sections of the site for this crawl,
though perhaps I can exclude more.
One reason I was running it on the 1 GB home machine is that after awhile, it
started timing out every ~250 URLs -- but only at work. I'm guessing if I
compared the path of ISP's between work & the host to the ones between home &
the host, the latter must have nicer peering agreements ... but I noticed the
advice on this group yesterday to un-check the option to fail all URLs on a
domain, so that should help me.
Thanks for the insights, y'all ... if anyone else would like to discuss their
experience crawling sites of this size, I'm curious to hear about it.
-Brandon