I've used Xenu for years, and it's an outstanding program. I've run into a
problem, though: when it gets much beyond 500,000 total URL's (only 15-35%
visited), my computer tells me the virtual memory is too full. I've got 1 GB of
RAM, and the task manager claims ~ 2 GB are being used.

I'd like to know what others have experienced when crawling a site of that size
(or larger). Is it my hardware? Should I increase the virtual memory?

I tend to always run it 1 thread at a time, so that my computer (and the server)
don't get too bogged down. I imagine if I ran it much higher, I'd have a bigger
problem with giant sites ... right?

Anyway, thanks to Tilman, and thanks to everyone who's helped with improvements
to the software.
- Brandon