Having problems with message search?
Hi Brandon,
if Tilman's hints don't help, you might try to spilt your site into different sections and check them one at a time.
You can use the "Do not check any URLs beginning with this:" to do this, if the site is structured in any way.
Cheers
Thomas
Von: xenu-usergroup@yahoogroups.com [mailto:xenu-usergroup@yahoogroups.com] Im Auftrag von brandonmbyers01
Gesendet: Mittwoch, 24. Juni 2009 22:52
An: xenu-usergroup@yahoogroups.com
Betreff: [xenu-usergroup] crawling a large siteI've used Xenu for years, and it's an outstanding program. I've run into a problem, though: when it gets much beyond 500,000 total URL's (only 15-35% visited), my computer tells me the virtual memory is too full. I've got 1 GB of RAM, and the task manager claims ~ 2 GB are being used.
I'd like to know what others have experienced when crawling a site of that size (or larger). Is it my hardware? Should I increase the virtual memory?
I tend to always run it 1 thread at a time, so that my computer (and the server) don't get too bogged down. I imagine if I ran it much higher, I'd have a bigger problem with giant sites ... right?
Anyway, thanks to Tilman, and thanks to everyone who's helped with improvements to the software.
- Brandon
Offline 
Send Email