Search the web
Sign In
New User? Sign Up
xenu-usergroup · Xenu Linkchecker Usergroup
? Already a member? Sign in to Yahoo!

Yahoo! Groups Tips

Did you know...
Show off your group to the world. Share a photo of your group with us.

Best of Y! Groups

   Check them out and nominate your group.
Having problems with message search? Fill out this form to ensure your group is one of the first to be migrated to the new message search system.

Messages

  Messages Help
Advanced
crawling a large site   Message List  
Reply | Forward Message #1163 of 1200 |
Re: crawling a large site

Having upgraded to 1.3c, it runs for awhile on the 4 GB machine, but soon shows
an "Out of memory" warning. Since I'd turned off the "fail domain" option, it
showed many of these warnings, and then Xenu abruptly disappeared.

I restarted, and tried opening the file but hitting pause immediately. Then I
tried sorting the listings, but nothing would happen. (I'd wait awhile & come
back to the computer, just in case) I also tried exporting to a tab-delimited
file, but nothing would happen. (again, I waited)

I suppose I just need to subdivide the site further.

Still, I can't complain; it's still the best program I've used for this, and all
the other sites I've crawled have worked out fine (usually 180-230,000 URLs in
those).

-Brandon




Thu Jun 25, 2009 5:22 pm

brandonmbyers01
Offline Offline
Send Email Send Email

Forward
Message #1163 of 1200 |
Expand Messages Author Sort by Date

I've used Xenu for years, and it's an outstanding program. I've run into a problem, though: when it gets much beyond 500,000 total URL's (only 15-35% visited),...
brandonmbyers01
Offline Send Email
Jun 24, 2009
8:52 pm

If you're using a version before 1.3b, update it - I made several changes to save memory. 1 GB of RAM isn't much... Consider buying new RAM, its really ...
Tilman Hausherr
geo4497
Offline Send Email
Jun 25, 2009
6:55 am

Hi Brandon, if Tilman's hints don't help, you might try to spilt your site into different sections and check them one at a time. You can use the "Do not check...
Thomas Fischer
thgb.fischer
Offline Send Email
Jun 25, 2009
9:35 am

Thank you for the advice. I was running it on my home machine with 1 GB of RAM. At work, one of my machines has 4 GB. It looks like I'm running version 1.3 in...
brandonmbyers01
Offline Send Email
Jun 25, 2009
2:48 pm

Having upgraded to 1.3c, it runs for awhile on the 4 GB machine, but soon shows an "Out of memory" warning. Since I'd turned off the "fail domain" option, it...
brandonmbyers01
Offline Send Email
Jun 25, 2009
5:23 pm
Advanced

Copyright © 2009 Yahoo! Inc. All rights reserved.
Privacy Policy - Terms of Service - Guidelines - Help