Search the web
Sign In
New User? Sign Up
xenu-usergroup · Xenu Linkchecker Usergroup
? Already a member? Sign in to Yahoo!

Yahoo! Groups Tips

Did you know...
Want your group to be featured on the Yahoo! Groups website? Add a group photo to Flickr.

Best of Y! Groups

   Check them out and nominate your group.
Having problems with message search? Fill out this form to ensure your group is one of the first to be migrated to the new message search system.

Messages

  Messages Help
Advanced
crawling a large site   Message List  
Reply | Forward Message #1162 of 1200 |
Re: crawling a large site

Thank you for the advice. I was running it on my home machine with 1 GB of RAM.
At work, one of my machines has 4 GB. It looks like I'm running version 1.3 in
all locations, so I'll try upgrading it.

And the funny thing is I am excluding large sections of the site for this crawl,
though perhaps I can exclude more.

One reason I was running it on the 1 GB home machine is that after awhile, it
started timing out every ~250 URLs -- but only at work. I'm guessing if I
compared the path of ISP's between work & the host to the ones between home &
the host, the latter must have nicer peering agreements ... but I noticed the
advice on this group yesterday to un-check the option to fail all URLs on a
domain, so that should help me.

Thanks for the insights, y'all ... if anyone else would like to discuss their
experience crawling sites of this size, I'm curious to hear about it.
-Brandon




Thu Jun 25, 2009 2:48 pm

brandonmbyers01
Offline Offline
Send Email Send Email

Forward
Message #1162 of 1200 |
Expand Messages Author Sort by Date

I've used Xenu for years, and it's an outstanding program. I've run into a problem, though: when it gets much beyond 500,000 total URL's (only 15-35% visited),...
brandonmbyers01
Offline Send Email
Jun 24, 2009
8:52 pm

If you're using a version before 1.3b, update it - I made several changes to save memory. 1 GB of RAM isn't much... Consider buying new RAM, its really ...
Tilman Hausherr
geo4497
Offline Send Email
Jun 25, 2009
6:55 am

Hi Brandon, if Tilman's hints don't help, you might try to spilt your site into different sections and check them one at a time. You can use the "Do not check...
Thomas Fischer
thgb.fischer
Offline Send Email
Jun 25, 2009
9:35 am

Thank you for the advice. I was running it on my home machine with 1 GB of RAM. At work, one of my machines has 4 GB. It looks like I'm running version 1.3 in...
brandonmbyers01
Offline Send Email
Jun 25, 2009
2:48 pm

Having upgraded to 1.3c, it runs for awhile on the 4 GB machine, but soon shows an "Out of memory" warning. Since I'd turned off the "fail domain" option, it...
brandonmbyers01
Offline Send Email
Jun 25, 2009
5:23 pm
Advanced

Copyright © 2009 Yahoo! Inc. All rights reserved.
Privacy Policy - Terms of Service - Guidelines - Help