archive-crawler : archive-crawler

Discussion group for the Heritrix open-source archival web crawler project.

which Crawl Scope should I use? I am using heritrix 1.14.3.. I just want use heritrix to crawl some wsdl document,but after several attempts，I found it too hard. my problem: Posted - Wed Nov 25, 2009 10:07 am	zhongkem@... zhongkem... Offline Send Email
Exporting Edited Heritrix 2 Hi all, This is not a "real" Heritrix issue but maybe somebody has an answer to this: I programmed 2 processors for Heritrix 2 and put my classes in some Posted - Tue Nov 24, 2009 11:16 pm	sendaman69 Offline Send Email
Re: (subject edited) Recrawling In Heritrix3.0.0-RC1 I see two potential issues in your order: - There's no FetchHistoryProcessor, which is still necessary to collect deduplication-relevant information and insert Posted - Tue Nov 24, 2009 7:48 pm	Gordon Mohr gojomo Offline Send Email
Re: (subject edited) Recrawling In Heritrix3.0.0-RC1 sorry about the broken link sent before. http://cs.odu.edu/~pramo_p/crawler-beans.cxml ... From: Pranay Pandey <sspranay@...> Subject: [archive-crawler] Posted - Tue Nov 24, 2009 3:44 pm	Pranay Pandey sspranay Offline Send Email
Re: (subject edited) Recrawling In Heritrix3.0.0-RC1 Hello Matt and Gordon, Following Gordon's advice and assuming HER-1706 to be fixed, I am using only two of the persistProcessors: load and store. As suggested, Posted - Tue Nov 24, 2009 3:36 pm	Pranay Pandey sspranay Offline Send Email
Add archive-crawler to your personalized My Yahoo! page What's This?

Related Link:	http://crawler.archive.org
Post message:	archive-crawler@yahoogroups.com
Subscribe:	archive-crawler-subscribe@yahoogroups.com
Unsubscribe:	archive-crawler-unsubscribe@yahoogroups.com
List owner:	archive-crawler-owner@yahoogroups.com

Questions in Computers & Internet