bixo-dev : Bixo Web Mining Toolkit

Activity within 7 days:

Description

Bixo is an open source vertical web crawler toolkit. It consists of an easily customizable set of components that can be piped together via Cascading to create complex workflows.

The primary use case for Bixo is data mining the web, where the scope of pages to be fetched and processed is 100K to 100M.

More sources of information about Bixo include:

A web site at openbixo.org
Source code hosted at GitHub.
Issue tracking using GitHub Issues.

Most Recent Messages

(View All)
(Group by Topic)

Advanced

Start Topic

Re: restarting a crawl and adding to a previous crawl When the crawler loops it fetches every url found in the previous loop, right? So the crawl time will likely increase exponentially with each loop, right? So Posted - Sat May 26, 2012 4:54 pm	Pat Ferrel reallyreally... Offline Send Email
Re: restarting a crawl and adding to a previous crawl ... The most recent loop dir has an up-to-date snapshot of the crawlDB, which is regenerated after each loop. ... Yes, exactly. You've hit upon a fundamental Posted - Fri May 25, 2012 7:24 pm	Ken Krugler kkrugler Offline Send Email
Re: restarting a crawl and adding to a previous crawl OK, so if I understand correctly every time I restart a crawl on an existing one, it will extend the original crawl with newly fetched data. It never recrawls Posted - Fri May 25, 2012 7:07 pm	Pat Ferrel reallyreally... Offline Send Email
Re: restarting a crawl and adding to a previous crawl Hi Pat, See below. But in general the SimpleCrawlTool is a demo of Bixo, not a complete crawler, thus much of the functionality you're asking about is missing. Posted - Fri May 25, 2012 5:24 pm	Ken Krugler kkrugler Offline Send Email
restarting a crawl and adding to a previous crawl If, for some reason, a crawl fails to finish properly. What is the recommended way to restart it where it left off, or somewhere close. I tried deleting what Posted - Fri May 25, 2012 5:15 pm	Pat Ferrel reallyreally... Offline Send Email
Add bixo-dev to your personalized My Yahoo! page What's This?

Message History

	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec
2012	29	30	15	50	19
2011	21	18	38	17	6	6	4	1	24	4	7	58
2010	47	7	2	5	16	42	54	94	7	10	45	29
2009					39	12	18	19	6	57	28	125

Yahoo! Groups

Group Information

Yahoo! Groups Tips

Did you know...

Home

Activity within 7 days:

Description

Most Recent Messages

Message History

Questions in Computers & Internet > Software

Want to help answer other questions? Go to Yahoo! Answers