Hi Sanjoy,
Re how to integrate indexing into SimpleCrawlTool - I would create a separate SimpleIndexTool, which can be run on the output directories of SimpleCrawlTool.
You could clone the SimpleStatusTool, since that's very similar, but with the change (as per a previous email) of processing the data using Cascading versus directly opening taps & iterating.
I just wrote some utility code to help create a Cascading source tap that's the set of crawl output subdirs you'd want to process for something like building the Lucene index.
Re how to output the Lucene index - since this is an output format, conceptually I think the right place is in a Cascading Schema, which already exists as IndexSchema in Bixo.
But I think I'd need to see your new code to better understand what looks different when handling this as a LuceneFunction.
Thanks,
-- Ken
On Nov 1, 2009, at 11:32am, sanjoy wrote:
Hi Ken,
This is what I am thinking. I want to get Lucene indexing done before I attempt Solr.
1) I am using Lucene 2.9.0. Nothing against 2.4.1 which seems to be the version in Maven, but I couldn't download the sources for Lucene 2.4.1. The tar and zip seem to have a problem on the Lucene site.
2) I have added a couple of lines to SiteCrawler.java.
Tap luceneSink = new Hfs(new SequenceFile( FetchedDatum.FIELDS.append(MetaData.FIELDS)), curCrawlDirName + "/lucene");
LucenePipe lucenePipe = new LucenePipe( fetchPipe.getContentTailPipe(), MetaData.FIELDS);
sinkMap.put( LucenePipe.LUCENE_PIPE_NAME, luceneSink);
Flow flow = flowConnector.connect(inputSource, sinkMap, fetchPipe, urlPipe, lucenePipe);
3) Adding a class named bixo.pipes.LucenePipe modeled on ParsePipe.
4) Adding a class named bixo.operations.LuceneFunction modeled on ParseFunction. The writing of the Lucene index will happen in the operate() function.
I have these done. I will borrow the code to write the index from existing test code.
Let me know what you think,
Sanjoy
--------------------------------------------
Ken Krugler
+1 530-210-6378
e l a s t i c w e b m i n i n g
Offline 
Send Email