Have Geek Brain

Opening lucene index stored in hdfs

How to read a lucene index stored in HDFS, such as how to get an IndexReader for an index stored in HDFS. IndexReader must be open in a map task.

Something like: IndexReader reader = IndexReader.open ("hdfs / path / to / index / directory");

Thanks, Akhil

+2


a source to share


2 answers


If you want to open a Lucene index that is stored in HDFS for searching, you're out of luck. AFAIK, there is no Directory HDFS implementation that allows lookups. One of the reasons for this is that HDFS is optimized for sequential reads of large blocks rather than the small random reads that Lucene carries.



There is an HDFSDirectory implementation in the Nutch project that you can use to create an IndexReader, but only to remove operations. Nutch uses only HDFSDirectory to perform document deduplication.

+2


a source


I think the Katta project might be what you are looking for. I haven't used it myself, but have recently looked into solutions like this and it seems to be in line with the bill.

This is a distributed version of lucene using shard indexes.



http://katta.sourceforge.net/

+1


a source






More articles:


All Articles