Python extended search library

Question

I have about 80,000 text files, and I want to be able to perform advanced searches on them. Let's say I have two lists of keywords, and I want to return all files containing at least one of the keywords in the first list and at least one in the second list. Is there already a library that would do this? I don’t want to rewrite it if it exists.

Answer 1

Since you need to search documents many times, you will most likely want to index text files to perform such searches as quickly as possible.

Implementing a reasonable index is certainly possible, but a quick search led me to:

• https://pypi.python.org/pypi/Whoosh/
• http://pythonhosted.org/Whoosh/

Take a look at the documentation. Hope this will be pretty trivial to achieve the desired behavior.

Answer 2

I just feel that you want to use the MapReduce processing type to search. It must be very scalable, Python must have MapReduce packages.