All geek questions in one place

Which of the best search engine searches

I have 10,000,000 entries that will be the best method of searching for entries, I am currently using full-text search, but it is slow, please suggest.

+4
source share
4 answers

There is no one-time solution, but you can try:

Sphinx

How to implement full-text search for this 10+ millionth row table, keep it loaded, and stay relevant? Sphinx is good at such Riddles.

Sphinx is a full-text search engine distributed under the GPL 2 version. A commercial license is also available for built-in use.

Typically, this is a stand-alone search engine designed for quick, size-efficient and relevant full-text search functions for other applications. Sphinx has been specially designed to integrate well with SQL databases and scripting languages. Currently, built-in data sources support receiving data through a direct connection to MySQL or PostgreSQL or using an XML feed mechanism (a feed for the indexer in a special XML-based format that Sphinx recognizes).

As for the name, Sphinx is an abbreviation that is officially decoded as an SQL Phrase Index. Yes, I know about the CMU Sphinx Project.

http://www.sphinxsearch.com/

Lucene PHP (Part of Zend Framework):

Zend_Search_Lucene is a generic text search engine entirely in PHP 5. Since it stores its index in the file system and does not require a database server, it can add search capabilities to almost any Web site managed by PHP. Zend_Search_Lucene supports the following functions:

  • Ranked Search - Top Results Returned First
  • Many powerful query types: phrase queries, logical queries, wildcard queries, proximity queries, range
    requests and many others.
  • Search for a specific field (e.g. title, author, content)

http://framework.zend.com/ http://framework.zend.com/manual/en/zend.search.lucene.overview.html

+7
source

It depends on a few simple questions:

  • what data is processed? (simple records such as "First Name, Last Name" or more complex data sets?
  • How is it structured? (normal database table? partitioned?)
  • what are you looking for? (i.e. search for names in the phone book)
+1
source

Since I have not worked with large datasets like this, here are some ideas that might work:

The first question is whether these entries are static (e.g. geoip) or not?

  • I would try to optimize my database as much as I can (try using EXPLAIN if you are using MySQL)
  • Look at all the possible queries, try to optimize your database for these queries.
  • If the indices are ok, I will go with some kind of cache where I would save my previous results. It will be convenient if your database is not updated regularly.
  • You can complete the task above (for example: the most frequently used search queries and their results can also be removed)
  • Try optimizing these ideas for your needs.

If you can provide more information, I can clarify my advice.

0
source

Use Solr . This is lucene with some add-ons easily accessible via http protocol. It flashes quickly compared to any full text searches from mysql.

0
source

Source: https://habr.com/ru/post/1314301/

More articles:


All Articles