How we use Solr:
We save 7K documents with 3k attributes attached to one document in solr. Each attribute is indexed by solr to enable searching / sorting by this attribute. We retrieve data from solr based on search / filter criteria with 400+ attribute attached to one document. Therefore, when we try to find some text in solr with 1 attribute (by setting fl = "projectId"), it takes hardly 1 second to display the result on the solr console, which is great.
However, if we try to get the 100+ attribute for the same search criteria (which returns the string ~ 100 * 3 * there is no matching document (~ 50) from the 7K document with the 4K attribute), it will take ~ 20 seconds. But we need to get the 400+ attribute with the matching document, it takes a lot of time ~ 90 seconds, before it crashed due to outOfMemoryError, which we fixed by increasing the RAM size and the JVM heap size.
Mysql data synchronization with Solr:
We are currently using MySql DB as the primary database and Solr Server as the secondary database. We synced mysql data daily with the Solr server. We also update the solr server whenever we update any attribute in Mysql DB.
Using solr result data in the application:
The application toolbar will contain a document with pre-configured user columns (attributes). The user can apply search / filter criteria to fill out the desired result on his dashboard. Therefore, our application tries to obtain data with search / filtering criteria from the solr server by contacting it.
We tried a lot of things, such as increasing heap size, RAM size and lack of processor, but no luck. Data is growing every day, which causes many problems. It works for small projects or small attributes, but whenever we try to extract more attributes, it takes too much time, once it crashed.
I'm not sure if we use indexes correctly?
- / ?
.