Tools for finding problems with solr performance

Question

I have a solr server that randomly encounters a performance issue. Queries that result in longer qtimes do not take the same time if I try them again. A performance problem even occurs sometimes when the load on the server is small.

So, I wanted to know if there are any tools or if there are any options in solr to indicate a problem leading to performance problems.

some of the options in solrconfig.xml are

<filterCache class="solr.LRUCache" size="1024" initialSize="512" autowarmCount="1000"/> <queryResultCache class="solr.LRUCache" size="512" initialSize="128" autowarmCount="0"/> <documentCache class="solr.LRUCache" size="1024" initialSize="512" autowarmCount="0"/> <enableLazyFieldLoading>true</enableLazyFieldLoading> <queryResultWindowSize>30</queryResultWindowSize> <queryResultMaxDocsCached>100</queryResultMaxDocsCached> <HashDocSet maxSize="10000" loadFactor="0.75"/> 

Answer 1

If you are looking for long-term monitoring, you can also look at tools / services such as Scalable Performance Monitoring (SPM) from my company, Sematext or others. http://sematext.com/spm/index.html

This service will expose requests, latency, all Solr cache information, JVM memory, GC, CPU, load, disk and network I / O, etc.

We eat our own dog food and use this tool to monitor Solr performance with Solr instances compared to search-lucene.com and search-hadoop.com/, and we use them regularly with our customers when we need to help them with work Solr setting.

If you do not want the “correct” performance monitoring to be higher, you can use tools like vmstat, iostat, sar, top, jstack, etc. to solve problems with Solr, provided that you run it under UNIX.

Answer 2

I think you should start by looking at Solr Performance Resistance on the Solr wiki page . You can also use the SolrMeter tool to help stress-test your changes.

Answer 3

The sar utility from the sysstat package is great for debugging disk I / O problems. If you notice that a particular drive gets more than 30% or so of use sequentially, you are probably related to IO.

Answer 4

Perhaps this tool can help you: LucidGaze .

I have no experience, but it looks promising. People from Lucid have also published some interesting articles that are really worth reading.

Answer 5

I conducted stress testing of the application, simultaneously running several requests to the solr server (requests were executed by subsequent wgets)

We began to see the impact of performance and narrowed our problem for two reasons:

• The cache size was too small, so there were too many evictions and inserts in the cache.
• We used solrversion1.3, which apparently uses blocking reads for index files. Switching to solr 1.4 removed this problem because it uses java nio, which uses non-blocking reads on index files.

[we also tried to put index files in ram and use it directly there, which improved performance (but obviously not scalable)]

Answer 6

I’ve tested SOLR memory leaks for more than a year and had to redistribute the server for five days as a workaround to the problem, in the end I found that the RELIC monitoring tool was causing the problem, we removed the monitoring and the problem was resolved, so my advice is to be careful when using too many monitoring tools, this creates additional overhead.

Answer 7

We experienced the same thing with tools for three monitors on the server. A massive memory leak occurred during use. This almost caused a bottleneck effect. You need these tools, but it slowed them down a lot. I also recommend using a utility to debug the disk of any of these problems.