Why is this happening all the time? COOL limit exceeded OutOfMemoryError GC

Question

I am doing a load test for my Solr application. The index contains more than 200 million documents. I use the default Jetty server and have configured the maximum JVM memory as 4 GB. To test my application, I compose 5,000 text requests and send them to Solr one by one. However, after approximately 110 requests, the Jetty container throws an exception.

Why is this happening? How can I solve it?

SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.lucene.util.AttributeImpl.clone(AttributeImpl.java:196) at org.apache.lucene.util.AttributeSource$State.clone(AttributeSource.java:116) at org.apache.lucene.util.AttributeSource$State.clone(AttributeSource.java:119) at org.apache.lucene.util.AttributeSource.captureState(AttributeSource.java:349) at org.apache.solr.highlight.TokenOrderingFilter.incrementToken(DefaultSolrHighlighter.java:595) at org.apache.lucene.search.highlight.OffsetLimitTokenFilter.incrementToken(OffsetLimitTokenFilter.java:43) at org.apache.lucene.analysis.CachingTokenFilter.fillCache(CachingTokenFilter.java:78) at org.apache.lucene.analysis.CachingTokenFilter.incrementToken(CachingTokenFilter.java:50) at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:225) at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByHighlighter(DefaultSolrHighlighter.java:468) at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:379) at org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:116) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:194) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) 

Answer 1

Obviously, 4Gb RAM is very low for processing the load test at the 200M index. We ran performance tests for Solr 4.2 on 300M documents with an average document size of 1K. The goal was to find out the minimum configuration of the machine on which we can have a stable response time <3 seconds for irreversible requests. For 100 simultaneous requests, our result showed that the minimum configuration of the machine was 8 processor cores / 15 GB of RAM. Of course, the results will depend on many factors, however you can use this as a rule of thumb for your machine size.