Geek Answers Handbook

Httpclient 4.3 lock in connection pool

when i use httpclient 4.3 below

static { try { SSLContextBuilder builder = new SSLContextBuilder(); builder.loadTrustMaterial(null, new TrustSelfSignedStrategy()); SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(builder.build()); CookieSpecProvider easySpecProvider = new CookieSpecProvider() { public CookieSpec create(HttpContext context) { return new BrowserCompatSpec() { @Override public void validate(Cookie cookie, CookieOrigin origin) throws MalformedCookieException { // Oh, I am easy } }; } }; Registry<CookieSpecProvider> r = RegistryBuilder.<CookieSpecProvider> create() .register(CookieSpecs.BEST_MATCH, new BestMatchSpecFactory()) .register(CookieSpecs.BROWSER_COMPATIBILITY, new BrowserCompatSpecFactory()) .register("easy", easySpecProvider).build(); RequestConfig requestConfig = RequestConfig.custom().setConnectionRequestTimeout(5000) .setSocketTimeout(10000).setConnectTimeout(10000).setCookieSpec("easy").setRedirectsEnabled(false) .build(); PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(); cm.setMaxTotal(100); cm.setDefaultMaxPerRoute(10); client = HttpClients.custom().setConnectionManager(cm).setDefaultCookieSpecRegistry(r) .setSSLSocketFactory(sslsf).setDefaultRequestConfig(requestConfig).build(); } catch (Exception e) { logger.error("http client init fail!", e); } } public static String execute(HttpRequest httpRequest) { CloseableHttpResponse response = null; HttpGet httpGet = null; HttpEntity httpEntity = null; try { httpGet = new HttpGet(httpRequest.getUrl()); httpGet.setHeader("Connection", "close"); if (httpRequest.isUseGzip()) { httpGet.addHeader("Accept-Encoding", "gzip,deflate,sdch"); } if (!StringUtils.isEmpty(httpRequest.getContentType())) { httpRequest.setContentType(httpRequest.getContentType()); } httpGet.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63"); response = client.execute(httpGet); httpEntity = response.getEntity(); byte[] bytes = null; try { bytes = EntityUtils.toByteArray(httpEntity); } catch (Exception e) { return null; } if (response.getStatusLine().getStatusCode() != 200) { logger.warn("error! StatusCode: " + response.getStatusLine().getStatusCode() + ", url: " + httpRequest.getUrl()); return null; } @SuppressWarnings("deprecation") String charset = EntityUtils.getContentCharSet(httpEntity); if (StringUtils.isEmpty(charset)) { Matcher match = charsetPatterm.matcher(new String(bytes)); if (match.find()) { charset = match.group(1); } } if (!StringUtils.isEmpty(charset)) { String strUtf8 = new String(new String(bytes, charset).getBytes(), GlobalConfig.ENCODING); return StringEscapeUtils.unescapeHtml4(strUtf8); } } catch (Exception e) { logger.error("error! url [" + httpRequest.getUrl() + "]", e); } finally { try { if (httpEntity != null) { EntityUtils.consume(httpEntity); } if (response != null) { response.close(); } if (httpGet != null) { httpGet.abort(); } } catch (Exception e) { // ignore } } return null; }

the thread will block. show jstack like this. I just use it to scan some websites. This happens when statusCode is 404.
Using Java Apache PoolingClientConnectionManager memory leak, how to solve it? my problem is similar to this.

 "pool-1-thread-10" prio=10 tid=0x00007f7168003000 nid=0x3e4d waiting on condition [0x00007f717c398000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000e69d7350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133) at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282) at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64) at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177) at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170) at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.jav a:244) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:231) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:173) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)

how to decide?

+9
java
source share
3 answers

I had the same problem and this was the first answer. I used Ironluca's comment to solve my problem, but felt that this needed a complete answer.

The quick start guide provides a detailed description of how to configure and use the basic HttpClient.

 // In order to ensure correct deallocation of system resources // the user MUST call CloseableHttpResponse#close() from a finally clause. // Please note that if response content is not fully consumed the underlying // connection cannot be safely re-used and will be shut down and discarded // by the connection manager. try { System.out.println(response1.getStatusLine()); HttpEntity entity1 = response1.getEntity(); // do something useful with the response body // and ensure it is fully consumed EntityUtils.consume(entity1); } finally { response1.close(); }

Looking back at your question, it seems that you have a closing and consumption code.
You also have a custom connection pool. This is the same as the default value, but I think you configured it differently.

+7
source

I encountered a similar error. It seems we need to use HttpEntity before returning. In your case, I see that you do not do this when faced with non-200 answers. You just returned zero. You may need to use this before returning. In addition, I would recommend using HttpClientUtils.closeQuietly (response); this is a wrapper around EntityUtils.consume

+3
source

The condition field which getPoolEntryBlocking is a signal that the connection slot is available in the pool. HttpClient does not configure the connection timeout by default, so await is called in Future without a timeout, which can cause deadlocks if for some reason the connections cannot be returned to the pool. The most likely reason is the lack of timeouts, so configure the timeout of the sane connection to protect yourself from malfunctioning hosts.

If you call CloseableHttpResponse#close you should not see a connection leak. Failure to use the entity will make it impossible to reuse the connections only by the pool, which will lead to the appearance of more new connections than would otherwise be.

0
source

More articles:


All Articles