HttpURLConnection does not read the whole answer

Question

I use HttpURLConnection to perform an HTTP POST, but I do not always return the full response. I wanted to debug the problem, but when I went through each line, it worked. I thought it was a synchronization problem, so I added Thread.sleep and it really made my code work, but this is only a temporary solution. I wonder why this is happening and how to solve it. Here is my code:

public static InputStream doPOST(String input, String inputMimeType, String url, Map<String, String> httpHeaders, String expectedMimeType) throws MalformedURLException, IOException { URL u = new URL(url); URLConnection c = u.openConnection(); InputStream in = null; String mediaType = null; if (c instanceof HttpURLConnection) { //c.setConnectTimeout(1000000); //c.setReadTimeout(1000000); HttpURLConnection h = (HttpURLConnection)c; h.setRequestMethod("POST"); //h.setChunkedStreamingMode(-1); setAccept(h, expectedMimeType); h.setRequestProperty("Content-Type", inputMimeType); for(String key: httpHeaders.keySet()) { h.setRequestProperty(key, httpHeaders.get(key)); if (logger.isDebugEnabled()) { logger.debug("Request property key : " + key + " / value : " + httpHeaders.get(key)); } } h.setDoOutput(true); h.connect(); OutputStream out = h.getOutputStream(); out.write(input.getBytes()); out.close(); mediaType = h.getContentType(); logger.debug(" ------------------ sleep ------------------ START"); try { Thread.sleep(2000); } catch (InterruptedException e) { e.printStackTrace(); } logger.debug(" ------------------ sleep ------------------ END"); if (h.getResponseCode() < 400) { in = h.getInputStream(); } else { in = h.getErrorStream(); } } return in; } 

later I do the following to read the input stream

  ByteArrayOutputStream bos = new ByteArrayOutputStream(); while (is.available() > 0) { bos.write(is.read()); } is.close(); //is.read(bytes); if (logger.isDebugEnabled()) { logger.debug(" Response lenght is : " + is.available()); //logger.debug("RAW response is " + new String(bytes)); logger.debug("RAW response is " + new String(bos.toByteArray())); } 

It generates the following HTTP headers

 POST /emailauthentication/ HTTP/1.1 Accept: application/xml Content-Type: application/xml Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39",oauth_nonce="YnDb5eepuLm%2Fbs",oauth_signature="dbN%2FWeWs2G00mk%2BX6uIi3thJxlM%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276524919", oauth_token="", oauth_version="1.0" User-Agent: Java/1.6.0_20 Host: test:6580 Connection: keep-alive Content-Length: 1107 

Other posts have suggested disabling keep-alive using

 http.keepAlive=false 

I tried this and the headers changed to

 POST /emailauthentication/ HTTP/1.1 Accept: application/xml Content-Type: application/xml Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39", oauth_nonce="Eaiezrj6X4Ttt0", oauth_signature="ND9fAdZMqbYPR2j%2FXUCZmI90rSI%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276526608", oauth_token="", oauth_version="1.0" User-Agent: Java/1.6.0_20 Host: test:6580 Connection: close Content-Length: 1107 d872-42b9-030e-4e74b9b60e39", oauth_nonce = "Eaiezrj6X4Ttt0", oauth_signature = "ND9fAdZMqbYPR2j% 2FXUCZmI90rSI% 3D", oauth_signature_method = "HMAC-SHA1", oauth_timestamp = " POST /emailauthentication/ HTTP/1.1 Accept: application/xml Content-Type: application/xml Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39", oauth_nonce="Eaiezrj6X4Ttt0", oauth_signature="ND9fAdZMqbYPR2j%2FXUCZmI90rSI%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276526608", oauth_token="", oauth_version="1.0" User-Agent: Java/1.6.0_20 Host: test:6580 Connection: close Content-Length: 1107 

The connection header is "closed", but I still can not read the whole answer. Any idea what I'm doing wrong?

Answer 1

I think your problem is in this line:

 while (is.available() > 0) { 

According to javadoc, available does not block and does not wait until all the data is available, so you can get the first packet and then it will return false. The correct way to read from an InputStream is as follows:

 int len; byte[] buffer = new byte[4096]; while (-1 != (len = in.read(buffer))) { bos.write(buffer, 0, len); } 

Reading will return -1 when there is nothing left in the input stream or the connection is closed, and it will block and wait for the network to do so. Reading arrays is also much more efficient than using single bytes.

Answer 2

Maybe I skipped this, but what type of data "injects" into your code? Something strange about InputStreams as a whole is that read methods (...) tend to block until data is available and then only that data is returned. You really need to continue reading from your InputStream and add to ByteArrayInputStream or some other structure until you explicitly throw an EOF exception.

Answer 3

If you read the entire message at once, you can compare isr.available () with the expected length of the content. Here is how I did it:

 public byte[] readData(HttpURLConnection conn) throws IOException, InterruptedException { String _connlen = conn.getHeaderField("Content-Length"); int connlen = Integer.parseInt(_connlen); InputStream isr = null; byte[] bytes = new byte[connlen]; try { isr = conn.getInputStream(); //security count that it doesn't begin to hang int maxcounter = 0; //wait till all data is avalibal, max 5sec while((isr.available() != connlen) && (maxcounter < 5000)){ Thread.sleep(1); maxcounter++; } //Throw if not all data could be read if(maxcounter >= 5000) throw new IllegalAccessError(); //read the data if(isr.read(bytes, 0, connlen) < 0) throw new IllegalAccessError(); } finally { if (isr != null) isr.close(); } return bytes; }