URLConnection does not handle content length through proxy correctly

Question

I ran into the following problem: when URLConnectionused through a proxy, the length of the content is always set to -1.

First I checked if the proxy server really returns Content-Length( lynxand wgetalso works through the proxy server, there is no other Internet connection from the local network):

$ lynx -source -head ftp://ftp.wipo.int/pub/published_pct_sequences/publication/2003/1218/WO03_104476/WO2003-104476-001.zip
HTTP/1.1 200 OK
Last-Modified: Mon, 09 Jul 2007 17:02:37 GMT
Content-Type: application/x-zip-compressed
Content-Length: 30745
Connection: close
Date: Thu, 02 Feb 2012 17:18:52 GMT

$ wget -S -X HEAD ftp://ftp.wipo.int/pub/published_pct_sequences/publication/2003/1218/WO03_104476/WO2003-104476-001.zip
--2012-04-03 19:36:54--  ftp://ftp.wipo.int/pub/published_pct_sequences/publication/2003/1218/WO03_104476/WO2003-104476-001.zip
Resolving proxy... 10.10.0.12
Connecting to proxy|10.10.0.12|:8080... connected.
Proxy request sent, awaiting response...
  HTTP/1.1 200 OK
  Last-Modified: Mon, 09 Jul 2007 17:02:37 GMT
  Content-Type: application/x-zip-compressed
  Content-Length: 30745
  Connection: close
  Age: 0
  Date: Tue, 03 Apr 2012 17:36:54 GMT
Length: 30745 (30K) [application/x-zip-compressed]
Saving to: `WO2003-104476-001.zip'

In Java, I wrote:

URL url = new URL("ftp://ftp.wipo.int/pub/published_pct_sequences/publication/2003/1218/WO03_104476/WO2003-104476-001.zip");
int length = url.openConnection().getContentLength();
logger.debug("Got length: " + length);

and I get -1. I started debugging FtpURLConnection, and it turned out that the necessary information is in the base field HttpURLConnection.responses, however it is never filled out properly:

(in the headlines Content-Length: 30745). The length of the content is not updated when you start reading a stream or even after reading a stream. The code:

URL url = new URL("ftp://ftp.wipo.int/pub/published_pct_sequences/publication/2003/1218/WO03_104476/WO2003-104476-001.zip");
URLConnection connection = url.openConnection();

logger.debug("Got length (1): " + connection.getContentLength());

InputStream input = connection.getInputStream();

byte[] buffer = new byte[4096];
int count = 0, len;
while ((len = input.read(buffer)) > 0) {
    count += len;
}

logger.debug("Got length (2): " + connection.getContentLength() + " but wanted " + count);

Conclusion:

Got length (1): -1
Got length (2): -1 but wanted 30745

, JDK6, # 7168608.

• - , FTP-, FTP- URL- file:/. .
• JDK6, , , (Apache Http Client?).

Answer 1

, - . , -, , . , , Content-Length .

HTTP 1.1:

4.4

• ...
• ...
• Content-Length ( 14.13), OCTET , . Content-Length , ( , Transfer-Encoding ). Transfer-Encoding, Content-Length, .

14.41

Transfer-Encoding , ( ) , . , , .

Transfer-Encoding       = "Transfer-Encoding" ":" 1#transfer-coding

- 3.6. :

Transfer-Encoding: chunked

, , . ​​ , .

HTTP/1.0 Transfer-Encoding.

, URLConnection Content-Length ,

, Transfer-Encoding. , ...

- , lynx , lynx -head. Transfer-Encoding, .

-

Ξ▶ lynx -useragent='dummy' -source -head http://www.bbc.co.uk                                                                                                                  
HTTP/1.1 302 Found
Server: Apache
X-Cache-Action: PASS (non-cacheable)
X-Cache-Age: 0
Content-Type: text/html; charset=iso-8859-1
Date: Tue, 03 Apr 2012 13:33:06 GMT
Location: http://www.bbc.co.uk/mobile/
Connection: close

Ξ▶ wget -useragent='dummy' -S -X HEAD http://www.bbc.co.uk                                                                                                                 
--2012-04-03 14:33:22--  http://www.bbc.co.uk/
Resolving www.bbc.co.uk... 212.58.244.70
Connecting to www.bbc.co.uk|212.58.244.70|:80... connected.
HTTP request sent, awaiting response... 
HTTP/1.1 200 OK
Server: Apache
Cache-Control: private, max-age=15
Etag: "7e0f292b2e5e4c33cac1bc033779813b"
Content-Type: text/html
Transfer-Encoding: chunked
Date: Tue, 03 Apr 2012 13:33:22 GMT
Connection: keep-alive
X-Cache-Action: MISS
X-Cache-Age: 0
X-LB-NoCache: true
Vary: Cookie

, , , , , , , Transfer-Encoding .

Answer 2

, "" jdk, ftp-, . FtpURLConnection HttpURLConnection, -. , FtpURLConnection, , HttpURLConnection . , , , "", . ( openjdk 1,6, - ).

Answer 3

, , - ( , ):

URLConnection connection= url.openConnection();
InputStream input= connection.getInputStream();
byte[] buffer= new byte[4096];
while(input.read(buffer) > 0)
  ;
logger.debug("Got length: " + getContentLength());

, , , URLConnection , , .