Curl: transfer closed with other read data left

Question

I have a problem with a big jingle.

I get

• nread <= 0, closed connection to the server, bailing
• transmission closed with left read data left

and the content is partially delivered

GET /stats/?stats_breakdown=track__track&campaign=&search_criteria=2&period=0&date_month=11&date_day=03&date_year=2015&start_date_month=11&start_date_day=03&start_date_year=2015&end_date_month=12&end_date_day=31&end_date_year=2014 HTTP/1.1 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13 Host: domain.com Accept: */* Cookie: sessionid=xxg4gglsm7o3b224wihqz8od19wl31h1; csrftoken=JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C Cache-Control: no-cache Connection: Keep-Alive Keep-Alive: 600 Accept-Language: en-us X-CSRFToken: SeN9bHryRK8FWLTLJIs5c6u9AZ47a8pR Content-Type: application/x-www-form-urlencoded Origin: https://domain.com Referer: https://domain.com * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * additional stuff not fine transfer.c:1037: 0 0 * HTTP 1.1 or later with persistent connection, pipelining supported < HTTP/1.1 200 OK < Server: nginx/1.8.0 < Date: Wed, 04 Nov 2015 12:54:05 GMT < Content-Type: text/html; charset=utf-8 < Transfer-Encoding: chunked < Connection: keep-alive < Vary: Accept-Encoding < Vary: Cookie, Accept-Language < P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI" < Content-Language: en * Replaced cookie csrftoken="JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C" for domain domain.com, path /, expire 1478091245 < Set-Cookie: csrftoken=JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C; expires=Wed, 02-Nov-2016 12:54:05 GMT; Max-Age=31449600; Path=/; secure < * nread <= 0, server closed connection, bailing * transfer closed with outstanding read data remaining * Closing connection #0 

This is the php configuration I'm using

 function getHeaders() { $headers = array(); $headers[] = 'Cache-Control: no-cache'; $headers[] = 'Connection: Keep-Alive'; $headers[] = 'Keep-Alive: 600'; $headers[] = 'Accept-Language: en-us'; $headers[] = 'X-CSRFToken: SeN9bHryRK8FWLTLJIs5c6u9AZ47a8pR'; $headers[] = 'Content-Type: application/x-www-form-urlencoded'; $headers[] = 'Origin: https://domain.com'; $headers[] = 'Referer: https://domain.com'; return $headers; } curl_setopt($connection, CURLOPT_URL, $url); curl_setopt($connection, CURLOPT_RETURNTRANSFER, 1); curl_setopt($connection, CURLOPT_FOLLOWLOCATION, true); curl_setopt($connection, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13'); curl_setopt($connection, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($connection, CURLOPT_SSL_VERIFYHOST, false); curl_setopt($connection, CURLOPT_COOKIEFILE, 'cookie.txt'); curl_setopt($connection, CURLOPT_COOKIEJAR, 'cookie.txt'); curl_setopt($connection, CURLOPT_CONNECTTIMEOUT ,550000000); curl_setopt($connection, CURLOPT_TIMEOUT, 5500000000); //timeout in seconds curl_setopt($connection, CURLOPT_HTTPHEADER, getHeaders()); curl_setopt($connection, CURLOPT_VERBOSE, 1); 

Answer 1

Well, after some searching and IRC chat, I found a solution, but I'm not 100% sure what the reason is. It seems that keep-alives were not sent enough to keep in touch. Lay out the solution here, I hope I can help someone out.

What helped me add

 --keepalive-time 2 

Curl Explanation

 --keepalive-time <seconds> 

This option sets the time during which the connection should remain in standby mode before sending keepalive probes and the time between individual keepalive probes. It is currently effective for operating systems, TCP_KEEPIDLE and TCP_KEEPINTVL Parameters (which means Linux, recent AIX, HP-UX, etc.). This option does not work if --no-keepalive is used. (Added in 7.18.0)

If this parameter is used several times, the latter will be used. If unpecified, the default option is 60 seconds.

It looks like the default was too high to open my connection.

Here is the complete command I used to call me

 curl URL -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,et;q=0.6,nl;q=0.4' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Connection: keep-alive' --compressed -v --keepalive-time 2 

And I am running this version of curl on osx

 curl 7.43.0 (x86_64-apple-darwin15.0) libcurl/7.43.0 SecureTransport zlib/1.2.5 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets 

If someone would like this parameter to be used in PHP curl, the -keepalive-time option is available with PHP 5.5. You can use it as follows:

 curl_setopt($connection, CURLOPT_TCP_KEEPALIVE, 1); curl_setopt($connection, CURLOPT_TCP_KEEPIDLE, 2); 

Hope this helps someone who is struggling with the same problem!

Answer 2

A similar problem was detected, my server is located behind nginx. Curl can get an answer if it connects to the server directly, but if curl connects to the server via nginx, curl will give an error below

Session * transfer closed with remaining incomplete read data
* Connection closure 0 curl: (18) transfer closed with incomplete read remaining data

When I connect to the same nginx url through the browser, Response is displayed normally. It is very strange. When I tried to connect to the same nginx url using curl, it threw the error above.

After comparison, the headers are sent by the browser and curled. I found that the browser may receive a response due to the following header that curl does not send:

 'Accept-Encoding: gzip' 

Sending above the header using curl works fine. So what the above header does is compress the response to gzip, which reduces the size of the response.

After several more digits, I found that nginx could not send any payload> 80KB. After spending a lot of time, I discovered that the problem is with nginx buffering, and nginx worked like a miracle after adding the proxy_buffering property to nginx.conf:

  location / { proxy_buffering off; } 

The accepted answer did not solve my problem. Writing this answer so that no one needs to waste their time if they face the same problem as me.

Answer 3

libcurl only tells you that the connection was disconnected by the server in an unclean way when it did not deliver the data that it promised to do. It seems that the encoded coding did not signal the end of the transmission.

Browsers are notorious for being extremely liberal in what they get, so they overlook and deal with all kinds of protocol violations to a much greater extent than libcurl.