Geek Answers Handbook

Using PHP CURL to parse heavy load sites?

I use PHP CURL to parse a site with a heavy load (this site even rarely opens in a browser). As a result, I have a server response code of 503 or 0 (nothing). Maybe you can give me advice or tell me some CURL functions to get a normal server response?

There are my CURL options:

$options = array( CURLOPT_REFERER => $url, CURLOPT_TIMEOUT => 1800, CURLOPT_HEADER => true, CURLOPT_RETURNTRANSFER => true, CURLOPT_FOLLOWLOCATION => true, CURLOPT_SSL_VERIFYHOST => false, CURLOPT_SSL_VERIFYPEER => false, CURLOPT_HEADERFUNCTION => "curlHeaderCallback", CURLOPT_COOKIE => Cookies::arrayToString(Cookies::instance()->load()), CURLOPT_USERAGENT => "Opera/9.80 (Windows NT 6.1; U; ru) Presto/2.9.168 Version/11.50", CURLOPT_HTTPHEADER => $headers );

The problem is that I cannot get a response with the page code.

I have 2 options: 1. The server did not respond; 2. In the server response, I get a page with the code 503 "server is overloaded."

CurlHeaderCallback () code:

 `function curlHeaderCallback($ch, $str) { if (strncmp($str, "Set-Cookie:", 11) === 0) { $cookie = trim(substr($str, 11)); list($cookie, $options) = explode(";", $cookie, 2); list($key, $value) = explode("=", $cookie, 2); Cookies::instance()->set($key, $value); } if (trim($str) == "") { curl_setopt($ch, CURLOPT_COOKIE, Cookies::arrayToString(Cookies::instance()->load())); } return (strlen($str)); }`

My actions: $response = curl_exec($ch); $info = curl_getinfo($ch); $response = curl_exec($ch); $info = curl_getinfo($ch);

I do not have the answer and $info["http_code"] or the second option: in response, I have the page code 503 and $info["http_code"] = 503

Oh, another option:

 CURLOPT_CONNECTTIMEOUT => 30

The diagram is here: http://s61.radikal.ru/i172/1212/d6/33471472ee8e.png

+4
source share
1 answer

If you are only after the http code, you need to use curl_getinfo with CURLINFO_HTTP_CODE , for example:

 $handle = curl_init($url); curl_setopt($handle, CURLOPT_RETURNTRANSFER, TRUE); $response = curl_exec($handle); $httpCode = curl_getinfo($handle, CURLINFO_HTTP_CODE);
+2
source

More articles:


All Articles