Can cURL HTTP request fully mimic a browser based request?

Question

This is a two-part question.

Q1: Can a cURL based query 100% mimic a browser based query?

Q2: If so, which parameters should be set. If not what does a browser that cannot be simulated cURL do?

I have a website and I see thousands of requests made from one IP in a very short time. These requests collect all my data. When you looked through the log to identify the agent used, it looks like a request from the browser. Therefore, it was curious to find out if his bot is not a user.

Thanks in advance

Answer 1

R1: I believe that if you set all the correct headers, then yes, a curl-based request can mimic a browser: in the end, both send an HTTP request, which is just a few lines of text following a certain convention (namely : HTTP RFC)

R2: The best way to answer this question is to see what your browser sends; with Firefox, for example, you can use Firebug or LiveHTTPHeaders to get this.

For example, to get this page, Firefox sent these request headers:

GET /questions/1926876/can-a-curl-based-http-request-imitate-a-browser-based-request-completely HTTP/1.1 Host: stackoverflow.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2b4) Gecko/20091124 Firefox/3.6b4 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 115 Connection: keep-alive Referer: http://stackoverflow.com/questions/1926876/can-a-curl-based-http-request-imitate-a-browser-based-request-completely/1926889 Cookie: ....... Cache-Control: max-age=0 

(I just deleted a couple of information - but you get the idea ;-))

Using curl, you can work with curl_setopt to set HTTP headers; here you will probably have to use a combination of CURLOPT_HTTPHEADER , CURLOPT_COOKIE , CURLOPT_USERAGENT , ...

Answer 2

This page has all the answers to your questions . You can imitate things basically.