CURL - works in the browser, not in cURL

Question

I have been fighting for this task for almost three days, and I think that I am missing some basic cURL skills.

I'll start with:

In F12 IE, I see 2 POSTs on the first page: (I notice that the first gets 302, which should be redirected, and cURL I get only 200)

Filling captcha:

on the second page (after interception):

traffic:

This is my code (and I cannot move on because it does not work in the early stages):

I created a special form that goes to my own page with GET (cURL), which, in turn, accesses the website:

 $id=$_GET['id']; // getting the biznumber $humanCode=$_GET['nobot']; $curl = curl_init(); curl_setopt ($curl, CURLOPT_URL, "https://www.*******.******.***"); // setting some https to be able to access the website from my local computer. curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, true); curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 2); curl_setopt($curl, CURLOPT_CAINFO, "c:/xampp/htdocs/CAcerts/curl-ca-bundle.crt"); // I know the values for the ASPX vars like __EVENTTARGET, __EVENTARGUMENT, __VIEWSTATE are arbitrary now. I need to take care of that but I don't yet know how. $postarr= array ( "__EVENTTARGET"=>"", "__VIEWSTATE=" =>"%2FwEPDwULLTEzMzI2OTg4NDYPZBYCZg9kFgQCBA8PZBYCHgdvbmNsaWNrBQxnb1RvTWl2emFrKClkAgYPD2QWAh8ABQxnb1RvTWl2emFrKClkZM6iZZ0Qaf2CpfXoJJdZ0IqaWsDO", "__EVENTARGUMENT=" =>"", "__EVENTVALIDATION" =>"%2FwEWBQKgysLGCwL2r7SGDQLh4ri%2BAwLWws7NDwLWwpLPD%2F1HuCAFYzs2seaziWbYEXjDfigP", "hidUrlFileIshurim"=>"https%3A%2F, "cod"=>"3322" ); $fields_string=''; foreach($postarr as $key=>$value) { $fields_string .= $key.'='.$value.'&'; } rtrim($fields_string,'&'); curl_setopt($curl, CURLOPT_POST ,1); curl_setopt($curl, CURLOPT_POSTFIELDS, $fields_string); curl_setopt($curl, CURLOPT_TIMEOUT, 10); curl_setopt ($curl, CURLOPT_USERAGENT, "User-Agent Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; MAAU)"); // I made a cookie file and it seems to work $cookiefile = "d:/cookie.txt"; curl_setopt($curl, CURLOPT_COOKIEJAR, $cookiefile); curl_setopt($curl, CURLOPT_COOKIEFILE, $cookiefile); curl_setopt($curl, CURLOPT_FRESH_CONNECT , 1); curl_setopt($curl, CURLOPT_FOLLOWLOCATION ,1); curl_setopt($curl, CURLOPT_HEADER ,1); // DO NOT RETURN HTTP HEADERS curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); $temp=curl_exec($curl); $info = curl_getinfo($curl); $html = mb_convert_encoding($temp, 'HTML-ENTITIES', 'utf-8'); echo "ERRCODE: ".curl_error($curl); echo '<br /><br />'; echo "INFO : "; print_r($info); echo '<br /><br />'; $httpcode = curl_getinfo($curl, CURLINFO_HTTP_CODE); echo "CODE: ".$httpcode; echo '<br /><br />'; echo "CODE: ".$httpcode; echo '<br /><br />'; echo "VARS: ".$vars; echo '<br /><br />'; //echo $html; curl_setopt ($curl, CURLOPT_URL, "https://www.*******.******.***"); curl_setopt($curl, CURLOPT_FRESH_CONNECT , 0); echo "<br /><br /><b>2nd</b><br /><br />"; $temp=curl_exec($curl); $info = curl_getinfo($curl); $html = mb_convert_encoding($temp, 'HTML-ENTITIES', 'utf-8'); echo "ERRCODE: ".curl_error($curl); echo '<br /><br />'; echo "INFO : "; print_r($info); echo '<br /><br />'; echo $html; 

I can’t even start working on this. It starts by returning me 200 OK, not 302, and sometimes I also get 500.

I know that ASPX vars can be crucial, but if my browser can make these vars and send them to the server, can't cURL do the same?

Thanks for any help!

Answer 1

The problem is resolved.
It is a matter of using the correct headers. Following the reports from the browser, I went through all the steps, and the result appeared.

I went through every step using:

 curl_init curl_setopt() .. curl_setopt() curl_exec() curl_close() 

Thus, I had to manually install each request and go through the settings. This made the code longer, but much easier to understand.

I had thoughts about this site, using special javascript code to make the site work, so I was very worried about all the extra javascript code that turned out to be unnecessary.

The thing is to be more organized and follow the correct header settings.

Also, since it was an ASPX site, I had to read and remember the VIEWSTATE and VALIDATION last page in each iteration. This is the first and very important reason for reporting the internal server 500, which I used all the time.

I used Firebug and LiveHttpHeaders to come up with every step.

Answer 2

"It doesn’t work to even start working. It starts with returning 200 OK to me, not 302, and sometimes 500."

 curl_setopt($curl, CURLOPT_FOLLOWLOCATION ,1); 

You have Curl to follow any 302 redirects. They will execute internally inside Curl and will not be displayed by PHP.

also:

 curl_setopt($curl, CURLOPT_HEADER ,1); // DO NOT RETURN HTTP HEADERS 

A comment does exactly the opposite of what the code does .... which seems wrong.

Answer 3

before you do cURL, you need to look at the field of the requested field. usually HTTP 500 from aspx is not found, the field is sent.

 foreach($postarr as $key=>$value) { $fields_string .= $key.'='.$value.'&'; echo" $fields_string <br> "; } 

make sure this field is not dynamic when sending a request. Hope this helps.

Answer 4

I used this:

 curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER["HTTP_USER_AGENT"]); 

It seems to simulate curl, as it has a name and browser version.