Why is the Indy Project HttpClient Get () giving 500 code on some URLs that work fine in web browsers?

Question

I have several URLs that work fine in all browsers, but if I try to get the page content using Get () the Indy Http client, it will return error code 500, an internal server error. This is with the latest version of Indy SVN (4981).

Here is my sample code. All that is needed for this is Delphi with Indy components and a button and note form.

procedure TForm1.Button1Click(Sender: TObject); var HTTPCLIENT1: TIdHTTP; begin try try HTTPCLIENT1 := TIdHTTP.Create(nil); Memo1.Clear; with HTTPCLIENT1 do begin HandleRedirects := True; Request.UserAgent := 'Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.7.3) Gecko/20040924 Epiphany/1.4.4 (Ubuntu)'; Memo1.Text := Get('http://www.laredoute.fr/vente-machine-a-coudre-bernette-20-kit-couture--garantie-2-ans.aspx?productid=401225048&documentid=999999&categoryid=22918417&customertarget=0&offertype=0&prodcolor=1#pos=33_n_n_n_n_n_n&numberpage=2'); Caption := ResponseText; end; except On e: Exception do begin Memo1.Lines.Add('Exception: '+e.Message); end; end; finally HTTPCLIENT1.Free; end; end; 

This is not a connectivity problem on my side, as 99% of URLs return 200 or 404, only a few return 500, but each browser opens them within a second.

Answer 1

Such a failure usually indicates that the GET request is incorrect, which leads to a server code failure at its end. But, without seeing that the web browser requests really look for comparison with TIdHTTP requests, there is no way to know for sure what the server does not like.

Refresh . I see that when a web browser requests a URL, the server immediately sends a 200 response, however, when TIdHTTP requests a URL, the server sends a 301 redirect to the new URL, which then sends a 302 redirect to the error page when TIdHTTP requests this A URL that then sends a 500 response when TIdHTTP requests this URL.

The two differences between the web browser request and the initial TIdHTTP request, which will affect the web server, are:

• The URL you request using TIdHTTP includes an anchor tag at the end (all after the # character - #pos=33_n_n_n_n_n_n&numberpage=2 ), which usually come out of a web browser. Anchors are not really part of the URLs. They are intended to be used by web browsers to find locations in data that is retrieved from a URL.

• user agent. Some web servers are sensitive to different user agents and may send different responses to different types of user agents.

When I remove the binding from the URL, TIdHTTP.Get () no longer fails:

 Memo1.Text := Get('http://www.laredoute.fr/vente-machine-a-coudre-bernette-20-kit-couture--garantie-2-ans.aspx?productid=401225048&documentid=999999&categoryid=22918417&customertarget=0&offertype=0&prodcolor=1');