Post data to an ASP.NET page using Delphi and TIdHttp

Question

I have an Asp.net page, such a simple http://issamsoft.com/app2/page1.aspx and I want to post some data in it and extract the data from the response using TIdHttp. I tried to do this in Delphi2009 as follows:

Procedure TForm1.Button1Click(Sender: TObject); Const VIEWSTATE = '/wEPDwUKMjA3NjE4MDczNmRkSxPt/LdmgqMd+hN+hkbiqIZuGUk='; EVENTVALIDATION = '/wEWAwL40NXEDALs0bLrBgKM54rGBtmtdOYy+U7IFq8B25bYT1d4o1iK'; FORMPARAMS = 'TextBox1=Issam&Button1=Button'; URL = 'http://issamsoft.com/app2/page1.aspx'; var http: TIdHttp; lstParams: TStringList; begin http := TIdHTTP.Create(self); lstParams := TStringList.Create; try lstParams.Add('__VIEWSTATE='+VIEWSTATE); lstParams.Add('__EVENTVALIDATION='+EVENTVALIDATION); lstParams.Add(FORMPARAMS); http.Request.ContentType := 'application/x-www-form-urlencoded'; Memo1.Lines.Text := http.Post(url,lstParams); finally http.Free; lstParams.Free; end; end; 

but TIdhttp always gives an error (HTTP / 1.1 500 Internal Server Error.) I read a few comments in the idHttp module, talking about problems with the http protocol v 1.1 like this:

Currently, when sending a POST, IdHTTP will automatically install the protocol to version 1.0 regardless of the value that it was originally. This is because there are some servers that do not fully respect RFC. In particular, they do not respect sending / not sending Expectations: 100-Continue heading. Until we find the optimal solution that will NOT violate the RFC, we will limit POSTS to version 1.0.

Is there something wrong with my code or with TidHttp Bug? and if the problem is in TIdHttp, is there a workaround? or is there another solution using indy components?

in addition. I made a solution in C # using WebClient and it works very well.

  private void button1_Click(object sender, EventArgs e) { WebClient myClient = new WebClient(); string viewstate = HttpUtility.UrlEncodeUnicode(@"/wEPDwUKMjA3NjE4MDczNmRkSxPt/LdmgqMd+hN+hkbiqIZuGUk="); string eventvaildation = HttpUtility.UrlEncodeUnicode(@"/wEWAwL40NXEDALs0bLrBgKM54rGBtmtdOYy+U7IFq8B25bYT1d4o1iK"); string postdata = "__VIEWSTATE=" + viewstate + "&" + "__EVENTVALIDATION=" + eventvaildation + "&TextBox1=Issam&Button1=Button"; myClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded"); byte[] responce = myClient.UploadData("http://issamsoft.com/app2/page1.aspx", Encoding.ASCII.GetBytes(postdata)); txtResponse.Text = Encoding.ASCII.GetString(responce); } 

Where can I find a (good / trusted) class like WebClient in Delphi? is free:)

Edit: I hope the VIEWSTATE, EVENTVALIDATION mechanism is clear enough for you, they are the hash values ​​generated by the server, and they can change (already changed), my original project has a code snippet to extract the current VIEWSTATE, EVENTVALIDATION values, but I skip this part to make my example simple and straightforward, so when you want to try this code, you should take the values ​​of VIEWSTATE, EVENTVALIDATION from the current source of the page.

Answer 1

This is most likely due to the fact that in C # you use ampersand ( & ) characters to combine VIEWSTATE, EVENTVALIDATION and FORMPARAMS.

But in Delphi you use TStringList and Add. Add will not put an ampersand ( & ), but CR + LF between additions.

Therefore, you send different data to Delphi than to C #.

If you change the string concatenation logic in Delphi to match the logic in C #, it should work, no matter what type of WebClient you use on the Delphi side.

- Jeroen

Edit: 20090830

This code works; checking the output of C # and IE7 with Fiddler2, I found that you also need to avoid the characters '/' and '='.

Edit2: 20090831

Somehow CHANGE and EVENTVALIDATION on the site have changed since yesterday (although I'm not sure why). I changed the code and it works while posting my changes in this answer. This is the new request content shot on Fiddler when publishing a page from Internet Explorer 7:

 __VIEWSTATE=%2FwEPDwUKMjA3NjE4MDczNmRk5%2FC2iWwvlAB3L1wYzRpm3KZhRC0%3D&__EVENTVALIDATION=%2FwEWAwLXzuATAuzRsusGAoznisYGSYOqDGy4vMunY6A8xi6ahQEPI5Q%3D&TextBox1=Issam&Button1=Button 

This is a new request that was sent in the form of a Delphi example:

 __VIEWSTATE=%2FwEPDwUKMjA3NjE4MDczNmRk5%2FC2iWwvlAB3L1wYzRpm3KZhRC0%3D&__EVENTVALIDATION=%2FwEWAwLXzuATAuzRsusGAoznisYGSYOqDGy4vMunY6A8xi6ahQEPI5Q%3D&TextBox1=Issam&Button1=Button 

The following is an example of this result (note the " Issam " as a result):

 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" > <head><title> </title></head> <body> <form name="form1" method="post" action="page1.aspx" id="form1"> <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKMjA3NjE4MDczNg9kFgICAw9kFgICBQ8PFgIeBFRleHQFDVdlbGNvbWUgSXNzYW1kZGSCDMOkTMjkZJgqLkhpK99twpD5+A==" /> <input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWAwKr+O/BBQLs0bLrBgKM54rGBlueO5BU/6BAJMZfHNwh5fsQFuAm" /> <div> <input name="TextBox1" type="text" value="Issam" id="TextBox1" /> <input type="submit" name="Button1" value="Button" id="Button1" /> <br /> <span id="Label1">Welcome Issam</span> <br /> </div> </form> </body> </html> 

Edit 3: 20090831

Indeed, VIEWSTATE and EVENTVALIDATION have changed again: somehow they continue to change: it seems that there is a time factor.

So, I adapted the code, now it can extract VIEWSTATE and EVENTVALIDATION from the GET request, and then put it in the POST request.

In addition, it turned out that “+” and “:” also needed to be escaped. So I burst into the logic of ASP.NET page generation, found out that they use HttpUtility.UrlEncode to perform escaping, which ultimately checks for HttpUtility.IsSafe , whose characters should be removed. So I adapted the Reflector method, the reverse engineering code of Delphi, into the THttpUtility class.

This is the final code:

 {$DEFINE FIDDLER} unit HttpPostExampleFormUnit; interface uses Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms, Dialogs, StdCtrls, IdBaseComponent, IdComponent, IdTCPConnection, IdTCPClient, IdHTTP; type THttpPostExampleForm = class(TForm) Button1: TButton; Memo1: TMemo; procedure Button1Click(Sender: TObject); private procedure Log(const What: string; const Content: string); procedure LogClear; end; var HttpPostExampleForm: THttpPostExampleForm; implementation uses HttpUtilityUnit; {$R *.dfm} procedure AddFormParameter(const StringStream: TStringStream; const ParameterName: string; const ParameterValue: string); var EncodedParameterValue: string; begin StringStream.WriteString(ParameterName); StringStream.WriteString('='); EncodedParameterValue := THttpUtility.UrlEncode(ParameterValue); StringStream.WriteString(EncodedParameterValue); end; procedure AddFormParameterSeparator(const StringStream: TStringStream); begin StringStream.WriteString('&'); end; function ExtractHiddenParameter(const ParameterName: string; const Request: string): string; //<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKMjA3NjE4MDczNg9kFgICAw9kFgICBQ8PFgIeBFRleHQFDVdlbGNvbWUgSXNzYW1kZGSCDMOkTMjkZJgqLkhpK99twpD5+A==" /> const PrefixMask = 'input type="hidden" name="%s" id="%s" value="'; Suffix = '" />'; var Prefix: string; PrefixLength: Integer; PrefixPosition: Integer; SuffixPosition: Integer; begin Prefix := Format(PrefixMask, [ParameterName, ParameterName]); PrefixPosition := Pos(Prefix, Request); if PrefixPosition = 0 then Result := '' else begin PrefixLength := Length(Prefix); Result := Copy(Request, PrefixPosition + PrefixLength, 1 + Length(Request) - PrefixPosition - PrefixLength); SuffixPosition := Pos(Suffix, Result); if SuffixPosition = 0 then Result := '' else Delete(Result, SuffixPosition, 1 + Length(Result) - SuffixPosition); end; end; procedure THttpPostExampleForm.Button1Click(Sender: TObject); const DefaultVIEWSTATE = '/wEPDwUKMjA3NjE4MDczNmRk5%2FC2iWwvlAB3L1wYzRpm3KZhRC0='; DefaultEVENTVALIDATION = '/wEWAwLXzuATAuzRsusGAoznisYGSYOqDGy4vMunY6A8xi6ahQEPI5Q='; FORMPARAMS = 'TextBox1=Issam&Button1=Button'; URL = 'http://issamsoft.com/app2/page1.aspx'; __VIEWSATE = '__VIEWSTATE'; __EVENTVALIDATION = '__EVENTVALIDATION'; var VIEWSTATE: string; EVENTVALIDATION: string; getHttp: TIdHttp; getRequest: string; postHttp: TIdHttp; ParametersStringStream: TStringStream; postRequest: string; begin LogClear(); getHttp := TIdHTTP.Create(self); try getRequest := getHttp.Get(URL); Log('GET Request', getRequest); VIEWSTATE := ExtractHiddenParameter(__VIEWSATE, getRequest); EVENTVALIDATION := ExtractHiddenParameter(__EVENTVALIDATION, getRequest); Log('Extracted VIEWSTATE', VIEWSTATE); Log('Extracted EVENTVALIDATION', EVENTVALIDATION); finally getHttp.Free(); end; postHttp := TIdHTTP.Create(self); {$IFDEF FIDDLER} postHttp.ProxyParams.ProxyServer := '127.0.0.1'; postHttp.ProxyParams.ProxyPort := 8888; {$ENDIF FIDDLER} postHttp.HTTPOptions := postHttp.HTTPOptions + [hoKeepOrigProtocol]; ParametersStringStream := TStringStream.Create(''); try AddFormParameter(ParametersStringStream, __VIEWSATE, VIEWSTATE); AddFormParameterSeparator(ParametersStringStream); AddFormParameter(ParametersStringStream, __EVENTVALIDATION, EVENTVALIDATION); AddFormParameterSeparator(ParametersStringStream); ParametersStringStream.WriteString(FORMPARAMS); postHttp.Request.ContentType := 'application/x-www-form-urlencoded'; ParametersStringStream.Seek(0, soFromBeginning); Log('POST Parameters', ParametersStringStream.DataString); postRequest := postHttp.Post(url, ParametersStringStream); Log('POST Request', postRequest); finally postHttp.Free; ParametersStringStream.Free; end; end; procedure THttpPostExampleForm.Log(const What, Content: string); begin Memo1.Lines.Add(What + '='); Memo1.Lines.Add(Content); end; procedure THttpPostExampleForm.LogClear; begin Memo1.Lines.Clear; end; end. 

And the THttpUtlity class:

 unit HttpUtilityUnit; interface type THttpUtility = class private class function IsSafe(const ch: Char): boolean; public class function UrlEncode(s: string): string; end; implementation uses Classes, SysUtils; class function THttpUtility.IsSafe(const ch: Char): boolean; begin if ((((ch >= 'a') and (ch <= 'z')) or ((ch >= 'A') and (ch <= 'Z'))) or ((ch >= '0') and (ch <= '9'))) then Result := True else case ch of '''', '(', ')', '*', '-', '.', '_', '!': Result := True; else Result := False; end; end; class function THttpUtility.UrlEncode(s: string): string; var ch: Char; HexCh: string; StringStream: TStringStream; Index: Integer; Ordinal: Integer; begin StringStream := TStringStream.Create(''); try //Note: this is not yet UTF-16 compatible; check before porting to Delphi 2009 for Index := 1 to Length(s) do begin ch := s[Index]; if IsSafe(ch) then StringStream.WriteString(Ch) else begin Ordinal := Ord(Ch); HexCh := IntToHex(Ordinal, 2); StringStream.WriteString('%'+HexCh); end; end; Result := StringStream.DataString; finally StringStream.Free; end; end; end. 

That should make you.

- Jeroen

Answer 2

If you included the hoKeepOrigProtocol option in the HTTPOptions property of the TIdHTTP instance, it will not return to HTTP 1.0.

Some lines are documented below the comment text you provided :)

  // If hoKeepOrigProtocol is SET, is possible to assume that the developer // is sure in operations of the server if not (hoKeepOrigProtocol in FOptions) then begin FProtocolVersion := pv1_0; end; 

Answer 3

You need to update the VIEWSTATE and EVENTVALIDATION values.

Running this code causes a 500 error, but if I open the page and use the new __VIEWSTATE and __EVENTVALIDATION values ​​from the web page source, it no longer causes a 500 error.

I don’t know why the C # client would work when Delphi did not do this, but maybe it doesn’t work anymore?

It looks like an ASP problem, not Delphi. Both Indy (TIdHTTP) and Synapse are good socket libraries for Delphi.