ProtocolError when calling HttpWebRequest.GetResponse ()

Question

I have a page containing links to some files.

I basically need to access the source of the page to parse it and get all the file hyperlinks.

My code is something like this (part of the code I found in many places on the network ..):

"private static byte[] ReadImageFromUrl(string url) { var myReq = (HttpWebRequest)WebRequest.Create(url); myReq.Timeout = 10000; WebResponse myResp = myReq.GetResponse(); Stream stream = myResp.GetResponseStream(); List<byte> bytesList = new List<byte>(); using (var br = new BinaryReader(stream)) { try { while (true) { var b = br.ReadByte(); bytesList.Add(b); } } catch (Exception) {} br.Close(); } myResp.Close(); return bytesList.ToArray(); }" 

Now the problem is that I get "System.Net.WebException: the remote server responded with an error: (500) Internal server error". when calling "myReq.GetResponse ()" - when analyzing the error, I see that the status is "ProtocolError".

The response property of the WebException object contains some server error .. (although when opening it from the browser it opens correctly) ... also when I call this function with the URL of one of my files, I get the same protocolError status, but error 404. ..

Please give some advice on how I can solve it ... or any other opportunity to complete this task.

Thanks!

Answer 1

My new code after using Fiddler :

 private static byte[] ReadFileFromUrl(string url) { var myReq = (HttpWebRequest)WebRequest.Create(url); myReq.Accept = const_AcceptHeader; myReq.Headers.Set(const_AcceptLanguageHeaderName, const_AcceptLanguageHeader); myReq.UserAgent = const_AcceptUserAgentHeader; myReq.CookieContainer = new CookieContainer(); myReq.KeepAlive = true; myReq.Timeout = Int32.Parse(ConfigSettings.RequestPageTimeout) * 1000; WebResponse myResp = null; List<byte> bytesList = null; myResp = myReq.GetResponse(); Stream stream = myResp.GetResponseStream(); bytesList = new List<byte>(); using (var br = new BinaryReader(stream)) { try { while (true) { var b = br.ReadByte(); bytesList.Add(b); } } catch (Exception ex) { throw; } br.Close(); } return bytesList.ToArray(); } 

All variables starting with const _ are taken from Fiddler .

Answer 2

Well, I decided that using Fiddler ... I passed my request to the header object as I saw them in Fiddler ... & it worked, without errors