Receiving the error "The remote server returned an error: (403) Forbidden" when clearing the screen using HttpWebRequest.GetResponse ()

Question

We have a tool that checks if a given URL is active. If this URL is live, another part of our software may display fingerprints of content from it.

This is my code to check if the URL is alive

public static bool IsLiveUrl(string url) { HttpWebRequest webRequest = WebRequest.Create(url) as HttpWebRequest; webRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5"; webRequest.CookieContainer = new CookieContainer(); WebResponse webResponse; try { webResponse = webRequest.GetResponse(); } catch (WebException e) { return false; } catch (Exception ex) { return false; } return true; } 

This code works fine, but for a specific site hosted on apache, I get a web exception with the following message. "The remote server responded with an error: (403) Forbidden" Upon further inspection, I found the following data in the WebException object

Status = "ProtocolError" StatusDescription = "Bad behavior"

This is the request header "User-Agent: Mozilla / 5.0 (Windows; U; Windows NT 5.1; en-US; rv: 1.8.0.6). Gecko / 20060728 Firefox / 1.5 Host: scenicspares.co.uk Connection: Keep-Alive"

This is the response header "Keep-Alive: timeout = 4, max = 512 Connection: Keep-Alive Transmission-coding: chunked Content-Type: text / html Date: Thu, Jan 13, 2011 10:29:36 GMT Server: Apache"

I extracted these headers using a watch in vs2008. The used frame is 3.5.

Answer 1

It turned out that all I have to do is follow

  webRequest.Accept = "*/*"; webResponse = webRequest.GetResponse(); 

and it has been fixed.

Answer 2

I believe that there are quite a few such problems that depend on the server application. In my specific case, see: The remote server responded with an error: (403) Forbidden