Proxy works locally, but fails when loading into webhost

Question

I spent a good time configuring my proxy. I am currently using a service called proxybonanza. They provide me with a proxy server, which I use to receive web pages.

I am using HTMLAGILITYPACK

Now, if I run my code without a proxy server, there is no problem locally or when uploading to a web hosting server.

If I decide to use a proxy server, it takes a little longer, but it still works locally.

If I publish my solution to, to my webhost I get a SocketException (0x274c) "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 38.69.197.71:45623" 

I have been debugging this for a long time.

There are two entries in my app.config relevant to this

 httpWebRequest useUnsafeHeaderParsing="true" httpRuntime executionTimeout="180" 

This helped me solve a few problems.

Now this is my C # code.

  HtmlWeb htmlweb = new HtmlWeb(); htmlweb.PreRequest = new HtmlAgilityPack.HtmlWeb.PreRequestHandler(OnPreRequest); HtmlDocument htmldoc = htmlweb.Load(@"http://www.websitetofetch.com, "IP", port, "username", "password"); //This is the preRequest config static bool OnPreRequest(HttpWebRequest request) { request.KeepAlive = false; request.Timeout = 100000; request.ReadWriteTimeout = 1000000; request.ProtocolVersion = HttpVersion.Version10; return true; // ok, go on } 

What am I doing wrong? I turned on the tracer in appconfig, but I am not getting the log on my web host ...?

  Log stuff from app.config <system.diagnostics> <sources> <source name="System.ServiceModel.MessageLogging" switchValue="Warning, ActivityTracing" > <listeners> <add name="ServiceModelTraceListener"/> </listeners> </source> <source name="System.ServiceModel" switchValue="Verbose,ActivityTracing"> <listeners> <add name="ServiceModelTraceListener"/> </listeners> </source> <source name="System.Runtime.Serialization" switchValue="Verbose,ActivityTracing"> <listeners> <add name="ServiceModelTraceListener"/> </listeners> </source> </sources> <sharedListeners> <add initializeData="App_tracelog.svclog" type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" name="ServiceModelTraceListener" traceOutputOptions="Timestamp"/> </sharedListeners> </system.diagnostics> 

Can anyone detect the problem, I have this setting, like a thousand times.

  request.KeepAlive = false; System.Net.ServicePointManager.Expect100Continue = false; 

Charles

Answer 1

Try loading the page as a string first, and then pass it to the HtmlAgilityPack. This will allow you to isolate errors that occur during the boot process from those that occur during the html parsing process. If you have a problem with proxybonanza (see end of post), you can isolate this problem from the HtmlAgilityPack configuration problem.

Load the page using WebClient:

 // Download page System.Net.WebClient client = new System.Net.WebClient(); client.Proxy = new System.Net.WebProxy("{proxy address and port}"); string html = client.DownloadString("http://example.com"); // Process result HtmlAgilityPack.HtmlDocument htmlDoc = new HtmlAgilityPack.HtmlDocument(); htmlDoc.LoadHtml(html); 

If you need more control over the request, use System.Net.HttpWebRequest :

 // Create request HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://example.com/"); // Apply settings (including proxy) request.Proxy = new WebProxy("{proxy address and port}"); request.KeepAlive = false; request.Timeout = 100000; request.ReadWriteTimeout = 1000000; request.ProtocolVersion = HttpVersion.Version10; // Get response try { HttpWebResponse response = (HttpWebResponse)request.GetResponse(); Stream stream = response.GetResponseStream(); StreamReader reader = new StreamReader(stream); string html = reader.ReadToEnd(); } catch (WebException) { // Handle web exceptions } catch (Exception) { // Handle other exceptions } // Process result HtmlAgilityPack.HtmlDocument htmlDoc = new HtmlAgilityPack.HtmlDocument(); htmlDoc.LoadHtml(html); 

Also, make sure your proxy provider (proxybonanza) allows you to access your proxies from your production environment. Most providers will restrict access to proxy servers to specific IP addresses. They may have allowed access to the external IP address of the network where you work locally, but NOT the external IP address of your work environment.

Answer 2

It looks like your web host has disabled outgoing connections from ASP.NET applications for security, as this allowed other scripts / applications to perform malicious attacks from their servers.

You will need to ask them to unlock the connections in your account, but do not be surprised if they say no.