C # WebClient DownloadString returns gibberish

Question

I am trying to browse the source http://simpledesktops.com/browse/desktops/2012/may/17/where-the-wild-things-are/ with the code:

String URL = "http://simpledesktops.com/browse/desktops/2012/may/17/where-the-wild-things-are/"; WebClient webClient = new WebClient(); webClient.Headers.Add("user-agent", "Mozilla/5.0 (Windows; Windows NT 5.1; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4"); webClient.Encoding = Encoding.GetEncoding("Windows-1255"); string download = webClient.DownloadString(URL); webClient.Dispose(); Console.WriteLine(download); 

When I run this, the console returns empty bullshit that looks like it was improperly decoded.

I also tried adding headers to no avail:

 webClient.Headers.Add("user-agent", "Mozilla/5.0 (Windows; Windows NT 5.1; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4"); webClient.Headers.Add("Accept-Encoding", "gzip,deflate"); 

Other websites have returned the correct html source. I can also view the page source through Chrome. What's going on here?

Answer 1

The answer to this gzipped url is, you have to unpack it or set an empty Accept-Encoding header, you do not need this user-agent field.

  String URL = "http://simpledesktops.com/browse/desktops/2012/may/17/where-the-wild-things-are/"; WebClient webClient = new WebClient(); webClient.Headers.Add("Accept-Encoding", ""); string download = webClient.DownloadString(URL); 

Answer 2

I had the same thing as me today.

Using a WebClient object to verify that the URL is returning something.

But my experience is different. I tried to remove Accept-Encoding, mostly using the @Antonio Bakula code that gave in his answer. But every time I got the same error (InvalidOperationException)

So this did not work:

 WebClient wc = new WebClient(); wc.Headers.Add("Accept-Encoding", ""); string result = wc.DownloadString(url); 

But adding "any" text as a User Agent instead did the trick. This works great:

 WebClient wc = new WebClient(); wc.Headers.Add(HttpRequestHeader.UserAgent, "My User Agent String"); System.IO.Stream stream = wc.OpenRead(url); 

Your mileage may vary, obviously, also in the note. I am using ASP.NET 4.0.30319.