Async NancyFX with HttpClient - GetAsync for fbcdn returns 403?

Question

NOTE. I pretty much changed this question, trying to point out a problem. The comments below do not reflect this question.

I am trying to get this image from fbcdn:

https://scontent.xx.fbcdn.net/v/t1.0-1/c15.0.50.50/p50x50/10354686_10150004552801856_220367501106153455_n.jpg?oh=6c801f82cd5a32fd6e5a4258ce00a314oe

The browser gets everything as best as possible. Here is my code:

public class ReverseProxyController : NancyModule { public ReverseProxyController() { Get["/", true] = async (parameters, ct) => { var result = await GetResult(parameters, ct); return result; }; } private async Task<Response> GetResult(dynamic parameters, CancellationToken ct) { var client = new HttpClient(); string url = Request.Query["url"].Value.ToString(); if (url == null) return null; client.DefaultRequestHeaders.Add("Access-Control-Allow-Origin", "*"); client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"); client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1"); client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"); client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, sdch, br"); client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.8,ru;q=0.6"); var response = await client.GetAsync(url, ct); ct.ThrowIfCancellationRequested(); switch (response.StatusCode) { case HttpStatusCode.OK: var stream = await response.Content.ReadAsStreamAsync(); return Response.FromStream(stream, response.Content.Headers.ContentType != null ? response.Content.Headers.ContentType.ToString() : "application/octet-stream"); default: return Response.AsText("\nError " + response.StatusCode); } } } 

I get a 403 Forbidden answer every time. I thought adding headers would make it work, but it wouldn't go away.

This code works for other images on other hosts, such as this:

https://s-media-cache-ak0.pinimg.com/564x/ff/f0/c9/fff0c988a4516659d4009f60e0694cb6.jpg

Answer 1

The problem is finding the URL from Nancy's request, not when retrieving data from HttpClient .

Suppose you send a request to nancy, for example:

http: // localhost /? url = ...

so for facebook it will be:

Http: // localhost /? url = https: //scontent.xx.fbcdn.net/v/t1.0-1/c15.0.50.50/p50x50/10354686_10150004552801856_220367501106153455_n.jpg? oh = 6c801f82cd5a32f148e5a5a5a5a5a5a5a5a5a5a004003e2ee4e5e5a5a004003e2ee4e5e5a0040014e2e8e3a00400142e2ece2e2e2e2e8e5e5a00400142e2ece2e2e2ce2e2e8e3

but for this url string url = Request.Query["url"].Value.ToString(); is incomplete and the last part is missing (& oe = 589AAD2F), so server responses are prohibited.

Here is a simple change to demonstrate the problem:

 private async Task<Response> GetResult(dynamic parameters, CancellationToken ct) { var client = new HttpClient(); var req = Request.Url.ToString(); var queryStart = req.IndexOf("url="); if (queryStart == -1) return Nancy.HttpStatusCode.BadRequest; var url = req.Substring(queryStart + 4); if (string.IsNullOrEmpty(url)) return Nancy.HttpStatusCode.BadRequest; client.DefaultRequestHeaders.Add("Access-Control-Allow-Origin", "*"); client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"); client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1"); client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"); client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, sdch, br"); client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.8,ru;q=0.6"); var response = await client.GetAsync(url, ct); ct.ThrowIfCancellationRequested(); switch (response.StatusCode) { case System.Net.HttpStatusCode.OK: var stream = await response.Content.ReadAsStreamAsync(); return Response.FromStream(stream, response.Content.Headers.ContentType != null ? response.Content.Headers.ContentType.ToString() : "application/octet-stream"); default: return Response.AsText("\nError " + response.StatusCode); } } 

Decision

We can actually encode the URL before sending it, and Nancy will automatically decrypt the URL for us, so there is no need to change anything on the server side.

Here is an example link generated using the HttpUtility.UrlEncode applied to

https://scontent.xx.fbcdn.net/v/t1.0-1/c15.0.50.50/p50x50/10354686_10150004552801856_220367501106153455_n.jpg?oh=6c801f82cd5a32fd6e5a4258ce00a314oe

Result:

HTTPS% 3a% 2f% 2fscontent.xx.fbcdn.net% 2fv% 2ft1.0-1% 2fc15.0.50.50% 2fp50x50% 2f10354686_10150004552801856_220367501106153455_n.jpg% 3foh% 3d6c8a2018f2f2d2f8d2f8d2f8d2f2a8f2a8f2a0f2a0f8a8f2a0f2

and the actual request for this particular link would look like this:

http: // localhost: 9876 /? url = https% 3a% 2f% 2fscontent.xx.fbcdn.net% 2fv% 2ft1.0-1% 2fc15.0.50.50% 2fp50x50% 2f10354686_10150004552801856_220367501106153455f4fjfjfjfj % 3d589AAD2F

Alternative solution

I personally prefer POST over GET in this situation, so here it is:

 public class ReverseProxyController : NancyModule { class ProxyRequest { public string Url { get; set; } } public ReverseProxyController() { Post["/", true] = async (parameters, ct) => { var result = await GetResult(parameters, ct); return result; }; } private async Task<Response> GetResult(dynamic parameters, CancellationToken ct) { var pReq = this.Bind<ProxyRequest>(); var url = pReq.Url; if (string.IsNullOrEmpty(url)) return null; var client = new HttpClient(); client.DefaultRequestHeaders.Add("Access-Control-Allow-Origin", "*"); client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"); client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1"); client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"); client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, sdch, br"); client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.8,ru;q=0.6"); var response = await client.GetAsync(url, ct); ct.ThrowIfCancellationRequested(); switch (response.StatusCode) { case System.Net.HttpStatusCode.OK: var stream = await response.Content.ReadAsStreamAsync(); return Response.FromStream(stream, response.Content.Headers.ContentType != null ? response.Content.Headers.ContentType.ToString() : "application/octet-stream"); default: return Response.AsText("\nError " + response.StatusCode); } } } 

Answer 2

as a console application whose code works great

  static void Main(string[] args) { GetStuff(); Console.ReadLine(); } private static async void GetStuff() { CancellationToken ct = new CancellationToken(); var client = new HttpClient(); //this is only a Simple Change to demonstrate the problem, and should not be considered as a proper solution string url = "https://scontent.xx.fbcdn.net/v/t1.0-1/c15.0.50.50/p50x50/10354686_10150004552801856_220367501106153455_n.jpg?oh=6c801f82cd5a32fd6e5a4258ce00a314&oe=589AAD2F"; if (url == null) return; client.DefaultRequestHeaders.Add("Access-Control-Allow-Origin", "*"); client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"); client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1"); client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"); client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, sdch, br"); client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.8,ru;q=0.6"); var response = await client.GetAsync(url, ct); ct.ThrowIfCancellationRequested(); switch (response.StatusCode) { case System.Net.HttpStatusCode.OK: var stream = await response.Content.ReadAsStreamAsync(); break; default: break; } } 

I get an OK 200 return message. Are these other images you use that may be required to log in?