Why do my REST .NET clients send each request without authentication headers and then replay it with an authentication header?

Question

We run the REST web service with the API, requiring clients to use basic authentication. We created a set of neat samples in different languages, demonstrating how to interact with our service. Now I look at the IIS service logs and see that the following pattern happens quite often:

• a request arrives, is rejected with HTTP code 401
• the same request is resubmitted and completed successfully

which looks like the first request is sent without authorization headers, and then the second one is sent with the correct headers and succeeds. In most cases, the log entry contains a "user agent", which is the same line that we installed in our .NET example.

Therefore, I assume that the problem is only with .NET programs. The problem cannot be reproduced using our sample code, so I assume that users have somehow modified the code or written their own from scratch.

We tried to contact users, but apparently they do not want to invest time in research. Therefore, it would be nice to find the most likely scenario that leads to this behavior of .NET programs.

Why would they do this? Why didn't they attach the headers on the first try?

Answer 1

This is the default behavior of the HttpClient and HttpWebRequest , which is displayed as follows.

Note. The text below explains the sub-optimal behavior causing the problem described in the question. Most likely, you should not write your code like this. Instead, scroll through the adjusted code below.

In both cases, create an instance of the NetworkCredenatial object and specify the username and password there.

 var credentials = new NetworkCredential( username, password ); 

If you use HttpWebRequest - set the .Credentials property:

 webRequest.Credentials = credentials; 

If you use HttpClient - pass the credential object to the HttpClientHandler (modified code from here ):

 var client = new HttpClient(new HttpClientHandler() { Credentials = credentials }) 

Then run Fiddler and run the query. You will see the following:

• the request is sent without an authorization header
• the service responds with HTTP 401 and WWW authentication: Basic realm = "UrRealmHere"
• the request is sent with the appropriate authorization header (and successfully)

This is explained here - the client does not know in advance that the service requires Basic and is trying to agree on an authentication protocol (and if the service requires a digest, sending the main headers in clear form is useless and can endanger the client).

Note: here the suboptimal explanation of the behavior ends and the best approach is explained. Most likely, you should use the code below instead of the code above.

In cases where it is known that the service requires Basic, an additional request can be eliminated as follows:

Do not install .Credentials , add headers manually instead, using the code here . Encode username and password:

 var encoded = Convert.ToBase64String( Encoding.ASCII.GetBytes( String.Format( "{0}:{1}", username, password ) ) ); 

When using HttpWebRequest add it to the headers:

 request.Headers.Add( "Authorization", "Basic " + encoded ); 

and when using HttpClient add it to the default headers:

 client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue( "Basic", encoded ); 

When you do this, the request is sent with the right authorization header each time. Please note that you must not install .Credentials , otherwise, if the username or password is incorrect, the same request will be sent twice both times with incorrect credentials and both times, of course, giving way to HTTP 401.