Retry 408 Timeout from Azure Table Storage Service

Question

We use the Azure table storage and periodically get 408 timeouts when performing the InsertOrMerge operation. In this case, we would like to try again, but it seems that the repetition policy is not respected for these errors.

This is the class that we use to handle table interactions. The GetFooEntityAsync method tries to retrieve an object from table storage. If it cannot, it creates a new FooEntity and adds it to the table (mapping to FooTableEntity).

public class FooTableStorageBase { private readonly string tableName; protected readonly CloudStorageAccount storageAccount; protected TableRequestOptions DefaultTableRequestOptions { get; } protected OperationContext DefaultOperationContext { get; } public CloudTable Table { get { return storageAccount.CreateCloudTableClient().GetTableReference(tableName); } } public FooTableStorage(string tableName) { if (String.IsNullOrWhiteSpace(tableName)) { throw new ArgumentNullException(nameof(tableName)); } this.tableName = tableName; storageAccount = CloudStorageAccount.Parse(ConnectionString); ServicePoint tableServicePoint = ServicePointManager.FindServicePoint(storageAccount.TableEndpoint); tableServicePoint.UseNagleAlgorithm = false; tableServicePoint.ConnectionLimit = 100; // Increasing connection limit from default of 2. DefaultTableRequestOptions = new TableRequestOptions() { PayloadFormat = TablePayloadFormat.JsonNoMetadata, MaximumExecutionTime = TimeSpan.FromSeconds(1), RetryPolicy = new OnTimeoutRetry(TimeSpan.FromMilliseconds(250), 3), LocationMode = LocationMode.PrimaryOnly }; DefaultOperationContext = new OperationContext(); DefaultOperationContext.Retrying += (sender, args) => { // This is never executed. Debug.WriteLine($"Retry policy activated in {this.GetType().Name} due to HTTP code {args.RequestInformation.HttpStatusCode} with exception {args.RequestInformation.Exception.ToString()}"); }; DefaultOperationContext.RequestCompleted += (sender, args) => { if (args.Response == null) { // This is occasionally executed - we want to retry in this case. Debug.WriteLine($"Request failed in {this.GetType().Name} due to HTTP code {args.RequestInformation.HttpStatusCode} with exception {args.RequestInformation.Exception.ToString()}"); } else { Debug.WriteLine($"{this.GetType().Name} operation complete: Status code {args.Response.StatusCode} at {args.Response.ResponseUri}"); } }; Table.CreateIfNotExists(DefaultTableRequestOptions, DefaultOperationContext); } public async Task<FooEntity> GetFooEntityAsync() { var retrieveOperation = TableOperation.Retrieve<FooTableEntity>(FooTableEntity.GenerateKey()); var tableEntity = (await Table.ExecuteAsync(retrieveOperation, DefaultTableRequestOptions, DefaultOperationContext)).Result as FooTableEntity; if (tableEntity != null) { return tableEntity.ToFooEntity(); } var fooEntity = CalculateFooEntity(); var insertOperation = TableOperation.InsertOrMerge(new FooTableEntity(fooEntity)); var executeResult = await Table.ExecuteAsync(insertOperation); if (executeResult.HttpStatusCode == 408) { // This is never executed. Debug.WriteLine("Got a 408"); } return fooEntity; } public class OnTimeoutRetry : IRetryPolicy { int maxRetryAttempts = 3; TimeSpan defaultRetryInterval = TimeSpan.FromMilliseconds(250); public OnTimeoutRetry(TimeSpan deltaBackoff, int retryAttempts) { maxRetryAttempts = retryAttempts; defaultRetryInterval = deltaBackoff; } public IRetryPolicy CreateInstance() { return new OnTimeoutRetry(TimeSpan.FromMilliseconds(250), 3); } public bool ShouldRetry(int currentRetryCount, int statusCode, Exception lastException, out TimeSpan retryInterval, OperationContext operationContext) { retryInterval = defaultRetryInterval; if (currentRetryCount >= maxRetryAttempts) { return false; } // Non-retryable exceptions are all 400 ( >=400 and <500) class exceptions (Bad gateway, Not Found, etc.) as well as 501 and 505. // This custom retry policy also retries on a 408 timeout. if ((statusCode >= 400 && statusCode <= 500 && statusCode != 408) || statusCode == 501 || statusCode == 505) { return false; } return true; } } } 

When calling GetFooEntityAsync (), the string "Request Failure" is sometimes executed. When checking the values ​​of args.RequestInformation.HttpStatusCode = 408. However:

• Debug.WriteLine("Got a 408"); in the GetFooEntity method is never executed.

• Debug.WriteLine($"Retry policy activated... in the DefaultOperationContext.Retrying DefaultOperationContext.Retrying never executed (I would expect it to be done twice - is this not a repeat?).

• DefaultOperationContext.RequestResults contains a long list of results (mostly with a status of 404, about 204 s).

According to this (rather old) blog post, exceptions with codes between 400 and 500, as well as 501 and 505, are not subject to retry. However, the timeout (408) is exactly the situation in which we would like to retry. Perhaps I need to write a custom retry policy for this case.

I don’t quite understand where the 408 comes from, since I cannot find it in the code, except when the RequestCompleted delegate is called. I am not trying to use different settings for my retry policy. What am I missing here? How can I make the operation retry on 408 from the table store?

EDIT: I updated the code to show a custom retry policy that I implemented in order to repeat error 408. However, it seems that my breakpoints on restarting still do not fall, so it seems like the attempt to restart does not start. What could be the reason my repeat policy is not activated?