Using The Exponential Back-Off Transient Error Detection Strategy

Microsoft released The Transient Fault Handling Application Block as part of the Microsoft Enterprise Library, which targets various Windows Azure services. An interesting aspect of this block is that its easy to extend and use.

The Transient Fault Handling Application Block is a product of the collaboration between the Microsoft patterns & practices team and the Windows Azure Customer Advisory Team. It is based on the initial detection and retry strategies, and the data access support from the Transient Fault Handling Application Framework. The new block now includes enhanced configuration support, enhanced support for wrapping asynchronous calls, provides integration of the block’s retry strategies with the Windows Azure Storage retry mechanism, and works with the Enterprise Library dependency injection container. The new Transient Fault Handling Application Block supersedes the Transient Fault Handling Framework and is now a recommended approach to handling transient faults in the cloud.

Targeted services include Windows Azure SQL Database, Windows Azure Service Bus, Windows Azure Storage, and Windows Azure Caching Service. Although These are all Cloud services, it is easy to define your own detection strategies to identify known transient error conditions.

The following is an example of the Exponential Back-Off Transient Error Detection Strategy working with Entity Framework

Retry four times, waiting two seconds before the first retry, then four seconds before the second retry, then eight seconds before the third retry, and sixteen seconds before the fourth retry.

This retry strategy also introduces a small amount of random variation into the intervals. This can be useful if the same operation is being called multiple times simultaneously by the client application.

 

With the above database, I am adding an Asset. Then I query the table to find out if my insert was successful.

[TestMethod]
public void UsingTheBackoffStrategy()
{
    ReliableModel.Do<FaultTestModel>(model =>
    {
        var asset = new Asset
        {
            Name = "Indexed",
            Created = DateTime.Now,
        };

        model.Assets.Add(asset);
        model.SaveChanges();
    });

    var count = ReliableModel
                    .Query<FaultTestModel, int>(model =>
                    {
                        return model.Assets.Count();
                    });

    Console.WriteLine("Number of Assets " + count);
    Assert.IsTrue(count == 1);
}

Preparing the database before each test

[TestInitialize]
public void PrepareDatabase()
{
    //Clear tables
    ReliableModel.Do<FaultTestModel>(model =>
    {
        model.Database.ExecuteSqlCommand("DELETE FROM AssetIndex");
        model.Database.ExecuteSqlCommand("DELETE FROM Asset");
    });
}

The following is a wrapper I wrote to package and standardize the way I work with Windows Azure SQL Database with Entity Framework. When I modify the state of the database, I use the Do method which wraps my modifications with a transaction. The Do method also uses the Exponential Back-Off Transient Error Detection Strategy. The Query method does not need a transaction, but it does required a retry policy because the connection to the database is not considered reliable. Therefore it also uses the Exponential Back-Off Transient Error Detection Strategy. TModel is the Entity Framework model type. It will get instantiated using the configurations found in the .config file of your project. Then its instance will get passed to your Action or Func.

public class ReliableModel
{
    public static void Do<TModel>(
        Action<TModel> action,
        int maxRetries = 4,
        int minBackoffDelayInMilliseconds = 2000,
        int maxBackoffDelayInMilliseconds = 8000,
        int deltaBackoffInMilliseconds = 2000)
           
        where TModel : IDisposable, new()
    {
        var policy = new RetryPolicy<SqlAzureTransientErrorDetectionStrategy>(
            maxRetries,
            TimeSpan.FromMilliseconds(minBackoffDelayInMilliseconds),
            TimeSpan.FromMilliseconds(maxBackoffDelayInMilliseconds),
            TimeSpan.FromMilliseconds(deltaBackoffInMilliseconds));

        policy.ExecuteAction(() =>
        {
            var tso = new TransactionOptions
            {
                IsolationLevel = IsolationLevel.ReadCommitted
            };

            using (var ts = new TransactionScope(TransactionScopeOption.Required, 
                                                 tso))
            {
                using (var model = new TModel())
                {
                    action(model);
                }
                ts.Complete();
            }
        });
    }

    public static void Do<TModel>(
        Action<TModel> action,
        Func<TModel> createModel,
        int maxRetries = 4,
        int minBackoffDelayInMilliseconds = 2000,
        int maxBackoffDelayInMilliseconds = 8000,
        int deltaBackoffInMilliseconds = 2000)
          
        where TModel : IDisposable, new()
    {
        var policy = new RetryPolicy<SqlAzureTransientErrorDetectionStrategy>(
            maxRetries,
            TimeSpan.FromMilliseconds(minBackoffDelayInMilliseconds),
            TimeSpan.FromMilliseconds(maxBackoffDelayInMilliseconds),
            TimeSpan.FromMilliseconds(deltaBackoffInMilliseconds));

        policy.ExecuteAction(() =>
        {
            var tso = new TransactionOptions
            {
                IsolationLevel = IsolationLevel.ReadCommitted,
                Timeout = TimeSpan.FromHours(24)
            };
            using (var ts = new TransactionScope(TransactionScopeOption.Required,
                                                tso))
            {
                using (var model = createModel())
                {
                    action(model);
                }
                ts.Complete();
            }
        });
    }

    public static void DoWithoutTransaction<TModel>(
        Action<TModel> action,
        Func<TModel> createModel,
        int maxRetries = 4,
        int minBackoffDelayInMilliseconds = 2000,
        int maxBackoffDelayInMilliseconds = 8000,
        int deltaBackoffInMilliseconds = 2000)

        where TModel : IDisposable, new()
    {
        var policy = new RetryPolicy<SqlAzureTransientErrorDetectionStrategy>(
            maxRetries,
            TimeSpan.FromMilliseconds(minBackoffDelayInMilliseconds),
            TimeSpan.FromMilliseconds(maxBackoffDelayInMilliseconds),
            TimeSpan.FromMilliseconds(deltaBackoffInMilliseconds));

        policy.ExecuteAction(() =>
        {
            using (var model = createModel())
            {
                action(model);
            }
        });
    }

    public static async Task<TResult> QueryAsync<TModel, TResult>(
        Func<TModel, TResult> query,
        Func<TModel> createModel,
        int maxRetries = 4,
        int minBackoffDelayInMilliseconds = 2000,
        int maxBackoffDelayInMilliseconds = 8000,
        int deltaBackoffInMilliseconds = 2000)

        where TModel : IDisposable, new()
    {
        var policy = new RetryPolicy<SqlAzureTransientErrorDetectionStrategy>(
            maxRetries,
            TimeSpan.FromMilliseconds(minBackoffDelayInMilliseconds),
            TimeSpan.FromMilliseconds(maxBackoffDelayInMilliseconds),
            TimeSpan.FromMilliseconds(deltaBackoffInMilliseconds));

        return await policy.ExecuteAsync(() => Task.Factory.StartNew(() =>
        {
            using (var model = createModel())
            {
                return query(model);
            }
        })).ConfigureAwait(false); ;
    }


    public static TResult Query<TModel, TResult>(
        Func<TModel, TResult> query,
        Func<TModel> createModel,
        int maxRetries = 4,
        int minBackoffDelayInMilliseconds = 2000,
        int maxBackoffDelayInMilliseconds = 8000,
        int deltaBackoffInMilliseconds = 2000)
            
        where TModel : IDisposable, new()
    {
        var policy = new RetryPolicy<SqlAzureTransientErrorDetectionStrategy>(
            maxRetries,
            TimeSpan.FromMilliseconds(minBackoffDelayInMilliseconds),
            TimeSpan.FromMilliseconds(maxBackoffDelayInMilliseconds),
            TimeSpan.FromMilliseconds(deltaBackoffInMilliseconds));

        return policy.ExecuteAction(() =>
        {
            using (var model = createModel())
            {
                return query(model);
            }
        });
    }
}