Query Level Caching For Entity Framework And Windows Azure SQL Database Including Transient Fault Handling

Caching with Entity Framework has always been a challenge. Out of the box Entity Framework doesn’t have second level caching support. Even though, many open source solutions like the Scalable Object Persistence (SOP) Framework exist, I decided to implement a query level cache that uses the Transient Fault Handling Application Block to execute retry policies to transparently handle transient faults.

Keep in mind that transient faults are normal and that its not a question of if they will occur, it’s really a question of when they will occur. SQL Database instances are continuously being shifted around to prevent the service’s performance from degrading.

The Database fluent API I created executes reusable queries. Reusable queries greatly simplify the application’s design by encapsulating query logic in named concepts. They are like Stored Procedures in SQL Server, where team members can discover functionality by reading file names from the query folders. More details about reusable queries can be found in the reference section at the bottom of this post.

The code from this Post is part of the Brisebois.WindowsAzure NuGet Package

To install Brisebois.WindowsAzure, run the following command in the Package Manager Console

PM> Install-Package Brisebois.WindowsAzure

Get more details about the Nuget Package.

Asynchronously Calling REST Services From a Windows Azure Role

I regularly consume REST services from code running in Web Roles and Worker Roles. The best practice when it comes to anything going over the network, is to use the The Transient Fault Handling Application Block. The RestClient from this post is a Fluent REST Client that uses the HttpTransientErrorDetectionStrategy and a RetryPolicy to be tolerant of transient faults. Find out more about the HttpTransientErrorDetectionStrategy in an earlier post.

Take some time to get to know the characteristics of good REST services. They are part of an established convention that help us to rapidly understand and consume new REST services. 

The code from this Post is part of the Brisebois.WindowsAzure NuGet Package

To install Brisebois.WindowsAzure, run the following command in the Package Manager Console

PM> Install-Package Brisebois.WindowsAzure

Get more details about the Nuget Package.

 

My Code Has Moved! It’s Now Available on NuGet!

I have decided to centralize my code  on GitHub. As of today, you can find the code from my Windows Azure posts, bundled in Brisebois.WindowsAzure.

https://github.com/brisebois/Brisebois.WindowsAzure

I also packaged my code and made it available through Nuget.org 

https://nuget.org/packages/Brisebois.WindowsAzure/ 

To install Brisebois.WindowsAzure, run the following command in the Package Manager Console

PM> Install-Package Brisebois.WindowsAzure

Issues can be logged on GitHub

https://github.com/brisebois/Brisebois.WindowsAzure/issues

Defining an HTTP Transient Error Detection Strategy For REST Calls

public class HttpTransientErrorDetectionStrategy
    : ITransientErrorDetectionStrategy
{
    private readonly List<HttpStatusCode> statusCodes = 
        new List<HttpStatusCode>
        {
            HttpStatusCode.GatewayTimeout,
            HttpStatusCode.RequestTimeout,
            HttpStatusCode.ServiceUnavailable,
        };

    public HttpTransientErrorDetectionStrategy(bool isNotFoundAsTransient = false)
    {
        if (isNotFoundAsTransient)
            statusCodes.Add(HttpStatusCode.NotFound);
    }

    public bool IsTransient(Exception ex)
    {
        var we = ex as WebException;
        if (we == null)
            return false;

        var response = we.Response as HttpWebResponse;

        var isTransient = response != null 
                                   && statusCodes.Contains(response.StatusCode);
        return isTransient;
    }
}

Windows Azure Blob Storage Polling Task

The code below uses a BlobContainerWorker which implements PollingTask<TWorkItem> to read blobs from a container and compress uncompressed blobs. If it does not find uncompressed blobs, it uses an exponential back off strategy to reduce the number of transactions made on Windows Azure Blob Storage Service. Without the back off strategy, the service would continuously make requests to check if files needed to be compressed.

Consequences of making too many requests to Blob Storage Service Include

• The service might throttle your requests
• The cost of operation would be significantly higher than it should be

The screenshot above is of my test console, showing that the PollingTask is effectively applying the back off strategy. If the BlobContainerWorker finds new files to convert the back off delay will be reset to zero.

The code from this Post is part of the Brisebois.WindowsAzure NuGet Package

To install Brisebois.WindowsAzure, run the following command in the Package Manager Console

PM> Install-Package Brisebois.WindowsAzure

Get more details about the Nuget Package.

Windows Azure Queue Storage Service Polling Task

The screenshot above is of my test console, showing that the PollingTask is effectively applying the back off strategy. If the QueueWorker finds new messages to process the back off delay will be reset to zero.

The code from this Post is part of the Brisebois.WindowsAzure NuGet Package

To install Brisebois.WindowsAzure, run the following command in the Package Manager Console

PM> Install-Package Brisebois.WindowsAzure

Get more details about the Nuget Package.

Polling Tasks Are Great Building Blocks For Windows Azure Roles

Polling Tasks are building blocks for my Worker Roles. They empower me to easily organize work loads and limit costs by backing off from services when there is no work to be done. The back off delay is calculated from the number of attempts where there was no work to be done.

Polling Tasks can be used to execute work from Windows Azure Queue Storage Service or from Windows Azure Blob Storage Service. It can also be used to pull work for any external system like from email severs, FTP servers or network drives.

The code from this Post is part of the Brisebois.WindowsAzure NuGet Package

To install Brisebois.WindowsAzure, run the following command in the Package Manager Console

PM> Install-Package Brisebois.WindowsAzure

Get more details about the Nuget Package.

 

Examples

• The QueueWorker implements the PollingTask in order to read messages from a Queue Storage Service. Working with Queues in Windows Azure is quite common. In many circumstances I try to make specialized Queues. This means that I store messages that can be acted upon by a single consumer. Doing so requires me to create multiple QueueWorker instances in the same Worker Role. Having many specialized QueueWorker implementations running side by side also has it benefits. Each QueueWorker can be tuned differently. Some can process 32 messages in parallel while others can process 4, this gives you a lot of free way when it comes to consuming a maximum amount of the available resources. Furthermore, if the demand for your services grow, you can easily break apart the collection of QueueWorker instances and redistribute and reorganize them over more Worker Roles.
• The BlobContainerWorker implements the PollingTask in order to read blobs from Blob Storage Service. Working with blobs in Windows Azure is very efficient. In most circumstances I try to keep private blobs compressed and publicly accessible blobs uncompressed. Keeping the private blobs compressed will ensure faster network communication and less idle time for your Worker Role CPUs.
• The BlobSnapshotCleaner implements the PollingTask in order to deleted expired blob Snapshots from Blob Storage Service. Woking with blob Snapshots is a great way to version your data. The BlobSnapshotCleaner allows you to specify the maximum age for a Snapshot and it will take care of the rest.

Calculating an Exponential Back Off Delay Based on Failed Attempts

I implemented the following formula found on Wikipedia in C#.

Furthermore, I added the ability to set a maximum delay that is quite useful when you want to control your delays. The calculated delay is in seconds.

public static class DelayCalculator
{
    public static Int64 ExponentialDelay(int failedAttempts,
                                    int maxDelayInSeconds = 1024)
    {
        //Attempt 1     0s     0s
        //Attempt 2     2s     2s
        //Attempt 3     4s     4s
        //Attempt 4     8s     8s
        //Attempt 5     16s    16s
        //Attempt 6     32s    32s

        //Attempt 7     64s     1m 4s
        //Attempt 8     128s    2m 8s
        //Attempt 9     256s    4m 16s
        //Attempt 10    512     8m 32s
        //Attempt 11    1024    17m 4s
        //Attempt 12    2048    34m 8s

        //Attempt 13    4096    1h 8m 16s
        //Attempt 14    8192    2h 16m 32s
        //Attempt 15    16384   4h 33m 4s

        var delayInSeconds = ((1d / 2d) * (Math.Pow(2d, failedAttempts) - 1d));

        return maxDelayInSeconds < delayInSeconds
            ? Convert.ToInt64(maxDelayInSeconds)
            : Convert.ToInt64(delayInSeconds);
    }
}

 

The code from this Post is part of the Brisebois.WindowsAzure NuGet Package

To install Brisebois.WindowsAzure, run the following command in the Package Manager Console

PM> Install-Package Brisebois.WindowsAzure

Get more details about the Nuget Package.

Using The Exponential Back-Off Transient Error Detection Strategy

Microsoft released The Transient Fault Handling Application Block as part of the Microsoft Enterprise Library, which targets various Windows Azure services. An interesting aspect of this block is that its easy to extend and use.

The Transient Fault Handling Application Block is a product of the collaboration between the Microsoft patterns & practices team and the Windows Azure Customer Advisory Team. It is based on the initial detection and retry strategies, and the data access support from the Transient Fault Handling Application Framework. The new block now includes enhanced configuration support, enhanced support for wrapping asynchronous calls, provides integration of the block’s retry strategies with the Windows Azure Storage retry mechanism, and works with the Enterprise Library dependency injection container. The new Transient Fault Handling Application Block supersedes the Transient Fault Handling Framework and is now a recommended approach to handling transient faults in the cloud.

Targeted services include Windows Azure SQL Database, Windows Azure Service Bus, Windows Azure Storage, and Windows Azure Caching Service. Although These are all Cloud services, it is easy to define your own detection strategies to identify known transient error conditions.

The following is an example of the Exponential Back-Off Transient Error Detection Strategy working with Entity Framework

Retry four times, waiting two seconds before the first retry, then four seconds before the second retry, then eight seconds before the third retry, and sixteen seconds before the fourth retry.

This retry strategy also introduces a small amount of random variation into the intervals. This can be useful if the same operation is being called multiple times simultaneously by the client application.