Archives For Queue Storage


Has Something Gone Wrong?

Generally, we choose to leverage Read-Access Geo-Redundant Azure Storage Accounts (RA-GRS) because we can use it as part of our disaster recovery (DR) plan. And sometimes, we forget that our devil is in the details. Disaster recovery (DR) plans are rarely tested and can cause headaches when they are. So let’s relieve some of those headaches.

Headache…

“Geo Replication Lag” for GRS and RA-GRS Accounts is the time it takes for data stored in the Primary Region of the storage account to replicate to the Secondary Region of the storage account. Because GRS and RA-GRS Accounts are replicated asynchronously to the Secondary Region, data written to the Primary Region of the storage account will not be immediately available in the Secondary Region. Customers can query the Geo Replication Lag for a storage account, but Microsoft does not provide any guarantees as to the length of any Geo Replication Lag under this SLA.

The Recovery Time Objective (RTO) and Recovery Point Objective (RPO) are the first items that come up in DR discussions. When we use RA-GRS we control the RTO because we decide when to read from the secondary location. The RPO is a bit different because that can vary due to physics and load. The best way get current Recovery Point (RP) is to get the last sync time for the RA-GRS in question. This post is all about getting the right information, when we need it, because we need facts to make the right decisions. Continue Reading…


The Challenge

As developers, we deal with lots of complexity, and this is a good thing. It forces us to be creative, and sometimes to go beyond our known universe to overcome challenges.

Microsoft Azure is designed to help us make the right choices. It imposes performance targets through a multitude of mechanisms like throttling and quotas. One of which, I’m sure you have come to know, is that we cannot scale a Cloud Service to zero instances. Let’s stop for a moment and think about this limitation for a second. How would you creatively overcome this challenge? Continue Reading…


Moving to Microsoft Azure

Every time I dig a little deeper into Azure, I’m amazed at how much there is to know. Having done a few projects with Cloud Services in the past, I thought it would be interesting to see if it was possible to lift and shift a Console Application into an Azure Worker Role.

Lift and Shift: The action of moving a workload to a new environment, without altering the application’s code.

Continue Reading…


Working with Microsoft Azure Resources

On September 22 2014, I had the pleasure of speaking to the MSDEVMTL community about working with Microsoft Azure Resources.

Microsoft Azure Resources include Blob Storage, Table Storage, Queue Storage, Service Bus, Virtual Machines, Cloud Services and SQL Database. During my talk, I introduced a couple of tools that allow us to work with these resources. Some tools are built by Microsoft others are built by companies like Cerebrata, Cloud Berry and Zudio. Continue Reading…


Updated May 1st 2015

Azure-Storage-Queue-vs-Azure-ServiceBus-Queue There are two kinds of queues in Microsoft Azure. The first is Azure Storage Queues and the second is Azure Service Bus Queues. Both are quite different.

Meet the Azure Queues

The Azure Storage Queue is capable of handling 2000 message transactions per second and each message has a maximum time to live of 7 days. Billing is calculated by storage transaction and size of the storage used by the queue.

The Azure Service Bus Queue is capable of handling 2000 message transaction per second and messages can live forever! Billing is calculated by the number of 64kb message transactions. This Queue flavor is feature rich and is especially interesting in Hybrid scenarios where part of the application is Cloud based and the other part is on-premise in your data center.

Continue Reading…


Have you tried forcing an unexpected reboot?

Imagine a Worker Role that continuously polls a Windows Azure Storage Queue. It probably got it’s configuration settings when the Role instance started and it might not use the CloudConfigurationManager to refresh its configuration for every pull. This is where the Changing and Changed events come in. They allow you to build a strategy to deal efficiently with configuration changes.

Details

The Changing event and the Changed event are used together to identify and manage configuration changes to the service model. By using the Changing event, an instance can respond to a configuration change in one of the following ways:

  • Accept the configuration change while it is running, without going offline.

  • Set the Cancel property of RoleEnvironmentChangingEventArgs to true to take the instance offline, apply the configuration change, and then bring the instance back online.

By using the Cancel property, you can ensure that the instance proceeds through an orderly shutdown sequence and is taken offline before the configuration change is applied. During the shutdown process, Windows Azure raises the Stopping event, and then runs any code in the OnStop method.

You have a limited amount of time to accept or cancel the Changing event. Make sure your event handler can return in a timely manner.

You may want to cancel the Changing event if:

  • Your role instance does not support configuration changes while it is running, and requires recycling in order to apply the change.
  • Your role instance is performing work that should not be disrupted by a configuration change, and needs to proceed through the shutdown sequence before applying the change.

The RoleEnvironmentChangingEventArgs class provides a Changes property that returns a collection of the configuration changes that are about to be applied to the instance. Objects in this collection can be one of the following types:

Continue Reading…

Poison Queues Are A Must!

August 14, 2013 — 1 Comment

poisonAlong with including a readable copy of the original queue message along with the stack trace in your application’s diagnostics, it’s absolutely imperative that you implement a poison queue.

Poison or dead letter queues are essential in pull based systems, because they allow us to relieve the system from having to keep processing the same message over and over again.

A typical pull based system will use queues to absorb and protect services from peak loads. Allowing them to run at their own pace. Furthermore, it allows us to distribute the queue processing load over many compute nodes. Adding and removing compute nodes can be achieved by using the Auto Scaling features which can be found in the Windows Azure Management Portal.

Continue Reading…


server-caching-page-wordpressWindows Azure Storage Queues were designed with a performance target of 2,000 messages per second. I personally don’t see the queue’s performance target as a limitation. I see the queue as a partition that allows me to scale across logical and physical boundaries.

By scaling across many physical machines, the limitations imposed by the hardware becomes irrelevant.

This experiment has the goal of demonstrating how we can stitch together queues from multiple Windows Azure Storage Accounts in order to augment the maximum throughput of our Cloud Services.

Continue Reading…


IMG_2010_05_26_000018719_DxOIf your Cloud Service is composed of many Web Roles, Worker Roles or Virtual Machines, then you may be looking to cut your operational costs. When designing a Windows Azure solution, details like the way you consume Windows Azure services, are especially important. Using too many resources can generate surprising monthly bills! Consequently, it greatly influences whether the application survives its infancy.

Web & Worker Role Definitions

web role: A web role provides a dedicated Internet Information Services (IIS) web-server used for hosting front-end web applications.

worker role: Applications hosted within worker roles can run asynchronous, long-running or perpetual tasks independent of user interaction or input.

Utilizing The Full Potential of Your Cloud Services

In this example, data flows from an input queue, to converter Worker Roles that put messages on a persistence queue. The persistence Worker Role then reads from the persistence queue and inserts the data into a Windows Azure SQL Database.

Since the number of converter Worker Roles varies from 1 to 10, the persistence Worker Role has been put in place to protect the system from being throttled by the Windows Azure SQL Database Service.

Continue Reading…


experts-kid-einsteinToday I was watching the "Windows Azure Storage: What’s Coming, Best Practices, and Internals" session from Build 2013. These are the best practices that were presented.

    As I previously mentioned in “Why Are WebRequests Throttled? I Want More Throughput!” the Windows Azure Storage team recommends that we disable Nagle for small messages that are less than 14 kilobytes. They also recommend that we augment the default connection limit. By default the limit is set to 2, which isn’t much for applications on the cloud. Another recommendation made by the team was to disable the expect 100 Continue response for requests you expect to succeed.

  • ServicePointManager.UseNagleAlgorithm = false;
  • ServicePointManager.Expect100Continue = false;
  • ServicePointManager.DefaultConnectionLimit = 100 (Or more)

The use of the .Net 4.5 framework is greatly encouraged. A lot of work has gone into the Garbage Collector in order to drastically improve it. Using the latest framework is especially important for Worker Roles that continuously processes data.

Continue Reading…