Sharded Queue – Handling up to 60,000 Messages per Second

Windows Azure Storage Queues were designed with a performance target of 2,000 messages per second. I personally don’t see the queue’s performance target as a limitation. I see the queue as a partition that allows me to scale across logical and physical boundaries.

By scaling across many physical machines, the limitations imposed by the hardware becomes irrelevant.

This experiment has the goal of demonstrating how we can stitch together queues from multiple Windows Azure Storage Accounts in order to augment the maximum throughput of our Cloud Services.

What if 20,000 Windows Azure Storage Transactions Per Second Isn’t Enough?

The scalability targets for a single Windows Azure Storage Account created after June 7^th 2012 are as follows:

• Capacity – Up to 200 TBs
• Transactions – Up to 20,000 entities/messages/blobs per second
• Bandwidth for a Geo Redundant storage account
  • Ingress – up to 5 gigabits per second
  • Egress – up to 10 gigabits per second
• Bandwidth for a Locally Redundant storage account
  • Ingress – up to 10 gigabits per second
  • Egress – up to 15 gigabits per second

Why Does This Matter?

Within a Windows Azure Storage Account, all of the objects are grouped in partitions as described here. Therefore, it is important to understand the performance targets of a single partition for each storage service.