What if 20,000 Windows Azure Storage Transactions Per Second Isn’t Enough?

July 9, 2013 — 6 Comments

High SpeedThe scalability targets for a single Windows Azure Storage Account created after June 7th 2012 are as follows:

  • Capacity – Up to 200 TBs
  • Transactions – Up to 20,000 entities/messages/blobs per second
  • Bandwidth for a Geo Redundant storage account
    • Ingress – up to 5 gigabits per second
    • Egress – up to 10 gigabits per second
  • Bandwidth for a Locally Redundant storage account
    • Ingress – up to 10 gigabits per second
    • Egress – up to 15 gigabits per second

Why Does This Matter?

Within a Windows Azure Storage Account, all of the objects are grouped in partitions as described here. Therefore, it is important to understand the performance targets of a single partition for each storage service.

The below Queue and Table throughputs were achieved using an object size of 1KB:

  • Single Queue– all of the messages in a queue are accessed via a single queue partition. A single queue is targeted to be able to process:
    • Up to 2,000 messages per second
  • Single Table Partition– a table partition are all of the entities in a table with the same partition key value, and usually tables have many partitions. The throughput target for a single table partition is:
    • Up to 2,000 entities per second
    • Note, this is for a single partition, and not a single table. Therefore, a table with good partitioning, can process up to the 20,000 entities/second, which is the overall account target described above.
  • Single Blob– the partition key for blobs is the “container name + blob name”, therefore we can partition blobs down to a single blob per partition to spread out blob access across our servers. The target throughput of a single blob is:
    • Up to 60 MBytes/sec

When your application reaches the limits of its Windows Azure Storage Account, it will start to receive
503 Server Busy” or “500 Operation Timeout” responses. At this point you have a few options available.

Sharding is a concepts that usually comes up when we talk about databases and is normally achieved by splitting data horizontally over multiple instances of the data store. Fortunately, this concept translates well to Windows Azure Storage and it can become quite handy when your application needs more transactions per second.

To a certain extent, best practices mention horizontal partitioning of your data in order to maintain and ensure consistent performance for your end users. Best practices for SQL Server on a Virtual Machine mentions that it is possible to strip VHDs from multiple Windows Azure Storage Accounts to benefit from a higher IOPS (IO Operations per Second). Furthermore, it is recommended to use different Windows Azure Storage Accounts for your data and the application’s diagnostics.

Services like the Windows Azure Table Storage Service have a certain amount of auto scaling built-in to ensure consistent performance per partition. Although this usually provides more than enough performance for the majority of applications, it’s still limited to 20,000 transaction per second by the Windows Azure Storage Account.

The Windows Azure Blob Storage Service can be scaled by leveraging the power of the CDN, but keep in mind that the CDN is read only and perfect for data that does not change very often.

The Windows Azure Queues Storage Service is partitioned by individual queues. Scaling can be achieved by multiplying the number of queues and by consuming them in a round robin fashion. To augment the possible number of transactions per second, each Queue needs to reside on a different Windows Azure Storage Account.

The key here is to partition your data over multiple Windows Azure Storage Accounts in order to benefit from the combined through put.

In this post I will deal more specifically with sharding using the Windows Azure Blob Storage Service.

The Concept

I needed a way to spread data across 10 Windows Azure Storage Accounts using a round robin strategy. To achieve this I created the ShardedBlobContainer. It’s quite basic, but the purpose of this example is to give you ideas about how to go about solving these kinds of issues. The code below is an example of how to use the ShardedBlobContainer in order to upload 20 images in parallel. The files are distributed evenly over the available Windows Azure Storage Accounts.

The example below first makes sure that the container exists in all Windows Azure Storage Accounts. Then it reads from all the accounts to find out how many blobs are currently in the containers. This gives me the baseline for my test. Now I start uploading twenty 2 megabyte files in parallel. The  Upload method takes care of the distribution. Once the upload is complete, I then list all the files from the containers.

static void Main(string[] args)
{
    var connectionStrings = new List<string>
    {
        "DefaultEndpointsProtocol=https;AccountName=brisebois01;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois02;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois03;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois04;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois05;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois06;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois07;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois08;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois09;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois010;AccountKey=abcdefg",
    };

    var accounts = connectionStrings.Select(CloudStorageAccount.Parse)
                                    .ToList();

    var container = new ShardedBlobContainer("documents", accounts);

    container.CreateIfNotExists();
    PrintNumberOfBlobs(container);

    int file = 1;

    for (int cycle = 1; cycle <= 4; cycle++)
    {
        Enumerable.Range(0, 5).ToList().ForEach(i =>
        {   
            var fs = File.OpenRead(@"C:\Users\Brisebois\Pictures\IMG_1002.jpg");
            container.Upload("file" + file, fs, new AsyncCallback(ar =>
            {
                if (ar.IsCompleted)
                    Console.WriteLine(((CloudBlockBlob)ar.AsyncState).Uri.ToString());
            }));
            file++;
        });
    }

    Console.ReadLine();
    PrintNumberOfBlobs(container);
    Console.ReadLine();
}

public static void PrintNumberOfBlobs(ShardedBlobContainer container)
{
    var totalBlobs = container.Find(null, true).Count();

    Console.WriteLine(totalBlobs + " |> blobs were found");
    Console.WriteLine("");
    foreach (var blob in container.Find(null, true))
    {
        Console.WriteLine(((CloudBlockBlob)blob).Uri.ToString());
    }
    Console.WriteLine("");
}

The Console below shows the trace for my example. It first notes that none my containers have any data. Then it upload the twenty files, the URIs show that they are uploaded in parallel and that they do not have a predefined order. Then once the files have been uploaded, I query the containers and list their contents in order to validate the uploads.

2013-07-08_20h25_19

The ShardedBlobContainer takes care of selecting a random last location when it is instantiated. This is done in order to ensure a better distribution. If this was not in place, the first containers would contain the majority of the blob and the containers would be unbalanced. By ensuring an even distribution of blobs in the containers, I am hoping for an optimal number of transactions per second for the overall sharded container.

The code below is by no means complete. It is an example that can be used to build upon.

public class ShardedBlobContainer
{
    private readonly string name;
    
    private readonly List<CloudBlobContainer> containers;
    private readonly int count;
    private int lastLocation;

    public ShardedBlobContainer(string name,
                                IEnumerable<CloudStorageAccount> storagAccounts)
    {
        this.name = name;

        var accounts = storagAccounts.ToList();

        count = accounts.Count();

        var random = new Random(DateTime.Now.Millisecond);
        lastLocation = random.Next(count);

        var clients = accounts.Select(a => a.CreateCloudBlobClient())
                                .ToList();
            
        containers = clients.Select(c => c.GetContainerReference(name))
                            .ToList();
    }

    public string Name
    {
        get { return name; }
    }

    public void CreateIfNotExists()
    {
        containers.AsParallel().ForAll(c => c.CreateIfNotExists());
    }

    public ICancellableAsyncResult Upload(string blobName,
                                            string content,
                                            AsyncCallback callback)
    {
        var stream = new MemoryStream();
        var writer = new StreamWriter(stream);

        writer.Write(content);
        writer.Flush();
            
        return Upload(blobName, stream, callback);
    }

    readonly object lockObject = new object();

    public ICancellableAsyncResult Upload(string blobName,
                                            Stream stream,
                                            AsyncCallback callback)
    {
        lock (lockObject)
            lastLocation = (lastLocation + 1)%count;

        var container = containers[lastLocation];
        var blobReferenceFromServer = container.GetBlockBlobReference(blobName);
        return blobReferenceFromServer
                .BeginUploadFromStream(stream,
                                        callback,
                                        blobReferenceFromServer);
    }

    public IEnumerable<ICloudBlob> Find(string prefix, bool useFlatBlobListing)
    {
        return containers.AsParallel()
            .SelectMany(c => c.ListBlobs(prefix, useFlatBlobListing))
            .Cast<ICloudBlob>()
            .ToList();
    }
}

In Summary

By distributing blobs over 10 different Windows Azure Storage Accounts the number of potential transactions per second jumps from 20,000 to 200,000. This can be quite valuable when you need to modify a large amount of blobs simultaneously.

Horizontally partitioning your data over multiple Windows Azure Storage Account by implementing a sharding mechanism may not be trivial, but it may be necessary for large applications that consume an impressive amount of transactions.

The important lesson to pull from this post, is that whether you choose to use sharding or not, it’s crucial to take time to think about your data partitioning strategy. Because, in order to provide an acceptable end user experience, you need to ensure that your application will operate within the prescribed Windows Azure Storage Account performance targets.

6 responses to What if 20,000 Windows Azure Storage Transactions Per Second Isn’t Enough?

  1. 

    Great article. I am currently experiencing problems due to hitting the transaction limits. My storage account was created before June 2012 so I am on the lower limit of 5000 transactions per second. I am currently waiting for support from Microsoft because I believe the limit is not being applied by Windows Azure Storage Account .. but by Subscription. All my blobs, tables and queues were in the one storage account when the problem hit. I have now moved the queues (~40% of all load) into their own, new Storage Account .. I am still experiencing the problem but it affects BOTH storage accounts at the same time. Queues AND Tables will fail in separate accounts at the exact same time. I’ll leave an update here when I get a response in case it helps any one else in future!

    Like

    • 

      Steven, that’s great feedback. Please let us know how things turn out, your experience may prove to be invaluable to others (I’m personally included in that group of very interested people)

      Like

    • 
      Raistlin Majere January 23, 2014 at 7:51 PM

      Steven, your storage accounts might be on the same hardware, in that case you might have to recreate them, asking Azure support to distribute the accounts better.

      Like

  2. 
    Cristian Diaconescu July 18, 2016 at 5:21 AM

    Hi, you have a broken link: [https://azure.microsoft.com/en-us/blog/windows-azure-s-flat-network-storage-and-2012-scalability-targets/] should be [https://azure.microsoft.com/en-us/blog/windows-azures-flat-network-storage-and-2012-scalability-targets/]. They must have slightly changed how the permalinks are generated.

    Like

Trackbacks and Pingbacks:

  1. Dew Drop – July 10, 2013 (#1,582) | Alvin Ashcraft's Morning Dew - July 10, 2013

    […] What if 20,000 Windows Azure Storage Transactions Per Second Isn’t Enough? (Alexandre Brisebois) […]

    Like

  2. What if 20,000 Windows Azure Storage Transactions Per Second Isn’t Enough? | Program In .NET - July 10, 2013

    […] Read more at What if 20,000 Windows Azure Storage Transactions Per Second Isn’t Enough? […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s