What if 20,000 Windows Azure Storage Transactions Per Second Isn’t Enough?

The scalability targets for a single Windows Azure Storage Account created after June 7^th 2012 are as follows:

• Capacity – Up to 200 TBs
• Transactions – Up to 20,000 entities/messages/blobs per second
• Bandwidth for a Geo Redundant storage account
  • Ingress – up to 5 gigabits per second
  • Egress – up to 10 gigabits per second
• Bandwidth for a Locally Redundant storage account
  • Ingress – up to 10 gigabits per second
  • Egress – up to 15 gigabits per second

Why Does This Matter?

Within a Windows Azure Storage Account, all of the objects are grouped in partitions as described here. Therefore, it is important to understand the performance targets of a single partition for each storage service.

The below Queue and Table throughputs were achieved using an object size of 1KB:

• Single Queue– all of the messages in a queue are accessed via a single queue partition. A single queue is targeted to be able to process:
  • Up to 2,000 messages per second
• Single Table Partition– a table partition are all of the entities in a table with the same partition key value, and usually tables have many partitions. The throughput target for a single table partition is:
  • Up to 2,000 entities per second
  • Note, this is for a single partition, and not a single table. Therefore, a table with good partitioning, can process up to the 20,000 entities/second, which is the overall account target described above.
• Single Blob– the partition key for blobs is the “container name + blob name”, therefore we can partition blobs down to a single blob per partition to spread out blob access across our servers. The target throughput of a single blob is:
  • Up to 60 MBytes/sec

When your application reaches the limits of its Windows Azure Storage Account, it will start to receive
“503 Server Busy” or “500 Operation Timeout” responses. At this point you have a few options available.

• Apply a back off strategy in order to relieve the pressure on the service.
• Vertically partition your data over multiple Windows Azure Storage Account. In other words, have a Windows Azure Storage Account for each type of data. For example, have user profiles in one account and your diagnostics in a different account.
• Horizontally partition your data over multiple Windows Azure Storage Account by  implementing a sharding mechanism.

Sharding is a concepts that usually comes up when we talk about databases and is normally achieved by splitting data horizontally over multiple instances of the data store. Fortunately, this concept translates well to Windows Azure Storage and it can become quite handy when your application needs more transactions per second.

To a certain extent, best practices mention horizontal partitioning of your data in order to maintain and ensure consistent performance for your end users. Best practices for SQL Server on a Virtual Machine mentions that it is possible to strip VHDs from multiple Windows Azure Storage Accounts to benefit from a higher IOPS (IO Operations per Second). Furthermore, it is recommended to use different Windows Azure Storage Accounts for your data and the application’s diagnostics.

Services like the Windows Azure Table Storage Service have a certain amount of auto scaling built-in to ensure consistent performance per partition. Although this usually provides more than enough performance for the majority of applications, it’s still limited to 20,000 transaction per second by the Windows Azure Storage Account.

The Windows Azure Blob Storage Service can be scaled by leveraging the power of the CDN, but keep in mind that the CDN is read only and perfect for data that does not change very often.

The Windows Azure Queues Storage Service is partitioned by individual queues. Scaling can be achieved by multiplying the number of queues and by consuming them in a round robin fashion. To augment the possible number of transactions per second, each Queue needs to reside on a different Windows Azure Storage Account.

The key here is to partition your data over multiple Windows Azure Storage Accounts in order to benefit from the combined through put.

In this post I will deal more specifically with sharding using the Windows Azure Blob Storage Service.

The Concept

I needed a way to spread data across 10 Windows Azure Storage Accounts using a round robin strategy. To achieve this I created the ShardedBlobContainer. It’s quite basic, but the purpose of this example is to give you ideas about how to go about solving these kinds of issues. The code below is an example of how to use the ShardedBlobContainer in order to upload 20 images in parallel. The files are distributed evenly over the available Windows Azure Storage Accounts.

The example below first makes sure that the container exists in all Windows Azure Storage Accounts. Then it reads from all the accounts to find out how many blobs are currently in the containers. This gives me the baseline for my test. Now I start uploading twenty 2 megabyte files in parallel. The  Upload method takes care of the distribution. Once the upload is complete, I then list all the files from the containers.

static void Main(string[] args)
{
    var connectionStrings = new List<string>
    {
        "DefaultEndpointsProtocol=https;AccountName=brisebois01;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois02;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois03;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois04;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois05;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois06;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois07;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois08;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois09;AccountKey=abcdefg",
        "DefaultEndpointsProtocol=https;AccountName=brisebois010;AccountKey=abcdefg",
    };

    var accounts = connectionStrings.Select(CloudStorageAccount.Parse)
                                    .ToList();

    var container = new ShardedBlobContainer("documents", accounts);

    container.CreateIfNotExists();
    PrintNumberOfBlobs(container);

    int file = 1;

    for (int cycle = 1; cycle <= 4; cycle++)
    {
        Enumerable.Range(0, 5).ToList().ForEach(i =>
        {   
            var fs = File.OpenRead(@"C:\Users\Brisebois\Pictures\IMG_1002.jpg");
            container.Upload("file" + file, fs, new AsyncCallback(ar =>
            {
                if (ar.IsCompleted)
                    Console.WriteLine(((CloudBlockBlob)ar.AsyncState).Uri.ToString());
            }));
            file++;
        });
    }

    Console.ReadLine();
    PrintNumberOfBlobs(container);
    Console.ReadLine();
}

public static void PrintNumberOfBlobs(ShardedBlobContainer container)
{
    var totalBlobs = container.Find(null, true).Count();

    Console.WriteLine(totalBlobs + " |> blobs were found");
    Console.WriteLine("");
    foreach (var blob in container.Find(null, true))
    {
        Console.WriteLine(((CloudBlockBlob)blob).Uri.ToString());
    }
    Console.WriteLine("");
}

The Console below shows the trace for my example. It first notes that none my containers have any data. Then it upload the twenty files, the URIs show that they are uploaded in parallel and that they do not have a predefined order. Then once the files have been uploaded, I query the containers and list their contents in order to validate the uploads.

The ShardedBlobContainer takes care of selecting a random last location when it is instantiated. This is done in order to ensure a better distribution. If this was not in place, the first containers would contain the majority of the blob and the containers would be unbalanced. By ensuring an even distribution of blobs in the containers, I am hoping for an optimal number of transactions per second for the overall sharded container.

The code below is by no means complete. It is an example that can be used to build upon.

public class ShardedBlobContainer
{
    private readonly string name;
    
    private readonly List<CloudBlobContainer> containers;
    private readonly int count;
    private int lastLocation;

    public ShardedBlobContainer(string name,
                                IEnumerable<CloudStorageAccount> storagAccounts)
    {
        this.name = name;

        var accounts = storagAccounts.ToList();

        count = accounts.Count();

        var random = new Random(DateTime.Now.Millisecond);
        lastLocation = random.Next(count);

        var clients = accounts.Select(a => a.CreateCloudBlobClient())
                                .ToList();
            
        containers = clients.Select(c => c.GetContainerReference(name))
                            .ToList();
    }

    public string Name
    {
        get { return name; }
    }

    public void CreateIfNotExists()
    {
        containers.AsParallel().ForAll(c => c.CreateIfNotExists());
    }

    public ICancellableAsyncResult Upload(string blobName,
                                            string content,
                                            AsyncCallback callback)
    {
        var stream = new MemoryStream();
        var writer = new StreamWriter(stream);

        writer.Write(content);
        writer.Flush();
            
        return Upload(blobName, stream, callback);
    }

    readonly object lockObject = new object();

    public ICancellableAsyncResult Upload(string blobName,
                                            Stream stream,
                                            AsyncCallback callback)
    {
        lock (lockObject)
            lastLocation = (lastLocation + 1)%count;

        var container = containers[lastLocation];
        var blobReferenceFromServer = container.GetBlockBlobReference(blobName);
        return blobReferenceFromServer
                .BeginUploadFromStream(stream,
                                        callback,
                                        blobReferenceFromServer);
    }

    public IEnumerable<ICloudBlob> Find(string prefix, bool useFlatBlobListing)
    {
        return containers.AsParallel()
            .SelectMany(c => c.ListBlobs(prefix, useFlatBlobListing))
            .Cast<ICloudBlob>()
            .ToList();
    }
}

In Summary

By distributing blobs over 10 different Windows Azure Storage Accounts the number of potential transactions per second jumps from 20,000 to 200,000. This can be quite valuable when you need to modify a large amount of blobs simultaneously.

Horizontally partitioning your data over multiple Windows Azure Storage Account by implementing a sharding mechanism may not be trivial, but it may be necessary for large applications that consume an impressive amount of transactions.

The important lesson to pull from this post, is that whether you choose to use sharding or not, it’s crucial to take time to think about your data partitioning strategy. Because, in order to provide an acceptable end user experience, you need to ensure that your application will operate within the prescribed Windows Azure Storage Account performance targets.