time distortion

Using Time-based Partition Keys in #Azure Table Storage

In a previous post about storing Azure Storage Table entities in descending order I combined a time-based key with a guid in order to create a unique key. This is practical when you need to use combined keys for the Row Keys or Partition Key. But it’s not practical for logs.

A better solution for logs, is to generate a Partition Key based on time. This allows you to query for logs by time periods. There are many ways to generate time-based partitions, so I will cover the two that I use the most.

The first method comes in two flavors. It consists of taking the current time value in Ticks as the Partition Key value. Some round it up to the minute, which makes it possible to do batch inserts. Remember that batch inserts are limited to 100 entities and that they must all belong to the same table partition.

var time = new DateTime(2014,06,16,2,40,0,DateTimeKind.Utc);

// Some logging libraries add an extra zero in front of the ticks
var firstForm = "0" + time.Ticks;
// First Form: 0635384832000000000

// Other services use the ticks as-is
var secondForm = time.ToUniversalTime().Ticks;
// Second Form: 635384832000000000

The second method sorts entities in descending order. Building on my previous example, let’s calculate a Partition Key that’s rounded up to the minute.

var time = new DateTime(2014, 06, 16, 2, 40, 0, DateTimeKind.Utc);

var key = String.Format("{0:D19}", (DateTime.MaxValue - time.ToUniversalTime()).Ticks);
//key: 2519994143999999999

usin

This method, can be used to calculate the Partition Keys required to query against logs persisted by the Semantic Logging Application Block (SLAB). The following is a query against Azure Table Storage, that I would use to extract events that were persisted by SLAB between June 1 2014 at 5AM UTC and  June 1 2014 at 6AM UTC.

var fromTime = new DateTime(2014, 06, 1, 5, 0, 0, DateTimeKind.Utc);
var fromTicks = (DateTime.MaxValue - fromTime.ToUniversalTime()).Ticks;
var fromKey = fromTicks.ToString("d19", CultureInfo.InvariantCulture);

var from = TableQuery.GenerateFilterCondition("PartitionKey",
                                               QueryComparisons.LessThanOrEqual,
                                               fromKey);

var toTime = new DateTime(2014, 06, 1, 6, 0, 0, DateTimeKind.Utc);
var toTicks = (DateTime.MaxValue - toTime.ToUniversalTime()).Ticks;
var toKey = toTicks.ToString("d19", CultureInfo.InvariantCulture);

var to = TableQuery.GenerateFilterCondition("PartitionKey",
                                            QueryComparisons.GreaterThan,
                                            toKey);

var query = TableQuery.CombineFilters(from, TableOperators.And, to);

// Generated query string:
// (PartitionKey le '2520007019999999999') and (PartitionKey gt '2520006983999999999')

This query string can be used in Cerebrata’s Azure Management Studio or in Visual Studio to get the desired data set. It can also be used to retrieve data through the Azure Storage Client library.

Trackbacks and Pingbacks:

  1. Dew Drop – June 17, 2014 (#1799) | Morning Dew - June 17, 2014

    […] Using Time-based Partition Keys in #Azure Table Storage (Alexandre Brisebois) […]

    Like

  2. Reading Notes 2014-06-23 | Matricis - June 25, 2014

    […] Using Time-based Partition Keys in #Azure Table Storage (Alexandre Briseboi) – Nice tips about saving our logs in Azure. […]

    Like

  3. End of Month Research Roundup – June 2014 | endjin blog - July 5, 2014

    […] Using Time-based Partition Keys in Azure Table Storage [Alexandre Brisebois] […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s