The Challenge

As developers, we deal with lots of complexity, and this is a good thing. It forces us to be creative, and sometimes to go beyond our known universe to overcome challenges.

Microsoft Azure is designed to help us make the right choices. It imposes performance targets through a multitude of mechanisms like throttling and quotas. One of which, I’m sure you have come to know, is that we cannot scale a Cloud Service to zero instances. Let’s stop for a moment and think about this limitation for a second. How would you creatively overcome this challenge?

Scaling Azure Cloud Services

When we build applications on the cloud, we usually favor scaling-out over scaling-up. Consequently, scaling in and out is supported out of the box on Microsoft Azure. On rare occasions, there is a business need to scale-up during peak hours and to scale-down for the quiet hours. This blog post will show you how to achieve this with the help of Microsoft Azure Automation and with PowerShell.

Keep in mind, that scaling up and down requires us to be creative.

Dealing With Worker Role Failovers by Scheduling Duplicate Jobs

This question came from Rajat on a post I did about using using Quartz.Net to schedule jobs in Windows Azure Worker roles.

When we are using quartz with multiple instances. If one instance that schedules the job goes down, how will the other instances maintain it. Because if one instance goes down then the triggers of that instance also goes down. [source]

Back then I didn’t have a great answer for him and I’m sorry about that.

Since then, I ran into the same problem.  While one of my long running jobs was executing, my worker role was updated by the Windows Azure Fabric Controller. Consequently, my job never completed and left the system with an incomplete data import.

In order to resolve this issue, I decided to track job completions in Windows Azure Table Storage. When a job runs to completion, it logs the current date and time along with the job’s name. Subsequent job instances check whether they’re the first or second attempt. If the first attempt has failed then the second job will run otherwise it will terminate itself.

This solution is great for jobs that run once or twice a day. This may not be the best solution for jobs that run every hour. When jobs run on a tight schedule, it may be preferable not to schedule duplicates, but to drop the failed jobs and wait for the next job instance to run.

Scheduling duplicate jobs isn’t natural, but it’s a necessity on Windows Azure. Failovers happen regularly, whether it’s for scheduled updates or critical failures, we need to equip ourselves in order to deal with these interruptions.

For this to be effective, you will need to calculate a time offset using the average time a job takes to run to completion. Add a slight buffer to this average and schedule the second job using the time from the first job plus the resulting offset. This will ensure that the second job doesn’t try to execute before the first job runs to completion.

Try it out and let me know how it turns out in your Cloud Services.

More on Windows Azure Table Storage Service

• Inserting & Modifying Large Amounts of Data in Windows Azure Table Storage Service
• Querying Over Windows Azure Table Storage Service