Don’t be Fooled, Blob Lease Management is Tricky Business

Leasing Blobs on Azure Blob Storage Service establishes a lock for write and delete operations. The lock duration can be 15 to 60 seconds, or can be infinite.

Personally prefer to lease a blob for a short period of time, which I renew until I decide to release the lease. To some, it might seem more convenient to lease a blob indefinitely and to release it when they’re done. But on Azure, Role instances can be taken offline for a number of reasons like maintenance updates. Blobs that are indefinitely leased can eventually be leased by a process that no longer exists.

Granted that leasing a blob for a short period of time and renewing the lease costs more, but it’s essential for highly scalable Azure Cloud Services. In situations where a process that had the original lease is prevented from completing its task, an other Role can take over.

Renewing blob leases requires extra work, but it’s really worth it when you think about it.

To help me deal with the complexity created by the necessity of continuously renewing the Blob lease, I created a Blob Lease Manager. It uses Reactive Extensions (Rx) to renew the Blob lease on a regular schedule that is based on the number of seconds specified when the lease was originally acquired.

Interesting Information About Blob Leases

Once a lease has expired, the lease ID is maintained by the Blob service until the blob is modified or leased again. A client may attempt to renew or release their lease using their expired lease ID and know that if the operation is successful, the blob has not been changed since the lease ID was last valid.

If the client attempts to renew or release a lease with their previous lease ID and the request fails, the client then knows that the blob was modified or leased again since their lease was last active. The client must then acquire a new lease on the blob.

If a lease expires rather than being explicitly released, a client may need to wait up to one minute before a new lease can be acquired for the blob. However, the client can renew the lease with their lease ID immediately if the blob has not been modified.

Using the Blob Lease Manager

1) Get a CloudBlockBlob reference

var cs = CloudConfigurationManager.GetSetting("StorageCS");

var account = CloudStorageAccount.Parse(cs);

var client = account.CreateCloudBlobClient();

var container = client.GetContainerReference("maps");
container.CreateIfNotExists();

var blob = container.GetBlockBlobReference("canada.map");

2) Create a BlobLeaseManager instance and try to acquire a lease on the blob

var manager = new BlobLeaseManager();

const double renewLeaseEveryThirtySeconds = 30d;

if (!manager.TryAcquireLease(blob, renewLeaseEveryThirtySeconds))
    throw new Exception(“Was unable to acquire a lease on the blob”);

3) If you were successful at acquiring a lease, you can now manipulate the blob

var content = blob.DownloadText();

// Work with blob content
// …

var leaseId = manager.GetLeaseId(blob);

blob.UploadText(content, Encoding.UTF8, new AccessCondition()
{
    LeaseId = leaseId
});

4) Then be sure to release the lease

manager.ReleaseLease(blob);

The Code

public class BlobLeaseManager : IDisposable
{
    private const string FAILED_ATTEMPT_AT_RELEASING_BLOB_LEASE =
        @”|> failed attempt at releasing blob lock on {0} using lease id {1}”;

    private const string FAILED_TO_ACQUIRE_LEASE = @”|> failed to acquire lease {0}”;

    private const string REMOVED_LEASE_FROM_ACQUIREDLEASES
        = @”|> removed lease from acquired leases for blob {0}”;

    private const string RENEWED_LEASE = @”|> renewed lease for {0}”;

    private readonly Dictionary<string, Lease> acquiredLeases = new Dictionary<string, Lease>();

    public void Dispose()
    {
        acquiredLeases.ForEach(pair => pair.Value.KeepAlive.Dispose());
    }

    public bool HasLease(CloudBlockBlob blob)
    {
        return acquiredLeases.ContainsKey(blob.Name);
    }

    public string GetLeaseId(CloudBlockBlob blob)
    {
        return HasLease(blob) ? acquiredLeases[blob.Name].LeaseId : string.Empty;
    }

    public void ReleaseLease(CloudBlockBlob blob)
    {
        if (!HasLease(blob)) return;

        string leaseId = GetLeaseId(blob);

        ClearLease(blob);

        try
        {
            blob.ReleaseLease(new AccessCondition
            {
                LeaseId = leaseId
            });
        }
        catch (StorageException)
        {
            Trace.WriteLine(string.Format(FAILED_ATTEMPT_AT_RELEASING_BLOB_LEASE,
                                          blob.Name,
                                          leaseId));
        }
    }

    public bool TryAcquireLease(CloudBlockBlob blob, double leaseTimeInSeconds)
    {
        if(leaseTimeInSeconds < 15 || leaseTimeInSeconds > 60)
            throw new ArgumentException(@”value must be greater than 15 and smaller than 60″, 
                                        “leaseTimeInSeconds”);
        try
        {
            string proposedLeaseId = Guid.NewGuid().ToString();

            var leaseTime = TimeSpan.FromSeconds(leaseTimeInSeconds);
            string leaseId = blob.AcquireLease(leaseTime,
                                                        proposedLeaseId);

            UpdateAcquiredLease(blob, leaseId, leaseTimeInSeconds);
                
            return true;
        }
        catch (StorageException)
        {
            Trace.WriteLine(string.Format(FAILED_TO_ACQUIRE_LEASE, blob.Name));
            return false;
        }
    }

    private void UpdateAcquiredLease(CloudBlockBlob blob,
                                     string leaseId,
                                     double lockTimeInSeconds)
    {
        var name = blob.Name;

        if (IsAcquiredLeaseMissMatched(name, leaseId))
            ClearLease(blob);
        else
            acquiredLeases.Add(name, MakeLease(blob, leaseId, lockTimeInSeconds));
    }

    private bool IsAcquiredLeaseMissMatched(string name, string leaseId)
    {
        return acquiredLeases.ContainsKey(name) &&
                acquiredLeases[name].LeaseId != leaseId;
    }

    private void ClearLease(CloudBlockBlob blob)
    {
        var name = blob.Name;

        Lease lease = acquiredLeases[name];
        lease.KeepAlive.Dispose();
        acquiredLeases.Remove(name);
    }

    private Lease MakeLease(CloudBlockBlob blob,
                            string leaseId,
                            double lockTimeInSeconds)
    {
        TimeSpan interval = TimeSpan.FromSeconds(lockTimeInSeconds – 1);

        return new Lease
        {
            LeaseId = leaseId,
            KeepAlive = Observable.Interval(interval)
                .Subscribe(l => RenewLease(blob))
        };
    }

    private bool RenewLease(CloudBlockBlob blob)
    {
        if (!HasLease(blob))
            return false;

        var name = blob.Name;

        try
        {
            blob.RenewLease(new AccessCondition
            {
                LeaseId = acquiredLeases[name].LeaseId
            });

            Trace.WriteLine(string.Format(RENEWED_LEASE, name));

            return true;
        }
        catch (StorageException)
        {
            acquiredLeases.Remove(name);

            Trace.WriteLine(string.Format(REMOVED_LEASE_FROM_ACQUIREDLEASES, name));

            return false;
        }
    }

    private struct Lease
    {
        internal string LeaseId { get; set; }
        internal IDisposable KeepAlive { get; set; }
    }
}

More About Blob Leases

• Lease Blob (REST API) – This resource should answer most of your questions about blob leasing