Copy Blobs With StartCopyFromBlob and Track its Progression Using CopyState

Copying a large blob between Windows Azure Storage Account and datacenters, will probably require a Shared Access Signature so that the destination Windows Azure Storage Account will be able to read from the source blob. Without proper authorization the source Windows Azure Storage Account will return a 404 NOT Found Http Status Code.

Once you have a Shared Access Signature, I strongly recommend using the StartCopyFromBlob method from the destination Block Blob reference. This method provides you with an ID that can be used to cancel the queued copy operation.

Then you can follow the operation’s progression through the CopyState property by pulling for an updated Block Blob reference by using the GetBlobReferenceFromServer method form the destination container.

To illustrate this, I have put together an example where I attempt to move a 575 MB file from the East US datacenter to the West US datacenter. As can be observed in the trace below, the file was moved in a little under 2 minutes.

Small blobs are usually very quick to copy and you will usually not get to see such a nice trace. The CopyState property comes in handy when something goes wrong. It can be used to identify blobs that failed to copy.

When blobs fail to copy, they are usually listed in the destination container with a length of zero and a failed CopyState status. If you try to copy from the source without deleting the failed blob from the destination container, you will receive a conflict error message.

Trying to copy blob that is not publicly visible without a Shared Access Signature will result in a not found exception.

The code below demonstrates how to copy a Block Blob from one datacenter to another. First it gets references to the source and destination containers. Then it creates a Shared Access Signature for the “db-2013-7-9-19-44.bacpac” blob, which provides read access for a period of 360 minutes. This should be enough for the copy operation.

Then a reference to the destination blob is created and the StartCopyFromBlob method is called. At this point a tasks is queued and the destination Windows Azure Storage Account will initiate the copying process.

It’s progression can be monitored by pulling updates of the destination blob from the server by using the GetBlockBlobReferenceFromServer method. Then by using the CopyState property of this updated Block Blob we are able to peer into it’s progression.

The CopyState status can be Pending, Failed, Success, Invalid and Aborted.

private void CopyLargeBlob()
{
    var eastContainer = MakeEastContainer();

    var westContainer = MakeWestContainer();

    var shareAccessUri = GetShareAccessUri("db-2013-7-9-19-44.bacpac", 
                                            360, 
                                            eastContainer);
    var blob = new Uri(shareAccessUri);

    const string blobName = "db-2013-7-9-19-44.bacpac";

    var backupBlob = westContainer.GetBlockBlobReference(blobName);
            
    var taskId = backupBlob.StartCopyFromBlob(blob);

    Console.WriteLine();
    Console.WriteLine("\tTask ID: " + taskId);

    PrintTaskProgress(westContainer, blobName);
}

private static void PrintTaskProgress(CloudBlobContainer westContainer,
                                      string blobName)
{
    CloudBlockBlob blob;
    blob = (CloudBlockBlob) westContainer.GetBlobReferenceFromServer(blobName);

    while (blob.CopyState.Status == CopyStatus.Pending)
    {
        PrintStatus(blob);
        Task.Delay(TimeSpan.FromSeconds(20d)).Wait();
        blob = (CloudBlockBlob) westContainer
                        .GetBlobReferenceFromServer(blobName);
    }
    PrintStatus(blob);
}

private static CloudBlobContainer MakeWestContainer()
{
    const string westCs = "StorageConnectionStringWest";
    var wcs = CloudConfigurationManager.GetSetting(westCs);
    var westAccount = CloudStorageAccount.Parse(wcs);

    var westClient = westAccount.CreateCloudBlobClient();

    var westContainer = westClient.GetContainerReference("destination");
    westContainer.CreateIfNotExists();
    return westContainer;
}

private static CloudBlobContainer MakeEastContainer()
{
    const string eastCs = "StorageConnectionString";
    var cs = CloudConfigurationManager.GetSetting(eastCs);
    var eastAccount = CloudStorageAccount.Parse(cs);

    var eastClient = eastAccount.CreateCloudBlobClient();
    var eastContainer = eastClient.GetContainerReference("databases");
    return eastContainer;
}

private string GetShareAccessUri(string blobname,
                                 int validityPeriodInMinutes,
                                 CloudBlobContainer container)
{
    var toDateTime = DateTime.Now.AddMinutes(validityPeriodInMinutes);

    var policy = new SharedAccessBlobPolicy
    {
        Permissions = SharedAccessBlobPermissions.Read,
        SharedAccessStartTime = null,
        SharedAccessExpiryTime = new DateTimeOffset(toDateTime)
    };

    var blob = container.GetBlockBlobReference(blobname);
    var sas = blob.GetSharedAccessSignature(policy);
    return blob.Uri.AbsoluteUri + sas;
}

private static void PrintStatus(CloudBlockBlob backupBlob)
{
    Console.WriteLine();
    Console.WriteLine("\t{0} |> Status:{1} |> {2}", 
                      DateTime.Now.ToString(), 
                      backupBlob.CopyState.Status, 
                      backupBlob.CopyState.StatusDescription);

    double bytesCopied = 0;
    if (backupBlob.CopyState.BytesCopied.HasValue)
        bytesCopied = backupBlob.CopyState.BytesCopied.Value;

    var totalBytes = backupBlob.CopyState.TotalBytes;

    if (totalBytes.HasValue)
    {
        Console.WriteLine("\t{0}/{1} bytes copied",
                          (bytesCopied)
                          .ToString(CultureInfo.InvariantCulture),
                          totalBytes.Value.ToString());
        Console.WriteLine("\tProgress : {0} %",
                          (bytesCopied / totalBytes.Value) * 100);
    }
              
    Console.WriteLine();
}