Copy Blobs With StartCopyFromBlob and Track its Progression Using CopyState

June 25, 2013 — 1 Comment

image (3)Copying a large blob between Windows Azure Storage Account and datacenters, will probably require a Shared Access Signature so that the destination Windows Azure Storage Account will be able to read from the source blob. Without proper authorization the source Windows Azure Storage Account will return a 404 NOT Found Http Status Code.

Once you have a Shared Access Signature, I strongly recommend using the StartCopyFromBlob method from the destination Block Blob reference. This method provides you with an ID that can be used to cancel the queued copy operation.

Then you can follow the operation’s progression through the CopyState property by pulling for an updated Block Blob reference by using the GetBlobReferenceFromServer method form the destination container.

To illustrate this, I have put together an example where I attempt to move a 575 MB file from the East US datacenter to the West US datacenter. As can be observed in the trace below, the file was moved in a little under 2 minutes.

2013-06-25_20h40_17

Small blobs are usually very quick to copy and you will usually not get to see such a nice trace. The CopyState property comes in handy when something goes wrong. It can be used to identify blobs that failed to copy.

When blobs fail to copy, they are usually listed in the destination container with a length of zero and a failed CopyState status. If you try to copy from the source without deleting the failed blob from the destination container, you will receive a conflict error message.

2013-06-25_18h42_13

Trying to copy blob that is not publicly visible without a Shared Access Signature will result in a not found exception.

2013-06-25_18h31_51

The code below demonstrates how to copy a Block Blob from one datacenter to another. First it gets references to the source and destination containers. Then it creates a Shared Access Signature for the “db-2013-7-9-19-44.bacpac” blob, which provides read access for a period of 360 minutes. This should be enough for the copy operation.

Then a reference to the destination blob is created and the StartCopyFromBlob method is called. At this point a tasks is queued and the destination Windows Azure Storage Account will initiate the copying process.

It’s progression can be monitored by pulling updates of the destination blob from the server by using the GetBlockBlobReferenceFromServer method. Then by using the CopyState property of this updated Block Blob we are able to peer into it’s progression.

The CopyState status can be Pending, Failed, Success, Invalid and Aborted.

private void CopyLargeBlob()
{
    var eastContainer = MakeEastContainer();

    var westContainer = MakeWestContainer();

    var shareAccessUri = GetShareAccessUri("db-2013-7-9-19-44.bacpac", 
                                            360, 
                                            eastContainer);
    var blob = new Uri(shareAccessUri);

    const string blobName = "db-2013-7-9-19-44.bacpac";

    var backupBlob = westContainer.GetBlockBlobReference(blobName);
            
    var taskId = backupBlob.StartCopyFromBlob(blob);

    Console.WriteLine();
    Console.WriteLine("\tTask ID: " + taskId);

    PrintTaskProgress(westContainer, blobName);
}

private static void PrintTaskProgress(CloudBlobContainer westContainer,
                                      string blobName)
{
    CloudBlockBlob blob;
    blob = (CloudBlockBlob) westContainer.GetBlobReferenceFromServer(blobName);

    while (blob.CopyState.Status == CopyStatus.Pending)
    {
        PrintStatus(blob);
        Task.Delay(TimeSpan.FromSeconds(20d)).Wait();
        blob = (CloudBlockBlob) westContainer
                        .GetBlobReferenceFromServer(blobName);
    }
    PrintStatus(blob);
}

private static CloudBlobContainer MakeWestContainer()
{
    const string westCs = "StorageConnectionStringWest";
    var wcs = CloudConfigurationManager.GetSetting(westCs);
    var westAccount = CloudStorageAccount.Parse(wcs);

    var westClient = westAccount.CreateCloudBlobClient();

    var westContainer = westClient.GetContainerReference("destination");
    westContainer.CreateIfNotExists();
    return westContainer;
}

private static CloudBlobContainer MakeEastContainer()
{
    const string eastCs = "StorageConnectionString";
    var cs = CloudConfigurationManager.GetSetting(eastCs);
    var eastAccount = CloudStorageAccount.Parse(cs);

    var eastClient = eastAccount.CreateCloudBlobClient();
    var eastContainer = eastClient.GetContainerReference("databases");
    return eastContainer;
}

private string GetShareAccessUri(string blobname,
                                 int validityPeriodInMinutes,
                                 CloudBlobContainer container)
{
    var toDateTime = DateTime.Now.AddMinutes(validityPeriodInMinutes);

    var policy = new SharedAccessBlobPolicy
    {
        Permissions = SharedAccessBlobPermissions.Read,
        SharedAccessStartTime = null,
        SharedAccessExpiryTime = new DateTimeOffset(toDateTime)
    };

    var blob = container.GetBlockBlobReference(blobname);
    var sas = blob.GetSharedAccessSignature(policy);
    return blob.Uri.AbsoluteUri + sas;
}

private static void PrintStatus(CloudBlockBlob backupBlob)
{
    Console.WriteLine();
    Console.WriteLine("\t{0} |> Status:{1} |> {2}", 
                      DateTime.Now.ToString(), 
                      backupBlob.CopyState.Status, 
                      backupBlob.CopyState.StatusDescription);

    double bytesCopied = 0;
    if (backupBlob.CopyState.BytesCopied.HasValue)
        bytesCopied = backupBlob.CopyState.BytesCopied.Value;

    var totalBytes = backupBlob.CopyState.TotalBytes;

    if (totalBytes.HasValue)
    {
        Console.WriteLine("\t{0}/{1} bytes copied",
                          (bytesCopied)
                          .ToString(CultureInfo.InvariantCulture),
                          totalBytes.Value.ToString());
        Console.WriteLine("\tProgress : {0} %",
                          (bytesCopied / totalBytes.Value) * 100);
    }
              
    Console.WriteLine();
}

One response to Copy Blobs With StartCopyFromBlob and Track its Progression Using CopyState

  1. 

    Thanks for this post. I used some of your code and built a small console app to enable blob copying. It’s here: github.com/city41/azure-copy-blobs

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s