Geo-HA Service Fabric Cluster

One of the biggest challenges that we face when we build an Internet-scale solution, is high availability across geographic locations (Geo-HA). Why is this important? Well, there can be a few different reasons. The most common reason, is to be able to survive data center outages. Another reason, is to bring services closer to end users so that we can provide good user experiences.

Geo-HA brings challenges to the table. For example, should we use an Active-Passive or Active-Active strategy for data across regions? Keeping in mind that Active-Active is difficult to get right, we need to take time to analyze and to make the correct choices. We need to consider our Disaster recovery (DR) plan, target RPO and RTO. Azure has a whole bunch of mechanisms for replication, backup and monitoring, so how do we decide what’s the right combination?

Today’s Internet-scale services are built using microservices. Service Fabric is a next-generation middleware platform used for building enterprise-class, Tier-1 services. This microservices platform allows us to build scalable, highly available, reliable, and easy to manage solutions. It addresses the significant challenges in developing and managing stateful services. The Reliable Actors API is one of two high-level frameworks provided by Service Fabric, and it is based on the Actor pattern. This API gives us an asynchronous, single-threaded programming model that simplifies our code while still providing the advantages of scalability and reliability guarantees offered by Service Fabric.

A Service Fabric cluster is HA within its geographic region by default. Thinking about our heritage of on premise data centers, we’ve poured thousands of man-hours to deploy Disaster Recovery sites in secondary physical locations, because we know that everything is possible. Over the past few years, we’ve experienced many interesting scenarios, for example, a cut cable, or a faulty DNS entry broke the Internet. So why should we do anything differently in the cloud? We must treat each region as we treat our own data centers and think about Geo-HA.

The rest of this post is about taking high availably to the next level by deploying a Geo-HA Service Fabric cluster.This is actually easier than you think, because Service Fabric is region aware. This means that if you build the cluster from Virtual Machines located in multiple data centers, Service Fabric will automatically take this into consideration when it replicates data and allocates resources for our service instances.

Top of Mind

In the world of quorum based platforms like Service Fabric, the number 3 is a extremely important, because decisions are made by the majority. 50% just doesn’t cut it! For Geo-HA we need at a minimum of 3 regions, each containing a minimum of 3 nodes. Why 3? Because quorum is satisfied by 2 out of 3. Otherwise, the cluster is degraded and becomes read-only.

Here be Dragons

At the time of exploring this type of deployment, all the logs and diagnostics flow back to a single region. This can cause significant traffic over the site-to-site connections and results in one region being hit with more bandwidth requirements. This is done because it would be difficult to have an all up view of what is happening in the cluster if everything was region bound. The cluster can reorganize itself at anytime causing logs to be generated from multiple nodes during the lifetime of a service. Reconstructing this is a challenge in itself.

Scaling to demand also needs special consideration. In order to maintain quorum, each region needs to be scaled together. This means that if I add 2 nodes in one regions, I must add 2 nodes in each region.

Getting Started

Let’s start by defining the following variables.

$clusterName = 'sfcbrisebois'

# Used to generate the Certificate
$clusterfqdn = 'sfcbrisebois.trafficmanager.net'

# Primary Region
$clusterRegion1 = 'East US'
# Secondary Region
$clusterRegion2 = 'West US'
# Secondary Region
$clusterRegion3 = 'South Central US'

Creating the Resource Group

New-AzureRmResourceGroup -Name $clusterName `
                         -Location $clusterRegion1

Create Azure Key Vaults in Target Regions

The primary reason that we need three Azure Key Vaults is that, we can only access secrets stored in the same data center. Therefore, we will be storing the same certificate in both data centers. 

$keyVaultRegion1Name = 'kvbrisebois1'

New-AzureRmKeyVault -VaultName $keyVaultRegion1Name `
                    -ResourceGroupName $clusterName `
                    -Location $clusterRegion1 `
                    -EnabledForDeployment


$keyVaultRegion2Name = 'kvbrisebois2'
                   
New-AzureRmKeyVault -VaultName $keyVaultRegion2Name `
                    -ResourceGroupName $clusterName `
                    -Location $clusterRegion2 `
                    -EnabledForDeployment

$keyVaultRegion3Name = 'kvbrisebois3'

New-AzureRmKeyVault -VaultName $keyVaultRegion3Name `
                    -ResourceGroupName $clusterName `
                    -Location $clusterRegion3 `
                    -EnabledForDeployment

Create and Secure a Self-Signed Certificate

#
# Once the certificate is created and stored in the vault, the script will provide the parameter values needed for template deployment
# 

$CertPassword = '<Password>'
$CertDNSName = $clusterfqdn

$SecurePassword = ConvertTo-SecureString -String $CertPassword `
                                         -AsPlainText `
                                         -Force

$CertFileFullPath = $(Join-Path (Split-Path -Parent 'C:\Users\brise\Desktop\') "$CertDNSName.pfx")

$NewCert = New-SelfSignedCertificate -CertStoreLocation Cert:\CurrentUser\My `
                                     -DnsName $CertDNSName 

Export-PfxCertificate -FilePath $CertFileFullPath `
                      -Password $SecurePassword `
                      -Cert $NewCert

$Bytes = [System.IO.File]::ReadAllBytes($CertFileFullPath)
$Base64 = [System.Convert]::ToBase64String($Bytes)

$JSONBlob = @{
    data = $Base64
    dataType = 'pfx'
    password = $CertPassword
} | ConvertTo-Json

$ContentBytes = [System.Text.Encoding]::UTF8.GetBytes($JSONBlob)
$Content = [System.Convert]::ToBase64String($ContentBytes)

$SecretValue = ConvertTo-SecureString -String $Content `
                                      -AsPlainText -Force

$KeyVaultSecretName = 'cert'

$NewSecretRegion1 = Set-AzureKeyVaultSecret -VaultName $keyVaultRegion1Name `
                                            -Name $KeyVaultSecretName `
                                            -SecretValue $SecretValue `
                                            -Verbose

Write-Host
Write-Host "Source Vault Resource Id: "$(Get-AzureRmKeyVault -VaultName $keyVaultRegion1Name).ResourceId
Write-Host "Certificate URL : "$NewSecretRegion1.Id
Write-Host "Certificate Thumbprint : "$NewCert.Thumbprint


$NewSecretRegion2 = Set-AzureKeyVaultSecret -VaultName $keyVaultRegion2Name `
                                            -Name $KeyVaultSecretName `
                                            -SecretValue $SecretValue `
                                            -Verbose

Write-Host
Write-Host "Source Vault Resource Id: "$(Get-AzureRmKeyVault -VaultName $keyVaultRegion2Name).ResourceId
Write-Host "Certificate URL : "$NewSecretRegion2.Id
Write-Host "Certificate Thumbprint : "$NewCert.Thumbprint


$NewSecretRegion3 = Set-AzureKeyVaultSecret -VaultName $keyVaultRegion3Name `
                                            -Name $KeyVaultSecretName `
                                            -SecretValue $SecretValue `
                                            -Verbose
Write-Host
Write-Host "Source Vault Resource Id: "$(Get-AzureRmKeyVault -VaultName $keyVaultRegion3Name).ResourceId
Write-Host "Certificate URL : "$NewSecretRegion3.Id
Write-Host "Certificate Thumbprint : "$NewCert.Thumbprint

Testing the ARM Template

Before we deploy the ARM Template, it’s wise to test it out. Running through a deployment cycle can easily take 30 minutes of your precious time. Take this opportunity to validate the template and find various types of possible errors. 

Test-AzureRmResourceGroupDeployment -ResourceGroupName $clusterName `
                                    -TemplateFile 'ServiceFabricCluster.json' `
                                    -TemplateParameterObject @{
                                        clusterRegion1 = $clusterRegion1
                                        clusterRegion2 = $clusterRegion2
                                        clusterRegion3 = $clusterRegion3
                                        adminUserName = 'brisebois'
                                        adminPassword = '<Password>'
                                        certificateThumbprint = $NewCert.Thumbprint
                                        sourceVaultRegion1ResourceId = $(Get-AzureRmKeyVault -VaultName $keyVaultRegion1Name).ResourceId
                                        sourceVaultRegion2ResourceId = $(Get-AzureRmKeyVault -VaultName $keyVaultRegion2Name).ResourceId
                                        sourceVaultRegion3ResourceId = $(Get-AzureRmKeyVault -VaultName $keyVaultRegion3Name).ResourceId
                                        certificateRegion1UrlValue= $NewSecretRegion1.Id
                                        certificateRegion2UrlValue = $NewSecretRegion2.Id
                                        certificateRegion3UrlValue = $NewSecretRegion3.Id
                                        sharedKey = '33aaca0d4fcb457ca3f443354c4829a9'
                                        clusterName = $clusterName
                                        } `
                                    -Verbose

Deploying the ARM Template

New-AzureRmResourceGroupDeployment -ResourceGroupName $clusterName `
                                   -TemplateFile 'ServiceFabricCluster.json' `
                                   -TemplateParameterObject @{
                                        clusterRegion1 = $clusterRegion1
                                        clusterRegion2 = $clusterRegion2
                                        clusterRegion3 = $clusterRegion3
                                        adminUserName = 'brisebois'
                                        adminPassword = '<Password>'
                                        certificateThumbprint = $NewCert.Thumbprint
                                        sourceVaultRegion1ResourceId = $(Get-AzureRmKeyVault -VaultName $keyVaultRegion1Name).ResourceId
                                        sourceVaultRegion2ResourceId = $(Get-AzureRmKeyVault -VaultName $keyVaultRegion2Name).ResourceId
                                        sourceVaultRegion3ResourceId = $(Get-AzureRmKeyVault -VaultName $keyVaultRegion3Name).ResourceId
                                        certificateRegion1UrlValue= $NewSecretRegion1.Id
                                        certificateRegion2UrlValue = $NewSecretRegion2.Id
                                        certificateRegion3UrlValue = $NewSecretRegion3.Id
                                        sharedKey = '33aaca0d4fcb457ca3f443354c4829a9'
                                        clusterName = $clusterName
                                        } `
                                   -Verbose

VERBOSE: 2:22:12 PM - Create template deployment 'ServiceFabricCluster'.
VERBOSE: 2:22:18 PM - Resource Microsoft.Network/publicIPAddresses 'Vnet3GwIP' provisioning status is running
VERBOSE: 2:22:18 PM - Resource Microsoft.Network/publicIPAddresses 'Vnet2GwIP' provisioning status is running
VERBOSE: 2:22:18 PM - Resource Microsoft.Storage/storageAccounts 'vmssfcbrisebois3' provisioning status is running
VERBOSE: 2:22:18 PM - Resource Microsoft.Network/virtualNetworks 'westusVNet' provisioning status is running
VERBOSE: 2:22:18 PM - Resource Microsoft.Network/virtualNetworks 'southcentralusVNet' provisioning status is running
VERBOSE: 2:22:18 PM - Resource Microsoft.Storage/storageAccounts 'logssfcbrisebois' provisioning status is running
VERBOSE: 2:22:18 PM - Resource Microsoft.Storage/storageAccounts 'vmssfcbrisebois1' provisioning status is running
VERBOSE: 2:22:18 PM - Resource Microsoft.Storage/storageAccounts 'vmssfcbrisebois2' provisioning status is running
VERBOSE: 2:22:18 PM - Resource Microsoft.Storage/storageAccounts 'diagssfcbrisebois' provisioning status is running
VERBOSE: 2:22:24 PM - Resource Microsoft.Network/publicIPAddresses 'PublicIP-LB-FE-3' provisioning status is running
VERBOSE: 2:22:24 PM - Resource Microsoft.Network/publicIPAddresses 'Vnet1GwIP' provisioning status is running
VERBOSE: 2:22:24 PM - Resource Microsoft.Network/publicIPAddresses 'PublicIP-LB-FE-1' provisioning status is running
VERBOSE: 2:22:24 PM - Resource Microsoft.Network/publicIPAddresses 'PublicIP-LB-FE-2' provisioning status is running
VERBOSE: 2:22:24 PM - Resource Microsoft.Network/virtualNetworks 'eastusVNet' provisioning status is running
VERBOSE: 2:22:29 PM - Resource Microsoft.Network/publicIPAddresses 'Vnet3GwIP' provisioning status is succeeded
VERBOSE: 2:22:29 PM - Resource Microsoft.Network/publicIPAddresses 'Vnet2GwIP' provisioning status is succeeded
VERBOSE: 2:22:29 PM - Resource Microsoft.Network/virtualNetworks 'westusVNet' provisioning status is succeeded
VERBOSE: 2:22:29 PM - Resource Microsoft.Network/virtualNetworks 'southcentralusVNet' provisioning status is succeeded
VERBOSE: 2:22:34 PM - Resource Microsoft.Network/loadBalancers 'LoadBalancer-sfcbrisebois3-nt1vm' provisioning status is succeeded
VERBOSE: 2:22:34 PM - Resource Microsoft.Network/loadBalancers 'LoadBalancer-sfcbrisebois2-nt1vm' provisioning status is succeeded
VERBOSE: 2:22:34 PM - Resource Microsoft.Network/virtualNetworkGateways 'Vnet1Gw' provisioning status is running
VERBOSE: 2:22:34 PM - Resource Microsoft.Network/virtualNetworkGateways 'Vnet2Gw' provisioning status is running
VERBOSE: 2:22:34 PM - Resource Microsoft.Network/loadBalancers 'LoadBalancer-sfcbrisebois1-nt1vm' provisioning status is succeeded
VERBOSE: 2:22:34 PM - Resource Microsoft.Network/virtualNetworkGateways 'Vnet3Gw' provisioning status is running
VERBOSE: 2:22:34 PM - Resource Microsoft.Network/publicIPAddresses 'PublicIP-LB-FE-3' provisioning status is succeeded
VERBOSE: 2:22:34 PM - Resource Microsoft.Network/publicIPAddresses 'Vnet1GwIP' provisioning status is succeeded
VERBOSE: 2:22:34 PM - Resource Microsoft.Network/publicIPAddresses 'PublicIP-LB-FE-1' provisioning status is succeeded
VERBOSE: 2:22:34 PM - Resource Microsoft.Network/publicIPAddresses 'PublicIP-LB-FE-2' provisioning status is succeeded
VERBOSE: 2:22:34 PM - Resource Microsoft.Network/virtualNetworks 'eastusVNet' provisioning status is succeeded
VERBOSE: 2:22:39 PM - Resource Microsoft.Network/trafficManagerProfiles 'sfcbriseboistm' provisioning status is succeeded
VERBOSE: 2:22:45 PM - Resource Microsoft.Storage/storageAccounts 'vmssfcbrisebois3' provisioning status is succeeded
VERBOSE: 2:22:45 PM - Resource Microsoft.Storage/storageAccounts 'vmssfcbrisebois1' provisioning status is succeeded
VERBOSE: 2:22:45 PM - Resource Microsoft.Storage/storageAccounts 'vmssfcbrisebois2' provisioning status is succeeded
VERBOSE: 2:22:45 PM - Resource Microsoft.Storage/storageAccounts 'diagssfcbrisebois' provisioning status is succeeded
VERBOSE: 2:22:50 PM - Resource Microsoft.ServiceFabric/clusters 'sfcbrisebois' provisioning status is succeeded
VERBOSE: 2:22:50 PM - Resource Microsoft.Storage/storageAccounts 'logssfcbrisebois' provisioning status is succeeded
VERBOSE: 2:22:50 PM - Resource Microsoft.Storage/storageAccounts 'diagssfcbrisebois' provisioning status is succeeded
VERBOSE: 2:22:50 PM - Resource Microsoft.Storage/storageAccounts 'logssfcbrisebois' provisioning status is succeeded
VERBOSE: 2:22:55 PM - Resource Microsoft.Compute/virtualMachineScaleSets 'nt1vm3' provisioning status is running
VERBOSE: 2:22:55 PM - Resource Microsoft.Compute/virtualMachineScaleSets 'nt1vm1' provisioning status is running
VERBOSE: 2:22:55 PM - Resource Microsoft.Compute/virtualMachineScaleSets 'nt1vm2' provisioning status is running
VERBOSE: 2:32:16 PM - Resource Microsoft.Compute/virtualMachineScaleSets 'nt1vm2' provisioning status is succeeded
VERBOSE: 2:34:02 PM - Resource Microsoft.Compute/virtualMachineScaleSets 'nt1vm3' provisioning status is succeeded
VERBOSE: 2:37:57 PM - Resource Microsoft.Compute/virtualMachineScaleSets 'nt1vm1' provisioning status is succeeded
VERBOSE: 2:51:19 PM - Resource Microsoft.Network/virtualNetworkGateways 'Vnet2Gw' provisioning status is succeeded
VERBOSE: 2:51:51 PM - Resource Microsoft.Network/connections 'Vnet23Connection' provisioning status is running
VERBOSE: 2:51:51 PM - Resource Microsoft.Network/virtualNetworkGateways 'Vnet3Gw' provisioning status is succeeded
VERBOSE: 2:51:57 PM - Resource Microsoft.Network/connections 'Vnet32Connection' provisioning status is running
VERBOSE: 2:52:18 PM - Resource Microsoft.Network/connections 'Vnet12Connection' provisioning status is running
VERBOSE: 2:52:18 PM - Resource Microsoft.Network/connections 'Vnet31Connection' provisioning status is running
VERBOSE: 2:52:18 PM - Resource Microsoft.Network/virtualNetworkGateways 'Vnet1Gw' provisioning status is succeeded
VERBOSE: 2:52:23 PM - Resource Microsoft.Network/connections 'Vnet21Connection' provisioning status is running
VERBOSE: 2:52:23 PM - Resource Microsoft.Network/connections 'Vnet13Connection' provisioning status is running
VERBOSE: 2:52:39 PM - Resource Microsoft.Network/connections 'Vnet32Connection' provisioning status is succeeded
VERBOSE: 2:52:55 PM - Resource Microsoft.Network/connections 'Vnet23Connection' provisioning status is succeeded
VERBOSE: 2:53:00 PM - Resource Microsoft.Network/connections 'Vnet31Connection' provisioning status is succeeded
VERBOSE: 2:53:05 PM - Resource Microsoft.Network/connections 'Vnet21Connection' provisioning status is succeeded
VERBOSE: 2:54:41 PM - Resource Microsoft.Network/connections 'Vnet13Connection' provisioning status is succeeded
VERBOSE: 2:54:41 PM - Resource Microsoft.Network/connections 'Vnet12Connection' provisioning status is succeeded


DeploymentName    : ServiceFabricCluster
ResourceGroupName : sfcbrisebois
ProvisioningState : Succeeded
Timestamp         : 3/28/2016 6:54:45 PM
Mode              : Incremental
TemplateLink      : 
Parameters        : 
                    Name             Type                       Value     
                    ===============  =========================  ==========
                    clusterRegion1   String                     East US   
                    clusterRegion2   String                     West US   
                    clusterRegion3   String                     South Central US
                    adminUserName    String                     brisebois 
                    adminPassword    SecureString                         
                    certificateThumbprint  String                     92EEAD1A74992B7A20260A531E899C6DE9FCC692
                    sourceVaultRegion1ResourceId  String                     
                    /subscriptions/64668628-d5e6-4c5b-9be4-9e577d5b8fd0/resourceGroups/sfcbrisebois/providers/Microsoft.KeyVault/vaults/kvbrisebois1
                    sourceVaultRegion2ResourceId  String                     
                    /subscriptions/64668628-d5e6-4c5b-9be4-9e577d5b8fd0/resourceGroups/sfcbrisebois/providers/Microsoft.KeyVault/vaults/kvbrisebois2
                    sourceVaultRegion3ResourceId  String                     
                    /subscriptions/64668628-d5e6-4c5b-9be4-9e577d5b8fd0/resourceGroups/sfcbrisebois/providers/Microsoft.KeyVault/vaults/kvbrisebois3
                    certificateRegion1UrlValue  String                     https://kvbrisebois1.vault.azure.net:443/secrets/cert/f37f5629e79043c3b008886cac36f0f2
                    certificateRegion2UrlValue  String                     https://kvbrisebois2.vault.azure.net:443/secrets/cert/5105cfed7680403fa954e0692d602c10
                    certificateRegion3UrlValue  String                     https://kvbrisebois3.vault.azure.net:443/secrets/cert/ec7a771da482402cbb815c4177186223
                    sharedKey        String                     33aaca0d4fcb457ca3f443354c4829a9
                    clusterName      String                     sfcbrisebois

Template Output: clusterProperties

{
  "provisioningState": "Succeeded",
  "clusterId": "a6a59893-fc81-4bed-a3ba-df66e0687aed",
  "clusterCodeVersion": "4.5.192.9590",
  "clusterState": "WaitingForNodes",
  "managementEndpoint": "https://sfcbrisebois.trafficmanager.net:19080",
  "clusterEndpoint": "https://eastus.servicefabric.azure.com/runtime/clusters/a6a59893-fc81-4bed-a3ba-df66e0687aed",
  "certificate": {
    "thumbprint": "92EEAD1A74992B7A20260A531E899C6DE9FCC692",
    "x509StoreName": "My"
  },
  "fabricSettings": [
    {
      "name": "Security",
      "parameters": [
        {
          "name": "ClusterProtectionLevel",
          "value": "EncryptAndSign"
        }
      ]
    }
  ],
  "diagnosticsStorageAccountConfig": {
    "storageAccountName": "logssfcbrisebois",
    "protectedAccountKeyName": "StorageAccountKey1",
    "blobEndpoint": "https://logssfcbrisebois.blob.core.windows.net/",
    "queueEndpoint": "https://logssfcbrisebois.queue.core.windows.net/",
    "tableEndpoint": "https://logssfcbrisebois.table.core.windows.net/"
  },
  "nodeTypes": [
    {
      "name": "nt1vm",
      "clientConnectionEndpointPort": 19000,
      "httpGatewayEndpointPort": 19080,
      "applicationPorts": {
        "startPort": 20000,
        "endPort": 30000
      },
      "ephemeralPorts": {
        "startPort": 49152,
        "endPort": 65534
      },
      "isPrimary": true,
      "vmInstanceCount": 15
    }
  ]
}

Template Output: trafficManagerProperties

{
  "profileStatus": "Enabled",
  "trafficRoutingMethod": "Weighted",
  "dnsConfig": {
    "relativeName": "sfcbrisebois",
    "fqdn": "sfcbrisebois.trafficmanager.net",
    "ttl": 30
  },
  "monitorConfig": {
    "profileMonitorStatus": "CheckingEndpoints",
    "protocol": "HTTP",
    "port": 80,
    "path": "/"
  },
  "endpoints": [
    {
      "id": "/subscriptions/64668628-d5e6-4c5b-9be4-9e577d5b8fd0/resourceGroups/sfcbrisebois/providers/Microsoft.Network/trafficManagerProfiles/sfcbriseboistm/azureEndpoints/East
      US",
      "name": "East US",
      "type": "Microsoft.Network/trafficManagerProfiles/azureEndpoints",
      "properties": {
        "endpointStatus": "Enabled",
        "endpointMonitorStatus": "CheckingEndpoint",
        "targetResourceId": "/subscriptions/64668628-d5e6-4c5b-9be4-9e577d5b8fd0/resourceGroups/sfcbrisebois/providers/Microsoft.Network/publicIPAddresses/PublicIP-LB-FE-1",
        "target": "sfcbrisebois1.eastus.cloudapp.azure.com",
        "weight": 1,
        "priority": 1,
        "endpointLocation": "East US"
      }
    },
    {
      "id": "/subscriptions/64668628-d5e6-4c5b-9be4-9e577d5b8fd0/resourceGroups/sfcbrisebois/providers/Microsoft.Network/trafficManagerProfiles/sfcbriseboistm/azureEndpoints/West
      US",
      "name": "West US",
      "type": "Microsoft.Network/trafficManagerProfiles/azureEndpoints",
      "properties": {
        "endpointStatus": "Enabled",
        "endpointMonitorStatus": "CheckingEndpoint",
        "targetResourceId": "/subscriptions/64668628-d5e6-4c5b-9be4-9e577d5b8fd0/resourceGroups/sfcbrisebois/providers/Microsoft.Network/publicIPAddresses/PublicIP-LB-FE-2",
        "target": "sfcbrisebois2.westus.cloudapp.azure.com",
        "weight": 1,
        "priority": 2,
        "endpointLocation": "West US"
      }
    },
    {
      "id": "/subscriptions/64668628-d5e6-4c5b-9be4-9e577d5b8fd0/resourceGroups/sfcbrisebois/providers/Microsoft.Network/trafficManagerProfiles/sfcbriseboistm/azureEndpoints/South Central
      US",
      "name": "South Central US",
      "type": "Microsoft.Network/trafficManagerProfiles/azureEndpoints",
      "properties": {
        "endpointStatus": "Enabled",
        "endpointMonitorStatus": "CheckingEndpoint",
        "targetResourceId": "/subscriptions/64668628-d5e6-4c5b-9be4-9e577d5b8fd0/resourceGroups/sfcbrisebois/providers/Microsoft.Network/publicIPAddresses/PublicIP-LB-FE-3",
        "target": "sfcbrisebois3.southcentralus.cloudapp.azure.com",
        "weight": 1,
        "priority": 3,
        "endpointLocation": "South Central US"
      }
    }
  ]
}

Navigating to the https://sfcbrisebois.trafficmanager.net:19080/Explorer/ presents us with a dashboard that we can use to explore our brand new Service Fabric Cluster.

2016-03-28_15h58_50

Clicking on the Cluster Map provides us with a table that shows us how each Virtual Machine is distributed across Update Domains and Fault Domains.

2016-03-28_15h59_22

Building the Azure Resource Manager (ARM) Template

This template has many parts so I will try to speak to each of them.

  • VMStorageAccount Region 1: Storage account used to store Virtual Machine VHDs
  • VMStorageAccount Region 2: Storage account used to store Virtual Machine VHDs
  • VMStorageAccount Region 3: Storage account used to store Virtual Machine VHDs
  • SupportLogStorageAccount: This is located in Region 1 and stores the logs from all Service Fabric Nodes. Note that logs are streamed from Region 2 and 3 as well and incur cross data center transfer costs.
  • DiagnosticsStorageAccount: This is located in Region 1 and stores the diagnostics from all Service Fabric Nodes. Note that logs are streamed from Region 2 and 3 as well and incur cross data center transfer costs.
  • Per Region:
    • VirtualNetwork: This VNet contains the Service Fabirc Nodes
    • PublicIPAddress: The public load balanced IP load balancer
    • LoadBalancer: The load balancer used by the Service Fabric Nodes
    • Cluster scale set: Used to make the Service Fabric Nodes
    • Gatewaty Public IP: This IP is reserved for the VNet’s VPN Gateway
    • Virtual Network Gateway: This VPN Gateway is used to bridge VNets from all regions
    • Connections: This is the VPN connection that bridges both VNets
  • Cluster: This is the Service Fabric Cluster Configuration
  • Traffic Manager: This is the public facing endpoint. It load balances traffic across both public facing load balanced IPs

To build it, I merged the Service Fabric Template from Visual Studio and the VNet to VNet template form the Quick Start Templates on GitHub. The diagram at the top of this post, depicts a high-level perspective of the Geo-HA Service Fabric Cluster. Below I provide you with a materialized view of the resulting ARM Template deployed across multiple Microsoft Azure data centers.

Geo-HA Service Fabric Cluster

The ARM Template

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "clusterRegion1": {
      "type": "string",
      "allowedValues": [
        "East US",
        "West US",
        "South Central US"
      ],
      "metadata": {
        "description": "Location of the Cluster"
      }
    },
    "clusterRegion2": {
      "type": "string",
      "allowedValues": [
        "East US",
        "West US",
        "South Central US"
      ],
      "metadata": {
        "description": "Cluster Region Locations"
      }
    },
    "clusterRegion3": {
      "type": "string",
      "allowedValues": [
        "East US",
        "West US",
        "South Central US"
      ],
      "metadata": {
        "description": "Cluster Region Locations"
      }
    },
    "adminUserName": {
      "type": "string",
      "metadata": {
        "description": "Remote desktop user Id"
      }
    },
    "adminPassword": {
      "type": "securestring",
      "metadata": {
        "description": "Remote desktop user password"
      }
    },
    "certificateThumbprint": {
      "type": "string"
    },
    "sourceVaultRegion1ResourceId": {
      "type": "string",
      "metadata": {
        "description": "Resource ID of the key vault where cluster certificate is stored. The format is /subscriptions/<subscription ID>/resourceGroups/<resource group name>/providers/Microsoft.KeyVault/vaults/<vault name>"
      }
    },
    "sourceVaultRegion2ResourceId": {
      "type": "string",
      "metadata": {
        "description": "Resource ID of the key vault where cluster certificate is stored. The format is /subscriptions/<subscription ID>/resourceGroups/<resource group name>/providers/Microsoft.KeyVault/vaults/<vault name>"
      }
    },
    "sourceVaultRegion3ResourceId": {
      "type": "string",
      "metadata": {
        "description": "Resource ID of the key vault where cluster certificate is stored. The format is /subscriptions/<subscription ID>/resourceGroups/<resource group name>/providers/Microsoft.KeyVault/vaults/<vault name>"
      }
    },
    "certificateRegion1UrlValue": {
      "type": "string",
      "metadata": {
        "description": "Refers to the location URL in your key vault where the cluster certificate was uploaded. The format is https://<name of the vault>.vault.azure.net:443/secrets/<exact location>"
      }
    },
    "certificateRegion2UrlValue": {
      "type": "string",
      "metadata": {
        "description": "Refers to the location URL in your key vault where the cluster certificate was uploaded. The format is https://<name of the vault>.vault.azure.net:443/secrets/<exact location>"
      }
    },
    "certificateRegion3UrlValue": {
      "type": "string",
      "metadata": {
        "description": "Refers to the location URL in your key vault where the cluster certificate was uploaded. The format is https://<name of the vault>.vault.azure.net:443/secrets/<exact location>"
      }
    },
    "sharedKey": {
      "type": "string",
      "metadata": {
        "description": "The shared key used to establish connection between the two VirtualNetworkGateways."
      }
    },
    "clusterName": {
      "type": "string",
      "metadata": {
        "description": "Cluster Name"
      }
    }
  },
  "variables": {
    "clusterName": "[parameters('clusterName')]",
    "clusterfqdn": "[concat(parameters('clusterName'),'.trafficmanager.net')]",
    "vmStorageAccountName": "[concat('vms', parameters('clusterName'))]",

    "virtualNetworkNameRegion1": "[concat(toLower(replace(parameters('clusterRegion1'),' ','')),'VNet')]",

    "gatewayPublicIPName1": "Vnet1GwIP",
    "gatewayName1": "Vnet1Gw",

    "addressPrefix1": "192.168.100.0/24",

    "subnet1Name1": "WebTier1",
    "SubnetRef1": "[concat(variables('vnetID1'),'/subnets/',variables('subnet1Name1'))]",

    "subnet1Prefix1": "192.168.100.16/28",

    "gatewaySubnetPrefix1": "192.168.100.0/28",

    "connectionName12": "Vnet12Connection",
    "connectionName13": "Vnet13Connection",

    "vnetID1": "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualNetworkNameRegion1'))]",

    "gatewaySubnetRef1": "[concat(variables('vnetID1'),'/subnets/','GatewaySubnet')]",

    "virtualNetworkNameRegion2": "[concat(toLower(replace(parameters('clusterRegion2'),' ','')),'VNet')]",

    "gatewayPublicIPName2": "Vnet2GwIP",
    "gatewayName2": "Vnet2Gw",

    "addressPrefix2": "192.168.101.0/24",

    "subnet1Name2": "WebTier2",
    "SubnetRef2": "[concat(variables('vnetID2'),'/subnets/',variables('subnet1Name2'))]",

    "subnet1Prefix2": "192.168.101.16/28",

    "gatewaySubnetPrefix2": "192.168.101.0/28",

    "connectionName21": "Vnet21Connection",
    "connectionName23": "Vnet23Connection",

    "vnetID2": "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualNetworkNameRegion2'))]",
    "gatewaySubnetRef2": "[concat(variables('vnetID2'),'/subnets/','GatewaySubnet')]",

    "virtualNetworkNameRegion3": "[concat(toLower(replace(parameters('clusterRegion3'),' ','')),'VNet')]",

    "gatewayPublicIPName3": "Vnet3GwIP",

    "gatewayName3": "Vnet3Gw",

    "addressPrefix3": "192.168.102.0/24",

    "subnet1Name3": "WebTier2",
    "SubnetRef3": "[concat(variables('vnetID3'),'/subnets/',variables('subnet1Name3'))]",

    "subnet1Prefix3": "192.168.102.16/28",

    "gatewaySubnetPrefix3": "192.168.102.0/28",

    "connectionName31": "Vnet31Connection",
    "connectionName32": "Vnet32Connection",

    "vnetID3": "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualNetworkNameRegion3'))]",
    "gatewaySubnetRef3": "[concat(variables('vnetID3'),'/subnets/','GatewaySubnet')]",

    "supportLogStorageAccountName": "[concat('logs', parameters('clusterName'))]",
    "applicationDiagnosticsStorageAccountName": "[concat('diags', parameters('clusterName'))]",

    "vmNodeType0Name": "nt1vm",

    "lbName1": "[concat('LoadBalancer-', variables('clusterName'), '1-', variables('vmNodeType0Name'))]",
    "lbIPName1": "PublicIP-LB-FE-1",

    "lbName2": "[concat('LoadBalancer-', variables('clusterName'), '2-', variables('vmNodeType0Name'))]",
    "lbIPName2": "PublicIP-LB-FE-2",

    "lbName3": "[concat('LoadBalancer-', variables('clusterName'), '3-', variables('vmNodeType0Name'))]",
    "lbIPName3": "PublicIP-LB-FE-3",

    "lbID1": "[resourceId('Microsoft.Network/loadBalancers', variables('lbName1'))]",
    "lbIPConfig1": "[concat(variables('lbID1'), '/frontendIPConfigurations/LoadBalancerIPConfig')]",
    "lbPoolID1": "[concat(variables('lbID1'), '/backendAddressPools/LoadBalancerBEAddressPool')]",
    "lbProbeID1": "[concat(variables('lbID1'), '/probes/FabricGatewayProbe')]",
    "lbHttpProbeID1": "[concat(variables('lbID1'), '/probes/FabricHttpGatewayProbe')]",
    "lbNatPoolID1": "[concat(variables('lbID1'), '/inboundNatPools/LoadBalancerBEAddressNatPool')]",

    "lbID2": "[resourceId('Microsoft.Network/loadBalancers', variables('lbName2'))]",
    "lbIPConfig2": "[concat(variables('lbID2'), '/frontendIPConfigurations/LoadBalancerIPConfig')]",
    "lbPoolID2": "[concat(variables('lbID2'), '/backendAddressPools/LoadBalancerBEAddressPool')]",
    "lbProbeID2": "[concat(variables('lbID2'), '/probes/FabricGatewayProbe')]",
    "lbHttpProbeID2": "[concat(variables('lbID2'), '/probes/FabricHttpGatewayProbe')]",
    "lbNatPoolID2": "[concat(variables('lbID2'), '/inboundNatPools/LoadBalancerBEAddressNatPool')]",

    "lbID3": "[resourceId('Microsoft.Network/loadBalancers', variables('lbName3'))]",
    "lbIPConfig3": "[concat(variables('lbID3'), '/frontendIPConfigurations/LoadBalancerIPConfig')]",
    "lbPoolID3": "[concat(variables('lbID3'), '/backendAddressPools/LoadBalancerBEAddressPool')]",
    "lbProbeID3": "[concat(variables('lbID3'), '/probes/FabricGatewayProbe')]",
    "lbHttpProbeID3": "[concat(variables('lbID3'), '/probes/FabricHttpGatewayProbe')]",
    "lbNatPoolID3": "[concat(variables('lbID3'), '/inboundNatPools/LoadBalancerBEAddressNatPool')]",

    "fabricTcpGatewayPort": "19000",
    "fabricHttpGatewayPort": "19080",

    "loadBalancedAppPort1": "80",
    "loadBalancedAppPort2": "8081",

    "certificateStoreValue": "My",

    "apiVersion": "2015-05-01-preview"
  },
  "resources": [
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[concat(variables('vmStorageAccountName'),'1')]",
      "location": "[parameters('clusterRegion1')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "VMStorageAccount Region 1"
      },
      "properties": {
        "accountType": "Standard_LRS"
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[concat(variables('vmStorageAccountName'),'2')]",
      "location": "[parameters('clusterRegion2')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "VMStorageAccount Region 2"
      },
      "properties": {
        "accountType": "Standard_LRS"
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[concat(variables('vmStorageAccountName'),'3')]",
      "location": "[parameters('clusterRegion3')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "VMStorageAccount Region 3"
      },
      "properties": {
        "accountType": "Standard_LRS"
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[variables('supportLogStorageAccountName')]",
      "location": "[parameters('clusterRegion1')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "SupportLogStorageAccount"
      },
      "properties": {
        "accountType": "Standard_LRS"
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[variables('applicationDiagnosticsStorageAccountName')]",
      "location": "[parameters('clusterRegion1')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "DiagnosticsStorageAccount"
      },
      "properties": {
        "accountType": "Standard_LRS"
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/virtualNetworks",
      "name": "[variables('virtualNetworkNameRegion1')]",
      "location": "[parameters('clusterRegion1')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "VirtualNetwork Region 1"
      },
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "[variables('addressPrefix1')]"
          ]
        },
        "subnets": [
          {
            "name": "[variables('subnet1Name1')]",
            "properties": {
              "addressPrefix": "[variables('subnet1Prefix1')]"
            }
          },
          {
            "name": "GatewaySubnet",
            "properties": {
              "addressPrefix": "[variables('gatewaySubnetPrefix1')]"
            }
          }
        ]
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/virtualNetworks",
      "name": "[variables('virtualNetworkNameRegion2')]",
      "location": "[parameters('clusterRegion2')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "VirtualNetwork Region 2"
      },
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "[variables('addressPrefix2')]"
          ]
        },
        "subnets": [
          {
            "name": "[variables('subnet1Name2')]",
            "properties": {
              "addressPrefix": "[variables('subnet1Prefix2')]"
            }
          },
          {
            "name": "GatewaySubnet",
            "properties": {
              "addressPrefix": "[variables('gatewaySubnetPrefix2')]"
            }
          }
        ]
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/virtualNetworks",
      "name": "[variables('virtualNetworkNameRegion3')]",
      "location": "[parameters('clusterRegion3')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "VirtualNetwork Region 3"
      },
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "[variables('addressPrefix3')]"
          ]
        },
        "subnets": [
          {
            "name": "[variables('subnet1Name3')]",
            "properties": {
              "addressPrefix": "[variables('subnet1Prefix3')]"
            }
          },
          {
            "name": "GatewaySubnet",
            "properties": {
              "addressPrefix": "[variables('gatewaySubnetPrefix3')]"
            }
          }
        ]
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/publicIPAddresses",
      "name": "[variables('lbIPName1')]",
      "location": "[parameters('clusterRegion1')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "PublicIPAddress Region 1"
      },
      "properties": {
        "dnsSettings": {
          "domainNameLabel": "[concat(variables('clusterName'),'1')]"
        },
        "publicIPAllocationMethod": "Dynamic"
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/publicIPAddresses",
      "name": "[variables('lbIPName2')]",
      "location": "[parameters('clusterRegion2')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "PublicIPAddress Region 2"
      },
      "properties": {
        "dnsSettings": {
          "domainNameLabel": "[concat(variables('clusterName'),'2')]"
        },
        "publicIPAllocationMethod": "Dynamic"
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/publicIPAddresses",
      "name": "[variables('lbIPName3')]",
      "location": "[parameters('clusterRegion3')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "PublicIPAddress Region 3"
      },
      "properties": {
        "dnsSettings": {
          "domainNameLabel": "[concat(variables('clusterName'),'3')]"
        },
        "publicIPAllocationMethod": "Dynamic"
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/loadBalancers",
      "name": "[variables('lbName1')]",
      "location": "[parameters('clusterRegion1')]",
      "dependsOn": [
        "[variables('lbIPName1')]"
      ],
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "LoadBalancer Region 1"
      },
      "properties": {
        "frontendIPConfigurations": [
          {
            "name": "LoadBalancerIPConfig",
            "properties": {
              "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses', variables('lbIPName1'))]"
              }
            }
          }
        ],
        "backendAddressPools": [
          {
            "name": "LoadBalancerBEAddressPool"
          }
        ],
        "loadBalancingRules": [
          {
            "name": "LBRule",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID1')]"
              },
              "backendPort": "[variables('fabricTcpGatewayPort')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig1')]"
              },
              "frontendPort": "[variables('fabricTcpGatewayPort')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[variables('lbProbeID1')]"
              },
              "protocol": "Tcp"
            }
          },
          {
            "name": "LBHttpRule",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID1')]"
              },
              "backendPort": "[variables('fabricHttpGatewayPort')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig1')]"
              },
              "frontendPort": "[variables('fabricHttpGatewayPort')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[variables('lbHttpProbeID1')]"
              },
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortLBRule1",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID1')]"
              },
              "backendPort": "[variables('loadBalancedAppPort1')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig1')]"
              },
              "frontendPort": "[variables('loadBalancedAppPort1')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[concat(variables('lbID1'), '/probes/AppPortProbe1')]"
              },
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortLBRule2",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID1')]"
              },
              "backendPort": "[variables('loadBalancedAppPort2')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig1')]"
              },
              "frontendPort": "[variables('loadBalancedAppPort2')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[concat(variables('lbID1'), '/probes/AppPortProbe2')]"
              },
              "protocol": "Tcp"
            }
          }
        ],
        "probes": [
          {
            "name": "FabricGatewayProbe",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('fabricTcpGatewayPort')]",
              "protocol": "Tcp"
            }
          },
          {
            "name": "FabricHttpGatewayProbe",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('fabricHttpGatewayPort')]",
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortProbe1",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('loadBalancedAppPort1')]",
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortProbe2",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('loadBalancedAppPort2')]",
              "protocol": "Tcp"
            }
          }
        ],
        "inboundNatPools": [
          {
            "name": "LoadBalancerBEAddressNatPool",
            "properties": {
              "backendPort": 3389,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig1')]"
              },
              "frontendPortRangeEnd": 4500,
              "frontendPortRangeStart": 3389,
              "protocol": "Tcp"
            }
          }
        ]
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/loadBalancers",
      "name": "[variables('lbName2')]",
      "location": "[parameters('clusterRegion2')]",
      "dependsOn": [
        "[variables('lbIPName2')]"
      ],
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "LoadBalancer Region 2"
      },
      "properties": {
        "frontendIPConfigurations": [
          {
            "name": "LoadBalancerIPConfig",
            "properties": {
              "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses', variables('lbIPName2'))]"
              }
            }
          }
        ],
        "backendAddressPools": [
          {
            "name": "LoadBalancerBEAddressPool"
          }
        ],
        "loadBalancingRules": [
          {
            "name": "LBRule",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID2')]"
              },
              "backendPort": "[variables('fabricTcpGatewayPort')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig2')]"
              },
              "frontendPort": "[variables('fabricTcpGatewayPort')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[variables('lbProbeID2')]"
              },
              "protocol": "Tcp"
            }
          },
          {
            "name": "LBHttpRule",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID2')]"
              },
              "backendPort": "[variables('fabricHttpGatewayPort')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig2')]"
              },
              "frontendPort": "[variables('fabricHttpGatewayPort')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[variables('lbHttpProbeID2')]"
              },
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortLBRule1",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID2')]"
              },
              "backendPort": "[variables('loadBalancedAppPort1')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig2')]"
              },
              "frontendPort": "[variables('loadBalancedAppPort1')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[concat(variables('lbID2'), '/probes/AppPortProbe1')]"
              },
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortLBRule2",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID2')]"
              },
              "backendPort": "[variables('loadBalancedAppPort2')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig2')]"
              },
              "frontendPort": "[variables('loadBalancedAppPort2')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[concat(variables('lbID2'), '/probes/AppPortProbe2')]"
              },
              "protocol": "Tcp"
            }
          }
        ],
        "probes": [
          {
            "name": "FabricGatewayProbe",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('fabricTcpGatewayPort')]",
              "protocol": "Tcp"
            }
          },
          {
            "name": "FabricHttpGatewayProbe",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('fabricHttpGatewayPort')]",
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortProbe1",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('loadBalancedAppPort1')]",
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortProbe2",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('loadBalancedAppPort2')]",
              "protocol": "Tcp"
            }
          }
        ],
        "inboundNatPools": [
          {
            "name": "LoadBalancerBEAddressNatPool",
            "properties": {
              "backendPort": 3389,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig2')]"
              },
              "frontendPortRangeEnd": 4500,
              "frontendPortRangeStart": 3389,
              "protocol": "Tcp"
            }
          }
        ]
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/loadBalancers",
      "name": "[variables('lbName3')]",
      "location": "[parameters('clusterRegion3')]",
      "dependsOn": [
        "[variables('lbIPName3')]"
      ],
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "LoadBalancer Region 3"
      },
      "properties": {
        "frontendIPConfigurations": [
          {
            "name": "LoadBalancerIPConfig",
            "properties": {
              "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses', variables('lbIPName3'))]"
              }
            }
          }
        ],
        "backendAddressPools": [
          {
            "name": "LoadBalancerBEAddressPool"
          }
        ],
        "loadBalancingRules": [
          {
            "name": "LBRule",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID3')]"
              },
              "backendPort": "[variables('fabricTcpGatewayPort')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig3')]"
              },
              "frontendPort": "[variables('fabricTcpGatewayPort')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[variables('lbProbeID3')]"
              },
              "protocol": "Tcp"
            }
          },
          {
            "name": "LBHttpRule",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID3')]"
              },
              "backendPort": "[variables('fabricHttpGatewayPort')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig3')]"
              },
              "frontendPort": "[variables('fabricHttpGatewayPort')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[variables('lbHttpProbeID3')]"
              },
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortLBRule1",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID3')]"
              },
              "backendPort": "[variables('loadBalancedAppPort1')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig3')]"
              },
              "frontendPort": "[variables('loadBalancedAppPort1')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[concat(variables('lbID3'), '/probes/AppPortProbe1')]"
              },
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortLBRule2",
            "properties": {
              "backendAddressPool": {
                "id": "[variables('lbPoolID3')]"
              },
              "backendPort": "[variables('loadBalancedAppPort2')]",
              "enableFloatingIP": false,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig3')]"
              },
              "frontendPort": "[variables('loadBalancedAppPort2')]",
              "idleTimeoutInMinutes": 5,
              "probe": {
                "id": "[concat(variables('lbID3'), '/probes/AppPortProbe2')]"
              },
              "protocol": "Tcp"
            }
          }
        ],
        "probes": [
          {
            "name": "FabricGatewayProbe",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('fabricTcpGatewayPort')]",
              "protocol": "Tcp"
            }
          },
          {
            "name": "FabricHttpGatewayProbe",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('fabricHttpGatewayPort')]",
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortProbe1",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('loadBalancedAppPort1')]",
              "protocol": "Tcp"
            }
          },
          {
            "name": "AppPortProbe2",
            "properties": {
              "intervalInSeconds": 5,
              "numberOfProbes": 2,
              "port": "[variables('loadBalancedAppPort2')]",
              "protocol": "Tcp"
            }
          }
        ],
        "inboundNatPools": [
          {
            "name": "LoadBalancerBEAddressNatPool",
            "properties": {
              "backendPort": 3389,
              "frontendIPConfiguration": {
                "id": "[variables('lbIPConfig3')]"
              },
              "frontendPortRangeEnd": 4500,
              "frontendPortRangeStart": 3389,
              "protocol": "Tcp"
            }
          }
        ]
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Compute/virtualMachineScaleSets",
      "name": "[concat(variables('vmNodeType0Name'),'1')]",
      "location": "[parameters('clusterRegion1')]",
      "dependsOn": [
        "[concat(variables('vmStorageAccountName'),'1')]",
        "[variables('virtualNetworkNameRegion1')]",
        "[variables('supportLogStorageAccountName')]",
        "[variables('applicationDiagnosticsStorageAccountName')]"
      ],
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Cluster scale set Region 1"
      },
      "properties": {
        "upgradePolicy": {
          "mode": "Automatic"
        },
        "virtualMachineProfile": {
          "extensionProfile": {
            "extensions": [
              {
                "name": "[concat('ServiceFabricNodeVmExt', '_', variables('vmNodeType0Name'))]",
                "properties": {
                  "type": "ServiceFabricNode",
                  "autoUpgradeMinorVersion": true,
                  "protectedSettings": {
                    "StorageAccountKey1": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('supportLogStorageAccountName')),'2015-06-15').key1]",
                    "StorageAccountKey2": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('supportLogStorageAccountName')),'2015-06-15').key2]"
                  },
                  "publisher": "Microsoft.Azure.ServiceFabric",
                  "settings": {
                    "clusterEndpoint": "[reference(variables('clusterName')).clusterEndpoint]",
                    "nodeTypeRef": "[variables('vmNodeType0Name')]",
                    "certificate": {
                      "thumbprint": "[parameters('certificateThumbprint')]",
                      "x509StoreName": "[variables('certificateStoreValue')]"
                    }
                  },
                  "typeHandlerVersion": "1.0"
                }
              },
              {
                "name": "[concat('VMDiagnosticsVmExt', '_', variables('vmNodeType0Name'))]",
                "properties": {
                  "type": "IaaSDiagnostics",
                  "autoUpgradeMinorVersion": true,
                  "protectedSettings": {
                    "storageAccountName": "[variables('applicationDiagnosticsStorageAccountName')]",
                    "storageAccountKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('applicationDiagnosticsStorageAccountName')),'2015-06-15').key1]",
                    "storageAccountEndPoint": "https://core.windows.net/"
                  },
                  "publisher": "Microsoft.Azure.Diagnostics",
                  "settings": {
                    "WadCfg": {
                      "DiagnosticMonitorConfiguration": {
                        "overallQuotaInMB": "50000",
                        "EtwProviders": {
                          "EtwEventSourceProviderConfiguration": [
                            {
                              "provider": "Microsoft-ServiceFabric-Actors",
                              "scheduledTransferKeywordFilter": "1",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricReliableActorEventTable"
                              }
                            },
                            {
                              "provider": "Microsoft-ServiceFabric-Services",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricReliableServiceEventTable"
                              }
                            }
                          ],
                          "EtwManifestProviderConfiguration": [
                            {
                              "provider": "cbd93bc2-71e5-4566-b3a7-595d8eeca6e8",
                              "scheduledTransferLogLevelFilter": "Information",
                              "scheduledTransferKeywordFilter": "4611686018427387904",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricSystemEventTable"
                              }
                            }
                          ]
                        }
                      }
                    },
                    "StorageAccount": "[variables('applicationDiagnosticsStorageAccountName')]"
                  },
                  "typeHandlerVersion": "1.5"
                }
              }
            ]
          },
          "networkProfile": {
            "networkInterfaceConfigurations": [
              {
                "name": "NIC-0",
                "properties": {
                  "ipConfigurations": [
                    {
                      "name": "NIC-0-config",
                      "properties": {
                        "loadBalancerBackendAddressPools": [
                          {
                            "id": "[variables('lbPoolID1')]"
                          }
                        ],
                        "loadBalancerInboundNatPools": [
                          {
                            "id": "[variables('lbNatPoolID1')]"
                          }
                        ],
                        "subnet": {
                          "id": "[variables('SubnetRef1')]"
                        }
                      }
                    }
                  ],
                  "primary": true
                }
              }
            ]
          },
          "osProfile": {
            "adminPassword": "[parameters('adminPassword')]",
            "adminUsername": "[parameters('adminUsername')]",
            "computerNamePrefix": "[concat(variables('vmNodeType0Name'),'2')]",
            "secrets": [
              {
                "sourceVault": {
                  "id": "[parameters('sourceVaultRegion1ResourceId')]"
                },
                "vaultCertificates": [
                  {
                    "certificateStore": "[variables('certificateStoreValue')]",
                    "certificateUrl": "[parameters('certificateRegion1UrlValue')]"
                  }
                ]
              }
            ]
          },
          "storageProfile": {
            "imageReference": {
              "publisher": "MicrosoftWindowsServer",
              "offer": "WindowsServer",
              "sku": "2012-R2-Datacenter",
              "version": "latest"
            },
            "osDisk": {
              "vhdContainers": [
                "[concat('http://',variables('vmStorageAccountName'),'1','.blob.core.windows.net/vhds')]"
              ],
              "name": "vmssosdisk",
              "caching": "ReadOnly",
              "createOption": "FromImage"
            }
          }
        }
      },
      "sku": {
        "name": "Standard_A2",
        "capacity": 5,
        "tier": "Standard"
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Compute/virtualMachineScaleSets",
      "name": "[concat(variables('vmNodeType0Name'),'2')]",
      "location": "[parameters('clusterRegion2')]",
      "dependsOn": [
        "[concat(variables('vmStorageAccountName'),'2')]",
        "[variables('virtualNetworkNameRegion2')]",
        "[variables('supportLogStorageAccountName')]",
        "[variables('applicationDiagnosticsStorageAccountName')]"
      ],
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Cluster scale set Region 2"
      },
      "properties": {
        "upgradePolicy": {
          "mode": "Automatic"
        },
        "virtualMachineProfile": {
          "extensionProfile": {
            "extensions": [
              {
                "name": "[concat('ServiceFabricNodeVmExt', '_', variables('vmNodeType0Name'))]",
                "properties": {
                  "type": "ServiceFabricNode",
                  "autoUpgradeMinorVersion": true,
                  "protectedSettings": {
                    "StorageAccountKey1": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('supportLogStorageAccountName')),'2015-06-15').key1]",
                    "StorageAccountKey2": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('supportLogStorageAccountName')),'2015-06-15').key2]"
                  },
                  "publisher": "Microsoft.Azure.ServiceFabric",
                  "settings": {
                    "clusterEndpoint": "[reference(variables('clusterName')).clusterEndpoint]",
                    "nodeTypeRef": "[variables('vmNodeType0Name')]",
                    "certificate": {
                      "thumbprint": "[parameters('certificateThumbprint')]",
                      "x509StoreName": "[variables('certificateStoreValue')]"
                    }
                  },
                  "typeHandlerVersion": "1.0"
                }
              },
              {
                "name": "[concat('VMDiagnosticsVmExt', '_', variables('vmNodeType0Name'))]",
                "properties": {
                  "type": "IaaSDiagnostics",
                  "autoUpgradeMinorVersion": true,
                  "protectedSettings": {
                    "storageAccountName": "[variables('applicationDiagnosticsStorageAccountName')]",
                    "storageAccountKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('applicationDiagnosticsStorageAccountName')),'2015-06-15').key1]",
                    "storageAccountEndPoint": "https://core.windows.net/"
                  },
                  "publisher": "Microsoft.Azure.Diagnostics",
                  "settings": {
                    "WadCfg": {
                      "DiagnosticMonitorConfiguration": {
                        "overallQuotaInMB": "50000",
                        "EtwProviders": {
                          "EtwEventSourceProviderConfiguration": [
                            {
                              "provider": "Microsoft-ServiceFabric-Actors",
                              "scheduledTransferKeywordFilter": "1",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricReliableActorEventTable"
                              }
                            },
                            {
                              "provider": "Microsoft-ServiceFabric-Services",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricReliableServiceEventTable"
                              }
                            }
                          ],
                          "EtwManifestProviderConfiguration": [
                            {
                              "provider": "cbd93bc2-71e5-4566-b3a7-595d8eeca6e8",
                              "scheduledTransferLogLevelFilter": "Information",
                              "scheduledTransferKeywordFilter": "4611686018427387904",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricSystemEventTable"
                              }
                            }
                          ]
                        }
                      }
                    },
                    "StorageAccount": "[variables('applicationDiagnosticsStorageAccountName')]"
                  },
                  "typeHandlerVersion": "1.5"
                }
              }
            ]
          },
          "networkProfile": {
            "networkInterfaceConfigurations": [
              {
                "name": "NIC-0",
                "properties": {
                  "ipConfigurations": [
                    {
                      "name": "NIC-0-config",
                      "properties": {
                        "loadBalancerBackendAddressPools": [
                          {
                            "id": "[variables('lbPoolID2')]"
                          }
                        ],
                        "loadBalancerInboundNatPools": [
                          {
                            "id": "[variables('lbNatPoolID2')]"
                          }
                        ],
                        "subnet": {
                          "id": "[variables('subnetRef2')]"
                        }
                      }
                    }
                  ],
                  "primary": true
                }
              }
            ]
          },
          "osProfile": {
            "adminPassword": "[parameters('adminPassword')]",
            "adminUsername": "[parameters('adminUsername')]",
            "computerNamePrefix": "[concat(variables('vmNodeType0Name'), '2')]",
            "secrets": [
              {
                "sourceVault": {
                  "id": "[parameters('sourceVaultRegion2ResourceId')]"
                },
                "vaultCertificates": [
                  {
                    "certificateStore": "[variables('certificateStoreValue')]",
                    "certificateUrl": "[parameters('certificateRegion2UrlValue')]"
                  }
                ]
              }
            ]
          },
          "storageProfile": {
            "imageReference": {
              "publisher": "MicrosoftWindowsServer",
              "offer": "WindowsServer",
              "sku": "2012-R2-Datacenter",
              "version": "latest"
            },
            "osDisk": {
              "vhdContainers": [
                "[concat('http://',variables('vmStorageAccountName'),'2','.blob.core.windows.net/vhds')]"
              ],
              "name": "vmssosdisk",
              "caching": "ReadOnly",
              "createOption": "FromImage"
            }
          }
        }
      },
      "sku": {
        "name": "Standard_A2",
        "capacity": 5,
        "tier": "Standard"
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Compute/virtualMachineScaleSets",
      "name": "[concat(variables('vmNodeType0Name'),'3')]",
      "location": "[parameters('clusterRegion3')]",
      "dependsOn": [
        "[concat(variables('vmStorageAccountName'),'3')]",
        "[variables('virtualNetworkNameRegion3')]",
        "[variables('supportLogStorageAccountName')]",
        "[variables('applicationDiagnosticsStorageAccountName')]"
      ],
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Cluster scale set Region 3"
      },
      "properties": {
        "upgradePolicy": {
          "mode": "Automatic"
        },
        "virtualMachineProfile": {
          "extensionProfile": {
            "extensions": [
              {
                "name": "[concat('ServiceFabricNodeVmExt', '_', variables('vmNodeType0Name'))]",
                "properties": {
                  "type": "ServiceFabricNode",
                  "autoUpgradeMinorVersion": true,
                  "protectedSettings": {
                    "StorageAccountKey1": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('supportLogStorageAccountName')),'2015-06-15').key1]",
                    "StorageAccountKey2": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('supportLogStorageAccountName')),'2015-06-15').key2]"
                  },
                  "publisher": "Microsoft.Azure.ServiceFabric",
                  "settings": {
                    "clusterEndpoint": "[reference(variables('clusterName')).clusterEndpoint]",
                    "nodeTypeRef": "[variables('vmNodeType0Name')]",
                    "certificate": {
                      "thumbprint": "[parameters('certificateThumbprint')]",
                      "x509StoreName": "[variables('certificateStoreValue')]"
                    }
                  },
                  "typeHandlerVersion": "1.0"
                }
              },
              {
                "name": "[concat('VMDiagnosticsVmExt', '_', variables('vmNodeType0Name'))]",
                "properties": {
                  "type": "IaaSDiagnostics",
                  "autoUpgradeMinorVersion": true,
                  "protectedSettings": {
                    "storageAccountName": "[variables('applicationDiagnosticsStorageAccountName')]",
                    "storageAccountKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('applicationDiagnosticsStorageAccountName')),'2015-06-15').key1]",
                    "storageAccountEndPoint": "https://core.windows.net/"
                  },
                  "publisher": "Microsoft.Azure.Diagnostics",
                  "settings": {
                    "WadCfg": {
                      "DiagnosticMonitorConfiguration": {
                        "overallQuotaInMB": "50000",
                        "EtwProviders": {
                          "EtwEventSourceProviderConfiguration": [
                            {
                              "provider": "Microsoft-ServiceFabric-Actors",
                              "scheduledTransferKeywordFilter": "1",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricReliableActorEventTable"
                              }
                            },
                            {
                              "provider": "Microsoft-ServiceFabric-Services",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricReliableServiceEventTable"
                              }
                            }
                          ],
                          "EtwManifestProviderConfiguration": [
                            {
                              "provider": "cbd93bc2-71e5-4566-b3a7-595d8eeca6e8",
                              "scheduledTransferLogLevelFilter": "Information",
                              "scheduledTransferKeywordFilter": "4611686018427387904",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricSystemEventTable"
                              }
                            }
                          ]
                        }
                      }
                    },
                    "StorageAccount": "[variables('applicationDiagnosticsStorageAccountName')]"
                  },
                  "typeHandlerVersion": "1.5"
                }
              }
            ]
          },
          "networkProfile": {
            "networkInterfaceConfigurations": [
              {
                "name": "NIC-0",
                "properties": {
                  "ipConfigurations": [
                    {
                      "name": "NIC-0-config",
                      "properties": {
                        "loadBalancerBackendAddressPools": [
                          {
                            "id": "[variables('lbPoolID3')]"
                          }
                        ],
                        "loadBalancerInboundNatPools": [
                          {
                            "id": "[variables('lbNatPoolID3')]"
                          }
                        ],
                        "subnet": {
                          "id": "[variables('subnetRef3')]"
                        }
                      }
                    }
                  ],
                  "primary": true
                }
              }
            ]
          },
          "osProfile": {
            "adminPassword": "[parameters('adminPassword')]",
            "adminUsername": "[parameters('adminUsername')]",
            "computerNamePrefix": "[concat(variables('vmNodeType0Name'), '3')]",
            "secrets": [
              {
                "sourceVault": {
                  "id": "[parameters('sourceVaultRegion3ResourceId')]"
                },
                "vaultCertificates": [
                  {
                    "certificateStore": "[variables('certificateStoreValue')]",
                    "certificateUrl": "[parameters('certificateRegion3UrlValue')]"
                  }
                ]
              }
            ]
          },
          "storageProfile": {
            "imageReference": {
              "publisher": "MicrosoftWindowsServer",
              "offer": "WindowsServer",
              "sku": "2012-R2-Datacenter",
              "version": "latest"
            },
            "osDisk": {
              "vhdContainers": [
                "[concat('http://',variables('vmStorageAccountName'),'3','.blob.core.windows.net/vhds')]"
              ],
              "name": "vmssosdisk",
              "caching": "ReadOnly",
              "createOption": "FromImage"
            }
          }
        }
      },
      "sku": {
        "name": "Standard_A2",
        "capacity": 5,
        "tier": "Standard"
      }
    },
    {
      "apiVersion": "2016-01-01-beta",
      "type": "Microsoft.ServiceFabric/clusters",
      "name": "[variables('clusterName')]",
      "location": "[parameters('clusterRegion1')]",
      "dependsOn": [
        "[variables('supportLogStorageAccountName')]"
      ],
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Cluster"
      },
      "properties": {
        "certificate": {
          "thumbprint": "[parameters('certificateThumbprint')]",
          "x509StoreName": "[variables('certificateStoreValue')]"
        },
        "clientCertificateCommonNames": [ ],
        "clientCertificateThumbprints": [ ],
        "clusterState": "Default",
        "diagnosticsStorageAccountConfig": {
          "blobEndpoint": "[concat('https://', variables('supportLogStorageAccountName'), '.blob.core.windows.net/')]",
          "protectedAccountKeyName": "StorageAccountKey1",
          "queueEndpoint": "[concat('https://', variables('supportLogStorageAccountName'), '.queue.core.windows.net/')]",
          "storageAccountName": "[variables('supportLogStorageAccountName')]",
          "tableEndpoint": "[concat('https://', variables('supportLogStorageAccountName'), '.table.core.windows.net/')]"
        },
        "fabricSettings": [
          {
            "parameters": [
              {
                "name": "ClusterProtectionLevel",
                "value": "EncryptAndSign"
              }
            ],
            "name": "Security"
          }
        ],
        "managementEndpoint": "[concat('https://', variables('clusterfqdn') , ':', variables('fabricHttpGatewayPort'))]",
        "nodeTypes": [
          {
            "name": "[variables('vmNodeType0Name')]",
            "applicationPorts": {
              "endPort": "30000",
              "startPort": "20000"
            },
            "clientConnectionEndpointPort": "[variables('fabricTcpGatewayPort')]",
            "ephemeralPorts": {
              "endPort": "65534",
              "startPort": "49152"
            },
            "httpGatewayEndpointPort": "[variables('fabricHttpGatewayPort')]",
            "isPrimary": true,
            "vmInstanceCount": 15
          }
        ],
        "provisioningState": "Default"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/publicIPAddresses",
      "name": "[variables('gatewayPublicIPName1')]",
      "location": "[parameters('clusterRegion1')]",
      "properties": {
        "publicIPAllocationMethod": "Dynamic"
      },
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Gatewaty Public IP Region 1"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/publicIPAddresses",
      "name": "[variables('gatewayPublicIPName2')]",
      "location": "[parameters('clusterRegion2')]",
      "properties": {
        "publicIPAllocationMethod": "Dynamic"
      },
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Gateway Public IP Region 2"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/publicIPAddresses",
      "name": "[variables('gatewayPublicIPName3')]",
      "location": "[parameters('clusterRegion3')]",
      "properties": {
        "publicIPAllocationMethod": "Dynamic"
      },
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Gatewaty Public IP Region 3"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/virtualNetworkGateways",
      "name": "[variables('gatewayName1')]",
      "location": "[parameters('clusterRegion1')]",
      "dependsOn": [
        "[concat('Microsoft.Network/publicIPAddresses/', variables('gatewayPublicIPName1'))]",
        "[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkNameRegion1'))]"
      ],
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Virtual Network Gateway Region 1"
      },
      "properties": {
        "ipConfigurations": [
          {
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "subnet": {
                "id": "[variables('gatewaySubnetRef1')]"
              },
              "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('gatewayPublicIPName1'))]"
              }
            },
            "name": "vnetGatewayConfig1"
          }
        ],
        "gatewaySize": "HighPerformance",
        "gatewayType": "Vpn",
        "vpnType": "RouteBased",
        "enableBgp": "false"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/virtualNetworkGateways",
      "name": "[variables('gatewayName2')]",
      "location": "[parameters('clusterRegion2')]",
      "dependsOn": [
        "[concat('Microsoft.Network/publicIPAddresses/', variables('gatewayPublicIPName2'))]",
        "[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkNameRegion2'))]"
      ],
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Virtual Network Gateway Region 2"
      },
      "properties": {
        "ipConfigurations": [
          {
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "subnet": {
                "id": "[variables('gatewaySubnetRef2')]"
              },
              "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('gatewayPublicIPName2'))]"
              }
            },
            "name": "vnetGatewayConfig2"
          }
        ],
        "gatewaySize": "HighPerformance",
        "gatewayType": "Vpn",
        "vpnType": "RouteBased",
        "enableBgp": "false"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/virtualNetworkGateways",
      "name": "[variables('gatewayName3')]",
      "location": "[parameters('clusterRegion3')]",
      "dependsOn": [
        "[concat('Microsoft.Network/publicIPAddresses/', variables('gatewayPublicIPName3'))]",
        "[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkNameRegion3'))]"
      ],
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Virtual Network Gateway Region 3"
      },
      "properties": {
        "ipConfigurations": [
          {
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "subnet": {
                "id": "[variables('gatewaySubnetRef3')]"
              },
              "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('gatewayPublicIPName3'))]"
              }
            },
            "name": "vnetGatewayConfig3"
          }
        ],
        "gatewaySize": "HighPerformance",
        "gatewayType": "Vpn",
        "vpnType": "RouteBased",
        "enableBgp": "false"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/connections",
      "name": "[variables('connectionName12')]",
      "location": "[parameters('clusterRegion1')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Connections Region (1 - 2)"
      },
      "dependsOn": [
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName1'))]",
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName2'))]"
      ],
      "properties": {
        "virtualNetworkGateway1": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName1'))]"
        },
        "virtualNetworkGateway2": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName2'))]"
        },
        "connectionType": "Vnet2Vnet",
        "routingWeight": 3,
        "sharedKey": "[parameters('sharedKey')]"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/connections",
      "name": "[variables('connectionName13')]",
      "location": "[parameters('clusterRegion1')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Connections Region (1 - 3)"
      },
      "dependsOn": [
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName1'))]",
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName3'))]"
      ],
      "properties": {
        "virtualNetworkGateway1": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName1'))]"
        },
        "virtualNetworkGateway2": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName3'))]"
        },
        "connectionType": "Vnet2Vnet",
        "routingWeight": 3,
        "sharedKey": "[parameters('sharedKey')]"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/connections",
      "name": "[variables('connectionName21')]",
      "location": "[parameters('clusterRegion2')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Connections Region (2 - 1)"
      },
      "dependsOn": [
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName1'))]",
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName2'))]"
      ],
      "properties": {
        "virtualNetworkGateway1": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName2'))]"
        },
        "virtualNetworkGateway2": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName1'))]"
        },
        "connectionType": "Vnet2Vnet",
        "routingWeight": 3,
        "sharedKey": "[parameters('sharedKey')]"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/connections",
      "name": "[variables('connectionName23')]",
      "location": "[parameters('clusterRegion2')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Connections Region (2 - 3)"
      },
      "dependsOn": [
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName2'))]",
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName3'))]"
      ],
      "properties": {
        "virtualNetworkGateway1": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName2'))]"
        },
        "virtualNetworkGateway2": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName3'))]"
        },
        "connectionType": "Vnet2Vnet",
        "routingWeight": 3,
        "sharedKey": "[parameters('sharedKey')]"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/connections",
      "name": "[variables('connectionName31')]",
      "location": "[parameters('clusterRegion3')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Connections Region (3 - 1)"
      },
      "dependsOn": [
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName1'))]",
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName3'))]"
      ],
      "properties": {
        "virtualNetworkGateway1": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName3'))]"
        },
        "virtualNetworkGateway2": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName1'))]"
        },
        "connectionType": "Vnet2Vnet",
        "routingWeight": 3,
        "sharedKey": "[parameters('sharedKey')]"
      }
    },
    {
      "apiVersion": "[variables('apiVersion')]",
      "type": "Microsoft.Network/connections",
      "name": "[variables('connectionName32')]",
      "location": "[parameters('clusterRegion3')]",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Connections Region (3 - 2)"
      },
      "dependsOn": [
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName2'))]",
        "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName3'))]"
      ],
      "properties": {
        "virtualNetworkGateway1": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName3'))]"
        },
        "virtualNetworkGateway2": {
          "id": "[resourceId('Microsoft.Network/virtualNetworkGateways',variables('gatewayName2'))]"
        },
        "connectionType": "Vnet2Vnet",
        "routingWeight": 3,
        "sharedKey": "[parameters('sharedKey')]"
      }
    },
    {
      "apiVersion": "2015-04-28-preview",
      "type": "Microsoft.Network/trafficManagerProfiles",
      "name": "[concat(variables('clusterName'),'tm')]",
      "location": "global",
      "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[variables('clusterName')]",
        "displayName": "Traffic Manager"
      },
      "dependsOn": [
        "[resourceId('Microsoft.Network/publicIPAddresses',variables('lbIPName1'))]",
        "[resourceId('Microsoft.Network/publicIPAddresses',variables('lbIPName2'))]",
        "[resourceId('Microsoft.Network/publicIPAddresses',variables('lbIPName3'))]"
      ],
      "properties": {
        "profileStatus": "Enabled",
        "trafficRoutingMethod": "Weighted",
        "dnsConfig": {
          "relativeName": "[variables('clusterName')]",
          "ttl": 30
        },
        "monitorConfig": {
          "protocol": "http",
          "port": 80,
          "path": "/"
        },
        "endpoints": [
          {
            "name": "[parameters('clusterRegion1')]",
            "type": "Microsoft.Network/trafficManagerProfiles/azureEndpoints",
            "properties": {
              "targetResourceId": "[resourceId('Microsoft.Network/publicIPAddresses', variables('lbIPName1'))]",
              "endpointStatus": "Enabled",
              "weight": 1
            }
          },
          {
            "name": "[parameters('clusterRegion2')]",
            "type": "Microsoft.Network/trafficManagerProfiles/azureEndpoints",
            "properties": {
              "targetResourceId": "[resourceId('Microsoft.Network/publicIPAddresses', variables('lbIPName2'))]",
              "endpointStatus": "Enabled",
              "weight": 1
            }
          },
          {
            "name": "[parameters('clusterRegion3')]",
            "type": "Microsoft.Network/trafficManagerProfiles/azureEndpoints",
            "properties": {
              "targetResourceId": "[resourceId('Microsoft.Network/publicIPAddresses', variables('lbIPName3'))]",
              "endpointStatus": "Enabled",
              "weight": 1
            }
          }
        ]
      }
    }
  ],
  "outputs": {
    "clusterProperties": {
      "value": "[reference(variables('clusterName'))]",
      "type": "object"
    },
    "trafficManagerProperties": {
      "value": "[reference(concat(variables('clusterName'),'tm'))]",
      "type": "object"
    }
  }
}

NOTE: This was an Experiment

This post was written as the result of my personal experimentation. I heard it was possible, and I set out to build it. This ARM Template is certainly a good place to start. Keep in mind that I did not customize or optimize the Service Fabric cluster. That work still needs to be done.

Share your thoughts in the comments below

In Microsoft Azure , , , , , , , , , , , , , , , , ,
permalink 15 Comments

15 responses to Deploy a Geo-HA Service Fabric Cluster on Azure

  1. 
    Christian Weiss June 6, 2016 at 2:07 AM

    this is great, thank you very much!

    Like

    Reply
  2. 
    Oliver Tomlinson (@dotDestroyer) July 2, 2016 at 6:19 AM

    Could you achieve a Geo-HA with just 2 datacenters if you set the target replica set size to 7 and set the minimum replica set size to 3?

    As I understand it Service Fabric would balance the 7 replicas across the fault domains such that FD1 would have 3 replicas and FD2 would have 4 replicas.

    If for instance you lost FD1, you would still have 4 replicas on FD2 (which would give a quorum of 3 out of the 4) and similarly if you lost FD2, you would still have 3 replicas on FD 1 (which would give a quorum of 2 of the 3)

    Like

    Reply
  3. 
    Anders Lybecker August 24, 2016 at 8:48 AM

    Very interesting. Will it work (well) with statefull services, actors etc?

    Like

    Reply
  4. 
    Tom Austin September 1, 2016 at 8:08 AM

    Really interesting. Thanks!

    Like

    Reply
  5. 
    Jean-Francois November 21, 2016 at 12:37 AM

    Bonjour Alexandre, we will be looking at implementing a SF cluster that follows the same Geo HA architecture as you described in your article.. will be happy to let you know how it went.

    I have a use case that I would like to run by you:

    Let’s say a web application makes a request to a Stateless Web API hosted in Service Fabric. The traffic manager will redirect the request to the closest load balancer (say Region 3) which in turn will send the request to one of the 5 nodes in Region 3. Once the stateless Web Api instance receives the request, it creates a ServiceProxy which we will use to read data in a Stateful Service and then send that data back to the web application.

    Here is the question:

    Is it possible to force the ServiceProxy to create a proxy to a Stateful Service that is in the same region as itself (region 3)? The goal would to keep ALL remoting connection between services contained within the same Region.

    Merci,
    Jean-Francois

    Like

    Reply
    • 
      Jean-Francois November 21, 2016 at 12:23 PM

      Rephrasing the question:

      When the Stateless Service (Web Api) creates a ServiceProxy to call the Stateful service, is it possible to force the ServiceProxy to create a proxy to the Stateful Service replica that is the closest to itself (ie. region 3)? The goal would to keep ALL remoting connection between services contained within the same Region.

      Like

      Reply
  6. 
    Vijay Kumar Nagappa October 17, 2017 at 7:52 AM

    Hi Alexander,

    I have a question. how is data replicated across regions. Is it taken care by Azure Service Fabric itself. Wonderful post.

    Regards,
    -Vijay

    Like

    Reply
  7. 
    Alexander Stulov August 7, 2018 at 6:02 AM

    Hi Alexandre,

    Could you clarify if such setup works in case of Master node failure (Master node switch to another region)?

    Regards,
    Alexander

    Like

    Reply

Trackbacks and Pingbacks:

  1. Azure Update (2016.06.02) | ブチザッキ - June 1, 2016

    […] Geo-HA Service Fabric Cluster […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.