Available in release: Windows Server vNext Technical Preview
Author:
Amitabh P Tamhane
Senior Program Manager
Microsoft Corp.
Cloud Witness is a new type of Failover Cluster quorum witness being introduced in Windows Server Technical Preview. In this blog, I intend to give a quick overview of Cloud Witness and the steps required to configure it.
Consider an example multi-site stretched Failover Cluster quorum configuration with Windows Server 2012 R2:
In this example configuration, there are 2 nodes in 2 datacenters (referred as Sites). Note, it is possible for cluster to span more than 2 datacenters as well as each datacenter can have many more than 2 nodes. A typical cluster quorum configuration in this setup (automatic failover SLA) would give each node a vote. And then we need one extra vote of the quorum witness to allow cluster to keep running even if either one of the datacenter experiences power outage. Math is simple: There are 5 total votes, and you need 3 votes for the cluster to keep running.
In case of power outage in one datacenter, to give equal opportunity for the cluster in other datacenter to keep running, it is recommended to host the quorum witness in a location other than the two datacenters. This typically means requiring a 3rd separate datacenter (site) to host File Server that is backing the File Share which is used as the quorum witness (File Share Witness).
We received feedback from our customers, that most don’t have a 3rd separate datacenter that will host File Server backing the File Share Witness. This means customers host the File Server in one of the two datacenters, by extension making that datacenter the primary datacenter. In a scenario where there is power outage in the primary datacenter, the cluster would go down as the other datacenter would only have 2 votes which is below the quorum majority of 3 votes. For the customers that have 3rd separate datacenter to host the File Server, it is an overhead to maintain the highly available File Server backing the File Share Witness. Hosting VMs in public cloud that have the File Server for File Share Witness running in Guest OS is a significant overhead in terms of both setup & maintenance.
Introducing Cloud Witness
Cloud Witness is a new type of Failover Cluster quorum witness that leverages Microsoft Azure as the arbitration point. It uses Microsoft Azure Blob Storage to read/write a blob file which is then used as an arbitration point in case of split-brain resolution.
There are significant benefits which this approach:
- Leverages Microsoft Azure (no need for 3rd separate datacenter)
- Uses standard publically available Microsoft Azure Blob Storage (no extra maintenance overhead of VMs hosted in public cloud)
- Same Microsoft Azure Storage Account can be used for multiple clusters (one blob file per cluster; cluster unique id used as blob file name)
- Very low on-going $cost to the Storage Account (very small data written per blob file, blob file updated only once when cluster nodes’ state changes)
- Built-in Cloud Witness resource type
Multi-site stretched clusters with Cloud Witness:
Notice there is no 3rd separate site that is required. Cloud Witness, like any other quorum witness, gets a vote and can participate in quorum calculations.
Cloud Witness: Single Witness Type for most scenarios
If you have a Failover Cluster deployment, where all nodes can reach the internet (by extension Microsoft Azure), it is recommended to configure Cloud Witness as your quorum witness resource.
Scenarios including (there may be more scenarios beyond the list below):
- Disaster recovery stretched multi-site clusters (example above)
- Failover clusters without shared storage (SQL Always On, Exchange DAGs, etc.)
- Failover clusters running inside Guest OS hosted in Microsoft Azure VM Role (or any other public cloud)
- Failover clusters running inside Guest OS of VMs hosted in Private Clouds
- Storage clusters with or without shared storage (Scale-out File Server clusters, etc.)
- Small branch-office clusters (even 2-node clusters)
Starting Windows Server 2012 R2, we recommend to “Always configure a Witness” as cluster automatically manages the witness vote, nodes vote with Dynamic Quorum. Cloud Witness complements this recommendation with “Always configure a Cloud Witness”.
For Failover Clusters that do not have access to internet, we recommend to continue to configure File Share Witness or Disk Witness as per your deployment configuration setup.
Creating Microsoft Azure Storage Account
To configure Cloud Witness, you would need a valid Microsoft Azure Storage Account which would be used to store the blob file (used for arbitration). Cloud Witness creates a well-known Container “msft-cloud-witness” under the Microsoft Storage Account. Cloud Witness writes a single blob file with corresponding cluster’s unique Id used as the file name of the blob file under this “msft-cloud-witness” container. This means same Microsoft Azure Storage Account can be used to configure a Cloud Witness for multiple different clusters.
When using the same Microsoft Azure Storage Account for configuring Cloud Witness for multiple different clusters, then you would notice there is a single “msft-cloud-witness” container that gets automatically created. And this container will contain one-blob file per cluster.
Microsoft Azure Management Portal: Creating Storage Account
When creating Microsoft Azure Storage Account, it is very important to select “Locally Redundant” for Replication Type. Failover Cluster uses the blob file as the arbitration point, which requires some consistency guarantees when reading the data.
Microsoft Azure Management Portal: Managing Access Keys
Microsoft Azure Management Portal: URL Links
When you create a Storage Account, the following URLs are generated using the format: https://<Storage Account Name>.<Storage Type>.<Endpoint>
For Cloud Witness, it always uses “Blob” as the storage type. Microsoft Azure uses “.core.windows.net” as the Endpoint. When configuring Cloud Witness, it is possible that you configure it with a different Endpoint as per your scenario (for example the Microsoft Azure in China has a different Endpoint).
Note: This URL is generated automatically by Cloud Witness resource and there is no extra step of configuration necessary for the URL. I added this section in the blog to give some context of how Cloud Witness reaches Microsoft Azure.
Microsoft Azure Management Portal: Container view
If you look under the “msft-cloud-witness” Container that gets created as part of the Storage Account, you would notice a blob getting created corresponding to the cluster GUID:
Configuring Cloud Witness with Failover Cluster Manager GUI
Cloud Witness configuration is well-integrated within the existing Quorum Configuration Wizard built into the Failover Cluster Manager GUI.
Select “Configure Cluster Quorum Settings”:
Then, select “Select the quorum witness”:
Then, select “Configure a cloud witness”:
On the Cloud Witness configuration page, enter the following information:
- (Required parameter) Microsoft Storage Account Name
- (Required parameter) Access Key corresponding to the Storage Account
- When creating for the first time, use Primary Access Key (see above)
- When rotating the Primary Access Key, use Secondary Access Key (see above)
- (Optional parameter) If you intend to use a different Azure service endpoint (for example the Microsoft Azure service in China), then update the endpoint server name.
Configuring Cloud Witness with PowerShell
The existing Set-ClusterQuorum PowerShell command has new additional parameters corresponding to Cloud Witness.
You can configure Cloud Witness using PowerShell command:
Set-ClusterQuorum –CloudWitness –AccountName <StorageAccountName> -AccessKey <StorageAccountAccessKey>
In case you need to use a different endpoint (rare):
Set-ClusterQuorum –CloudWitness –AccountName <StorageAccountName> -AccessKey <StorageAccountAccessKey> -Endpoint <servername>
Failover Cluster Manager GUI: Cloud Witness
Upon successful configuration of Cloud Witness, you can view the newly created witness resource in the Failover Cluster Manager snap-in:
Microsoft Azure Storage Account considerations with Cloud Witness
- Failover Cluster will not store the Access Key, but rather it will generate a Shared Access Security Token that is generated using the Access Key provided and stores this SAS Token securely.
- The generated SAS Token is valid as long as the Access Key remains valid. When rotating the Primary Access Key, it is important to first update the Cloud Witness (on all your clusters that are using that Storage Account) with the Secondary Access Key before regenerating the Primary Access Key.
- Cloud Witness uses HTTPS REST interface of the Microsoft Azure Storage Account service. This means it requires the HTTPS port to be open on all cluster nodes.
Thank you