On a Windows Server Failover Cluster for each Network Name resource there is a logical name which has a corresponding computer object (CO) created. The computer object associated with the Cluster Name this is commonly referred to as the Cluster Name Object (CNO) and for all other Network Name resources these are commonly referred to as Virtual Computer Objects (VCO). The cluster service creates and manages these CNO and VCO computer objects.

Because CNO and VCO’s are managed by the cluster, they can at times have slightly different behavior than normal computer objects. Understanding these differences can be important when trying to identify when computer objects are stale and no longer being used, so that it is safe to delete them.

If you are running a Windows Server 2008 R2 functional level Active Directory forest, I recommend turning on the Active Directory Recycle Bin feature (which is disabled by default). Here is the step-by-step guide. Active Directory Recycle Bin helps minimize directory service downtime by enhancing your ability to preserve and restore accidentally deleted Active Directory objects without restoring Active Directory data from backups, restarting Active Directory Domain Services (AD DS), or rebooting domain controllers. This is critical for being able to quickly and seamlessly recover from accidental computer object deletion.

There are a couple different attributes commonly used by domain administrators as hints to identify computer objects that are no longer in use and are safe to be deleted:

pwdLastSet – This is the last time the password was changed.
- Windows Server 2003 – The computer object password is not periodically rotated on Windows Server 2003 and below clusters. This attribute should not be used to identify stale objects for deletion on Windows 2000 Server or Windows Server 2003 clusters which have the Enable Kerberos Authentication option turned on. Note that computer objects were not created by default for cluster Network Name resources in these legacy releases.
- Windows Server 2008 – A feature was added in Windows Server 2008 that failover clustering will rotate the CNO and VCO passwords exactly the same as a normal machine computer object. The default domain policy for MaximumPasswordAge will be honored, which is every 30 days by default. Failover Clustering will also honor if DisablePasswordChange is set, to disable password rotation. Password rotation for CNO and VCO's is initiated when the age reaches 75% of expiration to ensure passwords are rotated within the policy, so with the default settings cluster will rotate the password on the 23rd day.
LastLogonTimeStamp – This is the last time a logon occurred. This is a new attribute introduced with a Windows Server 2003 functional level domain.
- When the cluster Network Name resource comes online, it will logon and the LastLogonTimeStamp attributed will be updated. The credentials are cached by clustering and used for the lifespan of the system. If a failure or failover occurs, during the next Network Name resource online a logon will re-occur. When the password for a CNO or VCO is changed per the MaximumPasswordAge policy described above, the credentials are updated and a logon occurs.
  - Using the LastLogonTimeStamp attribute can be an inaccurate hint to identify if a cluster computer object is stale. This is because if a production cluster is up and stable no logon’s will occur and the LastLogonTimeStamp attribute will not be updated until 75% expiration of MaximumPasswordAge. Using LastLogonTimeStamp as a hint can result in accidental CNO and VCO deletion. It is recommended that LastLogonTimeStamp be greater than 30 days (at an absolute minimum) to determine if an object is stale.

Given that CNO and VCO computer objects are associated with mission critical high availability systems, your domain administrators may simply choose to avoid automated / scripted deletion of cluster computer objects. This can be accomplished by querying and excluding computer objects that contain a service principal name (SPN) of “MSClusterVirtualServer” from deletion. Another technique is to move all CNO and VCO's to a custom organizational unit (OU) and exclude that OU from running CO deletion scripts.

As an extra safety mechanism you can protect CNO and VCO's from accidental deletion by simply checking the "Protect object from accidental deletion" box on the objects in the Active Directory Users and Computers snap-in. See this blog for more details

Another hint which can be used to identify stale cluster computer objects is if the computer object is in a Disabled state. By default, when a cluster Network Name resource is deleted or if a cluster is destroyed, the CNO and VCO’s are placed in a disabled state. Any cluster computer object which is in a Disabled state are no longer being used by the cluster.

When destroying a cluster, you may wish to delete the computer objects instead of leaving them in a disabled state. This can be accomplished by passing the –CleanupAD switch to the Remove-Cluster PowerShell cmdlet when destroying the cluster.

If you are reading this after you have already run a script that accidently deleted a CNO or VCO computer object, here is a blog which has a step-by-step guide on how to recover the computer objects:

If you do not have the AD Recycle Bin feature, follow this blog:
http://blogs.technet.com/b/askcore/archive/2009/04/27/recovering-a-deleted-cluster-name-object-cno-in-a-windows-server-2008-failover-cluster.aspx

If you do have the AD Recycle Bin feature, follow this blog:
http://blogs.technet.com/b/askcore/archive/2011/05/18/recovering-a-deleted-cluster-name-object-cno-in-a-windows-server-2008-failover-cluster-part-2.aspx

Thanks!
Elden Christensen
Principal Program Manager Lead
Clustering & High-Availability
Microsoft

Hi Cluster Fans,

This blog post will show the configuration of IP Addresses on multi-subnet clusters for the cluster’s network name, or “Client Access Point” (CAP). This is different from the IP Addresses configuration for other clustered workloads as there is no way to add resource to the CAP via the GUI, it must be done using PowerShell (or Cluster.exe).

First, check out Part 1 of the series which showed how to configure IP Addresses for clustered workloads using the GUI. Part 2 covers some more advanced settings such as Possible Owners. To see a video walkthrough of the multi-site deployment for this cluster, visit: http://technet.microsoft.com/en-us/edge/disaster-recovery-cluster-deployment-demo-multi-site-failover-clustering.

In this scenario I have a 2-node cluster, DRCluster, and I have just added an additional two nodes at my second site. If all four nodes were available when the cluster was created, the IP Addresses on both subnets for the CAP would have automatically been created. However in this scenario the nodes in the second subnet were added later, so the new IP Address for the CAP must be manually created.

Overview

First let’s take a look at the GUI for the ‘Core Cluster Resources’ (CCR) which contains the CAP. This can be done by selecting the cluster in the left navigation pane and expanding the ‘Core Cluster Resources’ section.

Adding a new IP Address to the Group

To add a new IP Address to the CCR we must use PowerShell (or Cluster.exe). We will launch PowerShell as an administrator and import the clustering module:

PS > Import-Module FailoverClusters

Next we will find the correct name of the CCR by running the cmdlet:

PS > Get-ClusterGroup

We see this is simply referred to as ‘Cluster Group’:

Next we want to add a new IP Address which is a type of cluster resource. Do not forget that PowerShell has great built-in help with examples, so even if you are not sure what the correct syntax is, you can use Get-Help <cmdlet name> -full to see all the details. For just the examples, use Get-Help <cmdlet name> -examples. This is very helpful in this case, giving the exact example we want to use:

To add an IP Address to the CCR we use the Add-ClusterResource cmdlet:

PS > Add-ClusterResource –Name NewIP –ResourceType “IP Address” –Group “Cluster Group”

We see that this was successful and the resource is in the Offline state:

This new IP Address is now visible in the GUI:

Configuring the New IP Address

To continue configuring this using PowerShell, you would use the Set-ClusterParameter cmdlet and Set-ClusterResourceDependency. Once again you can use Get-Help <cmdlet name> to get additional guidance. However that is not necessary as it is now possible for us to continue configuring the IP Address and set the dependencies from the GUI.

Right-click on the new IP Address and select ‘Properties’. Under the ‘General’ tab you are able to select the subnet that it is on (10.* for this scenario), as well as whether it uses DHCP or static IP Addresses:

After clicking ‘Apply’ and exiting you will see that the IP Address remains offline. If a static IP Address has been assigned it should be displayed. If DHCP has been selected, then no IP Address will show.

To immediately request a DHCP IP Address, right-click on the IP Address and select ‘Bring this Resource Online’. It will fail and an error message will be displayed. This is expected since the CCR is not on that subnet, however it will grab the IP Address that it will want to use at this time. Alternatively you can just wait until the first failover to the other subnet to get this IP Address. In our case we have been given 10.121.27.199.

Configuring Dependencies

Finally we will make the CAP dependent on either of the two IP Addresses. This will be a ‘OR’ dependency because we want the CAP to be available so long as an IP address on either network is available.

To do this, right-click on the CAP and select ‘Properties’, then select the ‘Dependencies’ tab.

Select the ‘Click here to add a dependency’ text.

From the dropdown in the ‘AND/OR’ column select ‘OR’ since this will be an ‘OR’ dependency type, then from the dropdown in the ‘Resource’ column select the new IP Address:

Finally click Apply, and when you return to the Failover Cluster Manager GUI you can see that it has been added to the CAP group:

It is in an offline state which is expected since the CAP is still on the 157.* subnet.

Testing Failover

We want to make sure that the new IP Address for the CAP will come online on the new cluster. However in the GUI there is no way to move this resource group to another node (other than stopping the cluster service of the node it is currently on).

To do this in PowerShell we use the Move-ClusterGroup cmdlet and specify a node on the other subnet.

PS > Move-ClusterGroup “Cluster Group” –node DRCluster-2a

In the GUI you will now see that the CAP has changed subnets. It is now online on the 10.* subnet and offline on the 157.* subnet:

This should help you understand how to successfully figure IP Addresses for the CAP in multi-subnet clusters. Syntax for these operations using Cluster.exe is available here: http://technet.microsoft.com/en-us/library/cc736767(WS.10).aspx

Thanks!

Symon Perriman
Technical Evangelist

Private Cloud Technologies

Microsoft
@SymonPerriman

Virtualization Career Training

On Tuesday, October 4^th Technical Evangelists Symon Perriman and Rick Claus are hosting an online conference on Virtualization Career Training with Microsoft Learning. This half day virtual event (8am – 11am PST) will offer a Level 100 to 200 introduction for anyone who wants to learn more about Microsoft Virtualization and how it can help their career. It is free and public so sign up for this warm-up for the Jump Start event on October 6^th.

Module 1 – Technology: Learn about Microsoft’s virtualization technologies, how they work, and the future roadmap to the Cloud!
Module 2 – Career: Understand the importance of virtualization and Private Cloud, and how it can make or break an IT Professional’s career!
Module 3 – Certification: Get prepared for your next steps towards a virtualization career by understanding and preparing for the Microsoft 70-659 Technical Specialist exam, Windows Server 2008 R2, Server Virtualization.

Learn More: http://mctreadiness.com/MicrosoftCareerConferenceRegistration.aspx?pid=270
Register: http://mctreadiness.com/MicrosoftCareerConferenceRegistration.aspx?pid=288

Instructor Bios: http://mctreadiness.com/MicrosoftCareerConferenceRegistration.aspx?pid=287

Virtualization Exam 70-659 Training

On Thursday October 6^th Technical Evangelist Symon Perriman and Technical Instructor Philip Helsel will host an online 8-hour deep dive training event for the Microsoft 70-659 Technical Specialist exam, Windows Server 2008 R2, Server Virtualization. This virtual event runs from 8am to 5pm PST and will include presentations, demos and live Q&A with the attendees. It costs $99 to attend, but includes a free exam voucher worth $150! It is public so sign up for some great training to help improve your career here: http://mctreadiness.com/MicrosoftCareerConferenceRegistration.aspx?pid=272. Register early to make the most of a weekly virtualization coaching newsletter!

Clustering will be featured in Modules 1 and 4, where we will talk about cluster infrastructure planning, networking, storage, Cluster Shared Volumes, live migration, cluster integration with System Center Virtual Machine Manager and more!

Module 1 – Installing and Configuring Host and Parent Settings
Module 2 – Configuring Child Settings
Module 3 – Managing and Monitoring Virtual Environments
Module 4 – Ensuring High Availability and Recoverability
Module 5 – Performing Migration
Module 6 – Configuring Remote Desktop (RD) Role Services Infrastructure

Technologies that we will cover include: Windows Server 2008 R2, Hyper-V, System Center Virtual Machine Manager (VMM), System Center Operations Manager (OpsMgr), System Center Data Protection Manager (DPM), Windows Server Backup, Failover Clustering, Remote Desktop Services, Active Directory, Microsoft Assessment & Planning Toolkit (MAP), Virtual Machine Servicing Tool (VMST), Windows Server Update Services (WSUS) & more!

Learn More: http://mctreadiness.com/MicrosoftCareerConferenceRegistration.aspx?pid=274

Exam Information: http://www.microsoft.com/learning/en/us/Exam.aspx?ID=70-659

On Thursday December 1^st and Wednesday January 11^th, Technical Evangelist Symon Perriman and Technical Instructor Philip Helsel will host an online 8-hour deep dive training event for the Microsoft 70-659 Technical Specialist exam, Windows Server 2008 R2, Server Virtualization. This virtual event runs for 8 hours at different times and will include presentations, demos and live Q&A with the attendees. It costs $99 to attend, but includes a free exam voucher worth $150! It is public so sign up for some great training to help improve your career here: http://mctreadiness.com/MicrosoftCareerConferenceRegistration.aspx?pid=274. Register early to make the most of a weekly virtualization coaching newsletter!

· Thursday, December 1, 2011 from 10pm – 6am PST (perfect for students in Asia and Europe)
· Wednesday, January 11, 2011 from 12pm – 8pm PST
· More times coming soon…

Module 1 – Installing and Configuring Host and Parent Settings
Module 2 – Configuring Child Settings
Module 3 – Managing and Monitoring Virtual Environments
Module 4 – Ensuring High Availability and Recoverability
Module 5 – Performing Migration
Module 6 – Configuring Remote Desktop (RD) Role Services Infrastructure

Learn More & Register: http://mctreadiness.com/MicrosoftCareerConferenceRegistration.aspx?pid=274

Exam Information: http://www.microsoft.com/learning/en/us/Exam.aspx?ID=70-659

Thanks!
Symon Perriman
Technical Evangelist
Private Cloud Technologies
Microsoft

See videos on TechNet

Follow announcements @SymonPerriman

Next Tuesday, December 6^th from 8am-11am PST, we are offering a free training: Failover Clustering with Hyper-V: Designing a Highly-Available Infrastructure for the Private Cloud

Join Symon Perriman (Technical Evangelist for Private Cloud) and Elden Christensen (Principal Program Manager Lead for Clustering) to understand how Windows Server 2008 R2 Failover Clustering with Hyper-V provides critical infrastructure for any datacenter with VM high-availability and mobility. This event will provide an introduction to clustering, then focus on the design, deployment and management considerations for your virtualized datacenter or Private Cloud. Topics include hardware, validation, deployment, host clustering, guest clustering, virtualization, live migration, multi-site clustering, System Center Virtual Machine Manager 2008 & 2012, and offers a wealth of best practices. No prior clustering knowledge is needed.

Agenda:

· Introduction to Clustering – Learn the basics of Windows Server 2008 R2 Failover Clustering
· Hyper-V High-Availability – Dive into best practices, tips and tricks to provide VM high-availability and mobility
· Private Cloud High-Availability – Enhance the datacenter with integration from SCVMM (2008 R2 and 2012) and multi-site clustering

Thanks!
Symon Perriman

Technical Evangelist for IT Professionals

Private Cloud | Server | Virtualization | System Center

TechNet Edge Blog & Videos Twitter @SymonPerriman

Hi Cluster Fans,

We recently launched the free & public Microsoft Virtual Academy (www.MicrosoftVirtualAcademy.com). This training portal offers quick and easy access to presentations, webcasts, whitepapers and other technical content about Microsoft’s technologies.

This week we just released a brand new track on Understanding Microsoft’s High-Availability Solutions.

This course covers the best ways to provide continual availability to all applications, services, servers and VMs in your datacenter. The modules will cover all the planning, deployment and management considerations for bringing high-availability to every components of your datacenter or Private Cloud. By the end of this course you will have an understanding of the basics of each of Microsoft's HA solutions and when each of them should be used. You will understand all the different methods you can use to keep your services up and running. Technologies that will be covered in this course include: Hyper-V, Failover Clustering, Network Load Balancing, SQL Server, Exchange Server, IIS, DFS-R, DNS, and every System Center product (including SC 2012) with a focus on Virtual Machine Manager (VMM)

The course includes:

Understanding Microsoft’s High-Availability Solutions
- Infrastructure HA with Failover Clustering & Hyper-V
- Application HA with SQL, Exchange & Other Servers
- Management HA with System Center
Failover Clustering Overview
- Introduction to Failover Clustering
- Server High-Availability
High-Availability & Clustering
- Overview, Software & Hardware
- Deployment & Management
- Hyper-V HA
Disaster Recovery

Introduction to Disaster Recovery
Storage Considerations

Thanks!
Symon Perriman

Technical Evangelist for IT Professionals

On February 21 & 22 Microsoft Learning is running a 2-day virtual training event to help the world learn about the upcoming enhancements with the Creating & Managing a Private Cloud with System Center 2012 Jump Start.

It is 100% free and open to the public, so register now!

Event Overview

Adopting this exciting new computing paradigm provides a whole new landscape of technology and career direction for IT professionals. Microsoft Learning and the Microsoft System Center 2012 team have partnered to bring you an exciting opportunity to learn what you need to know to deploy, manage and maintain Microsoft’s private cloud solution. Leveraging the popular Jump Start virtual classroom approach, the industry’s most gifted cloud experts will show attendees why this new private cloud solution, based on System Center 2012 and Windows Server, has garnered so much attention. Presenters include Symon Perriman, Sean Christensen, Adam Hall, Kenon Owens, Prabu Rambadran & Chris Van Wesep and there will be a live Q&A during the event.

Event Agenda

Day 1: Deployment & Configuration (Feb. 21)

Part 1: Understanding the Microsoft Private Cloud
Part 2: Deploying the Infrastructure Components
Part 3: Deploying the Private Cloud Infrastructure
Part 4: Deploying the Service Layer
Part 5: Deploying the Applications & VMs

Day 2: Management & Operations (Feb. 22)

Part 6: Managing the Infrastructure Components
Part 7: Managing the Private Cloud Infrastructure
Part 8: Managing the Service Layer
Part 9: Managing the Applications & VMs

Jump Start Overview

This accelerated Jump Start sponsored by Microsoft Learning is tailored for IT professionals familiar with Windows Server technologies, Hyper-V virtualization, and the System Center management solutions. The course is designed to provide a fast-paced and technical understanding of how and why Microsoft’s approach to the private cloud delivers scalability, security, flexibility and control. Here are few unique benefits of this course:

Students have the opportunity to learn from and interact with the industry’s best cloud technologists!
This high-energy, demo-rich learning experience will help IT Professionals understand why Microsoft private cloud solutions are making a splash in the industry.
Students will see with their own eyes how Windows Server 2008 R2 and System Center 2012 work together to provide the best combination of security and scale.
Information-packed agenda! Day one of this two-day online course will focus on designing and deploying the right solutions for your organization, while day two will provide an in-depth look at the tools available to help monitor, secure and control the operational aspects of a private cloud.

Thanks!
Symon Perriman

Technical Evangelist, Microsoft

TechNet Edge Blog & Videos

Twitter @SymonPerriman

The beta of Windows Server “8” is now available for IT professionals and software developers around the world to download, to evaluate, and to give us feedback on.

Specifically for the Failover Clustering feature, we have delivered a large number of new features in Windows Server 2012 that we hope you will find exciting and compelling. Now is the time to discover the new capabilities and give us feedback.

To get you started, in this blog I will discuss some of the higher level themes for Failover Clustering features in Windows Server 2012. Following this blog, there will be a series of “How to…” blogs that will give some quick guidance in how to setup and configure some of these new features, to enable you to go try them out and give us feedback. We have some more comprehensive documentation coming later, but we wanted to help get you started today. Welcome to Windows Server 2012 Failover Clustering!

Scalability – With Windows Server 2012 you will be able to have the industry leading most scalable private cloud, with Failover Clustering having four times the scale over Windows Server 2008 R2 Failover Clustering. There is now support for 64-nodes in a single cluster, as well as 4,000 virtual machines running on a cluster.

Manageability – One of the major themes in Windows Server 2012 is multi-machine management. You will see a new Server Manager which will enable managing your private cloud and along with cluster integration. To manage a cluster of this scale you will see new management paradigm’s in the Failover Cluster Manager snap-in to search, sort, and filter views in the Failover Cluster Manager snap-in. This will deliver a highly scalable easy to manage platform.

VM Mobility – In Windows Server 2012 a virtual machine can seamless move anywhere in your datacenter. You will be able to migrate VMs from one cluster to another, and between clusters and stand-alone hosts. Virtualization and high availability go hand-in-hand, so you will see tight integration with all the new Hyper-V features on a cluster. This will give you incredible flexibility and allow you to rethink your cluster deployment models.

Monitoring Applications in your Private Cloud – With Windows Server 2012 you will be able to monitor application health and have application mobility in new and more flexible ways. VM Monitoring will enable you to monitor the health of applications running inside of VMs in a lightweight way, and bubble the health state down to the host layer to take recovery actions. With Guest Clustering there will greater flexibility in how you configure a solution that achieves not only application health monitoring, but also application mobility. You will have greater flexibility to create Guest Clusters with fibre channel, iSCSI, or File (SMB) based storage.

Dynamic Clusters – With Windows Server 2012 how clustering determines quorum and resiliency is dynamic to the state of the cluster. This will deliver a private cloud that is flexible, dynamic, and more resilient.

Cluster Shared Volumes – First introduced in Windows Server 2008 R2, CSV enables all nodes in a cluster to access a common volume. With Windows Server 2012 CSV has undergone many innovations that enable it to provide a highly scalable, increased performance, security, and flexible shared storage infrastructure for your private cloud. Additionally, CSV will be supported with more workloads beyond Hyper-V… such as with a new Scale-out File Server.

This is just a very small taste of some of the new Failover Clustering features available in Windows Server 2012. You can discover a feature breakdown at this link.

Once again, welcome to Windows Server 2012! Look for more information to follow on the Clustering and High Availability blog site that will provide preliminary guides in how to setup and get going with some of the new features.

We would love your feedback, I encourage you to post feedback to the High Availability (clustering) newsgroup. The product team will be actively monitoring it and will be helping to answer your questions and hear your feedback.

Thanks!
Elden Christensen
Principal Program Manager Lead
Clustering & High-Availability
Microsoft

Hi Cluster Fans,

In Windows Server 2012, Windows PowerShell® cmdlet help content has new functionality that enables it to be updatable over the life of the product. This allows us to get feedback from the community and update the help with enhanced explanations or additional examples even after Windows Server 2012 has released!

One point to note is that out of the box Windows PowerShell® ships with basic cmdlet help which has Syntax, Parameters, Inputs, Outputs and Aliases but no descriptive text, related links or examples. This minimizes the size of your ISOs and VHDs for those machines that you won’t be using the Windows PowerShell® help on. When you want the latest and greatest help, it can be installed on demand by following these steps:

1) Open an elevated Windows PowerShell® prompt

2) Notice what basic help provides

3) Update the help content for the Failover Clusters or Network Load Balancing module

Update-Help –Module FailoverClusters

Update-Help –Module NetworkLoadBalancingClusters

The help content for a module can also be saved to a file system directory:

Save-Help -DestinationPath <DestinationPathString> -Module FailoverClusters

Save-Help -DestinationPath <DestinationPathString> -Module NetworkLoadBalancingClusters

This saved content can then be used to update the help content on Windows Server 2012 machines that are not internet capable or do not have WinHTTP configured:

Update-Help –SourcePath <SourcePathString> –Module FailoverClusters

Update-Help –SourcePath <SourcePathString> –Module NetworkLoadBalancingClusters

Note: One of the great new Windows PowerShell® features in Windows Server 2012 is that you do not need to manually import Windows PowerShell® modules each time you open a Windows PowerShell® prompt. They are automatically loaded for you when you run a cmdlet from the module for the first time.

4) You now have access to the most recent cmdlet help

You can also find the entire cmdlet help for Failover Clustering and Network Load Balancing online.

Failover Clustering: http://go.microsoft.com/fwlink/?LinkID=233200

Network Load Balancing: http://go.microsoft.com/fwlink/?LinkID=238123

Thanks!

Subhasish Bhattacharya
Program Manager
Clustering & High Availability
Microsoft

Overview

Cluster Shared Volumes (CSV) has many new features in Windows Server 2012 that will deliver the most resilient and highest performance storage infrastructure for your private cloud. One of those new features is the CSV Cache.

CSV Cache is a feature that allows you to allocate system memory (RAM) as a write-through cache. The CSV Cache provides caching of read-only unbuffered I/O. This can improve performance for applications such as Hyper-V, which conducts unbuffered I/O when accessing a VHD file. Unbuffered I/O’s are operations which are not cached by the Windows Cache Manager. What CSV Block Cache delivers is caching which can boost the performance of read requests, with write-through for no caching of write requests.

CSV Cache delivers caching at the block level, which enables it to perform caching of pieces of data being accessed within the VHD file. This is fundamentally very similar in the value it provides to other solutions on the market you may already be familiar with. The primary difference is that CSV Block Cache reserves its cache from system memory, where most solutions on the market today deliver cache in the form of SSD’s on a PCI card that you add to the server.

CSV Cache is completely integrated into the Failover Clustering feature and handles orchestration across the sets of nodes in the cluster.

Deployment Considerations

CSV Cache will deliver the most value in scenarios where VMs are used primarily for read requests, and are less write intensive. Scenarios such as Pooled VDI VMs or also for reducing VM boot storms. Because the applicability of CSV Cache depends on the workload and your specific deployment considerations, it is disabled by default.

You can allocate up to 20% of the total physical RAM for CSV write-through cache, which will be consumed from non-paged pool memory. Our preliminary testing has found 512 MB to deliver excellent gain at minimal cost, and is the recommend default value if enabled. Then based on your specific deployment and the I/O characteristics of the workloads in the VMs you may wish to increase the amount of memory allocated.

There are two configuration settings that allow you to control CSV Cache.

CsvEnableBlockCache – This is a private property of the cluster Physical Disk resource. It allows you to enable CSV Cache on an individual disk. This gives you the flexibility to configure cache for read intensive VMs running on some disks, while allowing you to disable and prevent random I/O on other disks from purging the cache. For example parent VHD’s with high reads you would enable caching on Disk1, and high writes for differencing disks the CSV cache could be disabled on Disk2. The default setting is 0 for disabled, setting to a value of 1 enables CSV Block Cache on that disk.
SharedVolumeBlockCacheSizeInMB – This is a cluster common property that allows you to define how much memory (in megabytes) you wish to reserve for the CSV Cache on each node in the cluster. If a value of 512 is defined, then 512 MB of system memory will be reserved on each node in the Failover Cluster. Configuring a value of 0 disables CSV Block Cache.

How to Configure CSV Cache

1. Open an elevated Windows PowerShell prompt

2. Define the size of the size of the cache to be reserved

(Get-Cluster). SharedVolumeBlockCacheSizeInMB = 512

3. Enable CSV Cache on an individual disk

Get-ClusterSharedVolume “Cluster Disk 1” | Set-ClusterParameter CsvEnableBlockCache 1

Considerations: The cache size can be modified with no downtime, however for the Hyper-V root memory reserve in the parent partition to be modified to accommodate the memory allocated to the CSV cache it does require a server reboot. To ensure resource contention is avoided, it is recommended to reboot each node in the cluster after modifying the memory allocated to the CSV cache.
Enabling CSV Cache on an individual disk requires that the Physical Disk resource be recycled (taken Offline / Online) for it to take effect.

Optimizing CSV Cache

The CSV Cache also provides a set of counters you can use to monitor the performance of the cache. You can leverage the Performance Monitor tool (PerfMon.msc) to add the following counts to monitor different aspects of the CSV Cache.

Open Performance Monitor, and under Add Counters you will find “Cluster CSV Volume Cache” with the following counters.

I/O satisfied from cache:

Cache IO Read-Bytes
Cache IO Read-Bytes/Sec
Cache Read
Cache Read/Sec

I/O satisfied from disk:

Disk IO Read-Bytes
Disk IO Read-Bytes/Sec
Disk Read
Disk Read/Sec

Total I/O:

IO Read-Bytes
IO Read-Bytes/Sec
IO Read
IO Read/Sec

Thanks!
Elden Christensen
Principal Program Manager Lead
Clustering & High-Availability
Microsoft

In Windows Server 2012 there have been several enhancements to how Windows Server Failover Clusters integrate with the Active Directory. In this blog I am going to discuss some of the changes to help enable creating Failover Clusters in restrictive Active Directory environments where permissions to create computer objects is delegated to specific organizational units (OU).

In Windows Server 2008 R2, Failover Clustering created computer objects in the Active Directory under the default Computers container for cluster Network name resources. In Windows Server “8” this has changed to enable greater flexibility when setting up a Failover Cluster.

Cluster Name Object (CNO)

The CNO is the computer object associated with the cluster network name resource called “Cluster Name” that is created during initial setup of the cluster. Before running Create Cluster one of the requirements is that all nodes be members of a domain. Since all nodes are domain joined and have corresponding computer objects, the OU in which the nodes computer objects reside in is used as the location to create the CNO. If you had permissions to setup the node computer objects, then this will enable creating a cluster to ‘just work’ with no additional considerations needed. The default setup experience now has better heuristics.

For increased flexibility, if you wish to create the CNO in a different OU location, now with Windows Server 2012 you can do so by specifying the full distinguished name during either the Create Cluster wizard in Failover Cluster Manager or through the New-Cluster PowerShell cmdlet. The distinguished name includes the path to the OU under which you would like the computer object created.

Specifying a Custom OU with Failover Cluster Manager:

To create a cluster with the Failover Cluster manager Create Cluster wizard and for example have the CNO placed in the OU named "Cluster":

Specifying a Custom OU with PowerShell:

To create a cluster via PowerShell and for example have the CNO placed in the OU named “Cluster” it would be in the following syntax:

New-Cluster -Name CN=MyCluster,OU=Cluster,DC=Contoso,DC=com -Node node1,node2

Virtual Computer Object (VCO)

The VCO is the computer object associated with all other cluster network name resources that are created for highly available roles on the cluster. This would include roles such as for a highly available File Server or SQL Server for example.

The VCO’s will all be created in the same OU in which the CNO currently resides at creation time.

Additional Information:

The user credentials of the currently logged on user who is creating the Failover Cluster will be used to create the computer objects in Active Directory. The user must have Create Computer Objects permissions to the OU to create the computer objects. Additionally, the CNO must have Create Computer Objects privileges in the OU it currently resides in to be able to create VCO’s.

If you do not have Create Computer Objects permissions, your domain admin can manually pre-stage the CNO and VCO computer objects. See this step-by-step guide for information on how to configure cluster accounts in the Active Directory:
http://go.microsoft.com/fwlink/?LinkId=139147

If you wish to move the CNO or VCO’s to a different location than the one they are originally created in, it is safe to do so without impacting the functionality of the Failover Cluster.

Thanks!
Elden Christensen
Principal Program Manager Lead
Clustering & High-Availability
Microsoft

Windows Server 2012 Failover Clusters are easier to manage and maintain with the new “Node Drain” and “Resume with Failback” features. This enables nodes to be gracefully drained for planned maintenance. This functionality is part of the infrastructure that enables “Cluster Aware Updating” (CAU) for patching nodes in a cluster.

Overview

Bringing an individual node down for planned maintenance is a common administrative task, to for example install a Service Pack or hardware upgrades.

On a Windows Server 2008 R2 Failover Cluster, this is a manual process where you place a cluster node in PAUSED state, and then move individual Roles (workloads) to the other nodes in the cluster as outlined in this KB article.

In Windows Server 2012 conducting planned maintenance on Failover Clusters is dramatically simplified, as these steps are automated in the Node Drain (or Node Maintenance Mode) feature.

Node Drain

Using Node Drain you can automate moving the Roles (workloads) off of a cluster node. Think of Node Drain is to as an enhanced, workload aware Node Pause.

Steps automated by Node Drain:

1) The cluster node is put in a PAUSED state, which prevents other workloads hosted on other nodes from moving to the node.

2) The Roles (workloads) currently owned by the cluster node, are sorted according to their Priority order. (Priority of Roles is another new Failover Clustering functionality in Windows Server 2012.)

3) The Roles are then distributed to the other active nodes in the cluster in priority order. Node Drain works with all workloads running on the cluster. For virtual machines, it leverages live migrations and memory-aware intelligent placement.

4) When all the Roles are moved off of the cluster node, Node Drain operation is completed.

Initiating Node Drain through Failover Cluster Manager:

Initiating Node Drain through Failover Cluster Manager snap-in is a simple one-click operation:

Open Failover Cluster Manager (CluAdmin.msc)
On the left hand pane navigate to Nodes
Right-click on the node you wish to drain
Under Pause select Drain Roles

Note: If you select “Do Not Drain Roles”, then it would simply “PAUSE” the node similar to Windows Server 2008 R2.

Initiating Node Drain through PowerShell:

You can initiate Node Drain using the “Suspend-ClusterNode” PowerShell command.

There are additional advanced options available through PowerShell to manage draining nodes, which includes:

Parameter	Purpose
Drain	Initiates Node Drain
TargetNode	The destination node where all drained roles will be moved/live migrated to
ForceDrain	Moves the roles off of the draining node even if the Group cannot move either because no other node can host this group or it is in locked state
Wait	Defines an amount of time to wait for the Node Drain operation to begin

Status of Drained Node:

When a Node Drain is initiated, the command returns the NodeDrainStatus property, indicating that the cluster node has begun the node drain operation. You can track the status of the on-going node drain operation using these two cluster node common properties:

Node Common Property	Values	Purpose
NodeDrainStatus	0 – Not Initiated	This property indicates the current status of the Node Drain.
	1 – In Progress
	2 – Completed
	3 – Failed
NodeDrainTarget	Cluster Node Id	ID of the cluster node which all the workload will be moved to. This ID is set when you use the TargetNode parameter.

Node Drain Failure:

Node Drain will fail if a virtual machine’s Live Migration fails due to some reason, or if a Role cannot be moved as the node being drained is the last possible owner node for the Role.

Upon encountering an error with an individual role, the node drain operation will continue to drain the remaining roles hosted on the node. The status of node drain would be set to “3” only after the remaining roles are drained from the cluster node.

Restarting Node Drain and optionally you can specify “-ForceDrain” parameter to override any errors encountered during the initial node drain.

Rebooting a Drained Node:

Once a node is drained, it will remain in the PAUSED state across reboots to prevent any roles from moving to that node, until the node is resumed. This keeps the node drained for the duration of the maintenance window.

Node Resume with Failback

When a node is drained, the cluster will remember the workload(s) that were moved off of the node. When resuming the node after maintenance, you have the option of moving back all the workload(s) to the cluster node. This will restore the cluster back to the original state it was in before the maintenance.

Steps automated Node Resume with Failback:

1) The cluster node is removed from PAUSED state - this enables workload(s) to move to this node.

2) The workload(s) that were originally drained from the node are moved back using Failback.

If a failback policy is configured to only failback during a specific failback window, resume will honor the setting and the roles failback will be delayed until the failback window.

Resuming Node through Failover Cluster Manager:

Open Failover Cluster Manager (CluAdmin.msc)
On the left hand pane navigate to Nodes
Right-click on the node you wish to resume
Under Resume select Fail Roles Back

Note: If you select “Do Not Fail Roles Back”, then it would simply “RESUME” the node similar to Windows Server 2008 R2.

Resuming Node through PowerShell:

You can resume a node using the Resume-ClusterNode PowerShell command.

There are additional advanced options available through PowerShell to manage resuming nodes, which includes:

Name

Value

Purpose

Failback

NoFailback – Don’t Failback workload

Immediate – Failback immediately

Policy – Failback during configured Window

This defines the type of failback to expect after node is resumed.

Additional Information:

Cancelling Node Drain:

Draining a node may be a long running operation. A Node Drain that is in progress can be cancelled by initiating a Node Resume. This will cause the Node Drain operation to stop, and if Fail Roles Back is specified, the drained workloads which were moved will be moved back to the cluster node.

Configuring the Move Type for a Virtual Machine

Node Drain and Node Resume with Failback will leverage Live Migration for virtual machines so that a node can be drained with no downtime. Live Migration may at times be a long running operation, and there may be scenarios where you wish to quickly drain a node. Node draining provides the flexibility to allow configuration of how VMs should be moved, using either Live Migration or Quick Migration.

You also have the granular control to configure the move type to be used based on the priority setting of the VM. This is configured with the Virtual Machine Resource Type property private property NodeDrainMoveTypeThreshold:

Name

Value

Purpose

NodeDrainMoveTypeThreshold

(Private Property)

Priority of Virtual Machines

Virtual Machines with Priority equal to or higher than the specified priority will be moved using Live Migration.

Virtual Machines with Priority lower than the specified priority will be moved using Quick Migration.

Example PowerShell commands to view or modify this private property:

Creating property:
Get-ClusterResourceType "Virtual Machine" | Set-ClusterParameter -Create @{"NodeDrainMoveTypeThreshold"="3000"}

Modifying created property:
Get-ClusterResourceType "Virtual Machine" | Set-ClusterParameter -Multiple @{"NodeDrainMoveTypeThreshold"="3000"}

Reading property:
Get-ClusterResourceType "Virtual Machine" | Get-ClusterParameter NodeDrainMoveTypeThreshold

Conclusion:

Node Drain is a great new time-saving feature in Windows Server 2012 Failover Clustering for conducting planned maintenance. Using this feature, you can easily drain the workload(s) off of a cluster node in a single click, and easily restore them when maintenance operations are completed on the cluster node.

Thanks!

Amitabh Tamhane                                                                                                           Lokesh Koppolu
Program Manager II                                                                                                        Principal Development Lead
Clustering & High Availability                                                                                       Clustering & High Availability
Microsoft                                                                                                                          Microsoft

In Windows Server 2012 Cluster Shared Volumes (CSV) has been more tightly integrated into the Failover Clustering feature. The process for a cluster Physical Disk Resource (PDR) to be enabled for CSV has been simplified and streamlined. In this blog, I will show you the new experience of adding storage from the Available Storage pool of your cluster to Clustered Shared Volumes. The Available Storage pool contains disks that have been added to your cluster but not assigned to a specific use in your cluster.

Failover Cluster Manager

To add storage to Clustered Shared Volumes follow these steps:

1) Launch the Failover Cluster Manager (CluAdmin.msc)

2) Select the Storage node

3) Select the Disks that you want to add to Clustered Shared Volumes.

Note: A great new Failover Cluster Manager feature in Windows Server 2012 is support for multi-select and the ability to enable CSV across a number of disks all at once!

4) Right click on your selection and choose the Add to Cluster Shared Volumes option.

5) Your disks are now added to Clustered Shared Volumes! Yes, it is that easy in Windows Server 2012!

CSV provides a single consistent file name space. Files have the same name and path when viewed from any node in the cluster. CSV volumes are exposed as directories and subdirectories under the “ClusterStorage” root directory: C:\ClusterStorage\VolumeX\<root>

CSV enabled volumes now appear as “CSVFS”. CSVFS is the NTFS file system under the covers and volumes are still formatted with the NTFS file system. However, this change enables applications to be aware that they are running on CSV and allows them to ensure compatibility.

PowerShell

You can also use the Failover Clustering Windows PowerShell® cmdlet, Add-ClusterSharedVolume, to add storage to Clustered Shared Volumes. This cmdlet accepts disks from the Available Storage pool.

Note: Another awesome Windows Server 2012 Failover Clustering PowerShell® feature is the support for Wildcard characters! The screenshot above shows you an example of how you can use wildcards to select multiple PDRs to add to CSV at once.

Thanks!

Subhasish Bhattacharya
Program Manager
Clustering & High Availability
Microsoft

The installation procedure for Failover Clustering in Windows Server 2012 is very similar to that found in Windows Server 2008 and Windows Server 2008 R2. In this blog we will discuss the changes in Windows Server 2012 as well as the different options for installing the Failover Clustering feature.

Windows Server 2012 continues with the Roles and Features model. All clustering technologies are considered Features, as they are infrastructure that enables Roles to be made highly available.

The Failover Cluster feature consists of the following components as shown by the Windows Server PowerShell CmdLet Get-WindowsFeature:

Display Name	Name	Description
Failover Clustering	Failover-Clustering	This installs the core Failover Clustering feature and all fundamental requirements. It does not include any management tools.
Failover Cluster Management Tools	RSAT-Clustering-Mgmt	Includes the Failover Cluster Manager snap-in and the Cluster-Aware Updating interface.
Failover Cluster Module for Windows PowerShell	RSAT-Clustering-PowerShell	Includes Windows Powershell cmdlets for managing failover clusters. It also includes the Cluster-Aware Updating module for Windows PowerShell, for installing software updates on failover clusters.
Failover Cluster Automation Server	RSAT-Clustering-AutomationServer	Includes the deprecated Component Object Model (COM) programmatic interface, MSClus.
Failover Cluster Command Interface	RSAT-Clustering-CmdInterface	Includes the deprecated cluster.exe command-line tool for Failover Clustering. This tool has been replaced by the Failover Clustering module for Windows PowerShell.

Installing the Failover Clustering feature using Server Manager

The Failover Clustering feature can be installed with either Server Manager or Windows PowerShell cmdlets. In Server Manager, the Add Roles and Features Wizard is used to add roles and\or features. The Add Roles and Features Wizard are accessed in the Server Manager Menu bar by choosing Add Roles and Features from the list.

This starts the Add Roles and Features Wizard. The Installation Type is Role-based or feature-based installation.

Be sure the correct server is selected in the Server Selection screen

In the Features screen, select Failover Clustering

A pop-up screen appears listing additional requirements for the feature

If you wish to install the Failover Cluster Manager snap-in and Failover Cluster PowerShell cmdlets, management tools, then click Add Features

Confirm the selections and click Install

The installation of the Failover Clustering feature does not require a reboot, checking the Restart the destination server automatically if required check box is not necessary.

There are optional features available for the Failover Clustering feature administration tools.

These are deprecated features (Failover Cluster Command Interface (cluster.exe) and Failover Cluster Automation Server) in Windows Server 2012 but are made available, as there are still some applications that may need them, SQL Server being one of them. Installing it may be necessary for any legacy scripts you have built on the old Cluster.exe command line interface.

Once the Failover cluster feature is installed, the Failover Cluster Manager interface is available in the Tools category in the Menu bar.

Failover Cluster Manager is also available by right clicking on a node in the cluster in the All Servers view.

Choosing Failover Cluster Manager opens the snap-in.

You are now ready to fun Validate and create your cluster!

See this doc for the steps of setting up the cluster now that you have the feature installed and are ready to go:
http://technet.microsoft.com/en-us/library/cc731844(v=ws.10).aspx

Installing the Failover Cluster feature using Windows PowerShell

It is important to note that you must run these cmdlets in a PowerShell console that is opened with elevated privileges, which means opening it with the “Run as Administrator” option.

The following cmdlet will install the Failover Clustering feature and the management tools.
Note: If you do not specify the –IncludeManagementTools switch, the Failover Cluster Administrator and PowerShell cmdlets for cluster will not be installed.

PowerShell:

Install-WindowsFeature -Name Failover-Clustering –IncludeManagementTools

You can use the –ComputerName parameter to install the features on other servers without having to log into them. Here is an example of the cmdlet to install the failover cluster feature and tools on a specified server, in this case “ServerX”:

PowerShell:

Install-WindowsFeature -Name Failover-Clustering –IncludeManagementTools –ComputerName ServerX

If you would like to find the list of features and the names to specify in the Install-WindowsFeature cmdlet, you can use this cmdlet:

PowerShell:

Get-WindowsFeature

Wildcards can be helpful to narrow down the returned set of features:

PowerShell:

Get-WindowsFeature Failover*

The Get-WindowsFeature Failover* cmdlet will return the feature, but not the tools. To get the tools you can use the following:

PowerShell:

Get-WindowsFeature RSAT-Cluster*

Starting with Windows Server 2008, SMB file shares on a Failover Cluster are scoped so that they are only accessible by valid UNC paths associated with the network name they are bound to. See this blog for more information:
http://blogs.technet.com/b/askcore/archive/2009/01/09/file-share-scoping-in-windows-server-2008-failover-clusters.aspx

File Share Scoping delivered improved functionality that solved a number of issues, but introduced some side effects. Namely that you could no longer connect to SMB shares on a Failover Cluster by the IP address or in some scenarios you may wish to connect by a name other than the name associated with the cluster Network Name resource. For example, using DNS CNAME records to alias server names.

Now with Windows Server 2012 there is greater flexibility in connecting to clustered SMB file shares, where you can connect with the associated IP address and define aliases for the clustered shares to be available with. This can be helpful when dealing with applications that are hardcoded to specific names or directly to IP addresses.

Connecting using an IP Address:

Cluster Network Name resources have a dependency on one or more IP Address resources. Aliases will be automatically created in the cluster so that connecting using one of the IP Addresses will associate it with the network name that depends on it. You will be able to connect specifying the IP address with no additional configuration steps in the following format:
\\10.10.10.10\Share

Note: File based storage with the new Windows Server 2012 Scale-out File Server for Application Data feature does not support connecting via IP address, as Scale-Out File Server does not use cluster IP Address resources with those types of configurations. For more information on Scale-Out File Server, see this document.

Connecting using an Alias:

It is now possible to configure scoped SMB shares on a Clustered File Server to listen for aliases with Windows Server 2012. This is a two-step process involving both DNS and Failover Cluster configuration. The following steps outline how to configure an alias with an example name of “AliasName” for the Network Name resource called “MyClusterName” that is part of a highly available File Server role.

Configure DNS:

1. On the DNS server configure a CNAME record for AliasName. See this KB article for details: http://support.microsoft.com/kb/168322

Alternatively, provide some other name resolution mechanism that allows the client to resolve AliasName

Configure Cluster:

2. On the Cluster open an elevated Windows PowerShell® prompt

3. View the currently configured aliases on the MyClusterName Network Name resource:
Get-ClusterResource "MyClusterName" | Get-ClusterParameter Aliases

4. Add a new alias to the MyClusterName Network Name resource with the name of “AliasName”, type the following:
Get-ClusterResource "MyClusterName" | Set-ClusterParameter Aliases AliasName

Note: For the setting to take effect it requires recycling (taking Offline then Online) the cluster Network Name resource. The new alias will now appear for the Network Name resource.

5. Connect to a share by specifying the alias
\\AliasName\Share

Considerations:

Connecting to an SMB share by IP address, or an alias, on a clustered or stand-alone server does not support Kerberos authentication. Connections will be negotiated with NTLM. While connecting with aliases does bring flexibility, the security trade-offs should be taken into consideration.

Multiple aliases can also be configured for an individual Network Name resource. This can be configured by typing the following:
Get-ClusterResource "MyClusterName" | Set-ClusterParameter Aliases Alias1, Alias2

Thanks!
Elden Christensen
Principal Program Manager Lead
Clustering & High-Availability
Microsoft

The Microsoft Virtual Academy, our public online training portal, has just released several new courses on SQL HA:

Microsoft SQL Server 2012 Security enhancements
Microsoft SQL Server 2012 High Availability and Disaster Recovery Design Patterns, Architectures and Best Practices using AlwaysOn
T-SQL Cookbook - Microsoft SQL Server 2012 Enhancements
Mission Critical application testing with Distributed Replay
AlwaysOn Failover Cluster Instances and other High Availability enhancements
Improve Application Availability Using AlwaysOn - A Deep Dive Into Availability Groups
AlwaysOn - Active Secondary Part 1 - Readable Secondary
AlwaysOn - Active Secondary Part 2 - Enabling Backup On Secondary Replicas
Using The Power Of Extended Events For Tracing And Troubleshooting

This training is free for everyone - please pass this URL to your friends and colleauges !!!

http://www.microsoftvirtualacademy.com/tracks/mission-critical-confidence-using-microsoft-sql-server-2012

Thanks!

Rob Hindman

Program Manager

Clustering & High-Availability

Microsoft

Overview

Do you have a large number of virtualized workloads in your cluster? Have you been looking for a solution that allows you to detect if any of the virtualized workloads in your cluster are behaving abnormally? Would you like the cluster service to take recovery actions when these workloads are in an unhealthy state? In Windows Server 2012, there is a great new feature, in Failover Clustering called “VM Monitoring”, which does exactly that – it allows you monitor the health state of applications that are running within a virtual machine and then reports that to the host level so that it can take recovery actions. You can monitor any Windows service (such as SQL or IIS) in your virtual machine or ANY ETW event occurring in your virtual machine. When the condition you are monitoring gets triggered, the Cluster Service logs an event in the error channel on the host and takes recovery actions.

In this blog, I will provide a step by step guide of how you can configure VM Monitoring using the Failover Cluster Manager in Windows Server 2012.

Note: There are multiple ways to configure VM Monitoring. In this blog, I will cover the most common method. In a future blog, I will cover the many different flexible options for configuring VM Monitoring.

Configuring VM Monitoring

Pre-requisites

Before you can configure monitoring from the Failover Cluster Manager on a Management Console the following pre-steps are required:

1) Configure the guest operating system running inside the virtual machine

a) The guest operating system running inside the virtual machine must be running Windows Server 2012

b) Ensure that the guest OS is a member of a domain which is same as the host or a domain with a trust relationship with the host domain.

2) Grant the cluster administrator permissions to manage the guest

a) The administrator running Failover Cluster Manager must be a member of the local administrators group in the guest

3) Enable the “Virtual Machine Monitoring” firewall rule on the guest

a) Open the Windows Firewall console

b) Select “Allow an app or feature through Windows Firewall”

c) Click on “change settings” and enable the “Virtual Machine Monitoring” rule.

Note:

You can also enable the “Virtual Machine Monitoring” firewall rule using the Windows PowerShell® cmdlet Set-NetFirewallRule:

Set-NetFirewallRule -DisplayGroup "Virtual Machine Monitoring" -Enabled True

Configuration

VM Monitoring can be easily configured using the Failover Cluster Manager through the following steps:

1) Right click on the Virtual Machine role on which you want to configure monitoring

2) Select “More Actions” and then the “Configure Monitoring” options

3) You will then see a list of services that can be configured for monitoring using the Failover Cluster Manager.

Note:

You will only see services listed that run on their own process e.g. SQL, Exchange. The IIS and Print Spooler services are exempt from this rule. You can however setup monitoring for any NT service using Windows PowerShell® using the Add-ClusterVMMonitoredItem cmdlet – with no restrictions:

Add-ClusterVMMonitoredItem –VirtualMachine TestVM -Service spooler

How does VM Monitoring work?

When a monitored service encounters an unexpected failure, the sequence of recovery actions is determined by the Recovery actions on failure for the service. These recovery actions can be viewed and configured using Service Control Manager inside the guest. In the example below, on the first and second service failures, the service control manager will restart the service. On the third failure, the service control manager will take no action and defer recovery actions to the cluster service running in the host.

The cluster service monitors the status of clustered virtual machines through periodic health checks. When the cluster services determines that a virtual machine is in a “critical” state i.e. an application or service inside the virtual machine is in an unhealthy state, the cluster service takes the following recovery actions:

1) Event ID 1250 is logged on the host

a. This event can be monitored with tools such as System Center Operations Manager to trigger further customized actions

2) The virtual machine status in Failover Cluster Manager will indicate that the virtual machine is in an “Application Critical” state.

Note:

Verbose information is logged to the Cluster debug log for post-mortem analysis of failures.
The StatusInformation resource common property for a virtual machine in “Application Critical” state has the value 2 as compared to a value of 0 during normal operation. The Windows PowerShell® cmdlet Get-ClusterResource can be used to query this property.

Get-ClusterResource “TestVM” | fl StatusInformation

3) Recovery action is taken on the virtual machine in “Application Critical” state

a. The virtual machine is first restarted on the same node

Note: The restart of the virtual machine is forced but graceful

b. On the second failure, the virtual machine restarted and failed over to another node in the cluster.

Note: The decision on whether to failover or restart on the same node is configurable and determined by the failover properties for the virtual machine.

That’s the VM Monitoring feature in Windows Server 2012 in a nutshell!

Subhasish Bhattacharya
Program Manager
Clustering & High Availability
Microsoft

Storage migration is a new feature in Windows 8 Server that allows moving the files for a virtual machine (VM) to a new location while the VM is running.

For example, suppose your VM’s files are currently on a SAN volume, but the SAN storage device is being retired. A new SAN is deployed and ready to take over. In previous releases you would have to shut down the VM, and then copy the files to the new SAN, then either ensuring the drive mapping was exactly the same or fix-up the configuration for the VM for the new paths.

In Windows Server 2012, you can leave the VM running and let the system move the files and perform configuration changes without taking the VM out offline..! Think how useful this will be for scenarios such as a volume that is either running out of space, or is over utilized. Provided that you have another volume that has spare capacity, you can move your VMs files on-the-fly to load balance or optimize capacity utilization.

Failover Cluster Manager

For VMs that are managed by a cluster, you can move the VMs files by using the Move action in the Failover Cluster Manager and then select Virtual Machine Storage:

When you select Virtual Machine Storage, the Move Virtual Machine Storage dialog will open with the VM that you selected showing in the top pane:

The bottom-left pane shows the cluster managed storage available for placing VM files. The bottom-right pane is the contents of the volume/folder that is selected in the bottom left pain.

In the picture above, the bottom-left pane shows both Cluster Shared Volumes, I.E., Cluster Storage, and Cluster Disk 1. Cluster Disks will also be displayed if there is a cluster disk in the same cluster group as the VM. Since the disk is in the same group as the VM, the VM can use that disk to store its files. If you want to move a VM’s files to a disk, you need to first move the disk into the group with the VM, and then open the Move Virtual Machine Storage tool for the VM.

This tool provides a drag-and-drop experience: left-click and hold, and then drag the entire VM or just specific files of the VM to the bottom-right pane, and release. That will cause the Destination Folder Path column in the top pane to note the path where you want the VM file(s) to be moved to.

The picture below shows the VM contents expanded, which shows each file associated with the VM and the source column indicates where it currently exists. The bottom-left pane shows the volumes expanded to show folders where the VM’s files can be moved. Select the folder in the bottom-left pane, and drag the VM file(s), or the entire VM object, to the bottom-right pane to set the destination path for the files.

Once you have set the destination paths, click on the Start button - the dialog will close, and storage migration will start. You may notice in the Roles view of failover Cluster Manager the Information column will indicate that storage migration is running.

Storage migration can be a long running operation - the time it takes depends file size - the biggest files are usually the VHDs. Storage migration can take advantage of direct file copy, AKA Offload Data Transfer (ODX) - if the SAN supports it, or it will copy data using the network. If you close the Move Virtual Machine Storage tool, you’ll see the status of the migration in the Roles view of Failover Cluster Manager, enabling you to manage the cluster and do other tasks while the migration completes.

Once you press the Start button, the Hyper-V and Failover Cluster services will perform the storage migration - you can close the Failover Cluster Manager UI without interrupting the process.

Multiple VM Storage Migration

The Move Virtual Machine Storage tool can work with more than one VM at a time. Simply select multiple VMs in Failover Cluster Manager and then select the same Move action and Virtual Machine Storage option. The following picture shows two VMs in the tool:

Moving VMs to File Shares

The Add Share button allows adding file share locations that can be used as destination folder paths for the VM. When the Add Share button is pressed, the path to the file share can be specified and then it will be added to the navigation tree in the bottom left pain – as shown below. Note that this control doesn’t verify that the share and folder selected have permissions or connectivity for all the nodes of the cluster to access. It’s important to check that any file share and folder that the VM is configured to use is accessible and that permissions are configured correctly so that the cluster managed VM can migration or failover successfully – please see Jose Barreto’s excellent blog article enumerating Test cases for Hyper-V over SMB.

Checking the Results of Storage Migration

After completing the storage migration, you can re-open the Move Virtual Machine Storage tool and see the new path in the Source Folder Path column, which is an easy way to verify that the migration completed successfully and to check the currently file locations for your VMs:

Moving Storage of Cluster Managed VMs with Hyper-V Manager

Hyper-V Manager has a wizard to move a VMs files, but it can only be used for VMs that are not managed by a cluster. If a VM is managed by a cluster and the Move action is selected from Hyper-V Manager, the following dialog will presented to let you know that you should move the cluster managed VM using live migration for the VM, or use the Failover Cluster Manager’s storage migration feature:

In summary, Windows Server 2012 empowers you to not only to live migrate VMs between servers, but also to move the storage of running VMs to new paths or devices using the Failover Cluster Manager’s storage migration feature – more options and better agility !!

Creating a cluster on Windows Server 2012 is easy experience!

After installing the Failover Clustering feature, and validating a configuration, the next step is to create a new cluster. There are some minor changes in the create cluster experience in Windows Server 2012, but it is very similar to creating a cluster in Windows Server 2008 R2.

For details on installing the Failover Clustering feature, see this blog posting: http://blogs.msdn.com/b/clustering/archive/2012/04/06/10291601.aspx

To learn more about validation, see this TechNet article: http://technet.microsoft.com/en-us/library/cc731844(v=ws.10).aspx

You can create a cluster using the Failover Cluster Manager UI, or using PowerShell. Both of these can be installed on Windows Server 2012 or Remote Server Administration Tools (RSAT) for Windows Server 2012.

Creating a Failover Cluster using Failover Cluster Manager

1. Open Failover Cluster Manager - it can be opened from Server Manager using the Tools menu:

2. In the Failover Cluster Manager, choose the “Create Cluster…” action, which can be found in 3 places:

3. The Create Cluster Wizard initializes. Review the information on the Before You Begin screen. Click Next

4. Enter the names of all the servers that will be part of the cluster. Note: More that none node can be specified at a time using comma separation.

Example: MyServer1, MyServer2, MyServer3

5. If the nodes specified have not been validated, the following page in the wizard will be shown. It’s highly recommended to validate the configuration before you create the cluster. This will help ensure that the servers are connected and configured correctly and that it can be supported by Microsoft:

6. In the “Cluster Name” field, provide a NetBIOS name to be used as the cluster name. This cluster name is also the name that can be used to connect to the cluster to manage it. During cluster creation, a computer object will also be created in the Active Directory domain and Organizational Unit where the cluster nodes computer objects are located. If the servers have no NICs configured for DHCP, then this page will also prompt for a static IP address. If any of the networks are configured for DHCP, then this will not be shown and an IPv4 DHCP assigned address will be used. Click Next:

Note: If you do not want the Active Directory object for the cluster to be placed in the same Organizational Unit (OU) as the servers, the specific OU can be designated by specifying the full distinguished name like screen shot below:

For additional details on using a full distinguished name, please see “How to Create a Cluster in a Restrictive Active Directory Environment” (http://blogs.msdn.com/b/clustering/archive/2012/03/30/10289577.aspx)

Review the Confirmation screen. If all eligible storage will be added to the cluster, check the box Add all eligible storage to the cluster. Click Next

Note: This ability to choose whether all eligible storage will be added to the cluster or not is new for Windows Server 2012. In previous versions all storage would always be added to the cluster. If you choose not to add all eligible storage to the cluster, you can add specific disks after the cluster is created:

7. The cluster should be successfully created. Review the Summary report if desired. Click Finish

8. A Failover Cluster Manager will automatically connect to the cluster when the wizard finishes:

Creating a Failover Cluster using PowerShell

An alternate way to create a Failover Cluster is to use PowerShell. This can be accomplished with the New-Cluster PowerShell cmdlet. The following command creates 2-Node cluster (Contoso-FC1) and it assumes that a DHCP assigned address can be assigned and all eligible storage is added.

PowerShell:

New-Cluster -Name Contoso-FC1 -Node Contoso-N1,Contoso-N2

The following command is an example of specifying a static IP address for cluster to use for its management connection, and if you don’t want any storage to be automatically added to the cluster.

PowerShell:

New-Cluster -Name Contoso-FC1 -Node Contoso-N1,Contoso-N2 –StaticAddress 10.0.0.14 -NoStorage

The following command is an example that would put the cluster account put into an existing Active Directory OU called “Clusters” that is in the Contoso.local domain.

PowerShell:

New-Cluster -Name CN=Contoso-FC1,OU=Clusters,DC=Contoso,DC=local -Node Contoso-N1,Contoso-N2

In this blog, I will outline the steps to troubleshoot “Create Cluster” failures.

Steps when Troubleshooting “Create Cluster” Failures

Step 1: Run the Cluster Validation Tool

The cluster validation tool runs a suite of tests to verify that your hardware and settings are compatible with failover clustering. The first thing to do when troubleshooting, and something you should do every time you create a cluster is to run the Validate tool. To run cluster validation:

1. Open the Failover Cluster Manager snap-in (CluAdmin.msc)

2. Select Validate Cluster:

Note:

You can also use the Failover Clustering Windows PowerShell® cmdlet, Test-Cluster, to validate your cluster.

3. Navigate to C:\Windows\Cluster\Reports directory and open the Validation Report .MHT file

4. Review any tests that report as Failed or Warning.

The validation summary provides a starting point to drill down further into the failure. For instance, in the example below we can detect an invalid Windows Firewall Configuration.

It is also useful to investigate the warnings flagged by validate. For example, the Active Directory Configuration test warning below flags a potential cluster creation problem:

Step 2: Examine the CreateCluster.mht file

If you cannot successfully create a cluster after all your validation tests are passing, the next step is to examine the CreateCluster.mht file. This file is created during the cluster creation process through the “Create Cluster” wizard in Failover Cluster Manager or the Create-Cluster Failover Clustering Windows PowerShell® cmdlet. The file can be found in the following location: C:\Windows\Cluster\Reports\CreateCluster.mht

The admin level logging in the CreateCluster.mht file can help you determine the step at which the cluster creation process failed. For example in the CreateCluster.mht snippet below you can infer that there was a problem with configuring a Cluster Name Object for the cluster.

Step 3: Turn on Cluster API debug tracing

If you are unable to pinpoint the root cause of the failure by neither the Validate report nor the Create Cluster log, then verbose debug logging can be enabled. Debug tracing can be turned on with the following steps:

1. Open Event Viewer (eventvwr.msc)

2. Click View then “Show Analytic and Debug Logs”

3. Browse down to Applications and Services Logs \ Microsoft \ Windows \ FailoverClustering-Client \ Diagnostic

4. Right-click on Diagnostic and select “Enable Log”

5. Attempt to create a cluster

6. Right-click on Diagnostic and select “Disable Log”.

Note: The debug tracing will be generated to the Diagnostic channel and viewable only after you disable logging.

7. Left-click on Diagnostic to view the logging captured.

The following are examples of events generated to the Diagnostic channel when cluster creation fails when the Cluster Name Object cannot be added to the clusterou container. In this case, the cluster administrator does not have the Read All Properties permission on the organizational unit (OU) in Active Directory.

Step 3b: Turn on Cluster API event log tracing programmatically

You can also turn on the Cluster API event log tracing programmatically. The debug information obtained will be the same as Step 3 but you are able to set this up using a script. The following are the steps to configure:

1. Run:

logman start clusapiLogs -p {a82fda5d-745f-409c-b0fe-18ae0678a0e0} -o clusapi.etl -ets

2. Attempt to create a cluster

3. Run: logman stop clusapiLogs -ets

4. Run: tracerpt clusapi.etl -of CSV –o c:\report.csv

5. Open the generated Comma Separated Value (CSV) dump file and examine the User Data column for potential issues. Note that the ‘-o’ parameter determines where the CSV dump file is generated.

The following are some examples of Cluster API event log traces found for a “create cluster” failure.

CreateCluster: Create cluster test-33 will be using a Read-Write DC \\subhatt-VM1.subhattcluster.com.

CreateClusterNameCOIfNotExists: Failed to create computer object test-33 on DC \\subhatt-VM1.subhattcluster.com with OU ou=clusterou

"CreateCluster: Create cluster failed with exception. Error = 8202

msg: Failed to create cluster name test-33 on DC \\subhatt-VM1.subhattcluster.com. Error 8202.

Step 4: Generate the cluster.log file

The cluster log provides verbose logging for the cluster service and allows advanced troubleshooting. The cluster log can be generated even when the cluster creation fails by specifying the node to collect the log on. You can generate the cluster log using the Failover Clustering Windows PowerShell® cmdlet Get-ClusterLog

Get-ClusterLog –Node <CreateClusterNode>

Note:

· The default verbosity level for the cluster log is 3. This proves to be sufficient for most debugging purposes. However, if this verbosity level is not capturing the data you need, you can increase the verbosity level

o On a Windows PowerShell® console run: (Get-Cluster).ClusterLogLevel = 5

o This generates significant spew so the default level should be restored once the troubleshooting is completed.

· The cluster log can be generated in local time using Failover Clustering Windows PowerShell®:

o Get-ClusterLog -UseLocalTime

Bonus Tip:

The number one reason for create cluster failures is due to misconfigured permissions in Active Directory environments resulting in failures while creating the Cluster Name Object (CNO).

Review: “How to Create a Cluster in a Restrictive Active Directory Environment”

“Failover Cluster Step-by-Step Guide: Configuring Accounts in Active Directory”

Did you really review the links above? Here’s a quick test… How would you fix the following “Create Cluster” errors?

1. An enabled computer account (object) for <cno> was found.

Answer:

1. Verify that the cluster name you attempting to use for the new cluster is not already being used by a cluster in production. If it is, you should chose another name for the cluster. In other words you need to ensure that you can take over the computer name with no adverse repurcussions.

2. On the Domain Controler, launch the Active Directory Users and Computers snap-in (type dsa.msc)

3. Navigate to the OU you which has the cluster name you are trying to use. In this case you are searching for “Test-8”. You might have to search multiple OUs to find the conflicting cluster name.

4. Delete the existing Cluster Name Object (CNO), “Test-8” or disable it by right-clicking on the CNO and selecting disable.

2. You do not have permissions to create a computer account (object) in Active Directory

Answer:

1. On the Domain Controler launch the Active Directory Users and Computers snap-in (type dsa.msc)

2. On the View menu, make sure that Advanced Features is selected

3. Navigate to the OU you are trying to create your Cluster Name Object (CNO) in. By default this will the same OU as that of the node you are trying to create a cluster from.

4. Right-click on the OU and select Properties and then the Security tab.

5. Ensure that the Cluster Administrator has Create all child objects permissions

6. Select the Advanced tab

7. Click Add, type the name of the cluster administrator account for the Principal

8. In the Permission container dialog box, locate the Create Computer objects and Read All Properties permissions, and make sure that the Allow check box is selected for each one.

A final note: In this blog I have focused on “Create Cluster” failures. However, the same troubleshooting steps can also be used for “Add node” failures (failures encountered while adding a node to a cluster).

Thanks!

Subhasish Bhattacharya
Program Manager
Clustering & High Availability
Microsoft