Build and run Windows Failover Clusters on VMware ESXi

Yes, you read the title right. We ca run Microsoft Windows Failover Clusters on VMware ESXi hosts, and believe it or not, a lot of companies are doing it nowadays. It’s cheap, and it does the job very well. All we need is at least one ESXi host and two Windows Server VMs that will run as nodes in the Failover Cluster and we are done; no more extra switches, no more buying extra network adapters and especially no more complicated configurations.

Now, as a bonus, VMware has made this possible in three “flavors”, and here they are:

Cluster in a Box (CiB) – Is the configuration where the VMs participating in a Failover Cluster are running on the same ESXi host. This is the easiest of the three configurations to set up, but it’s also the one that does not protect you against a hardware failure. As you will see later in the article, the VMs participating in the Failover Cluster cannot be spread across ESXi hosts, and with this, they are not recommended for production environments that need a high uptime since they do not benefit of the HA feature of vCenter. VMDK files are recommended for the shared disks on this type of cluster.

Cluster in a Box representation

Cluster across Boxes (CAB) – With this type of configuration, the VMs are running on different ESXi hosts, which means that VMs participating in the cluster are protected against a single host failure. Even tough this is good news, vMotion and vSphere Distributed Resource Scheduler (DRS) are still not supported (at the time of this writing) for Microsoft-clustered VMs.

UPDATE: Starting with VMware vSphere 7.0, VMDKs can now be used as a shared (clustered) resource. A VMDK can be used as a shared disk resource for a WSFC deployed on VMs hosted on different ESXi hosts. Using a ESXi hosts with a version lower then 7.0 can cause slow boot times and become unresponsive. A host with a version lower than ESXi 7.0 cannot mount a clustered VMDK datastore. This is because the ESXi hosts on which WSFC VMs run must have physical SCSI-3 WEAR type reservation on the LUN. For more details please check this document and the guide.

Taken from Recommendations for using Clustered VMDKs with WSFC

Cluster across boxes representation

Physical-to-Virtual – This is the type of configuration where at least one physical server and at least one VM are joined together in a Failover Cluster. Having this setup, gives us the best of both worlds; the only restriction is that you cannot use Virtual Compatibility mode with the RDMs.

Physical to virtual Failover Cluster representation

Before jumping into the configuration part, I think it’s important to list the setup requirements and limitations in order for the deployment to work and get VMware support when needed.

 

Component Requirement
Virtual SCSI adapter LSI Logic Parallel for Windows Server 2003.
LSI Logic SAS for Windows Server 2008 SP2 and above.
VMware Paravirtual for Windows Server 2008 SP2 and above.
Operating system Windows Server 2003 SP1 and SP2, Windows Server 2008 SP2 above releases.
I/O timeout Modify the Windows registry disk timeout value to 60 seconds or more.
Modify HKEY_LOCAL_MACHINESystemCurrentControlSetServicesDiskTimeOutValue.
Disk format Thick Provision in eagerzeroedthick format.
Number of nodes Windows Server 2003 SP1 and SP2 : two-node clustering
Windows Server 2008 SP2 and above: up to five-node clustering
NTP server Synchronize cluster nodes time with a common NTP server (domain controller), and disable host-based time synchronization when using clustering in the guest.

 

And for the disks type, we have to use the ones in the following table:

 

Storage Type Clusters on a single ESXi host
Cluster in a Box (CiB)
Clusters across ESXi hosts
Cluster across boxes (CAB)

Clusters of Physical and Virtual Machines

(Standby Host Clustering)

Virtual Disks (VMDK’s) Yes (recommended) No No
Pass-through RDM (physical compatibility mode) No Yes (recommended) Yes
Non-pass-through RDM (virtual compatibility mode) Yes No No

 

    1. Cluster in a box (CIB)

    1.1 Configuring the SCSI controller for the Windows Virtual Machines

As mention in the beginning of the article, this is the simplest cluster option of all three but also the less safer one. All we need is at least two VMs, one ESXi host, and a datastore. The configuration of the Windows VMs/nodes needs to be from the start on a single ESXi host and not spread around in your VMware cluster, or you will get the bellow error message when you try to put all the Windows VMs back on this one ESXi server. If you do not have a VMware cluster, ignore this and move forward with the configuration.

Virtual machine is configured to use a device that prevents the operation: Device ‘SCSI controller 1’ is a SCSI controller engaged in bus-sharing.

Migration error message of a VM with a shared SCSI bus

Thinking that you already deployed the number of VMs you want to put in the Windows Failover Cluster, right-click one of them and choose Edit Settings.

Editing the virtual machine configuration settings

Once the virtual machine Edit Settings window opens, click the Add New Device button then choose SCSI Controller. Now expand the New SCSI controller configuration and from the SCSI Bus Sharing drop-down-box choose Virtual. We need to add this new controller with this new configuration in order for the VMs to be able to share the same disk(s).

Repeat the operation for the other VMs that you want them to participate in your Windows Failover Cluster.

Adding a new SCSI controller to a virtual machine    Setting up the new SCSI controller for virtual bus sharing

Just a mention here! If your ESXi  server is not running in a VMware cluster environment, configuring the SCSI controller Bus Sharing while the VMs are running will not work, the option is disabled. You will have to shut down the VMs in order to configure the new SCSI controller.

New SCSI controller configuration unavailable for powered on VMs in a non-clustered ESXi environment

 

    1.2 Creating the cluster virtual disk(s) for the Windows Virtual Machines

Now open again the Settings for one of the VMs, click the Add New Device button and choose Hard Disk.

Creating a new virtual hard disk for the VM

Type a size for the new disk. This is going to be one of the Windows Failover Cluster disk resources, so make sure you create it with the right size to accommodate your data.

From the Location option we can choose to place the new virtual disk on a different datastore, or keep it within the virtual machine’s folder. This is not a must and it works either way.

From the Disk Provisioning option choose Thick Provision Eager Zeroed. I know many of you don’t want to use it because it provisions the entire disk size, taking datastore space, but this is the only option that works and is supported by VMware for a CiB deployment. If you try to use any of the other two disk provisioning options you  will get the bellow error message later on when configuring the rest of the Windows VMs, and the adding of the virtual disk will fail.

Incompatible device backing specified for device ‘0’.

Virtual disk error configuration for VMs not using Thick Provision Eager Zeroed disks in a CiB deployment

The last setting that we need to configure here is the Virtual Device Node one. From the drop-down-box choose the controller we added a few moments ago (section 1.1), SCSI controller 1 in my case. This is a must, because as you remember, this controller offers us the possibility to share the virtual disk(s) between multiple VMs. Click OK when you’re done.

Based on VMware’s documentation, the multi-writer feature must not be used for a clustered disk resource for WSFC, so set to No sharing on the Sharing drop-down box.

Depending on the size of the virtual disk you set up, the operation can take quite some time. I configured mine on a different LUN, a LUN that I know it’s fast, and by browsing the datastore we can see the virtual disk in its full size.

Locating the new virtual disk in the datastore

Feel free to create other virtual disks that you think you need in the Windows Failover Cluster using the above steps. The only thing that will be different is the SCSI ID number, which increase as you add new virtual disks. Here, for the second disk that I added to use for my quorum drive, the controller is still the same, but the ID has changed to 1:1 since 1:0 was already taken.

Configuring SCSI ID number for the second virtual disk on the VM

 

    1.3 Adding cluster virtual disk(s) for the rest of the Windows Virtual Machines

We are now done with the configuration of the first VM, so let’s move on to the other one(s) with the taught in mind that you already have added the second SCSI controller as discussed in section 1.1.

Open the settings for the second machine, click the Add New Device button and choose Existing Hard Disk.

Attaching an existing virtual disc to the second node (VM)

From the datastore browser window that opens, select the disk(s) we created on the first VM then click OK.

Locating the existing virtual disk in the datastore

We need to configure the disk(s) to be identical with the one(s) from the first VM. Set the the Virtual Device Node to SCSI controller 1 on ID 1:0. Off course, the next virtual disk that you add will be on SCSI ID 1:1, the third one on SCSI ID 1:2, and so on, but they need to be the same as the first VM that you configured.

Once you’re done with this, go ahead and build the Windows Failover Cluster. During the Validate a Configuration Wizard you might get a warning about the SCSI 3 persistent reservation. This is because your storage is not configured or does not support SCSI 3 persistent reservation. The only way to get rid of the warning is to re-configure your storage with the mentioned feature.

Test Disk 0 does not support SCSI-3 Persistent Reservations commands needed by clustered storage pools that use the Storage Spaces subsystem. Some storage devices require specific firmware versions or settings to function properly with failover clusters. Contact your storage administrator or storage vendor for help with configuring the storage to function properly with failover clusters that use Storage Spaces.

Windows Failover Cluster configuration validation report

 

    1.4 Creating VMware affinity rule for the clustered Windows Virtual Machines

We are almost done. The only thing left is to create an affinity rule so our VMs participating in the Windows Failover Cluster will be kept together, on the same ESXi host -for those that have a VMware Cluster-.

Go to Configure > Configuration > VM/Host Rules. Here, push the Add button.

Creating a new vCenter affinity rule

Give the rule a name then select Keep Virtual Machines Together from the Type drop-down box. Now we have to chose what VMs do we want to keep together/on the same host. For that, click the Add button and select the Windows nodes/VMs participating in the Windows Failover Cluster. Once you’re done, click OK twice.

Attaching Virtual Machines to the affinity rule    Configuring the vCenter affinity rule

And here we have our affinity rule that keeps our Windows VMs together.

View of the new affinity rule in vCenter

Based on VMware’s documentation we also need to configure an advanced setting on DRS so the affinity rule is strictly applied.

Select your VMware cluster and go to Configure > Services > vSphere DRS. From the Actions pane click the Edit button.

Editing the advanced DRS settings

The Edit Cluster Settings window opens. From the Advanced Options tab, click Add.

Adding a new configuration parameter to the DRS feature of the VMware cluster

In the Options column type ForceAffinePoweron, then in the Value one type 1. Click OK to save the changes.

New configuration parameter values to strictly apply the affinity and anti-affinity rules

We now have our CiB configured and ready. Off course, there is still the Windows Failover Cluster configuration part, but I will leave that up to you. Just let me know how it goes in the comments area of the article.

 

    2. Cluster across Boxes (CAB)

    2.1 Configuring the SCSI controller for the Windows Virtual Machines

This second type of cluster is the most popular for business critical applications because it allows us to put the Windows nodes on different ESXi hosts and take advantage of vCenter HA and vMotion. The configuration is pretty much the same as CiB, the only difference are the disks. We don’t have virtual disks anymore (VMDKs), we have RAW disks or RDMs. This means that we are going to map our LUNs from our storage device directly to the Windows VMs/nodes.

As mentioned in the begging of the article, starting with vSphere 7.0 we can use VMDKs as a clustered resource.

Go ahead, prepare your LUNs on you storage array but do not add them as datastores in vCenter, just leave them there un-formatted and untouched. Now right-click the first Windows VM and choose Settings.

Open the Edit Settings window for a VM

Once the Edit Settings window opens, add a new SCSI controller, but this time set the SCSI Bus Sharing to Physical. This is a must in order for the VMs to be able to sit on different ESXi hosts and share the same SCSI bus. Click OK when you are done.

Repeat the same operation for the rest of the VMs that participate in the Windows Failover Cluster.

Adding a new SCSI controller and configure it in a physical bus sharing mode

 

    2.2 Adding RDM (RAW) disks to the Windows Virtual Machines

Now let’s add the LUNs to our VMs. Open the Edit Settings window again on the first VM and add a new RDM Disk.

Adding a new RDM disk to the first Windows node

On the Select Target LUN window that pops-up, select one of your LUN(s) then click OK.

Choosing the LUN to be added as RDM disk

As in the CiB section, we have some settings to configure on the disk before we can actually use it.

On the Location drop-down box we can change the location of the virtual file. Yes, even if we use RDM disks, we still have a virtual file created, which is actually a pointer to the RAW disk. You can read more about it here. As before, set the Sharing option to No sharing and make sure the Compatibility Mode is set to Physical. On the Virtual Device Node, choose the second SCSI controller, the one we added just a moment ago then click OK.

Based on VMware’s documentation, the multi-writer feature must not be used for a clustered disk resource for WSFC, so set to No sharing on the Sharing drop-down box.

Repeat this operation for the rest of the RDM disks you need to add to this VM.

On the second VM, or the rest of them -if you are building a Windows Failover Cluster with more than two nodes- add an existing hard disk.

Adding an existing RDM disk to the other Virtual Machines

Choose the virtual file/pointer that was created with the first VM and set the Sharing option to No sharing mode. From the Virtual Device Node option, set the controller to SCSI controller 1 with the same SCSI ID as the first VM.

Click OK when you are done then add the rest of the disks you might have. Off course, repeat this operation for the rest of the VMs.

Choosing the already created RDM disk pointer

 

    2.3 Creating VMware anti-affinity rule for the clustered Windows Virtual Machines

In the CiB section, we created an affinity rule to keep the VMs together. Now we have to create an anti-affinity rule to keep the VMs apart from each other so in case one of the ESXi hosts fails, we still have at least one VM/node running and not making the Windows Failover Cluster unavailable.

Once again, click your VMware cluster name and go to Configure > Configuration > VM/Host Rules. From the Actions pane, hit the Add button.

Creating a new Anti-Affinity rule for the CAB Windows VMs

Name the rule, add your Windows VMs into the list, then from the Type drop-down box choose Separate Virtual Machines.

Creating the Anti-Affinity rule for the CAB Windows VMs

Once you click OK, you should have an anti-affinity rule created in the vCenter console similar to the one bellow.

View of the CAB Anti-Affinity rule in VMware vCenter

The last thing that needs to be done is to enforce this anti-affinity rule to strictly apply.

Select your VMware cluster again and go to Configure > Services > vSphere DRS. From the Actions pane click the Edit button.

Editing the advanced DRS settings

In the window that pops-up hit Add.

Adding a new configuration parameter to the DRS feature of the VMware cluster

Type ForceAffinePoweron in the Options column and give it a value of 1. Click OK to save the changes.

New configuration parameter values to strictly apply the affinity and anti-affinity rules

 

    2.4 Testing the Windows Cluster

Now when we go and build the Windows Failover Cluster, the storage checks should pass with no problems during the validation wizard, which is exactly what we wanted. If you get any errors or warnings, please post them in the comments area so the community can know about them.

Validating storage cluster configuration using the Validate wizard

If we want to migrate the VMs to a different ESXi host, no problem, we can do that, and vMotion is doing the job just fine. The validation succeeds,

Testing the vMotion migration of one of the Windows VMs

and transferring the data to a different ESXi host completes with no errors whatsoever.

View on the vMotion migration process of the Windows VM in vCenter    View on the vMotion migration process completion in vCenter

Also, if one the ESXi hosts fails, HA will start the Windows VM on a different host. But remember, we have an anti-affinity rule, so the VM will not be started on an ESXi server where other Windows machines that participate in the Windows Failover Cluster are running.

View of the vCenter High Availability (HA) in action for the Windows VM Node

    3. Physical to Virtual

This last type of cluster configuration is pretty much the same as Cluster Across Boxes (CAB), just that now we have at least one Windows physical host involved.

I will not go too deep into the subject here since most of the settings and configurations were covered in the previous section, the CAB section, so if you need extra information, please read that section.

The most important thing in making this work is to map the same LUNs to the physical and virtual machines. It doesn’t really matter if the physical host is connected to the storage device by Fiber Channel or iSCSI as long as the disks/LUNs are mapped to the server.

Here, on the physical box I have connected to my storage array using iSCSI

Connecting to the storage device from the Windows physical box using iSCSI    Listing LUNs available on the storage device for this server

then on the virtual machine, the same LUNs are mapped so both servers can see them.

Configuring the SCSI controller for Windows VMAdding and configuring RAW disks on the Windows VM

During the Windows Failover Cluster validation configuration wizard, everything should be good and green.

Running the storage validation test on the Windows Failover cluster    View of the storage validation report

Off course, the VM running on the VMware infrastructure can be migrated using vMotion but it also benefits of HA, so if an ESXi server goes down, the VM will be restarted on a different one.

Running the VMware Migration wizard for the Windows node   View of the Windows VM migration process in VMware vCenter

 

Summary

In the end, I will leave it up to you to choose which type of Windows clusters you want to build on your VMware infrastructure, but take advantage of this for your sensitive applications and services so you have at little as possible of downtime. As you just saw, building them it’s a piece-of-cake since no extra hardware and complicated configurations are needed, like in “the real world”.

Want content like this delivered right to your

email inbox?


25 thoughts on “Build and run Windows Failover Clusters on VMware ESXi

Leave a Reply

Your email address will not be published. Required fields are marked *

*

css.php