Replacing VMware ESX server default (self signed) certificate

I bet you got tired of those annoying certificate messages every time you connect to your ESX/ESXi servers. And is not just about the messages, is more about trusting the host, you or your users connect to. There are situations when you need to have a proper and a trusted certificate on you ESX/ESXi hosts, and in this guide I’m going to show you how to replace the default (self signed) certificate with a proper one.

The certificate is going to be issued by a Microsoft Windows Internal CA running on Windows Server 2008 R2. I also have a Windows 7 host joined to the domain, on which I installed the VMware vSphere Client to connect to my ESXi 5 server. If you don’t have an ESXi 5 host available for testing you can use ESX 4.0 or 4.1, and it will work just fine. There is no vCenter server present on this lab configuration, so the connection is made directly to the ESXi host. Now like I said we need to get rid of this messages:

    

and for this the certificate needs to be trusted; issued by a trusted CA. To get started first you need to download and install OpenSSL on you Windows 7 machine, then create the request. For this example I will create a SAN certificate, so I can use it on multiple hosts and connect to them using either the FQDN or the NetBIOS name of the server. A detailed post on how to request normal certificates and certificates with a SAN attribute can be found here, so go read it then come back. I presume you finished with the installation of OpenSSL, so let’s create the request by issuing the following commands, one by one:

C:\OpenSSL-Win64\bin\openssl genrsa 2048 > rui.key
C:\OpenSSL-Win64\bin\openssl req ‐new ‐key rui.key > rui.csr

Complete the required information and make sure on the Common Name you type the FQDN of you ESX/ESXi host. In my example here I typed a descriptive name of the certificate because all the names are in the SAN attribute of the certificate.

Now from your OpenSSL bin directory open with notepad or any other editor the rui.csr file and copy the request to your clipboard; be sure not to modify it in any way. Now open your Microsoft CA web page and choose Request a certificate > advanced certificate request > Submit a certificate request by using a base-64-encoded… then paste the content from you clipboard in the Saved Request box and choose Web Server from the Certificate Template. Click the Submit button, then YES on the warning message.

The certificate was issued. Check the Base 64 encoded radio button and download the certificate to the bin directory of OpenSSL.

We now need to change the format of the certificate so the ESXi host can recognize it. For this issue the following command:

openssl x509 ‐in ESXi.Certificate.cer ‐out rui.crt

where ESXi.Certificate.cer is the certificate issued by the Windows Internal CA, the one we just download it.

The next step is to upload the certificate and the private key (rui.crt and rui.key) to our ESXi host, and for this we will use WinSCP.

For you to be able to connect to the ESX/ESXi host using WinSCP, the SSH server service need to be enabled on the host, and port 22 opened.

Now follow the path etc/vmware/ssl and you should be able to see the self signed certificate of the ESXi host. Make a backup, then copy our certificate and private key in this directory, replacing the default ones. Reboot the ESXi host.

After the server is back online connect to it using the VMware vSphere Client. The certificate error should not appear any more. Now let’s try using a browser and connect to our ESXi host. As you can see the connection is now trusted.

And here is my certificate with that “fancy” name. On the SAN attribute you can see that I have multiple FQDN and NetBIOS names, so I’m able to use this certificate on multiple servers without requesting so many certificates, and connect to those servers using either the NetBIOS or the FQDN.