Removing vCenter or PSC from Linked Mode configuration

Having our vCenter servers in an enhanced link mode configuration is great, since everything is in one console, but there are situations where one of the vCenter servers or External Platform Service Controllers (PSC) just die, it’s improperly removed from the infrastructure during decommission, or you just want to break the Enhanced Link Mode configuration (ELM) between one or more vCenter servers because the company was sold/acquired by another one. If the vCenter or PSC are dead/offline, then we will have to remove the link mode so we don’t keep filling the logs with connection errors, login time-outs, and especially remove the annoying message banner that we get after login.

Could not connect to one or more vCenter Server Systems:<vCenter Server:443>/sdk

View of the error message when the linked remote vCenter server is not available

To simulate this so you can get a better idea, I have created an environment with two sites connected trough an IPSec VPN tunnel and both of them have a vCenter Server with an External PSC. The PSC’s replicate with one another and are joined to a single SSO domain (vsphere.local). After the ELM configuration is removed between the two, they will operate individually.

Graphical representation of the environment for this article

So right now, all of my vCenter servers are displayed in a single web console thanks to the ELM configuration.

View of a healthy ELM configuration

Using the GUI to remove the vCenter or PSC from an Enhanced Link Mode configuration is out of the question (for now), and if we have a failed vCenter server, we will not even be able to see it in the Linked vCenter Servers tab.

Linked vCenter server(s) not showing in the GUI when they are failed or offline

One thing that I want to point out here before moving to the practical stuff is that the process for removing a vCenter Server or a Platform Services Controller from the vSphere domain is irreversible. Once the appliance is removed from the SSO domain, the only way to re-join it is to perform a re-install or a re-deploy of vCenter Server or Platform Services Controller system.

 

1. Enabled SSH for the vCenter or PSC appliance

Since we cannot use the GUI, the only method we have left is from the CLI, but before that, we need to enable SSH and Bash Shell for our vCenter or PSC appliance. Open the Appliance Management portal (https://psc or vCenter FQDN:5480) and log in using the credentials set up during the deployment.

Primary PSC (headquarters) appliance management portal

Click Access, then on the upper right corner Edit.

PSC appliance SSH login options

In the window that pops-up, click the Enable SSH Login button and the Enable BASH Shell one and set a timeout. Hit OK when done.

Enabling SSH and Bash Shell for the PSC appliance

 

2. Removing the Enhanced Linked Mode configuration

2.1. Un-register the first PSC or vCenter appliance

Download Putty and do an SSH connection on the PSC or vCenter Server we just enabled console access to. Once connected, get into the shell by typing shell and pressing Enter.

SSH connection to the PSC appliance from the headquarters

Next, we need to list all the nodes so we can have a better view of what it’s registered in this SSO domain. And sure enough, once we hit the Enter key, we get the list of all the registered nodes in the SSO domain. In my case, I have two external PSC appliances with two vCenter servers. The HQ* ones are my primary ones and the S1* appliances are the ones in the remote site.

/usr/lib/vmware-vmafd/bin/dir-cli nodes list

Listing the nodes part of the ELM configuration on the primary (headquarters) PSC

Before removing the link mode, make sure you have a maintenance window since all the vCenter services will be restarted. Also, a backup it’s highly recommended.

If the remote PSC is fine and working, and all we want is to remove the linked mode configuration with it’s replication partner, we will have to prevent them talking to each other. Shutting down the remote PSC is not a great solution since we will need to connect to it later on, so instead we will disconnect its network adapter in order for the removal process to be successful. If communication between the two still happens, we will get the bellow error message and the un-register process will fail.

If the remote PSC or vCenter appliance cannot be reached anymore and failed, then we can go ahead and issue the command for the ELM configuration removal.

Solution users, computer account and service endpoints will be unregistered
2021-10-20T15:52:46.767Z Running command: ['/usr/lib/vmware-vmafd/bin/dir-cli', 'service', 'list', '--login', 'administrator@vsphere.local']
2021-10-20T15:52:46.836Z Done running command
2021-10-20T15:52:47.259Z RC = 1
Stdout = vdcleavefd offline for server s1-psc.vkernel.local
Leave federation cleanup failed. Error[1] - Operations error

Stderr = password:

ERROR unregistering computer account. Details {
"componentKey": null,
"detail": [
{
"id": "install.ciscommon.command.errinvoke",
"args": [
"Stderr: password: n"
],
"translatable": "An error occurred while invoking external command : '%(0)s'",
"localized": "An error occurred while invoking external command : 'Stderr: password: n'"
}
],
"problemId": null,
"resolution": null
}

Error message if the PSC set for un-registration from the ELM configuration is still up and running

Once the above concerns were taken care off, issue the bellow command line to un-register our vCenter or PSC from the SSO ELM configuration. When asked for the SSO password just type it in and the un-register process will begin. If everything worked out fine, we will get a Success message at the end.

In case of a vCenter with an external PSC, both will have to be un-registered. Just re-issue the command and replace the FQDN of the appliance.

cmsso-util unregister --node-pnid <vCenter or PSC FQDN> --username administrator@<your SSO domain>

vCenter un-register process from the ELM configuration on the primary PSC node PSC un-register process from the ELM configuration on the primary PSC node

List the nodes again, and this time our vCenter server or PSC -from the remote site- that we just un-registered should not be in the list anymore.

Listing nodes after the un-register process is done. The un-registered ones should not be displayed anymore.

Also, in vCenter we should not get the error message banner and the un-registered vCenter should not even be displayed in the inventory list.

vCenter view after remote nodes were un-registered.

If your remote PSC/vCenter appliance was a failed one, or one that you do not need anymore, then the un-register process is done, but if you are in a situation like in this article where you want to separate the PSC appliances into their own SSO domain, then read on, because it is not finished yet.

2.2. Un-register the second PSC or vCenter appliance

Just by removing the link mode from the primary vCenter/PSC appliance(s) (hq-vcsa, hq-psc) will only do half of the job, because the remote PSC is still configured to replicate with it’s partner from the headquarters.

To fix this, we will have to break the link mode configuration from the remote site PSC also. Since earlier we disconnected the network adapter for this PSC appliance we will have to use the ESXi console in order to issue the cmsso-util command line, well…sort of. We both know that by using the hypervisor console we will have to do a lot of typing, and I really hate typing those complex root passwords; better if we just use the Paste command in an SSH shell.

The easier solution will be to go and disconnect the network adapter from the primary PSC/replication partener(s) (HQ-PSC) then come back to the remote site PSC (S1-PSC) and enable its network adapter. If you are following this, make sure they do not talk to each other.

Enable SSH on the PSC appliance like we did on the headquarter one, connect to it and list the nodes. As you can see the remote PSC still has a replication partner configured.

/usr/lib/vmware-vmafd/bin/dir-cli nodes list

Listing the nodes part of the ELM configuration on the remote PSC

Un-register the headquartes/replication partener node(s) nodes.

cmsso-util unregister --node-pnid <vCenter or PSC FQDN> --username administrator@<your SSO domain>

PSC un-register process from the ELM configuration on the remote PSC node vCenter un-register process from the ELM configuration on the remote PSC node

List the nodes again, and sure enough everything looks good.

Listing nodes after the un-register process is done on the remote PSC. The un-registered ones should not be displayed anymore.

Now we can log in on the remote site vCenter server (s1-vcsa) where the PSC does not replicate anymore with the headquartes one. It is totally independent.

Login portal working on the remote site vCenter

After login, we should only see the infrastructure from the vCenter server(s) that are connected to this PSC appliance. Go back to the headquartes PSC and re-enable its network adapter.

View of the remote site vCenter after the un-registration from the ELM configuration

Summary

Having the option to break the enhanced link mode configuration, even from the CLI, it is great. With a simple command line we can easily separate two or more PSC/vCenter appliances or remove the offline/failed ones from an Enhanced Link Mode configuration. I’m just hopping that in a future vCenter release we can do this from the portal. Until then, test it out and let me know how this worked out for you in the comments area bellow.

Want content like this delivered right to your

email inbox?


5 thoughts on “Removing vCenter or PSC from Linked Mode configuration

  • 28/12/2023 at 20:51
    Permalink

    We ran into 1 issue. We were unable to re-enable the NIC on the vCenter server in our HQ because we are using a VDS now. The ESXi host was giving us an error: “Invalid Configuration for Device 20”.

    We connected 1 of the open 1 GB NICs on the ESXi host the vCenter server was running on to the 10GB switch and enabled the uplink in the old standard switch. Then we configured the vCenter NIC to the same VLAN on the standard switch. It started responding. Then we logged into vCenter configured its NIC back to the same VLAN on the VDS in vCenter.

    It was a bit stressful for about 10 minutes, but we got it fixed. lol

    Reply
    • 10/01/2024 at 20:39
      Permalink

      Hi Dave,
      Yes, I did not like it the way VMware implemented vCenter in the past with the PSCs and everything because I also had quite a few issues with those. This is one of the reasons I did not posted about them too often.
      Now with the new versions they are more stable and working as it should.

      Thanks for posting the issue, much appreciate it.

      Reply
  • 03/05/2023 at 17:49
    Permalink

    NVM, missed the part where it mentions that the secondary VCSA must be powered down prior to running the command on VC1 to unregister VC2.

    Thanks!

    Reply
  • 03/05/2023 at 17:16
    Permalink

    Great Post!

    Do we still need to disable nic for VCSA with embedded PSC ? or can we skip that part as the article mentions to disable nic on External PSC and not on VCSA.

    Thanks in Advance!

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

css.php