How do I get to vSphere 7.0 without dying in the process?

Hello Everyone,

After a long hiatus, I decided to write a new blog post (and hopefully improve the frequency of them :D) – This will be based on a 2-hour presentation that I did for VMUG (VMware User Group) Argentina last week, which was done in spanish, and I will link it down below

However, for all of the non-spanish Speakers, I will do a breakdown of everything you need to check before attempting a vSphere upgrade from the vCenter & PSC perspective to pass the upgrade wth flying colors! – Buckle up!

Where is our environment currently standing?

First of all, you need to assess the current situation of your vCenters and PSCs – Is replication working correctly for example? This article goes really really deep into checking that:

Pre-upgrade considerations in Multi-vCenter environments

If you have any replication issues, this is the first thing you need to fix, otherwise, as shown in the previous article (and the video) you risk completely destroying your environment.

The 2nd thing you need to check is your current topology – How many PSCs and vCenters are actually in my environment? Am I using PSC HA? Is everything converged? Depending on your current topology, it might be a pretty trivial migration or it would need multiple steps over the course of a weekend.

What happens in the upgrade process?

First of all, the external PSC is deprecated in vSphere 7.0 – That means that, as a part of the upgrade process, any environment with an external PSC is converged. Even though this process might be straightforward, it can cause multiple problems before, during and after the migration. It’s easier and more convenient to break it up in parts

So if we’re good with replication (check and re-check previous article, I can’t stress this enough) then we need to figure out an upgrade and migration plan

Planning the upgrade process based on our topology

Let’s start with something simple:

What would be the correct steps here?

Let’s break it down:

1: Offline snapshots of all three VCs (with embedded PSCs) – offline means with all the SSO domain powered off- this is done from the ESXi nodes that are hosting the VMs.

2: Upgrade vCenter 1

3: Check functionality and replication

4: Offline snapshots of all three VCs (with embedded PSCs)

5: Upgrade vCenter 2

6: Check Functionality and replication

7: Guess what?

8: Upgrade vCenter 3:

9: Check Functionality and replication

10: Delete all snapshots

Why am I taking snapshots at every step? Why don’t I just take a single round of snapshots and then upgrade all at once?

Well, because if you had any issue at any point of the 2nd or 3rd upgrade, you would have to roll back everything and start from scratch. If you do it this way, you have multiple points to go back and avoid having to re-do the upgrade process! This can get even worse if instead of 3 vCenters you have 9 or 10 – If let’s say, you had an issue with upgrade 7, you would have to revert everything!

Now let’s make this a little bit more complicated!

So let’s picture this scenario (which is not too uncommon, i’ve seen this is in the real world)

What do we have?

First of all, blue lines symbolize good replication and red lines symbolize that replication is not working – So, as discussed earlier, this will be the first thing to fix – in the process of fixing this (most likely with a GSS ticket), multiple rounds of offline snapshots will be taken!

Now, onto the topology:

• 6 External PSCs in a ring topology
• 3 PSC HA VIPs being used by 2 vCenters each
• 6 vCenters

So what should we do here? This not only involves the upgrade of the vSphere environment, but also, the re-pointing of 2nd and 3rd party tools to the new converged PSCs – Think of NSX and SRM for example.

The biggest pain point in this scenario, however, is PSC HA – how do we get rid of this prior to the upgrade?

Even though there is a KB for converging PSC HA (https://kb.vmware.com/s/article/65129) in practice, this is not the best approach due to how error prone it is.

What is the best approach? There are two ways to approach this, depending on downtime and operations.

The cleanest approach, would be to deploy 6 new PSCs, then repoint the vCenters to those 6 PSCs, and then decomission all the PSC HA nodes (as well as the VIP) – However, this might be complicated because of lack of IP addresses in the management segment, time, etc.

You could also leverage lsdoctor (https://kb.vmware.com/s/article/80469) to unconfigure PSC HA and then repoint the vCenters to each of the nodes – This introduces a little bit more downtime per vCenter (downtime when unconfiguring PSC HA + downtime until the repoint is complete) but removes the need of deploying new PSCs.

If you ask me, I recommend the first option, to make this as clean as possible.

So in this scenario, what would you do?

1. Offline snapshots of all vCenters and PSCs
2. Deploy PSC 7 pointed to PSC 6
3. Deploy PSC N pointed to PSC N-1 until all PSCs are deployed.
4. Check replication among the new PSCs

So now we have something like this

You can see that by deploying the PSCs in that order, we have a “semi-ring” already, with way less operational hassle than if we were deploying them pointed to a single PSC and then having to remake the replication agreements

So what’s next?

We need to repoint the vCenters to these new PSCs – Since the repoint is a pretty short process, you can get away with taking a single round of offline snapshots at the beginning and just repoint everything

1. Offline snapshots of all vCenters and PSCs
2. Repoint all vCenters to the new PSCs, 1:1
3. Check correct functioning

End result:

Lovely, right?

Now, we need to get rid of all the PSCs that were forming the PSC HA (nodes and VIPs)

1. Offline snapshots of all vCenters and PSCs
2. Decomission all PSCs and PSC HA VIP nodes using: https://kb.vmware.com/s/article/2106736
3. Check correct functioning

Now we’re here!

So we did all this and we haven’t even started upgrading or converging… but believe me, taking due diligence in doing this as clean as possible will save you from multiple headaches when you actually upgrade!

So what is left?

1. Form a ring creating an agreement between PSC12 and PSC7
2. Take a new round of offline snapshots
3. Converge PSC7
4. Check correct functioning
5. Take a new round of offline snapshots
6. Converge PSC8
7. ….
8. ….
9. Until all PSCs are converged

In case there is any issue with the convergence, you can just go back to the latest functioning snapshot so you don’t have to redo everything!

You should be here now:

And from here, you can finally do the upgrade process – as discussed previously and in the first scenario, you should take a round of offline snapshots per each upgrade, to avoid having to re-do upgrades

Last but not least, you should repoint all 2nd and 3rd party solutions to the new converged (and upgraded) PSCs that are now living inside the vCenter appliance!

Closing note

I hope you enjoyed this post – If you have even limited knowledge of spanish, I encourage you to watch the youtube video in which I go over this in detail, and also I analyze and fix replication issues the same way it would be done if you contacted GSS.

Feel free to share this with peers, customers, partners – If we generate awareness about these processes and a clean and correct way of doing them, we will have way more succesful upgrades!