Published on 28 Nov 2023.
Kinamo maintains several VMware vSphere clusters for our customers and for Kinamo. One of the crucial tasks in maintaining a healthy, high-performance vSphere cluster is keeping up with updates. In this blog post, we describe an approach to troubleshooting drivers when upgrading a VMware Esxi 7.0 Update 2 to WMware ESXI Update 2a.
We recently found ourselves in a situation where one of our vSphere clusters was running ESXi 7.0 Update 2 Build 17630552. This version was taken offline by VMware on March 12, 2021 due to upgrade issues. When upgrading these servers to ESXi 7.0 Update 2a (Build 17867351), we encountered a number of errors.
Please note that apart from the Dell EMC add-ons for PowerEdge servers provided by Lifecycle Manager itself, these images were not customized.
The problem here was the downgrading of an add-on component, namely "Mellanox Native OFED ConnectX-3 Drivers" (version 3.19.70.1). Downgrading is not supported, which caused a blocking problem in the vSphere cluster, as hosts could no longer be upgraded.
Kinamo maintains several VMware vSphere clusters for our customers and for Kinamo. One of the crucial tasks in maintaining a healthy, high-performance vSphere cluster is keeping up with updates. In this blog post, we describe an approach to troubleshooting drivers when upgrading a VMware Esxi 7.0 Update 2 to WMware ESXI Update 2a.
We recently found ourselves in a situation where one of our vSphere clusters was running ESXi 7.0 Update 2 Build 17630552. This version was taken offline by VMware on March 12, 2021 due to upgrade issues. When upgrading these servers to ESXi 7.0 Update 2a (Build 17867351), we encountered a number of errors.
Please note that apart from the Dell EMC add-ons for PowerEdge servers provided by Lifecycle Manager itself, these images were not customized.
The problem here was the downgrading of an add-on component, namely "Mellanox Native OFED ConnectX-3 Drivers" (version 3.19.70.1). Downgrading is not supported, which caused a blocking problem in the vSphere cluster, as hosts could no longer be upgraded.
esxcli software vib remove -n nmlx5-core -n nmlx5-rdma -n nmlx4-core -n nmlx4-fr -n nmlx4-rdmaThis will produce the following result:
Résultat de la suppression
Message : La mise à jour a été effectuée avec succès, mais le système doit être redémarré pour que les modifications soient prises en compte.
Redémarrage requis : true
VIB installé :
VIB supprimés : MEL_bootbank_nmlx4-core_3.19.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx4-en_3.19.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx4-rdma_3.19.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx5-core_4.19.70.1-1OEM.700.1.0.15525992, MEL_bootbank_nmlx5-rdma_4.19.70.1-1OEM.700.1.0.15525992
VIBs ignorés :The next and final step is to reboot the host. Either from the SSH command line with the "reboot" command, or from the GUI.
After following the above steps, this drive should no longer block the Lifecycle Manager upgrade.
In general, we don't recommend making changes to your hosts using SSH and esxcli, but in this case it's a solution. Manual modification of the host is overridden by the upgrade process we run immediately after the change, restoring compliance.
Bear in mind that executing esxcli commands from SSH can place a system in an unsupported state, according to VMware support.
As with all commands that are copied and pasted from the Internet, we recommend using them carefully and judiciously.
Remember to perform these actions when your host is in maintenance mode and no virtual machines are running.
WMware and Kinamo
Kinamo is based on VMware vSphere for server virtualization and Horizon for desktop virtualization. VMware offers scalability, high availability and a powerful server virtualization stack which, in turn, offers our customers the highest possible uptime, all in a cost-effective package. Business-focused management capabilities from an infrastructure perspective enable us to deliver high-quality service and reduce costs where necessary.
Kinamo has over 18 years' experience in cloud hosting, managed services, DevOps and development. With over 15 years' experience in VMware server virtualization, we're sure we know the ins and outs. We're not a traditional VMware solution provider. As with all Kinamo services, we like projects that require an "out of the box" approach. We're proud to offer our customers accessible, expert support from our dedicated team here in Antwerp, Belgium.
Do you have a question about this article? Or wondering if this accessible and expert support in Antwerp is real? Send us an e-mail to discuss.
