Archive for the ‘VMWare’ Category
“All aboard” the Nexus 1000v 1.5 train
Nexus 1000v release 4.2(1)SV1(5.1) finally hit the streets on Tuesday. You’re likely to see tons of information in the coming months regarding this release and its support of new features like VXLAN and vCloud Director. I’m happy to see these new features and can’t wait to test them, but I’ve got the 1.5 release on the “fast track” for production release for other reasons. The most important of these reasons is stability. For all intents and purposes, the Nexus 1000v 1.4 release train has been the most stable it its history and, admittedly, most bugs are “edge cases” that didn’t fall out of beta testing. Unfortunately for my environment, 1.4 included at least one “show stopping” bug, and a few others that were complete annoyances. The good news is that the 1000v team at Cisco is extremely open to feedback and dedicated to “squashing” bugs as quickly as possible. Much to my satisfaction, a quick glance at the 1.5 release notes this week revealed that all but one of my plaguing bugs has been resolved. If you haven’t seen these four below, take a hard look at your environment to see if you may need to implement a workaround or keep your fingers crossed until you can upgrade:
CSCtu05038 – Time change may result in unnecessary transmission of LACP PDUs. This is the showstopper. Although the bug title is accurate, it falls short of telling you that your upstream switch will be fatally flooded with those PDUs. If you’ve ever experienced a “network storm” you’ll understand what this bug can do to your environment. For me, this meant setting ESXi host time before adding them to the 1000v, and leaving NTP disabled.
CSCtl70759 – Addition of new NIC to port-channel fails if profile has queueing policy. I wrote about using WFQ QoS on the 1000v in an earlier post and it works great! Well, with one known exception. Thanks to this bug, after adding a queueing policy to an uplink port-profile, you’re prevented from properly adding any additional hosts utilizing port-channeled uplinks to the 1000v. Due to queuing policy limits in the 1.4 release, the second uplink in the port-channel will fail to properly apply the queuing policy. This one is also a bit tricky because it doesn’t expose itself anywhere within the VI client and host joins appear to complete successfully to the unsuspecting. The workaround before upgrading to 1.5 is to remove the queuing policy from the port-profile while adding hosts and to then add it back.
CSCsy75230 – Add support for the show port-channel load-bal forwarding-path command. So this one isn’t so much a bug, but rather a feature that’s been missing. It finally brings the 1000v up to standard with other NX-OS devices such that you can use this command to determine how traffic flows will be placed on specific uplinks in a port-channel. I wrote about this command and how to use it on the 5k platform a while back in this post.
CSCts58379 – The show running-config command displays the svs license volatile command as svs volatile license. The beauty of switch text configuration is that you can easily back it up, copy it, make changes, and re-apply it, right? Well, not in the case of this bug. Unfortunately, if you chose to use volatile licensing, copying the command from running-config meant that it wouldn’t properly apply if you used it to configure another switch. Its a simple case of command reversal. I’m glad this ones finally fixed.
Nexus 1010 HA configuration failure
I ran into a very weird issue today with a customer while setting up a pair of Cisco Nexus 1010 Virtual Services Appliances. If you’re not familiar, the appliance is a Cisco UCS C-Series server with a Cisco tuned KVM Linux hypervisor image. The appliance can run up to four Nexus 1000v VSMs (or other Virtual Service Blades) and is marketed as the hardware alternative to the VMware virtual machine based VSM option.
In our situation, after setting up redundant hardware appliances and building the initial configuration, we were unable to enable the secondary VSM in an HA configuration. Normally you’d simply type enable secondary within the Virtual Service Blade context to deploy the redundant supervisor. However, after entering the command, the secondary status continuously showed “VSB NOT PRESENT.” Review of the log on the secondary physical appliance revealed a “%CPPA_MGR-5-REMOTE_VB_IMG_FAILURE: Could not verify image in standby for virtual service blade 2 creation: image check operation timed out” message. As it turns out this is a known bug in the 4.0(4) SP1(1) software release of the 1010 and is fully described in the release notes. The issue occurs when you add a description to the configuration of the Virtual Service Blade. The fix is to rebuild the VSM pair without using a description. A check of the current bug report reveals that the issue is fixed in the next Nexus 1010 software release. I’ll update this post when it is released.
Customizing vmxnet3 MTU in Windows
Normally this is something you either already know how to do, or could figure out pretty quickly. Pop open the properties for the adapter, go to the Advanced tab and find the Jumbo Packet setting. Simple enough, right? Well, not exactly. As you see below, the setting allows just two values, “Standard 1500” and “Jumbo 9000.”
What if the physical network adapter in your host supports jumbo frames, but not at 9000 bytes? This is currently more common that you may think with 10GbE network adapters and CNAs. One such example is the Emulex OneConnect OCe10102-N. As stated in the Emulex Drivers for VMware ESX 4.0 guide, the maximum supported MTU for this adapter is 8179 bytes. Luckily, even though the vmxnet3 driver Advanced tab exposes only two values, the MTU can be fine-tuned in the Windows registry. I’ve only successfully modified the value in Windows Server 2008 R2, but I checked and the registry location seems to be the same in Server 2003 as well. The exact registry sub-key will vary based on the adapters in your system, but you can easily find the correct one by taking a look at the value for DriverDesc in each of the numbered sub-keys. You’ll find the settings at the following location in the registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}\
Once you find the vmxnet3 adapter, modify the *JumboPacket string value to the proper MTU. Then just disable and re-enable the adapter and you should now be running with the correct customized frame size.
NX-OS port-channel hashing
While working on the lab configuration for my upcoming VMware 10GbE networking performance comparison whitepaper I came across an interesting scenario. To test total throughput on dual port NICs, I needed to distribute VM load among the two ports using link aggregation. I knew IP hashing was the way to go, but I wanted to make sure that my VMs were configured with the appropriate IP addresses such that the load would actually be evenly distributed. Luckily, Scott Lowe, Ken Cline, and many others have written numerous articles about the VMware IP hashing algorithm. After carefully doing my math and setting my guest IP addresses, I found that ESX evenly distributed the transmit load onto the Nexus 5010 switch. However, even though I had configured the 5010 for IP hashing using the port-channel load-balance ethernet source-destination-ip command, uneven egress traffic distribution (70/30) told me Cisco was obviously using a different hashing algorithm. Scott had mentioned how to test the result of the hash calculation on the Catalyst line of switches in his earlier post, but the test etherchannel load-balance command no longer works in the NX-OS. Luckily, with some quick ? querying around the CLI, I finally uncovered the new command. You can find the full details of the command in the Nexus 5000 Series Command Reference, and here’s an example of how I used it:
show port-channel load-balance forwarding-path interface port-channel 2 vlan 902 dst-ip 192.168.110.212 src-ip 192.168.110.200
The command gives you the hash result along with the ethernet port that the packets will be placed. It took some work, but using the command I finally came up with a combination of source and destination addresses that created an even distribution of traffic on ingress and egress using both the VMware and Cisco algorithms.
vSphere 4 U2, ‘Troubleshooting Mode’ & esxupdate
I ran into an interesting issue the other day on an Update 2 host where I needed to uninstall a driver update that was causing a PSOD at boot (in relation to this issue, actually). I proceeded to boot into “Troubleshooting Mode” just as I’ve done many times in the past to uninstall the update. Unfortunately, when I tried to uninstall the driver using esxupdate, the host just hung. After waiting a few minutes, I eventually escaped the process using CTRL-C only to be presented with an error explaining that esxupdate couldn’t determine if the host was in maintenance mode. Now, in Troubleshooting Mode we could care less as to maintenance mode status. In fact, the issue occurs because the vmkernel components required for vimsh to run and check host status aren’t even available. This is pretty frustrating, but luckily for me I was able to manually run “rpm -e” to uninstall the driver. It appears at first glance that this potential issue slipped passed QA testing. Let’s hope that in the future, esxupdate either ignores host status in Troubleshooting Mode or a switch is provided to allow bypassing the maintenance mode requirement.
NetApp unified SRA requires NAS and SAN RBAC
Rawlinson Rivera blogged about a known issue with the 1.4.3 version of the NetApp SRA a while back. If you’re not familiar, this version marries the seperate NAS and SAN versions that previously existed. If you read through the comments you’ll discover that the documented issue occurs in NFS only environments where no iSCSI or FCP LUNs exist. As explained in the NetApp knowledgebase article (NOW access required), the workaround is to create a dummy VMware igroup:
igroup create -i -t vmware sra_dummy_igroup
After recently implementing the workaround, we continued to receive a LUN discovery error until we finally remembered that the filer permissions required for the older NAS and SAN SRAs were different. During our original setup, we had configured role based access control with the requirements for NAS only. Unfortunately, to complete a successful discovery with the new unified SRA, it appears that RBAC must now include the permissions for both NAS and SAN, even if you’re not using one or the other. Luckily for us, a quick look back at the NetApp SRA RBAC rights knowledgebase article (NOW access required) got us back on track.
Cisco UCS C-Series, VMware & RAID
A colleague of mine reminded me of a hardware configuration consideration regarding Cisco UCS C-Series servers today so I thought I’d share. As specifically noted in the Cisco UCS C-Series Servers VMware Installation Guide, VMware does not support the use of the onboard Intel ICH10R based software RAID controller. This isn’t anything new, as VMware has never supported software RAID. Just make sure you add an LSI 1064-based controller mezzanine card to C200 and C210 servers or an LSI 3081-based controller to UCS C250 servers when you order.
vSphere 10GbE networking
I began work this week on a vSphere 10GbE networking performance whitepaper that I hope to publish in time for VMworld in late August. The whitepaper will focus on specific tunable parameters and out of the box performance of 10GbE NICs and CNAs from a myriad of manufacturers. While doing some research in preparation, I stumbled across what I believe to be a little known Cisco whitepaper here. The whitepaper is extremely informative and covers very VM centric information that I think many admins will find useful. I’ll certainly use it as a basis for building my VMs to test with. Stay tuned for a series of posts as I put my whitepaper together. In the meantime, check out the Cisco doc for an interesting read.
update to Nexus 1000v, IP-based storage & jumbo frames
There’s been an issue for a while for 1000v users running jumbo frames on storage networks. The problem is that the 1000v VEMs do not retain MTU settings and boot with a default of 1500. Until the VSM is up and communicates with the VEM to change the MTU, IP-based storage is not available. If my VSM is on this storage, then this creates the “chicken and egg” scenario. For further details, check out Trey Layton’s explanation. NetApp offers an interesting workaround, and I’ve been having customers put one VSM in an HA pair on local storage to avoid the scenario. Cisco had told us that there was a fix coming, and apparently it has arrived. I recently installed 1000v version SV1(3) for a customer and noticed there’s now a command called “system mtu” that can be applied to ethernet port profiles. Just like you specify “system” vlans that can come up without the presence of the VSM, it appears that “system mtu” allows you to set the MTU of those VLANs as well. Finally, no more workarounds.
Configuring the NetApp SRA to use SSL
Cormac Hogan produced a great Proven Practice document a while back detailing the steps needed for configuring the NetApp SRA to use SSL communication instead of unencrypted http. The steps involve using PPM to download the Perl libraries needed for SSL communication. If the 27 page document seems a little daunting, and RTFM is not for you, then how about the two steps below:
1. Download the libraries here and drop them into C:\Program Files\VMware\VMware vCenter Site Recovery Manager\external\perl-5.8.8
2. Modify the ontap_config.txt file found in C:\Program Files\VMware\VMware vCenter Site Recovery Manager\scripts\SAN\ONTAP_NAS and set the SSL option to on.
Now, obviously this assumes you’ve already enabled SSL Secure Admin on your filer, etc. If not, take a look at Cormac’s guide. Either way, hopefully the libraries above will save you some time. 