Archive for the ‘Networking’ Category
“All aboard” the Nexus 1000v 1.5 train
Nexus 1000v release 4.2(1)SV1(5.1) finally hit the streets on Tuesday. You’re likely to see tons of information in the coming months regarding this release and its support of new features like VXLAN and vCloud Director. I’m happy to see these new features and can’t wait to test them, but I’ve got the 1.5 release on the “fast track” for production release for other reasons. The most important of these reasons is stability. For all intents and purposes, the Nexus 1000v 1.4 release train has been the most stable it its history and, admittedly, most bugs are “edge cases” that didn’t fall out of beta testing. Unfortunately for my environment, 1.4 included at least one “show stopping” bug, and a few others that were complete annoyances. The good news is that the 1000v team at Cisco is extremely open to feedback and dedicated to “squashing” bugs as quickly as possible. Much to my satisfaction, a quick glance at the 1.5 release notes this week revealed that all but one of my plaguing bugs has been resolved. If you haven’t seen these four below, take a hard look at your environment to see if you may need to implement a workaround or keep your fingers crossed until you can upgrade:
CSCtu05038 – Time change may result in unnecessary transmission of LACP PDUs. This is the showstopper. Although the bug title is accurate, it falls short of telling you that your upstream switch will be fatally flooded with those PDUs. If you’ve ever experienced a “network storm” you’ll understand what this bug can do to your environment. For me, this meant setting ESXi host time before adding them to the 1000v, and leaving NTP disabled.
CSCtl70759 – Addition of new NIC to port-channel fails if profile has queueing policy. I wrote about using WFQ QoS on the 1000v in an earlier post and it works great! Well, with one known exception. Thanks to this bug, after adding a queueing policy to an uplink port-profile, you’re prevented from properly adding any additional hosts utilizing port-channeled uplinks to the 1000v. Due to queuing policy limits in the 1.4 release, the second uplink in the port-channel will fail to properly apply the queuing policy. This one is also a bit tricky because it doesn’t expose itself anywhere within the VI client and host joins appear to complete successfully to the unsuspecting. The workaround before upgrading to 1.5 is to remove the queuing policy from the port-profile while adding hosts and to then add it back.
CSCsy75230 – Add support for the show port-channel load-bal forwarding-path command. So this one isn’t so much a bug, but rather a feature that’s been missing. It finally brings the 1000v up to standard with other NX-OS devices such that you can use this command to determine how traffic flows will be placed on specific uplinks in a port-channel. I wrote about this command and how to use it on the 5k platform a while back in this post.
CSCts58379 – The show running-config command displays the svs license volatile command as svs volatile license. The beauty of switch text configuration is that you can easily back it up, copy it, make changes, and re-apply it, right? Well, not in the case of this bug. Unfortunately, if you chose to use volatile licensing, copying the command from running-config meant that it wouldn’t properly apply if you used it to configure another switch. Its a simple case of command reversal. I’m glad this ones finally fixed.
Nexus 1010 HA configuration failure
I ran into a very weird issue today with a customer while setting up a pair of Cisco Nexus 1010 Virtual Services Appliances. If you’re not familiar, the appliance is a Cisco UCS C-Series server with a Cisco tuned KVM Linux hypervisor image. The appliance can run up to four Nexus 1000v VSMs (or other Virtual Service Blades) and is marketed as the hardware alternative to the VMware virtual machine based VSM option.
In our situation, after setting up redundant hardware appliances and building the initial configuration, we were unable to enable the secondary VSM in an HA configuration. Normally you’d simply type enable secondary within the Virtual Service Blade context to deploy the redundant supervisor. However, after entering the command, the secondary status continuously showed “VSB NOT PRESENT.” Review of the log on the secondary physical appliance revealed a “%CPPA_MGR-5-REMOTE_VB_IMG_FAILURE: Could not verify image in standby for virtual service blade 2 creation: image check operation timed out” message. As it turns out this is a known bug in the 4.0(4) SP1(1) software release of the 1010 and is fully described in the release notes. The issue occurs when you add a description to the configuration of the Virtual Service Blade. The fix is to rebuild the VSM pair without using a description. A check of the current bug report reveals that the issue is fixed in the next Nexus 1010 software release. I’ll update this post when it is released.
Customizing vmxnet3 MTU in Windows
Normally this is something you either already know how to do, or could figure out pretty quickly. Pop open the properties for the adapter, go to the Advanced tab and find the Jumbo Packet setting. Simple enough, right? Well, not exactly. As you see below, the setting allows just two values, “Standard 1500” and “Jumbo 9000.”
What if the physical network adapter in your host supports jumbo frames, but not at 9000 bytes? This is currently more common that you may think with 10GbE network adapters and CNAs. One such example is the Emulex OneConnect OCe10102-N. As stated in the Emulex Drivers for VMware ESX 4.0 guide, the maximum supported MTU for this adapter is 8179 bytes. Luckily, even though the vmxnet3 driver Advanced tab exposes only two values, the MTU can be fine-tuned in the Windows registry. I’ve only successfully modified the value in Windows Server 2008 R2, but I checked and the registry location seems to be the same in Server 2003 as well. The exact registry sub-key will vary based on the adapters in your system, but you can easily find the correct one by taking a look at the value for DriverDesc in each of the numbered sub-keys. You’ll find the settings at the following location in the registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}\
Once you find the vmxnet3 adapter, modify the *JumboPacket string value to the proper MTU. Then just disable and re-enable the adapter and you should now be running with the correct customized frame size.
NX-OS port-channel hashing
While working on the lab configuration for my upcoming VMware 10GbE networking performance comparison whitepaper I came across an interesting scenario. To test total throughput on dual port NICs, I needed to distribute VM load among the two ports using link aggregation. I knew IP hashing was the way to go, but I wanted to make sure that my VMs were configured with the appropriate IP addresses such that the load would actually be evenly distributed. Luckily, Scott Lowe, Ken Cline, and many others have written numerous articles about the VMware IP hashing algorithm. After carefully doing my math and setting my guest IP addresses, I found that ESX evenly distributed the transmit load onto the Nexus 5010 switch. However, even though I had configured the 5010 for IP hashing using the port-channel load-balance ethernet source-destination-ip command, uneven egress traffic distribution (70/30) told me Cisco was obviously using a different hashing algorithm. Scott had mentioned how to test the result of the hash calculation on the Catalyst line of switches in his earlier post, but the test etherchannel load-balance command no longer works in the NX-OS. Luckily, with some quick ? querying around the CLI, I finally uncovered the new command. You can find the full details of the command in the Nexus 5000 Series Command Reference, and here’s an example of how I used it:
show port-channel load-balance forwarding-path interface port-channel 2 vlan 902 dst-ip 192.168.110.212 src-ip 192.168.110.200
The command gives you the hash result along with the ethernet port that the packets will be placed. It took some work, but using the command I finally came up with a combination of source and destination addresses that created an even distribution of traffic on ingress and egress using both the VMware and Cisco algorithms.
vSphere 10GbE networking
I began work this week on a vSphere 10GbE networking performance whitepaper that I hope to publish in time for VMworld in late August. The whitepaper will focus on specific tunable parameters and out of the box performance of 10GbE NICs and CNAs from a myriad of manufacturers. While doing some research in preparation, I stumbled across what I believe to be a little known Cisco whitepaper here. The whitepaper is extremely informative and covers very VM centric information that I think many admins will find useful. I’ll certainly use it as a basis for building my VMs to test with. Stay tuned for a series of posts as I put my whitepaper together. In the meantime, check out the Cisco doc for an interesting read.
update to Nexus 1000v, IP-based storage & jumbo frames
There’s been an issue for a while for 1000v users running jumbo frames on storage networks. The problem is that the 1000v VEMs do not retain MTU settings and boot with a default of 1500. Until the VSM is up and communicates with the VEM to change the MTU, IP-based storage is not available. If my VSM is on this storage, then this creates the “chicken and egg” scenario. For further details, check out Trey Layton’s explanation. NetApp offers an interesting workaround, and I’ve been having customers put one VSM in an HA pair on local storage to avoid the scenario. Cisco had told us that there was a fix coming, and apparently it has arrived. I recently installed 1000v version SV1(3) for a customer and noticed there’s now a command called “system mtu” that can be applied to ethernet port profiles. Just like you specify “system” vlans that can come up without the presence of the VSM, it appears that “system mtu” allows you to set the MTU of those VLANs as well. Finally, no more workarounds.
HP ProCurve cross-stack EtherChannel
UPDATE 11/24/2010: After comments from a few readers and additional testing it appears that the ProCurve dt-lacp “trunks” do attempt to negotiate formal dynamic LACP. Oddly enough, typically, the “trunks” come up and forward traffic on both links even if LACP negotiation fails. This fact led me to my original findings and subsequently this blog post. However, after additional testing, I did find scenarios where the ports would sometimes fail if they both attempted to come online at the same time. Given this fact, I cannot recommend the configuration in the original post. Ironically, the only way ProCurve dt-lacp would be supported with ESX is in conjunction with the Cisco Nexus 1000v. The Nexus 1000v does support formal LACP port-channels and will work just fine in a dt-lacp configuration. Since original publication, I’ve also successfully tested HP LeftHand nodes with 10GbE networking and LACP teaming in conjunction with dt-lacp. More details to follow.
I’ve been working on a new vSphere deployment for a customer and have had to learn a few ProCurve tricks this week. The first of which is how to do what I’d normally refer to as “cross-stack EtherChannel.” It was really hard to track down this information so I thought I’d post it up (more info in the user guide here). First of all, HP refers to this feature (connecting a server via 802.3ad to multiple switches) as “Distributed Trunking.” I really hate the way certain manufacturers refer to link aggregation as trunking, but more on that in another post! The first thing you’ll want to make sure of is that you have at least ProCurve software release K14. Apparently, this release introduced the feature. HP switches don’t have stacking ports, but rather can be connected via standard uplinks. In this case, we connected a pair of ProCurve 6600′s via two “trunked” (static 802.3ad) ports. Once that is done, we configure the link as an “InterSwitch-Connect” (ISC) and we’re ready to begin configuring our server “distributed trunks.” Note that neither the ISC nor the server distributed trunks can have dynamic LACP enabled (this is fine since VMware doesn’t support dynamic LACP anyway). As the final step, we simply add the ports to the same numbered trunk (ie Trk5) on both switches and set the type to “dt-lacp.”
Here’s a rundown of the specific commands:
ProCurve Switch 1(config)# trunk 23-24 trk1 lacp
ProCurve Switch 1(config)# switch-interconnect trk1
ProCurve Switch 1(config)# trunk 1 trk5 dt-lacp
ProCurve Switch 2(config)# trunk 23-24 trk1 lacp
ProCurve Switch 2(config)# switch-interconnect trk1
ProCurve Switch 2(config)# trunk 1 trk5 dt-lacp