Archive for the ‘Cisco’ Category
“All aboard” the Nexus 1000v 1.5 train
Nexus 1000v release 4.2(1)SV1(5.1) finally hit the streets on Tuesday. You’re likely to see tons of information in the coming months regarding this release and its support of new features like VXLAN and vCloud Director. I’m happy to see these new features and can’t wait to test them, but I’ve got the 1.5 release on the “fast track” for production release for other reasons. The most important of these reasons is stability. For all intents and purposes, the Nexus 1000v 1.4 release train has been the most stable it its history and, admittedly, most bugs are “edge cases” that didn’t fall out of beta testing. Unfortunately for my environment, 1.4 included at least one “show stopping” bug, and a few others that were complete annoyances. The good news is that the 1000v team at Cisco is extremely open to feedback and dedicated to “squashing” bugs as quickly as possible. Much to my satisfaction, a quick glance at the 1.5 release notes this week revealed that all but one of my plaguing bugs has been resolved. If you haven’t seen these four below, take a hard look at your environment to see if you may need to implement a workaround or keep your fingers crossed until you can upgrade:
CSCtu05038 – Time change may result in unnecessary transmission of LACP PDUs. This is the showstopper. Although the bug title is accurate, it falls short of telling you that your upstream switch will be fatally flooded with those PDUs. If you’ve ever experienced a “network storm” you’ll understand what this bug can do to your environment. For me, this meant setting ESXi host time before adding them to the 1000v, and leaving NTP disabled.
CSCtl70759 – Addition of new NIC to port-channel fails if profile has queueing policy. I wrote about using WFQ QoS on the 1000v in an earlier post and it works great! Well, with one known exception. Thanks to this bug, after adding a queueing policy to an uplink port-profile, you’re prevented from properly adding any additional hosts utilizing port-channeled uplinks to the 1000v. Due to queuing policy limits in the 1.4 release, the second uplink in the port-channel will fail to properly apply the queuing policy. This one is also a bit tricky because it doesn’t expose itself anywhere within the VI client and host joins appear to complete successfully to the unsuspecting. The workaround before upgrading to 1.5 is to remove the queuing policy from the port-profile while adding hosts and to then add it back.
CSCsy75230 – Add support for the show port-channel load-bal forwarding-path command. So this one isn’t so much a bug, but rather a feature that’s been missing. It finally brings the 1000v up to standard with other NX-OS devices such that you can use this command to determine how traffic flows will be placed on specific uplinks in a port-channel. I wrote about this command and how to use it on the 5k platform a while back in this post.
CSCts58379 – The show running-config command displays the svs license volatile command as svs volatile license. The beauty of switch text configuration is that you can easily back it up, copy it, make changes, and re-apply it, right? Well, not in the case of this bug. Unfortunately, if you chose to use volatile licensing, copying the command from running-config meant that it wouldn’t properly apply if you used it to configure another switch. Its a simple case of command reversal. I’m glad this ones finally fixed.
Another Nexus 1000v WFQ QoS Example
Louis Watta did a great job of putting together an example of using Weighted Fair Queuing on the Nexus 1000v in his whitepaper on the subject a while back. In his example, and the example in the 1000v configuration guide, VMware Network I/O Control API dependent class-maps are used to classify traffic. This method definitely works, but classifies your traffic only within the 1000v. My preferred method is to actually utilize the CoS tagging capability of the 1000v so that QoS policies based on the tags can be used on upstream devices like the Nexus 5k and 7k as well. I’ll update this post with further explanation of each section, but for now here’s an example code snippet:
policy-map type qos class-cos1
description [Server & VMware VM traffic]
class class-default
set cos 1
policy-map type qos class-cos2
description [VMware vMotion traffic]
class class-default
set cos 2
policy-map type qos class-cos3
description [Storage traffic]
class class-default
set cos 3
policy-map type qos class-cos4
description [Cluster & VMware FT traffic]
class class-default
set cos 4
policy-map type qos class-cos5
description [Management traffic]
class class-default
set cos 5
class-map type queuing match-any class-que-cos0
match cos 0
class-map type queuing match-any class-que-cos1
match cos 1
class-map type queuing match-any class-que-cos2
match cos 2
class-map type queuing match-any class-que-cos3
match cos 3
class-map type queuing match-any class-que-cos4
match cos 4
class-map type queuing match-any class-que-cos5
match cos 5
policy-map type queuing policy-out-que-all
class type queuing class-que-cos0
bandwidth percent 10
class type queuing class-que-cos1
bandwidth percent 10
class type queuing class-que-cos2
bandwidth percent 10
class type queuing class-que-cos3
bandwidth percent 40
class type queuing class-que-cos4
bandwidth percent 20
class type queuing class-que-cos5
bandwidth percent 10
port-profile Uplink
service-policy type queuing output policy-out-que-all
Nexus 55xx & 50×0 vPC incompatibility
A Google search doesn’t return any definitive results regarding vPC compatibility between the older Nexus 50×0 series and the newer 55xx series so I thought I’d pop up a post confirming that it indeed does NOT work. I recently ran into this issue while trying to non-disruptively swap out a 5010 switch pair with 5548s. At first, vPCs between the switches appear to only show a Type 2 inconsistency and come up. Unfortunately, at closer inspection, you’ll see that STP Bridge Assurance on the vPC peer-link will block all VLANs. This means that traffic entering one switch that needs to span the peer-link to traverse upstream will be blocked. Thus, connectivity is sporadic at best. Official word from Cisco is that vPC between the two series doesn’t work because vPC was changed in the 5500 series to allow increases in VLAN scalability, etc. At closer inspection, the documentation does note the incompatibility. Maybe the document writers can put it in one of those handy “Note: (with a pencil)” boxes in a future document release.
Nexus 1010 HA configuration failure
I ran into a very weird issue today with a customer while setting up a pair of Cisco Nexus 1010 Virtual Services Appliances. If you’re not familiar, the appliance is a Cisco UCS C-Series server with a Cisco tuned KVM Linux hypervisor image. The appliance can run up to four Nexus 1000v VSMs (or other Virtual Service Blades) and is marketed as the hardware alternative to the VMware virtual machine based VSM option.
In our situation, after setting up redundant hardware appliances and building the initial configuration, we were unable to enable the secondary VSM in an HA configuration. Normally you’d simply type enable secondary within the Virtual Service Blade context to deploy the redundant supervisor. However, after entering the command, the secondary status continuously showed “VSB NOT PRESENT.” Review of the log on the secondary physical appliance revealed a “%CPPA_MGR-5-REMOTE_VB_IMG_FAILURE: Could not verify image in standby for virtual service blade 2 creation: image check operation timed out” message. As it turns out this is a known bug in the 4.0(4) SP1(1) software release of the 1010 and is fully described in the release notes. The issue occurs when you add a description to the configuration of the Virtual Service Blade. The fix is to rebuild the VSM pair without using a description. A check of the current bug report reveals that the issue is fixed in the next Nexus 1010 software release. I’ll update this post when it is released.
NX-OS port-channel hashing
While working on the lab configuration for my upcoming VMware 10GbE networking performance comparison whitepaper I came across an interesting scenario. To test total throughput on dual port NICs, I needed to distribute VM load among the two ports using link aggregation. I knew IP hashing was the way to go, but I wanted to make sure that my VMs were configured with the appropriate IP addresses such that the load would actually be evenly distributed. Luckily, Scott Lowe, Ken Cline, and many others have written numerous articles about the VMware IP hashing algorithm. After carefully doing my math and setting my guest IP addresses, I found that ESX evenly distributed the transmit load onto the Nexus 5010 switch. However, even though I had configured the 5010 for IP hashing using the port-channel load-balance ethernet source-destination-ip command, uneven egress traffic distribution (70/30) told me Cisco was obviously using a different hashing algorithm. Scott had mentioned how to test the result of the hash calculation on the Catalyst line of switches in his earlier post, but the test etherchannel load-balance command no longer works in the NX-OS. Luckily, with some quick ? querying around the CLI, I finally uncovered the new command. You can find the full details of the command in the Nexus 5000 Series Command Reference, and here’s an example of how I used it:
show port-channel load-balance forwarding-path interface port-channel 2 vlan 902 dst-ip 192.168.110.212 src-ip 192.168.110.200
The command gives you the hash result along with the ethernet port that the packets will be placed. It took some work, but using the command I finally came up with a combination of source and destination addresses that created an even distribution of traffic on ingress and egress using both the VMware and Cisco algorithms.
Cisco UCS C-Series, VMware & RAID
A colleague of mine reminded me of a hardware configuration consideration regarding Cisco UCS C-Series servers today so I thought I’d share. As specifically noted in the Cisco UCS C-Series Servers VMware Installation Guide, VMware does not support the use of the onboard Intel ICH10R based software RAID controller. This isn’t anything new, as VMware has never supported software RAID. Just make sure you add an LSI 1064-based controller mezzanine card to C200 and C210 servers or an LSI 3081-based controller to UCS C250 servers when you order.