vNephos.com

“All aboard” the Nexus 1000v 1.5 train

Andy Daniel — Sat, 04 Feb 2012 14:28:00 +0000

Nexus 1000v release 4.2(1)SV1(5.1) finally hit the streets on Tuesday. You’re likely to see tons of information in the coming months regarding this release and its support of new features like VXLAN and vCloud Director. I’m happy to see these new features and can’t wait to test them, but I’ve got the 1.5 release on the “fast track” for production release for other reasons. The most important of these reasons is stability. For all intents and purposes, the Nexus 1000v 1.4 release train has been the most stable it its history and, admittedly, most bugs are “edge cases” that didn’t fall out of beta testing. Unfortunately for my environment, 1.4 included at least one “show stopping” bug, and a few others that were complete annoyances. The good news is that the 1000v team at Cisco is extremely open to feedback and dedicated to “squashing” bugs as quickly as possible. Much to my satisfaction, a quick glance at the 1.5 release notes this week revealed that all but one of my plaguing bugs has been resolved. If you haven’t seen these four below, take a hard look at your environment to see if you may need to implement a workaround or keep your fingers crossed until you can upgrade:

CSCtu05038 – Time change may result in unnecessary transmission of LACP PDUs. This is the showstopper. Although the bug title is accurate, it falls short of telling you that your upstream switch will be fatally flooded with those PDUs. If you’ve ever experienced a “network storm” you’ll understand what this bug can do to your environment. For me, this meant setting ESXi host time before adding them to the 1000v, and leaving NTP disabled.

CSCtl70759 – Addition of new NIC to port-channel fails if profile has queueing policy. I wrote about using WFQ QoS on the 1000v in an earlier post and it works great! Well, with one known exception. Thanks to this bug, after adding a queueing policy to an uplink port-profile, you’re prevented from properly adding any additional hosts utilizing port-channeled uplinks to the 1000v. Due to queuing policy limits in the 1.4 release, the second uplink in the port-channel will fail to properly apply the queuing policy. This one is also a bit tricky because it doesn’t expose itself anywhere within the VI client and host joins appear to complete successfully to the unsuspecting. The workaround before upgrading to 1.5 is to remove the queuing policy from the port-profile while adding hosts and to then add it back.

CSCsy75230 – Add support for the show port-channel load-bal forwarding-path command. So this one isn’t so much a bug, but rather a feature that’s been missing. It finally brings the 1000v up to standard with other NX-OS devices such that you can use this command to determine how traffic flows will be placed on specific uplinks in a port-channel. I wrote about this command and how to use it on the 5k platform a while back in this post.

CSCts58379 – The show running-config command displays the svs license volatile command as svs volatile license. The beauty of switch text configuration is that you can easily back it up, copy it, make changes, and re-apply it, right? Well, not in the case of this bug. Unfortunately, if you chose to use volatile licensing, copying the command from running-config meant that it wouldn’t properly apply if you used it to configure another switch. Its a simple case of command reversal. I’m glad this ones finally fixed.

Another Nexus 1000v WFQ QoS Example

Andy Daniel — Tue, 29 Nov 2011 03:14:21 +0000

Louis Watta did a great job of putting together an example of using Weighted Fair Queuing on the Nexus 1000v in his whitepaper on the subject a while back. In his example, and the example in the 1000v configuration guide, VMware Network I/O Control API dependent class-maps are used to classify traffic. This method definitely works, but classifies your traffic only within the 1000v. My preferred method is to actually utilize the CoS tagging capability of the 1000v so that QoS policies based on the tags can be used on upstream devices like the Nexus 5k and 7k as well. I’ll update this post with further explanation of each section, but for now here’s an example code snippet:

policy-map type qos class-cos1 description [Server & VMware VM traffic] class class-default set cos 1 policy-map type qos class-cos2 description [VMware vMotion traffic] class class-default set cos 2 policy-map type qos class-cos3 description [Storage traffic] class class-default set cos 3 policy-map type qos class-cos4 description [Cluster & VMware FT traffic] class class-default set cos 4 policy-map type qos class-cos5 description [Management traffic] class class-default set cos 5


class-map type queuing match-any class-que-cos0

  match cos 0

class-map type queuing match-any class-que-cos1

  match cos 1

class-map type queuing match-any class-que-cos2

  match cos 2

class-map type queuing match-any class-que-cos3

  match cos 3

class-map type queuing match-any class-que-cos4

  match cos 4

class-map type queuing match-any class-que-cos5

  match cos 5
policy-map type queuing policy-out-que-all

  class type queuing class-que-cos0

    bandwidth percent 10

  class type queuing class-que-cos1

    bandwidth percent 10

  class type queuing class-que-cos2

    bandwidth percent 10

  class type queuing class-que-cos3

    bandwidth percent 40

  class type queuing class-que-cos4

    bandwidth percent 20

  class type queuing class-que-cos5

    bandwidth percent 10

port-profile Uplink service-policy type queuing output policy-out-que-all

Nexus 55xx & 50×0 vPC incompatibility

Andy Daniel — Thu, 07 Apr 2011 13:59:04 +0000

A Google search doesn’t return any definitive results regarding vPC compatibility between the older Nexus 50×0 series and the newer 55xx series so I thought I’d pop up a post confirming that it indeed does NOT work. I recently ran into this issue while trying to non-disruptively swap out a 5010 switch pair with 5548s. At first, vPCs between the switches appear to only show a Type 2 inconsistency and come up. Unfortunately, at closer inspection, you’ll see that STP Bridge Assurance on the vPC peer-link will block all VLANs. This means that traffic entering one switch that needs to span the peer-link to traverse upstream will be blocked. Thus, connectivity is sporadic at best. Official word from Cisco is that vPC between the two series doesn’t work because vPC was changed in the 5500 series to allow increases in VLAN scalability, etc. At closer inspection, the documentation does note the incompatibility. Maybe the document writers can put it in one of those handy “Note: (with a pencil)” boxes in a future document release.

Nexus 1010 HA configuration failure

Andy Daniel — Wed, 27 Oct 2010 01:16:48 +0000

I ran into a very weird issue today with a customer while setting up a pair of Cisco Nexus 1010 Virtual Services Appliances. If you’re not familiar, the appliance is a Cisco UCS C-Series server with a Cisco tuned KVM Linux hypervisor image. The appliance can run up to four Nexus 1000v VSMs (or other Virtual Service Blades) and is marketed as the hardware alternative to the VMware virtual machine based VSM option.

In our situation, after setting up redundant hardware appliances and building the initial configuration, we were unable to enable the secondary VSM in an HA configuration. Normally you’d simply type enable secondary within the Virtual Service Blade context to deploy the redundant supervisor. However, after entering the command, the secondary status continuously showed “VSB NOT PRESENT.” Review of the log on the secondary physical appliance revealed a “%CPPA_MGR-5-REMOTE_VB_IMG_FAILURE: Could not verify image in standby for virtual service blade 2 creation: image check operation timed out” message. As it turns out this is a known bug in the 4.0(4) SP1(1) software release of the 1010 and is fully described in the release notes. The issue occurs when you add a description to the configuration of the Virtual Service Blade. The fix is to rebuild the VSM pair without using a description. A check of the current bug report reveals that the issue is fixed in the next Nexus 1010 software release. I’ll update this post when it is released.

Customizing vmxnet3 MTU in Windows

Andy Daniel — Sun, 18 Jul 2010 14:28:37 +0000

Normally this is something you either already know how to do, or could figure out pretty quickly. Pop open the properties for the adapter, go to the Advanced tab and find the Jumbo Packet setting. Simple enough, right? Well, not exactly. As you see below, the setting allows just two values, “Standard 1500” and “Jumbo 9000.”

What if the physical network adapter in your host supports jumbo frames, but not at 9000 bytes? This is currently more common that you may think with 10GbE network adapters and CNAs. One such example is the Emulex OneConnect OCe10102-N. As stated in the Emulex Drivers for VMware ESX 4.0 guide, the maximum supported MTU for this adapter is 8179 bytes. Luckily, even though the vmxnet3 driver Advanced tab exposes only two values, the MTU can be fine-tuned in the Windows registry. I’ve only successfully modified the value in Windows Server 2008 R2, but I checked and the registry location seems to be the same in Server 2003 as well. The exact registry sub-key will vary based on the adapters in your system, but you can easily find the correct one by taking a look at the value for DriverDesc in each of the numbered sub-keys. You’ll find the settings at the following location in the registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}\

Once you find the vmxnet3 adapter, modify the *JumboPacket string value to the proper MTU. Then just disable and re-enable the adapter and you should now be running with the correct customized frame size.

NX-OS port-channel hashing

Andy Daniel — Thu, 15 Jul 2010 02:57:05 +0000

While working on the lab configuration for my upcoming VMware 10GbE networking performance comparison whitepaper I came across an interesting scenario. To test total throughput on dual port NICs, I needed to distribute VM load among the two ports using link aggregation. I knew IP hashing was the way to go, but I wanted to make sure that my VMs were configured with the appropriate IP addresses such that the load would actually be evenly distributed. Luckily, Scott Lowe, Ken Cline, and many others have written numerous articles about the VMware IP hashing algorithm. After carefully doing my math and setting my guest IP addresses, I found that ESX evenly distributed the transmit load onto the Nexus 5010 switch. However, even though I had configured the 5010 for IP hashing using the port-channel load-balance ethernet source-destination-ip command, uneven egress traffic distribution (70/30) told me Cisco was obviously using a different hashing algorithm. Scott had mentioned how to test the result of the hash calculation on the Catalyst line of switches in his earlier post, but the test etherchannel load-balance command no longer works in the NX-OS. Luckily, with some quick ? querying around the CLI, I finally uncovered the new command. You can find the full details of the command in the Nexus 5000 Series Command Reference, and here’s an example of how I used it:

show port-channel load-balance forwarding-path interface port-channel 2 vlan 902 dst-ip 192.168.110.212 src-ip 192.168.110.200

The command gives you the hash result along with the ethernet port that the packets will be placed. It took some work, but using the command I finally came up with a combination of source and destination addresses that created an even distribution of traffic on ingress and egress using both the VMware and Cisco algorithms.

vSphere 4 U2, ‘Troubleshooting Mode’ & esxupdate

Andy Daniel — Tue, 06 Jul 2010 11:50:47 +0000

I ran into an interesting issue the other day on an Update 2 host where I needed to uninstall a driver update that was causing a PSOD at boot (in relation to this issue, actually). I proceeded to boot into “Troubleshooting Mode” just as I’ve done many times in the past to uninstall the update. Unfortunately, when I tried to uninstall the driver using esxupdate, the host just hung. After waiting a few minutes, I eventually escaped the process using CTRL-C only to be presented with an error explaining that esxupdate couldn’t determine if the host was in maintenance mode. Now, in Troubleshooting Mode we could care less as to maintenance mode status. In fact, the issue occurs because the vmkernel components required for vimsh to run and check host status aren’t even available. This is pretty frustrating, but luckily for me I was able to manually run “rpm -e” to uninstall the driver. It appears at first glance that this potential issue slipped passed QA testing. Let’s hope that in the future, esxupdate either ignores host status in Troubleshooting Mode or a switch is provided to allow bypassing the maintenance mode requirement.

NetApp unified SRA requires NAS and SAN RBAC

Andy Daniel — Wed, 30 Jun 2010 03:25:06 +0000

Rawlinson Rivera blogged about a known issue with the 1.4.3 version of the NetApp SRA a while back. If you’re not familiar, this version marries the seperate NAS and SAN versions that previously existed. If you read through the comments you’ll discover that the documented issue occurs in NFS only environments where no iSCSI or FCP LUNs exist. As explained in the NetApp knowledgebase article (NOW access required), the workaround is to create a dummy VMware igroup:

igroup create -i -t vmware sra_dummy_igroup

After recently implementing the workaround, we continued to receive a LUN discovery error until we finally remembered that the filer permissions required for the older NAS and SAN SRAs were different. During our original setup, we had configured role based access control with the requirements for NAS only. Unfortunately, to complete a successful discovery with the new unified SRA, it appears that RBAC must now include the permissions for both NAS and SAN, even if you’re not using one or the other. Luckily for us, a quick look back at the NetApp SRA RBAC rights knowledgebase article (NOW access required) got us back on track.

Cisco UCS C-Series, VMware & RAID

Andy Daniel — Fri, 25 Jun 2010 17:31:02 +0000

A colleague of mine reminded me of a hardware configuration consideration regarding Cisco UCS C-Series servers today so I thought I’d share. As specifically noted in the Cisco UCS C-Series Servers VMware Installation Guide, VMware does not support the use of the onboard Intel ICH10R based software RAID controller. This isn’t anything new, as VMware has never supported software RAID. Just make sure you add an LSI 1064-based controller mezzanine card to C200 and C210 servers or an LSI 3081-based controller to UCS C250 servers when you order.

vSphere 10GbE networking

Andy Daniel — Thu, 24 Jun 2010 13:17:36 +0000

I began work this week on a vSphere 10GbE networking performance whitepaper that I hope to publish in time for VMworld in late August. The whitepaper will focus on specific tunable parameters and out of the box performance of 10GbE NICs and CNAs from a myriad of manufacturers. While doing some research in preparation, I stumbled across what I believe to be a little known Cisco whitepaper here. The whitepaper is extremely informative and covers very VM centric information that I think many admins will find useful. I’ll certainly use it as a basis for building my VMs to test with. Stay tuned for a series of posts as I put my whitepaper together. In the meantime, check out the Cisco doc for an interesting read.