Teaming NICs with ESX 3.5 and Cisco Switches in an aggregate.
Sam McGeownHere’s the setup. We have a core switch of 2 Cisco 3750s, connected together for fault tolerance as a single logical switch; we also have several ESX 3.5 hosts with 4 Gigabit Ethernet NICs installed each. The Virtual Machines will all be on VLAN 8 (reserved for internal servers) and the VMKernel will be on VLAN 107 (reserved for VMKernel traffic like VMotion). I want to create a load balanced, fault tolerant aggregate of these four NICs over the Core Switch.
Configure ESX server’s vSwitch
Configuring the vSwitch is actually pretty simple, but there are a couple of gotchas, so don’t skip this bit! First thing to note is that if you are making changes to the vSwitch and the Service Console is on that vSwitch you can quite easily lock yourself out. Make sure you configure this correctly, first time! In this setup, I am adding all 4 NICs to vSwitch0, which will be the only vSwitch. I’ll then use Port Groups to assign VLANs and Active/Passive configurations to the VMKernel/Service Console.
First things first then - assign the four NICs to the vSwitch. This is done in the Configuration Tab in VMware Infrastructure Client, then the Networking page. Edit the properties of your vSwitch, then select the Network Adaptor tab. Add all the NICs you wish to team in there (they may already be in there, depending on your setup). You should end up with something that looks like this (note that I’ve not assigned any VLAN yet):
Now you need to configure the NIC teaming, so edit the vSwitch Properties and under the Ports tab select the vSwitch. Click edit, and then go to the NIC teaming tab. Configure the teaming options like this:
That’s the easy part over and done with! Time to move onto the Cisco!
Configuring the Cisco Core Switch
Firstly, we need to log on to the switch and enter enable mode; I’m going to assume you know how to do this - if not, you really shouldn’t be attempting this setup!
Determine the switches trunk load balancing setup by using the command “show etherchannel load-balance”. It should look something like this:
If the protocol is NOT src-dst-ip, then you won’t be able to establish a trunk connection with the ESX server. If your protocol is not src-dst-ip, change it with the command “port-channel load-balance src-dst-ip”. This now matches the “Route based on IP hash” setting you configured in ESX. Although ESX has a setting for MAC based hashing, as does the Cisco, I was unable to get it to work.
Moving on. You need to create a Port-Channel interface for the trunk (this is a virtual interface that binds the 4 GigabitEthernet interfaces together). As i’ve got other Port-channels in use for connections to other switches, I’m setting up port-channel 40. Move to config mode (conf t) and then enter the setup:
interface Port-channel40
description VMTEST01 Aggregate
switchport trunk encapsulation dot1q
switchport trunk native vlan 8
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
end
Description simply adds a description, “switchport trunk encapsulation dot1q” sets the encapsulation of the trunk to 802.1Q. “switchport trunk native vlan 8” means that any traffic without a VLAN tag will be automatically assigned to VLAN 8. “switchport mode trunk” obviously designates that we want a trunk, rather than access. “switchport nonegotiate” means that it will not attempt to negotiate the protocol, and be a static trunk, rather than LCAP or PGaP. “spanning-tree portfast trunk” causes a Layer 2 LAN interface configured as an access port to enter the forwarding state immediately, bypassing the listening and learning states (i.e. if the link goes down and then comes back up, it will do so quickly).
With the Port-channel configured, you now need to edit your GigabitEthernet ports and assign them to the Port-channel. For each port in the trunk, enter the following config (this example is port 8 on the master switch in my stack, hence 1/0/8):
interface GigabitEthernet1/0/8
description VMTEST01 VMNIC1
switchport trunk encapsulation dot1q
switchport trunk native vlan 8
switchport mode trunk
switchport nonegotiate
channel-group 40 mode on
spanning-tree portfast trunk
end
The difference between that and the Port-channel setup? “channel-group 40 mode on” is simply assigning the port-channel in static mode.
Once all four NICs are assigned you might have to wait a few minutes for every layer of the connection to settle down before the trunk comes up. To check the status of the etherchannel you can use the command “show etherchannel 40 summary”, replacing the 40 for whichever number you assigned to your port-channel.
I hope this helps navigate the minefield that I found to be setting up the NIC teaming!