Configuring Software iSCSI multipathing on ESXi 4.1
Sam McGeown
Recently I had cause to configure iSCSI multipathing on a test ESXi server. The production environment servers use iSCSI HBAs to connect to the back end storage, so multipathing them is a straight-forward setup.
It’s good practice to separate VMotion, virtual machine and iSCSI traffic, it also helps you manage those logical and physical connections.
Connect to your ESXi server using the vSphere Client and select the host. Go to the configuration tab and click “Add Networking…”. Select a new VMKernel connection type.
Create a new virtual switch, but at this point don’t assign any network adaptors.
Create a new Port Group. If you are planning to use Jumbo Frames, you can’t use this Port Group – to specify a larger MTU you have to create the Port Group using the command line interface. If you aren’t using Jumbo Frames for whatever reason, you can go with the standard wizard-created Port Group.
Set the IP address for the VMKernel – this address needs to be on the same network as the iSCSI SAN because the iSCSI protocol does not like to be routed.
Click through the summary page and your new vSwitch and VMkernel is created
If you’re using Jumbo Frames, you need to delete the newly created Port Group, so open the vSwitch properties and Remove it
You should now have an empty vSwitch
Open up your command line interface – this is either a vMA instance, or if you have enabled SSH on the ESXi server, then the SSH terminal. Log in as a user with permission to sudo. I am using SSH.
Just like on a physical switch, the vSwitch needs to be told to allow Jumbo Frames by setting the MTU to 9000.
esxcfg-vswitch --MTU=9000 vSwitch4
Next we can add the Port Groups. iSCSI multipathing requires one for each pNIC we are planning to utilise. To add the new Port Groups, use the following commands:
esxcfg-vswitch --add-pg=iSCSI1 vSwitch4 esxcfg-vswitch --add-pg=iSCSI2 vSwitch4
If you need to specify the VLAN for the iSCSI traffic, use the following commands:
esxcfg-vswitch --vlan=105 --pg=iSCSI1 esxcfg-vswitch --vlan=105 --pg=iSCSI2
Now we need to add a VMKernel IP address per pNIC we are using to the Port Group on the iSCSI network. Make sure you use the next number in the vmk sequence, e.g. if you already have vmk0, vmk1 and vmk2 configured, create the next one as vmk3. To see your currently configured VMKernel ports, use esxcfg-vmknic –l.
esxcfg-vmknic --add vmk3 --ip 10.20.200.100 --netmask 255.255.255.0 ↓ --portgroup iSCSI1 --mtu=9000 esxcfg-vmknic --add vmk4 --ip 10.20.200.101 --netmask 255.255.255.0 ↓ --portgroup iSCSI2 --mtu=9000
If you switch to the vSphere Client and refresh, you should see the new config
Now we need to add the physical adaptors to the virtual switch. What we don’t want to do is add two pNICs to the iSCSI kernel, it doesn’t like this one bit! Open the properties for the vSwitch and go to the Network Adaptors tab. Click add, and select the pNICs you are going to use, and, this is important, set one pNIC to Active and the rest to Standby.
Now go to the iSCSI1 Port Group under the Ports tab and Edit the configuration – override the Failover order and set one pNIC to active, and the other to Unused. This gives you a one-to-one mapping from the Port Group to the pNIC. Edit the second iSCSI Port Group and map the second pNIC to active and the first to unused.
Obviously if you’re using more network adaptors you have more port groups and map them one-to-one. Switch back to the SSH session and bind the VMKernels to the Software HBA. You need to check the name of the Software iSCSI HBA from the Storage Adaptors
Add the vmk ports you created earlier
esxcli swiscsi nic add -n vmk3 -d vmhba34 esxcli swiscsi nic add -n vmk4 -d vmhba34
Add your iSCSI targets in the normal way, rescan the software adaptor to pick up all the paths. Modify each LUN to use Round Robin path selection.
You can verify both paths are being used using ESXTOP to in the SSH session and pressing “n” for network.
References:
iSCSI SAN Configuration Guide - http://www.vmware.com/pdf/vsphere4/r40/vsp_40_iscsi_san_cfg.pdf
Troubleshooting Software iSCSI on ESXi - http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008083