VMotion does not check vSwitch port availability

When I was reshuffling some VMs from one cluster to the other, I had a VM that was cut off from the network. When I did some investigation I found that the network adapter was disconnected in the VM Settings. Hmmmm strange.
How could that happen?

I did some more investigation on the ESX box to find a clue why the VM’s network adapter was disconnected due to the VMotion. Just before the VMotion I could still ping it.

I stumbled across this little error in the /var/log/vmkernel.

Feb  7 11:04:40 esx001 vmkernel: 57:22:46:06.825 cpu1:1864)Net: 1027: can’t connect device: VLAN0xxx: Out of resources

apparantly my vSwitch ran out of resources. So I double checked the vswitch with esxcfg-vswitch -l

Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch01         64               64               64                      1500     vmnic1,vmnic2

It looks that all my available vSwitch ports are in use. Why was there no warning before performing the VMotion?

Let’s do another test.

  1. First make sure that there are no more ports available on the vSwitch
    Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
    vSwitch01         64               64               64                      1500     vmnic1,vmnic2
  2. Ping the VM we are going to VMotion to the host which has no ports available
  3. Monitor the /var/log/vmkernel
  4. Peform the VMotion.

The proof is in the pudding!
No Warning, No Error, VMotion completed successful, VM down.

VMotion proof

VMotion proof

 
 
 
 
 
 
 
 
 
 
 
 
So what do we learn from this?
  • When upsizing your hardware, don’t forget to upsize your vSwitch too. You can set this in the properties of your vSwitch using the VIC or use this command on the ESX console.
    /usr/bin/vimsh -n -e “hostsvc/net/vswitch_setnumports <vswitch> <number_of_ports>”
    Remember that a reboot is necessary!!!
  • Some ports are reserved by the ESX kernel. I don’t know exactly why. In the VIC the properties of the vSwitch allways shows
    <number_of_ports-8>, but you can check your effective available ports using esxcfg-vswitch -l on the ESX console.

Related posts:

  1. VMware Storage Sudoku Tweet Last Friday I was brainstorming with Gabrie van Zanten about the optimal placement of the VMDKs across our LUNs. We tried to come up with an algorithm that could...
  2. Unable to login to your ESX server Tweet Ivo Beerens posted this article last week on the defunct cimservera processes that render an ESX Host unmanageable. See also this VMWare KB Article. Symptoms include: Unable to log...

3 Comments on “VMotion does not check vSwitch port availability”

  1. #1 Maish
    on Feb 10th, 2009 at 3:13 pm

    That is why one of the first things I do when setting up a new host is raise the number of ports to 512 and above for each vSwitch before I start doing anything on the machine..

  2. #2 Arnim van Lieshout
    on Feb 10th, 2009 at 9:33 pm

    That’s what we learn from this Maish!.
    Allthough vCenter should check this before performing th VMotion.

  3. #3 Chris
    on Nov 20th, 2009 at 10:16 pm

    I wish I had read this about an hour ago! I found out the hard way… lost 4 VMs… lucky for my it clicked about 15 seconds after I got the alerts that they were down.

Leave a Comment