VMware just updated their KB: Reserved or overhead ports for virtual switches (http://kb.vmware.com/kb/1008040) and we’ve run into this issue a number of times since upgrading to vSphere ESX 4. These new high memory hardware architectures allow an unprecedented number of virtual machine guests to be consolidated on a single ESX host.

By default a vswitch may not have enough ports to support the consolidation ratio your equipment can support. New ESX hosts can have 256 GB of RAM with 4 hex core processors and easily support 100 or more virtual machines. These virtual machines might have 1, 2, or more vNICs configured and each would need a port on the vswitch. One might imagine the need for 500 to 1000 ports needed per esx host. Why not just make it 2000 so we don’t have to worry about it later on?

Once you run out of vswitch ports you cannot power on any more vms on that host and even get errors about unplugged network cable.  Increasing the vSwitch port allocation seems easy enough, vmotion all workload off the host, put it in maintenance mode, change the vswitch config, reboot. Some system administrators run into this issue and decide to make the number of ports allocated to the vswitch really high to prevent this from ever being an issue. This can cause problems though.

There’s a limit of how many vswitch ports in total an ESX host has to hand out to it’s various vswitches. In addition, if security is a concern, you may start running firewall virtual appliances like vShield Zones or Catbird. WAN Accerators and Performance Monitoring tools like AppSpeed also require additional vSwitches to be created. Ports used on these vSwitches all take away from the total bucket of available ports.

Once 4096 ports are allocated to existing vSwitches you will not be able to add additional hosts to a vNetwork Distributed Switch either.

We also have the following Security Recommendation:

Only allocate vswitch ports to virtual machines on demand and as needed.

This will make it difficult if not impossible to “plug” a VM into the wrong network by accident. Testing for this can be done manually through the vSphere Client. If there are no ports available on a vSwitch then this is a positive test.

1. While connected to the vCenter Server Navigate to Home – Inventory – Networking in the vSphere Client and click on the vDS in question.
2. Click on the Ports Tab
3. If all of the ports in the list have a VM associated with it in the “connected”column then this is a positive test.

Deployment scenarios where a very large number of uplinks are teamed together on a single virtual switch might significantly impact the number of  ports on that virtual switch available for virtual machine use, and the overall size of the virtual switch might need to be adjusted accordingly.
 
The current port utilization data for virtual switches can be reviewed by using the esxcfg-vswitch –list command.
 
The current overhead utilization on a given virtual switch can be calculated by subtracting the Used Ports value for all PortGroups from the Used Ports value for that virtual switch.

Recommendation: Use VNDS vNetwork distributed Switches for all Virtual Machine traffic and limit the number of ports assigned to each standard vSwitch used for vmkernel and service console.

Standard vSwitch Procedure:

Note: A server reboot is required to apply the following configuration change.  Migrate the virtual machines off the ESX host to prevent any downtime.   On the vswitch there is an option to specify the number of ports the vswitch supports.  

To view this setting:

Reference Links

• Network cable of a virtual machine appears unplugged –
  http://kb.vmware.com/kb/1004883
• Unable to deploy configuration from –
  http://kb.vmware.com/kb/1007441
  Error: SysinfoException: Node (VSI_NODE_net_create) ; Status(bad0006)= Limit exceeded;
• Reserved or overhead ports for virtual switches –
  http://kb.vmware.com/kb/1008040