Glad to see this has been posted and we can talk about it now… please share your experiences and let us know if these tips work for you and what sort of performance benefits you’ve noticed when using this new driver.

We’ve been switching our Windows and Linux VMs to use “VMXNET Enhanced” for some time now and see public information on the new VMXNET3 NIC for guests…
This Thread has been started to help with procedures on the conversion of existing machines from older NIC to newer NIC as it is not 100% straightforward and there are some tricks to remove old hardware and change to new hardware. This would be similar in the physical world to changing from a 100 BaseT PCI Card to a GigE card. The old drivers need to be removed, new drivers installed, and IP Addresses moved over. If you just remove the old NIC and install the new one you may end up with a IP Address Conflict error saying the Address you are trying to use is already in use on another Network Interface. The problem is that when you open Device Manager the old NIC is hidden. See below for steps on how to overcome this.
Question: What is VMXNET3?
Answer: VMXNET3 builds upon VMXNET and Enhanced VMXNET as the third generation paravirtualized virtual networking NIC for guest operating systems.
New VMXNET3 features over previous version of Enhanced VMXNET include:
• MSI/MSI-X support (subject to guest operating system kernel support)
• Receive Side Scaling (supported in Windows 2008 when explicitly enabled through the device’s Advanced configuration tab)
• IPv6 checksum and TCP Segmentation Offloading (TSO) over IPv6
• VLAN off-loading
• Large TX/RX ring sizes (configured from within the virtual machine)

What’s New in vSphere 4.0

http://communities.vmware.com/viewwebdoc.jspa?documentID=DOC-9225&communityID=2701
http://www.vmware.com/files/pdf/VMW_09Q1_WP_vSphereNetworking_P8_R1.pdf

From the Cisco document:VMware vSphere 4 and Cisco Nexus 1000V Series

VMware vNetwork module that encompasses the vDS and VMXNET-3 enables inline monitoring and centralized firewall services and maintains the virtualmachine’s network run-time characteristics.

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/solution_overview_c22-529767-00.pdf

Tech Notes

Flexible shows up in Windows Device Manager as an “VMware
Accelerated AMD PCNet Adapter” and Enhanced vmxnet show up as “VMware
PCI Ethernet Adapter”.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001805
Flexible — The Flexible network adapter
identifies itself as a Vlance adapter when a virtual machine boots, but
initializes itself and functions as either a Vlance or a vmxnet
adapter, depending which driver initializes it. VMware Tools versions
recent enough to know about the Flexible network adapter include the
vmxnet driver but identify it as an updated Vlance driver, so the guest
operating system uses that driver. When using the Flexible network
adapter, you can have vmxnet performance when sufficiently recent
VMware tools are installed. When an older version of VMware Tools is
installed, the Flexible adapter uses the Vlance adapter (with Vlance
performance) rather than giving no network capability at all when it
can’t find the vmxnet adapter.
Enhanced vmxnet — The enhanced vmxnet adapter is
based on the vmxnet adapter but provides some high-performance features
commonly used on modern networks, such as jumbo frames. This virtual
network adapter is the current state-of-the-art device in virtual
network adapter performance, but it is available only for some guest
operating systems on ESX Server 3.5. This network adapter will become
available for additional guest operating systems in the future.

Solution
Under certain conditions, you may see the following error message from a Windows guest operating system:
The IP address XXX.XXX.XXX.XXX you have entered for this network
adapter is already assigned to another adapter Name of adapter. Name of
adapter is hidden from the network and Dial-up Connections folder
because it is not physically in the computer or is a legacy adapter
that is not working. If the same address is assigned to both adapters
and they become active, only one of them will use this address. This
may result in incorrect system configuration. Do you want to enter a
different IP address for this adapter in the list of IP addresses in
the advanced dialog box?
In this message, XXX.XXX.XXX.XXX is an IP address that you are
trying to set and Name of adapter is the name of a network adapter that
is present in the registry but hidden in Device Manager.
This can occur when you change a network connection’s TCP/IP configuration from DHCP to a static IP address if:

• You have upgraded VMware virtual network adapters (for example

when you migrate a virtual machine from an older to a new version of
VMware software.)

• You have added and removed network adapters multiple times.

The cause of the error is that a network adapter with the same IP
address is in the Windows registry but is hidden in the Device Manager
(My Computer > Properties > Hardware > Device Manager). This
hidden adapter is called a ghosted network adapter.

Using the Show hidden devices option in the Device Manager (View

(ghosted adapter) to which that IP Address is assigned

Microsoft addresses this issue in their Knowledge Base article
269155, which is available at the time of this writing at
http://support.microsoft.com/?kbid=269155.

To resolve this problem, follow these steps to make the ghosted
network adapter visible in the Device Manager and uninstall the ghosted
network adapter from the registry:
1. Select Start > Run.
2. Enter cmd.exe and press Enter.
3. At the command prompt, run this command:
set devmgr_show_nonpresent_devices=1
4. Enter Start DEVMGMT.MSC and press Enter to start Device Manager.
5. Select View > Show Hidden Devices.
6. Expand the Network Adapters tree (select the plus sign next to the Network adapters entry).
7. Right-click the dimmed network adapter, and then select Uninstall.
8. Close Device Manager.

How to remove these “phantom” NICs from Windows 2008 Server Core

1. Copy devcon.exe over to the server core server (extract devcon.exe from \SUPPORT\TOOLS\SUPPORT.CAB on a Windows 2003 R2 x64 disc).
2. Run devcon.exe findall =net (this should list all NICs on the system, including the phantoms). Example output:
   PCI\VEN_15AD&DEV_0720&SUBSYS_072015AD&REV_10\4&B70F118&0&0088: VMware PCI Ethernet Adapter #2
   PCI\VEN_15AD&DEV_0720&SUBSYS_072015AD&REV_10\3&18D45AA6&0&88: VMware PCI Ethernet Adapter
   PCI\VEN_15AD&DEV_07B0&SUBSYS_07B015AD&REV_01\FF565000EB16A3FE00: vmxnet3 Ethernet Adapter
   3 matching device(s) found.
   Observe that vmxnet3 was the active NIC and the others needed to be removed.
3. devcon -r remove “@PCI\VEN_15AD&DEV_0720&SUBSYS_072015AD&REV_10\3&18D45AA6&0&88″ removed the first one.
4. Repeat for the remaining unwanted NICs
5. Reboot the machine to restart all services

Reference: http://vmtoday.com/2009/11/vsphere-upgrade-breaks-active-directory/

Performance

http://blogs.vmware.com/networking/2009/04/considerations-for-maximum-network-performance.html

For UDP, use vmxnet3 to be able to configure a larger vNIC Rx ring size.  Because UDP can be a lot more bursty (due to lack of flow-control), having a larger Rx ring size helps to provide buffering/elasticity to better absorb the bursts.  The new vmxnet3 allows resizing the vNIC’s Rx ring size, up to around 1 to 2 thousand buffers.  As a side note, there is some negative performance impact with larger ring size due to larger memory foot print. The new vxmnet3 vNIC is more efficient than the e1000 vNIC.  Also in general, ESX 4 has some performance improvements over ESX 3.5.

Line Rate 10GigE

Choosing a Network Adapter for Your Virtual Machine KB Article 1001805 Updated May 05, 2009

Details

The Virtual Machine wizard’s Choose Networks window allows you to specify a network and a network adapter. The network adapter choices available depend on these factors: The version of the virtual machine, which depends on what host created it or most recently updated it Whether or not the virtual machine has been updated to the latest version for the current host The guest operating system The Choose Networks window makes available only those network adapters that make sense for the virtual machine you are creating. Each adapter type is discussed in some detail in “Available Network Adapters,” below. Here is an overview of what Choose Networks might offer you: For virtual machines native to VMware Workstation 4x, VMware GSX Server 3, or VMware ESX Server 2.x, you can explicitly choose between Vlance and vmxnet For most 32bit virtual machines native to VMware Workstation 5 or 6, VMware Server 2, or VMware ESX Server 3, only the Flexible adapter is available For most 64bit virtual machines and for 32bit Microsoft Windows Vista virtual machines, only the e1000 adapter is available For certain guest operating systems on VMware ESX Server 3.5 and later, you can choose the Enhanced vmxnet adapter in addition to the Flexible or e1000 adapter mentioned for that guest type in the previous bullets

Solution

Available Network Adapters   The following network adapters might be available for your virtual machine, depending on the factors discussed above: Vlance — Vlance (also called PCNet32) is a faithful virtual implementation of a common, if now somewhat aging, physical network adapter. Most 32bit guest operating systems, except for Windows Vista, have built-in support for this card so a virtual machine configured with this network adapter can use its network immediately. vmxnet — The vmxnet virtual network adapter has no physical counterpart. VMware makes vmxnet available because Vlance, a faithful implementation of a physical card, is far from optimal for network performance in a virtual machine. Vmxnet is highly optimized for performance in a virtual machine. Because there is no physical card of type vmxnet, operating system vendors do not provide built-in drivers for this card. You must install VMware Tools to have a driver for the vmxnet network adapter available. Flexible — The Flexible network adapter identifies itself as a Vlance adapter when a virtual machine boots, but initializes itself and functions as either a Vlance or a vmxnet adapter, depending which driver initializes it. VMware Tools versions recent enough to know about the Flexible network adapter include the vmxnet driver but identify it as an updated Vlance driver, so the guest operating system uses that driver. When using the Flexible network adapter, you can have vmxnet performance when sufficiently recent VMware tools are installed. When an older version of VMware Tools is installed, the Flexible adapter uses the Vlance adapter (with Vlance performance) rather than giving no network capability at all when it cannot find the vmxnet adapter. e1000 — e1000 is a faithful virtual implementation of a physical network adapter that is broadly supported by newer operating systems, specifically most 64bit operating systems and both 32 and 64bit Windows Vista. e1000 performance is intermediate between Vlance and vmxnet. Enhanced vmxnet — The enhanced vmxnet adapter is based on the vmxnet adapter but provides some high-performance features commonly used on modern networks, such as jumbo frames. This virtual network adapter is the current state-of-the-art device in virtual network adapter performance, but it is available only for some guest operating systems on ESX Server 3.5. This network adapter will become available for additional guest operating systems in the future. 32/64bit versions of Microsoft Windows 2003 (Enterprise and Datacenter Editions). You can use enhanced vmxnet adapters with other versions of the Microsoft Windows 2003 operating system, but a workaround is required to enable the option in the VI Client. For more information, see Enabling enhanced vmxnet adapters for Microsoft Windows Server 2003. 32bit version Microsoft Windows XP Professional 32/64bit versions Red Hat Enterprise Linux 5.0 32/64bit versions SUSE Linux Enterprise Server 10 64bit versions Red Hat Enterprise Linux 4.0 Enhanced VMXNET is supported only for a limited set of guest operating systems: Adapter Caveats    This section discusses some potential issues you might have.     Migrating virtual machines that use enhanced vmxnet. Enhanced vmxnet is new with ESX Server 3.5. Virtual machines configured to have enhanced vmxnet adapters cannot migrate to older ESX Server hosts, even though virtual machines can usually migrate freely between ESX Server 3.0 and ESX Server 3.0.1. Upgrading from ESX Server 2.x to ESX Server 3.x. When a virtual hardware upgrade operation transforms a virtual machine created on an ESX Server 2.x host to an ESX Server 3.x host, Vlance adapters are automatically upgraded to Flexible. In contrast, vmxnet adapters are not upgraded automatically because certain guest operating systems — specifically most or all Linux versions — do not reliably preserve network settings when a network adapter is replaced. Because the guest operating system thinks a Flexible adapter is still Vlance, it retains the settings in that case. If the upgrade were to replace a vmxnet adapter with a Flexible adapter, the guest operating system would erroneously discard the settings.   After the virtual hardware upgrade, the network adapter is still vmxnet, without the fallback compatibility of the Flexible adapter. Just as on the original older host, if VMware Tools is uninstalled on the virtual machine, it is unable to access its network adapters.   Network adapters on multi-boot Linux. The Virtual Machine Settings dialog box and New Virtual Machine wizard allow creation of only those virtual network adapters that are supported for the selected guest operating system. If you change the guest operating system, the existing network adapters are not affected. When you switch a multi-boot Linux system between 32bit mode and 64bit mode, a problem arises because most 32bit Linux versions do not support e1000 adapters while most 64bit Linux versions support only e1000 adapters. Consider configuring your virtual machine with one of each type of network adapter (e1000 and Flexible). You can then set up your guest operating system to use only the network adapter for which it has a driver in each mode.   You can add the second adapter any time the virtual machine is powered off, but you need to change the configured guest operating system type from 32bit to 64bit or vice-versa in order to be offered the other network adapter. Since changing that setting before rebooting into the other bit depth can potentially improve the efficiency of virtual machine scheduling, plan to change the guest operating system type setting before your first reboot into the other bit depth. Adding virtual disks. Adding an existing older (ESX Server 2.x) virtual disk to an ESX Server 3.x virtual machine results in a de-facto downgrade of that virtual machine to ESX Server 2.x. If you are using ESX Server 3.x features, such as enhanced vmxnet or Flexible network adapters, the virtual machine becomes inconsistent. When you add an existing ESX Server 2.x virtual disk to an ESX Server 3.x machine, you should immediately use the Upgrade Virtual Hardware command to restore the virtual machine to the ESX Server 3 version. Note: Executing Upgrade Virtual Hardware changes the ESX Server 2 virtual disk so it is no longer usable on an ESX Server 2 virtual machine. Consider making a copy of the disk before you upgrade one of the two copies to ESX Server 3 format. If you must migrate a virtual machine between newer and older hosts, do not choose enhanced vmxnet but instead one of the older adapter types. Flexible or e1000 are offered whenever enhanced vmxnet is offered.