Virtualization Adapted Adapting Business Processes for Virtual Infrastrcuture (and vice-versa)

2010/03/30

Application Performance Testing Method

Filed under: virtualization — Tags: , , , , , , , , , — iben @ 12:46

Are certain applications running slowly occasionally? Sometimes things are superfast and then they slow to a crawl. What’s going on?

First of all – do all you can to ensure the environment is configured according to established Best Practices. One of the benefits of VMware’s acquisition of the Zimbra email / collaboration server software is that they need to ensure users optimize the deployments on their Hypervisor. This document here covers the main settings to check on a Virtual Machine that needs to perform well under load: http://iben.users.sonic.net/wp//2011/05/performance-recommendations-for-virtualizing-anything-with-vmware-vsphere-4/

Any tool that uses SNMP to gather performance metrics can be used to baseline and stress test infrastructure and determine where the bottle necks are.

Basic methodology could go something like this…

1 – identify end to end system components from end user terminal through network to virtual machines, esx hosts, and storage.

2 – configure SNMP for all devices (keep in mind that the latest ESX/ESXi vSphere versions don’t have many performance counters exposed via SNMP and you’ll need to use their APIs)

3 – verify use patterns and confirm data collection over time (1 week or month). Tune alerts for normal use.

4 – schedule stress test for each component to determine performance ceiling and baseline throughput capacity.

5 – make changes as needed to improve end user experience.

6 – verify changes had desired effect.

Performance Troubleshooting for VMware vSphere

vsphere4-performance-troubleshooting.pdf (2.1 MB)

http://communities.vmware.com/docs/DOC-10352

Possible tools that could be used to poll for performance metrics include:

http://www.scriptlogic.com/Products/perspective/

http://www.vizioncore.com/products/vFoglight/features.php

http://www.whatsupgold.com/technology/network-management/monitoring-technologies/index.aspx

http://www.quest.com/Quest_Site_Assets/PDF/DSA-FoglightNetworkDevice-US-VC.pdf

http://network-optimisation.com/technology/network_monitoring/snmp_monitoring.php

http://www.microsoft.com/systemcenter/operationsmanager/en/us/default.aspx

http://www.manageengine.com/products/opmanager/index.html

http://www.managementsoftware.hp.com

http://www.solarwinds.com/products/orion/modules.aspx

http://www.veeam.com/vmware-esx-monitoring.html

http://www.monitorsnmp.com/

http://www.cisco.com/en/US/tech/tk869/tk769/technologies_white_paper09186a008011fde2.shtml

http://www.sage.org/lists/sage-members-archive/2002/msg01878.html

Do you know of a tool that should be added to this list? Please send it to me.

 

2010/03/19

Number of ports to use for standard and distributed virtual switches

VMware just updated their KB: Reserved or overhead ports for virtual switches (http://kb.vmware.com/kb/1008040) and we’ve run into this issue a number of times since upgrading to vSphere ESX 4. These new high memory hardware architectures allow an unprecedented number of virtual machine guests to be consolidated on a single ESX host.

By default a vswitch may not have enough ports to support the consolidation ratio your equipment can support. New ESX hosts can have 256 GB of RAM with 4 hex core processors and easily support 100 or more virtual machines. These virtual machines might have 1, 2, or more vNICs configured and each would need a port on the vswitch. One might imagine the need for 500 to 1000 ports needed per esx host. Why not just make it 2000 so we don’t have to worry about it later on?

Once you run out of vswitch ports you cannot power on any more vms on that host and even get errors about unplugged network cable.  Increasing the vSwitch port allocation seems easy enough, vmotion all workload off the host, put it in maintenance mode, change the vswitch config, reboot. Some system administrators run into this issue and decide to make the number of ports allocated to the vswitch really high to prevent this from ever being an issue. This can cause problems though.

There’s a limit of how many vswitch ports in total an ESX host has to hand out to it’s various vswitches. In addition, if security is a concern, you may start running firewall virtual appliances like vShield Zones or Catbird. WAN Accerators and Performance Monitoring tools like AppSpeed also require additional vSwitches to be created. Ports used on these vSwitches all take away from the total bucket of available ports.

Once 4096 ports are allocated to existing vSwitches you will not be able to add additional hosts to a vNetwork Distributed Switch either.

We also have the following Security Recommendation:

Only allocate vswitch ports to virtual machines on demand and as needed.

This will make it difficult if not impossible to “plug” a VM into the wrong network by accident. Testing for this can be done manually through the vSphere Client. If there are no ports available on a vSwitch then this is a positive test.

1. While connected to the vCenter Server Navigate to Home – Inventory – Networking in the vSphere Client and click on the vDS in question.
2. Click on the Ports Tab
3. If all of the ports in the list have a VM associated with it in the “connected”column then this is a positive test.

Deployment scenarios where a very large number of uplinks are teamed together on a single virtual switch might significantly impact the number of  ports on that virtual switch available for virtual machine use, and the overall size of the virtual switch might need to be adjusted accordingly.
 
The current port utilization data for virtual switches can be reviewed by using the esxcfg-vswitch –list command.
 
The current overhead utilization on a given virtual switch can be calculated by subtracting the Used Ports value for all PortGroups from the Used Ports value for that virtual switch.

Recommendation: Use VNDS vNetwork distributed Switches for all Virtual Machine traffic and limit the number of ports assigned to each standard vSwitch used for vmkernel and service console.

Standard vSwitch Procedure:

Note: A server reboot is required to apply the following configuration change.  Migrate the virtual machines off the ESX host to prevent any downtime.   On the vswitch there is an option to specify the number of ports the vswitch supports.  

To view this setting:

  1. Click the Configuration tab of the ESX host in the Virtual Infrastructure Client (VI Client).
  2. Click Networking.
  3. Click Properties.

  4. Click on vSwitch.
  5. Click Edit.

  6. On the General tab select the number of ports you want and click OK.

 

  • Reboot the ESX host for changes to take effect.
  • Reference Links

    Powered by WordPress