Virtualization Adapted Adapting Business Processes for Virtual Infrastrcuture (and vice-versa)


Application Performance Testing Method

Filed under: virtualization — Tags: , , , , , , , , , — iben @ 12:46

Are certain applications running slowly occasionally? Sometimes things are superfast and then they slow to a crawl. What’s going on?

First of all – do all you can to ensure the environment is configured according to established Best Practices. One of the benefits of VMware’s acquisition of the Zimbra email / collaboration server software is that they need to ensure users optimize the deployments on their Hypervisor. This document here covers the main settings to check on a Virtual Machine that needs to perform well under load:

Any tool that uses SNMP to gather performance metrics can be used to baseline and stress test infrastructure and determine where the bottle necks are.

Basic methodology could go something like this…

1 – identify end to end system components from end user terminal through network to virtual machines, esx hosts, and storage.

2 – configure SNMP for all devices (keep in mind that the latest ESX/ESXi vSphere versions don’t have many performance counters exposed via SNMP and you’ll need to use their APIs)

3 – verify use patterns and confirm data collection over time (1 week or month). Tune alerts for normal use.

4 – schedule stress test for each component to determine performance ceiling and baseline throughput capacity.

5 – make changes as needed to improve end user experience.

6 – verify changes had desired effect.

Performance Troubleshooting for VMware vSphere

vsphere4-performance-troubleshooting.pdf (2.1 MB)

Possible tools that could be used to poll for performance metrics include:

Do you know of a tool that should be added to this list? Please send it to me.


Using Cryptographic Hashes to verify file download integrity

Filed under: virtualization — Tags: , , , , , , , , , , — iben @ 10:58

The SHA hash functions are a set of cryptographic hash functions designed by the National Security Agency (NSA) and published by the NIST as a U.S. Federal Information Processing Standard. SHA stands for Secure Hash Algorithm.

Vendors provide a sha-1 hash for software downloads. This enables you to verify that your downloaded files are unaltered from the original.

To confirm file integrity, use an sha-1 utility on your computer to calculate your own hash for files downloaded from the VMware web site.

If your calculated hash matches the message digest we provide, you are assured that the file was downloaded intact.

sha-1 utilities are available for Windows and Linux and Mac. Most UNIX installations provide a sha1sum command for sha-1 hashes. You may need a newer linux kernel to calculate the checksums for larger files.

The File Checksum Integrity Verifier (FCIV) can be used on Windows based products to verify sha-1 values. Please see for details on FCIV.

Mac OS X: How to Verify a SHA-1 Digest

Instructions on checking an sha-1 checksum on a Mac:
In Finder, browse to /Applications/Utilities.
Double-click on the Terminal icon. A Terminal window will appear.
In the Terminal window, type: “openssl sha1 ” (sha1 followed by a space).
Drag the downloaded file from the Finder into the Terminal window.
Click in the Terminal window, press the Return key, and compare the checksum displayed to the screen to the one on the vendor’s download page.

From TechNet

Windows Server 2008 R2 Standard, Enterprise, Datacenter, and Web (x64) – DVD (English)
File Name: en_windows_server_2008_r2_standard_enterprise_datacenter_web_x64_dvd_x15-50365.iso
Size: 2,858 (MB)
Date Published (UTC): 8/31/2009 10:22:24 AM
Last Updated (UTC): 1/11/2010 4:31:40 PM
SHA1: A548D6743129F2A02C907D2758773A1F6BB1BCD7
 ISO/CRC: 8F94460B

About MD5

MD5 was designed by Ron Rivest in 1991 to replace an earlier hash function, MD4. In 1996, a flaw was found with the design of MD5. While it was not a clearly fatal weakness, cryptographers began recommending the use of other algorithms, such as SHA-1 (which has since been found also to be vulnerable). In 2004, more serious flaws were discovered, making further use of the algorithm for security purposes questionable; specifically, a group of researchers described how to create a pair of files that share the same MD5 checksum. Further advances were made in breaking MD5 in 2005, 2006, and 2007. In an attack on MD5 published in December 2008, a group of researchers used this technique to fake SSL certificate validity.

US-CERT says MD5 “should be considered cryptographically broken and unsuitable for further use,”and most U.S. government applications now require the SHA-2 family of hash functions.

VMware Data Recovery

Filed under: virtualization — Tags: , , , , , , — iben @ 10:49

VMware Data Recovery (CD ISO)
Released 11/19/09 | Version 1.1 | Size 418 MB | Binary (.iso)
Deploy VMware Data Recovery virtual appliance plus management components.
SHA1SUM 44dc0cd0c3e774d4912412b51dabeadf28d959b9


Host Profiles N1KV VDS

Filed under: virtualization — Tags: , , , , , , , , , , , — iben @ 06:21

Background to Using Host Profiles

The vDS UI also allows a phased migration of vmnics from vSS to vDS without disruption to an operational environment. VMs can be migrated from a vSS to a vDS on the fly so long as the vDS and vSS have connectivity to the same network at the same time and the origin Port Group on the vSS and destination DV Port Group on the vDS are configured to the same VLAN.

Host Profiles provide a way to migrate multiple hosts at one time. Host Profiles use a golden profile from a migrated host to propagate a configuration to a number of other hosts.

When applying a Host Profile to a host, the host must be in Maintenance Mode. This requires VMs to be either powered down or migrated to another host.

Host Profiles are most appropriate for new installations of similarly configured hosts (i.e. same number of vmnics, same vmnic to physical switch configuration, no active VMS).

The table below summarizes the deployment situations and suggested methods for migration from vSS to vDS. Note: These are suggestions only; both methods will work within the guidelines mentioned above.

Summary of Migration Methods

Table 1 – Summary of vSS to vDS Migration Methods

DeploymentSituation SuggestedMethod Details
New servers, same vmnic config, no active VMs vDS UI + HP Migrate first host with vDS UI. Take host profile and apply to remaining hosts
<5 Existing Servers, no active VMs vDS UI Small number of servers. Can use host profiles, but possibly easier to continue with vDS UI
>5 Existing servers, same vmnic configs, no active VMs vDS UI + HP Larger number of servers with similar vmnic configuration. No active VMs so can enter maintenance mode and use Host Profiles
Existing Servers, active/operational VMs vDS UI Cannot use Maintenance Mode as VMs active. Phased vmnic migration suggested to ensurecontinuity of VM communications
 Existing Servers, dissimilar vmnic configurations vDS UI Enables per host tailoring of vmnic to dvUplink PortGroup mapping
Ongoing Compliance Checking HP Non-disruptively check network settings are compliant with approved “golden” configuration

Note: vDS UI = Use vDS UI; HP = use Host Profiles; vDS + HP = use vDS UI to deploy first host and Host Profiles for remaining hosts.

Applying NIC Teaming Policies to DV Port Groups With a vSS, NIC teaming policies are defined on the virtual switch with an optional override on each Port Group definition.  With vDS, NIC teaming policies are only defined on the DV Port Groups and apply to dvUplinks, not vmnics.  The vmnics are mapped to the dvUplinks on a per host basis.  This enables each host to have a different vmnic to physical host configuration and yet use the same NIC teaming policy over all hosts spanned by the vDS.

Monitoring Hash vmnic Selection in NIC Teams

The esxtop command from the ESX console can reveal the physical NIC (vmnic) used by virtual port or VM within a NIC team.

Use esxtop to see the following information:

  • PORT-ID represents an internal port number on the virtual switch
  • USED-BY column shows what that port number is used by (e.g. VMkernel, VM, etc)
  • TEAM-PNIC column shows what physical nic (vmnic) is being used for traffic from that virtual port (the result of the hash within the NIC team)
  • The remaining columns indicate the Receive and Transmit traffic rates on those ports.

To use esxtop, type esxtop from the ESX console and then type n.

A list of commands for the ESX command line interface is published in Chapter 6 of the ESX 4.0 Configuration Guide (available at To control console output to one page at a time by adding the | more suffix to the commands. For example:
esxcfg-vswitch –l | more


(See page 8)


List of log files VMware vSphere ESX Classic version 4

Filed under: virtualization — Tags: , , , , , , , , — iben @ 11:02
The following log files contain information that needs to be track on a VMware vSphere ESX 4 Classic Host to be in compliance with many security standards and best practices such as CIS Benchmark, PCI-DSS, SOX section 404, HIPPA, CPNI, COSO, ISO 20001, COBIT, and so on.
You can use syslog or splunk lightweight forwarders for this purpose.














Table with Explanation of files to log for VMware vSphere ESX Classic version 4






 Records activities related to the virtual machines and ESX

VMkernel warnings


Records activities with the virtual machines

VMkernel summary


Used to determine uptime and availability statistics for ESX; comma separated

VMkernel summary human readable


Used to determine uptime and availability statistics for ESX; human‐readable summary

ESX host agent log


Contains information on the agent that manages and configures the ESX host and its virtual machines

vCenter agent



Contains information on the agent that communicates with vCenter

Web access

Log all the files in the directory /var/log/vmware/webAccess/*.log
client.log, proxy.log, unitTest.log, viewhelper.log, objectMonitor.log, timer.log, updateThread.log

Records information on Web-based access to ESX
(service vmware-webAccess start on ESX host to enable this)

Authentication log


Contains records of connections that require authentication, such as VMware daemons and actions initiated by the xinetd.

Service Console


Contain all general log messages used to troubleshoot virtual machines or ESX

Virtual machines

The same directory as the affected virtual machine’s configuration files; named vmware.log and vmware‐*.log



Contain Virtual Machine Power Events, system crashes, Tools status and activity, Time Sync, Virtual Hardware changes, VMotion Migrations, Machine Clones,

Table  – List of ESX Host Files to Log




Number of ports to use for standard and distributed virtual switches

VMware just updated their KB: Reserved or overhead ports for virtual switches ( and we’ve run into this issue a number of times since upgrading to vSphere ESX 4. These new high memory hardware architectures allow an unprecedented number of virtual machine guests to be consolidated on a single ESX host.

By default a vswitch may not have enough ports to support the consolidation ratio your equipment can support. New ESX hosts can have 256 GB of RAM with 4 hex core processors and easily support 100 or more virtual machines. These virtual machines might have 1, 2, or more vNICs configured and each would need a port on the vswitch. One might imagine the need for 500 to 1000 ports needed per esx host. Why not just make it 2000 so we don’t have to worry about it later on?

Once you run out of vswitch ports you cannot power on any more vms on that host and even get errors about unplugged network cable.  Increasing the vSwitch port allocation seems easy enough, vmotion all workload off the host, put it in maintenance mode, change the vswitch config, reboot. Some system administrators run into this issue and decide to make the number of ports allocated to the vswitch really high to prevent this from ever being an issue. This can cause problems though.

There’s a limit of how many vswitch ports in total an ESX host has to hand out to it’s various vswitches. In addition, if security is a concern, you may start running firewall virtual appliances like vShield Zones or Catbird. WAN Accerators and Performance Monitoring tools like AppSpeed also require additional vSwitches to be created. Ports used on these vSwitches all take away from the total bucket of available ports.

Once 4096 ports are allocated to existing vSwitches you will not be able to add additional hosts to a vNetwork Distributed Switch either.

We also have the following Security Recommendation:

Only allocate vswitch ports to virtual machines on demand and as needed.

This will make it difficult if not impossible to “plug” a VM into the wrong network by accident. Testing for this can be done manually through the vSphere Client. If there are no ports available on a vSwitch then this is a positive test.

1. While connected to the vCenter Server Navigate to Home – Inventory – Networking in the vSphere Client and click on the vDS in question.
2. Click on the Ports Tab
3. If all of the ports in the list have a VM associated with it in the “connected”column then this is a positive test.

Deployment scenarios where a very large number of uplinks are teamed together on a single virtual switch might significantly impact the number of  ports on that virtual switch available for virtual machine use, and the overall size of the virtual switch might need to be adjusted accordingly.
The current port utilization data for virtual switches can be reviewed by using the esxcfg-vswitch –list command.
The current overhead utilization on a given virtual switch can be calculated by subtracting the Used Ports value for all PortGroups from the Used Ports value for that virtual switch.

Recommendation: Use VNDS vNetwork distributed Switches for all Virtual Machine traffic and limit the number of ports assigned to each standard vSwitch used for vmkernel and service console.

Standard vSwitch Procedure:

Note: A server reboot is required to apply the following configuration change.  Migrate the virtual machines off the ESX host to prevent any downtime.   On the vswitch there is an option to specify the number of ports the vswitch supports.  

To view this setting:

  1. Click the Configuration tab of the ESX host in the Virtual Infrastructure Client (VI Client).
  2. Click Networking.
  3. Click Properties.

  4. Click on vSwitch.
  5. Click Edit.

  6. On the General tab select the number of ports you want and click OK.


  • Reboot the ESX host for changes to take effect.
  • Reference Links


    How to create a virtual appliance (OVF/OVA)

    Filed under: virtualization — Tags: , , , , , , , , — iben @ 16:08

    How to create a virtual appliance 


    The Open Virtualization Format (OVF) specification is a standard being developed within the Distributed Management Task Force (DMTF) association to promote an open, secure, portable, efficient, and extensible format for the packaging and distribution of software to be run in virtual machines.

    For use within an organization, Level 1 or Level 2 compatibility may be good enough, since the OVF package is distributed within a controlled environment where specific purchasing decisions of hardware or virtualization platforms can ensure consistency of the underlying feature set for the OVF.

    Level 1. Only runs on a particular virtualization product and/or CPU architecture and/or virtual hardware selection. This would typically be due to the OVF containing suspended virtual machines or snapshots of powered on virtual machines, including the current run-time state of the CPU and real or emulated devices. Such state ties the OVF to a very specific virtualization and hardware platform.


      Virtual machines created from OVF sources with SCSI LSI Logic disk controller might not start up after conversion to an ESX destination 
    OVF sources with LSI Logic disk controllers might fail to boot when imported to an ESX destination. This is because Converter Standalone might change the controller type to Bus Logic instead of preserving the source controller type. 
    Workaround: Using VI client, edit the settings of the imported virtual machine to change the controller type from Bus Logic back to LSI Logic. This will enable the virtual machine to boot.

    While exporting a virtual machine from an ESX 3.5 host to OVF “folder of files” format by using Converter Standalone, the vNICs are forcibly changed from their native type 
    While exporting a virtual machine source from an ESX 3.5 host to the OVF “folder of files” format, Converter Standalone changes the source vNICs from their native type (vmxnet, vlance, or e1000) to either PCNet32 (vlance) or E1000 (e1000). This might result in an unexpected lack of network connectivity when the OVF is imported. 
    Workaround: Edit the 
    .vmx file to manually modify the vNIC type after importing the virtual appliance. 

    NOTE: VMXNET3 is recommended for all vSphere Virtual Machine Guests.

    How to Make a Portable Virtual Appliance

    You can export a virtual machine to a virtual appliance, making it available to other users to import into their inventories. The resulting virtual appliance is an OVF 1.0 appliance and contains one virtual machine. OVF Virtual Appliances contain many files that are typically compressed into an archive that can be put on removable media or downloaded from a server. This file much be decompressed prior to being imported and is more cumbersome to use. Consider using OVA for internal enterprise use.

    OVA is also available – this format is a single file that is easier to distribute within an organization. The OVA format is not simply a tar. It places certain restrictions on the ordering and naming of files. These rules ensure that OVA archives are easy to stream – a tool or hypervisor does not need to download an entire OVA first and then unpack it.

    You cannot select a virtual appliance destination for physical machine sources or virtual appliance sources.

    The OVF created as a result of this conversion is not compatible with Workstation 6.5.x, nor with Converter 3.0.3.

    Install Converter Standalone in Windows

    You can install Converter Standalone onto a physical or a virtual machine. The Local setup installs the Converter Standalone server, Converter Standalone agent, and Converter Standalone client for local use. For remote access, you can create a Client-server installation. With remote access you can create and manage conversion tasks remotely.
    When you install the Converter Standalone agent and the Converter Standalone server, the local machine becomes a server for conversions, which you can manage remotely. When you use the local machine with the Converter Standalone client, you can convert the full range of machine types.

    Start the Wizard for a Conversion

    The Conversion wizard helps you specify your source machine, the destination for the machine, and to select the machine’s settings.
    1. Start the VMware vCenter Converter Standalone application.
    2. Click Convert Machine in the application menu.
    The Specify Source page introduces the conversion process: Specify Source, Specify Destination, View/Edit Options, and Ready to Complete.

    What to do next

    You can now select the source machine type to convert.

    Select a Source to Convert

    You can select from several source options for the type of machine to convert. If you are converting a virtual machine that runs on a VMware DRS cluster that vCenter Server manages, set VMware DRS Power Management (DPM) to manual to avoid DPM powering off the ESX hosts used by Converter Standalone. When the conversion process completes, restore DPM to its original settings. For information about how to change DPM settings, see the Resource Management Guide.
    • Select a VMware Infrastructure Virtual Machine Source
      You can convert a virtual machine that resides on an ESX host or ESX host that vCenter Server manages.

    What to do next

    You can now select the destination for your new virtual machine.

    Select a Destination for the New Virtual Machine


    The source virtual machine must be powered off.


    1. On the Destination page, select Virtual Appliance from the drop-down menu.
    2. In the Virtual appliance details pane, type the virtual appliance name in the Name text box.
    3. Click Browse to select a destination location.
      The destination folder can be local or a remote machine shared over the network.
    4. (Optional) If you are connected to a remote Converter Standalone server, click Connect as and provide the user credentials to be used when connecting to the destination machine.
      You must manually type the path to the destination.
    5. Select the Distribution format from the drop-down menu.
      You can create virtual appliance packages that contain monolithic compressed .vmdk files only. You can store the resulting files in an .ovf folder or place them in a single .ova tarred file.
    6. Click Next to customize the virtual appliance.

    You selected to export a virtual machine to a virtual appliance.

    What to do next

    On the View/Edit Options page, you can make more precise settings to the conversion task.

    Then begin the conversion. Once conversion is complete you can move the OVA file to a location where it can be accessed by an administrator with privileges to create virtual machines on the VMware vCenter Server.

    Use vCenter to import a virtual machine from OVF/OVA

    Start the Deploy OVF Template Wizard
    You deploy an OVF template with the Deploy OVF Template wizard.


    Select File > Deploy OVF Template

    On the Source page, you can specify to deploy an OVF template from a file or from a URL.

    • Deploy from a File
      You can deploy from a file that is either a OVF (.ovf file) or a OVA (.ova file) format. The OVF format is optimal for a web server or image library and deploys from a set of files. The OVA format is optimal for deploying from physical media and is packaged in a single file.
    • Deploy from a URL
      You may deploy the OVF template from a URL.


    VMFS versions and upgrade paths

    Filed under: virtualization — Tags: , , , , — iben @ 12:42

    SUMMARY: For best performance be sure to upgrade your VMFS Block Storage when you upgrade your ESX hosts to vSphere.

    VMFS 3 versions and upgrade paths


    It is not possible to upgrade an existing VMFS to a later version. However, all VMFS versions work with any version of ESX 3.0.0 and later. That is, ESX 3.0.0 can run a virtual machine from VMFS 3.33 and ESX 4 can run virtual machines from VMFS 3.21 volumes.


    VMFS3 which was released initially with ESX 3.0.0 as version 3.21 has since evolved with new minor versions:
    • ESX 3.0.0 is provided with 3.21 (initial release)
    • ESX 3.5.0 is provided with 3.31
    • vSphere (ESX 4.0) is provided with 3.33
    If for some reason you must upgrade your VMFS minor version:
    Warning: This removes the formatting of the LUN and all the data on the datastore. Relocate your virtual machines and files prior to removing the datastore.
    1. Migrate all the data off the VMFS datastore that you are upgrading.
    2. Delete the datastore from VI Client.
    3. On the VI Client connected to VirtualCenter, choose your ESX 3.5 or 4.0 host. Alternatively connect directly to the ESX host with the VI Client.
    4. Recreate the datastore from that ESX 3.5 or 4.0 host. Click Storage > Add Datastore.
    5. Migrate the data from step 1 to the newly formated datastore.

    Additional Information

    Features like VMFS grow in ESX 4 work regardless of the minor version.


    vSphere Network Connections and Ports

    Filed under: virtualization — Tags: , , , , — iben @ 11:57
    esx network ports

    esx network ports

    The amazing Dudley Smith, from VMware’s Technical Account Manager team has release a larger version of his vSphere Network Connections and Ports for ESX diagram and an accompanying excel spreadsheet listing all the TCP/IP ports for various communication purposes.

    Get them directly from the VMware blog site here:


    HyTrust Appliance 2.0 Released

    Filed under: virtualization — Tags: , , , , , — iben @ 09:32

    HyTrust Appliance 2.0 is available. Building on the successes of 2009, which included our initial product launch and numerous awards, we’re happy to see the streak continue into 2010 by delivering a major new release that will empower enterprises to capitalize on the wave of datacenter virtualization and accelerate efforts to virtualize tier-one applications. The features available in HyTrust Appliance 2.0 deliver true enterprise-class policy management and access control capabilities to virtual infrastructure. New features include the following:

    * Root Password Vault: Locks down privileged host accounts and provides passwords for temporary use to enable time-limited privileged account access. Root accounts on hypervisors are extremely powerful and, as a consequence, can create a significant liability if not kept out of the wrong hands. With the aid of Root Password Vault, all root account access is attributable to an individual and every action is logged, providing far greater visibility and accountability.
    * Federated Deployment: Secure distributed system architecture allows for automated replication of policies and templates across multiple HyTrust Appliances as well as geographic boundaries. For larger enterprises with multiple datacenters and collocation facilities, Federated Deployment of HyTrust Appliances ensures consistency of controls across the entire infrastructure.
    * Virtual Infrastructure Search: Enables quick and easy accessibility to all virtual infrastructure objects, policies, and logs within HyTrust Appliance.
    * Remote API: Interface to remotely access and automate the administration of the HyTrust Appliance. Provides the greater scalability demanded by large, enterprise-wide deployments of virtualization.
    * Object Policy Labels: Creates a policy categorization structure, similar to “Web 2.0 tagging” for virtual infrastructure objects, which enables better organization and tighter, more consistent controls. Object Policy Labels enable access, network segment, and zoning policies, which allows administrators to dictate which virtual machines are allowed to connect to which network segments or hosts via RuleSets and Constraints.
    * Router-Mode: a deployment option where all VMware management traffic is forced to flow through the HyTrust Appliance. HyTrust Appliance acts as a router for the “protected” management subnet and ESX/ESXi hosts and vCenter Server use HyTrust Appliance as their default gateway. This adds yet another flexible deployment option to the other existing options, ensuring the HyTrust Appliance will easily adapt to any enterprise architecture.

    Along with the new capabilities delivered in 2.0, we’d like to introduce you to the new editions of HyTrust Appliance:

    * Community Edition is a free version of the product that supports up to three hosts.
    * Standard Edition supports an unlimited number of hosts and offers more flexible deployment options.
    * Enterprise Edition supports an unlimited number of hosts, offers more flexible deployment options, supports federation of multiple HyTrust Appliances, enables privileged account management via Root Password Vault, allows two-factor authentication, and offers a remote API for additional management flexibility.

    You can download the Community Edition of HyTrust Appliance at

    Powered by WordPress