B2V Command Line Guide to VMware ESX Server 3

Last Updated 5th May 2009 by Alistair Sutherland

This guide has been compiled by the consultants & trainers at Taupo Consulting and is based upon their personal experiences with the VMware ESX Server 3 product and is updated frequently. For more information about Taupo Consulting, please click here. The information in this guide is not verified or sanctioned by VMware Inc. and we encourage our website visitors to use www.vmware.com/vmtn as their primary source of VMware product information. We are of course delighted if you find our shared experience documented in this guide of use in your environment and always appreciate being linked to. ESX Server 3 is the core component of VMware Virtual Infrastructure 3 and is generally available. The latest version is ESX 3.5 Update 2 on 25th July 2008. There is lots of new content coming to the command line guide, so please forgive the occasional gap as we work on updating the content! If you are using ESX server 2.x, you can click here for the command line guide to ESX 2.x

The esxcfg- Commands

esxcfgThere are a new set of command line tools in ESX 3.x which all start with "esxcfg-". These tools are used to configure each part of the ESX 3.x configuration. For example, esxcfg-firewall is used to manage the service console firewall while the esxcfg-nic is used to manage the physical Ethernet adapters present in the server. Watch out for vicfg- commands also. If you are using the RCLI tools for managing ESX 3i, then the esxcfg- tools are now prefixed with vicfg- although the esxcfg- prefix still works.

esxcfg-advcfg The esxcfg-advcfg command is interesting as there is not a huge amount of help about this command. However, we can figure out that it is meant to do advanced configuration and we can figure out some settings that can be made. The -g switch is used to "get" settings; the -s switch is used to "set" settings. Here are a few examples of some VMkernel parameters which can be interrogated.

[root@esx1host vmware]# esxcfg-advcfg -g /Misc/BlueScreenTimeout Value of BlueScreenTimeout is 0 [root@esx1host vmware]# esxcfg-advcfg -g /Misc/HostName Value of HostName is esx1.vmlab.net The question is, how much is configurable? To figure out what is configurable, we recommend that you look in the directory /proc/vmware/config which you will find in the service console command line and then you will see the following directories BufferCache Cpu Disk FileSystem Irq LVM Mem Migrate Misc Net NFS Numa Scsi User VMFS3 From these directories and the files within, you can work out the paths to be supplied to the esxcfg-advcfg command as parameters. Alternatively, you could also use the command esxcfg-info o to list the advanced options. We often see this tool used to make configuration changes relating to storage. For example, below, you can see we are checking to see if we are creating virtual disks in "eager zero" format by default, whether we will discover non-contiguous numbered LUNs, the maximum LUN number addressed,the SCSI conflict retry count and finally the logical volume manager (LVM) setting for resignaturing VMFS volumes. [root@esx1host vmware]# esxcfg-advcfg -g /VMFS3/ZeroedThickVirtualDisks Value of ZeroedThickVirtualDisks is 1 [root@esx1host vmware]# esxcfg-advcfg g /Disk/SupportSparseLUN Value of SupportSparseLUN is 1

[root@esx1host vmware]# esxcfg-advcfg g /Disk/MaxLUN

Value of MaxLUN is 255

[root@esx1host vmware]# esxcfg-advcfg g /Scsi/ConflictRetries

Value of ConflictRetries is [root@esx1host vmware]# esxcfg-advcfg g /LVM/EnableResignature Value of EnableResignature is

In this last example, we are again setting a parameter related to storage. This parameter limits the number of outstanding disk request for each VM. This is intended to equalise the disk access between virtual machines. [root@esx1host vmware]# esxcfg-advcfg -s 16 /Disk/SchedNumReqOutstanding When using the esxcfg-advcfg command, remember case sensitivity! Usage: esxcfg-advcfg <options> [<adv cfg Path>] -g|--get Get the value of the config option -s|--set <value> Set the value of the config option -d|--default Reset Config option to default -q|--quiet Suppress output -k|--set-kernel Set a VMkernel load time option value. -j|--get-kernel Get a VMkernel load time option value. -h|--help Show this message. -r|--restore Restore all advanced options from the configuration file. (FOR INTERNAL USE ONLY).

esxcfg-firewall The service console in ESX 3.x has a firewall enabled by default. The network packet filtering found in Red Hat Linux is called iptables. As the management of iptables is not entirely straightforward, the esxcfg-firewall command makes things a load easier. The firewall rules are stored in /etc/vmware/esx.conf, but we don't go editing this file, we use this command to ensure it is locked while we make our edits. If you are very interested in the iptables commands used behind the scenes, then you can inspect the log file /var/log/vmware/esxcfg-firewall.log We use the esxcfg-firewall command to view and configure the firewall rules. The most popular switch will be the -q switch to query the firewall for its current settings. [root@esxhost1 root]# esxcfg-firewall -q <output> The -s switch will allow you to enable or disable network services that may traverse the firewall successfully. The list of known services are shown below - very case sensitive!.... nfsClient ftpServer ntpClient dellom

nisClient vncServer tmpLicenseClient swISCSIClient CIMHttpsServer sshClient snmpd tmpAAMClient vpxHeartbeats smbClient hpim tmpHostVmdbServer tmpHostdSOAPServer ftpClient sshServer ibmdirector CIMHttpServer telnetClient The -l switch loads the firewall and enables the IP tables. The -u switch unloads the firewall and disables the IP tables. We use the -e switch to enable a particular known service, so if we wanted to enable ssh outbound connections from the service console we would simply enter [root@esxhost1 root]# esxcfg-firewall -e sshClient We use the -d switch to disable a service. In the following example, we prevent outbound connections [root@esxhost1 root]# esxcfg-firewall -d smbClient If we need to open a TCP or UDP port that is not described by a defined friendly name like "sshClient", then we can explicitly open that port with the -o switch. The service console firewall is bidirectional and so when opening a port you must also specify direction of incoming or outgoing. Equally, we can close an explicit port with the -c switch. [root@esxhost1 root]# esxcfg-firewall -o port,protocol,direction,name In the following example, we are opening a unique port which we are calling "MySQLclient". If we wanted to close a port that we had already opened, we would use the -c switch. [root@esxhost1 root]# esxcfg-firewall -o 3306,tcp,out,MySQLclient The service names such as sshClient and smbClient are defined in the file /etc/vmware/firewall/services.xml. It is strongly suggested that this file is not manually edited as changes are unlikely to survive host patch updates. A much better approach for defining services is to add a new XML file, for example the guys over at

Veeam very helpfully have already created one for you so you can enable their FastSCP tool - http://www.veeam.com/download/fastscp/FastSCP.xml. See more under the guide entry for services.xml. esxcfg-module This command is used to view and set options for start-up on the VMkernel modules (drivers). When this command is used with the list option, it produces an output similar to vmkload_mod -list [root@esx1host root]# esxcfg-module -l Module vmkapimod vmklinux cciss.o tg3.o qla2300_7xx.o Type vmkapimod linux scsi nic fc Enabled true true true true true Loaded true true false false false

This command is often used when we want to modify a VMkernel module behaviour, for example, if we wanted to change the queue depth of our fibre-channel host bus adapter. In the following example, we are setting the queue depth for our QLogic HBA to 64; up from it's default value of 32. [root@esx1host root]# esxcfg-module -s ql2xmaxdepth64 qla2300_707_vmw To do the same with an Emulex HBA, we would use something like [root@esx1host root]# esxcfg-module -s "lpfc0_lun_queue_depth=64" lpfcdd_7xx

esxcfg-rescan This command is used to perform a rescan of a host bus adapter (HBA). Specifically it scans a named vmkernel hba device, i.e. a vmhba. This command does a similar job to vmkfstools -rescan. In this example the esxcfg-rescan command is being used to rescan the VMkernel iSCSI software initiator vmhba. [root@esx1host]# esxcfg-rescan vmhba32

esxcfg-upgrade

esxcfg-upgrade -h --help

-g -f -r -o

--convert-grub --convert-fstab --upgrade-pre-vmkernel --upgrade-post-vmkernel

The -g option may only be used with the -r option.

esxcfg-vswitch This command is one of the most useful commands in the service console. This command allows you to list, add, modify or delete virtual Ethernet switches on an ESX host. The simplest option with this command is the -l option to list the virtual switches and portgroups defined on the host. [root@esx1host root]# esxcfg-vswitch -l If you are having problems with your ESX server after an in-place upgrade, this tool is invaluable in resolving the problems with service console networking. The output of this command is initially a little intimidating. It is best to keep in mind the network topology: Service Console IP Interface (vswif0) ---- connected to ----> Service Console Port on vSwitch ----- up-linked to ----> vmnic Where a vmnic is a physical Ethernet adapter. In following screenshot taken from the VI Client, we can see this ESX host has 2 connections to vSwitch0, the service console connection a VMkernel port connection.

If we wish to view the same information at the service console command line, we would use the esxcfg-vswitch command with the "-l" switch to list the defined virtual switches. [root@esx1host root]# esxcfg-vswitch -l Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch0

32

4 Internal ID portgroup0 portgroup1

32 VLAN ID 0 0 Used Ports 1 1

vmnic0 Uplinks vmnic0 vmnic0

PortGroup Name Service Console NFS access

If we wanted to add another virtual Ethernet switch, we would use esxcfg-vswitch command with the "-a" switch. Note that the -a is specified in lowercase. Take care to ensure you have specified lowercase because uppercase "A" performs a different function with this command. So, lets add a new virtual switch to our ESX host called vSwitch1 and then list the switches to check our command has worked ok. [root@esx1host root]# esxcfg-vswitch -a vSwitch1 [root@esx1host root]# esxcfg-vswitch -l Switch Name vSwitch0 Num Ports 32 Used Ports 4 Configured Ports 32 VLAN ID 0 0 Used Ports 1 1 Uplinks vmnic0 Uplinks vmnic0 vmnic0 Uplinks

PortGroup Name Service Console NFS access Switch Name vSwitch1

Internal ID portgroup0 portgroup1 Used Ports 0

Num Ports 64

Configured Ports 64 VLAN ID Used Ports

PortGroup Name

Internal ID

Uplinks

Notice that the number of ports on the virtual switch is 64 on the newly created switch. The original virtual switch has only 32. This difference arises between creating the switch in the VI Client or the command line. Notice also that the used port count doesn't immediately make sense. Each VM consumes a port, each vmnic consumes a port and the uplink itself consumes a port. Anyway, if you are like me and you can never remember which case of the letter "a" to use when adding a virtual switch, then use the esxcfg-vswitch command with the --add switch when creating a new switch like this: esxcfg-vswitch --add vSwitch2 which I think is a little clearer to understand. Now if we want to add a portgroup to the new virtual switch we have created, we can use the esxcfg-vswitch -A command. It does not matter whether you are creating a service console port, a VM port group or a VMkernel port when creating a port group; the way we create the connection to the virtual switch always starts out the same in the command line. Only after creating the port group do we then specify if it is to be anything other than a VM port group. In the following commands, we add a new portgroup called "Production" on the virtual switch vSwitch1. [root@esx1host root]# esxcfg-vswitch -A "Production" vSwitch1 [root@esx1host root]# esxcfg-vswitch -l

Switch Name vSwitch0

Num Ports 32

Used Ports 4

Configured Ports 32 VLAN ID 0 0 Used Ports 1 1

Uplinks vmnic0 Uplinks vmnic0 vmnic0 Uplinks

PortGroup Name Service Console NFS access Switch Name vSwitch1

Internal ID portgroup0 portgroup1 Used Ports 0

Num Ports 64

Configured Ports 64 VLAN ID 0 Used Ports 0

PortGroup Name Production

Internal ID portgroup2

Uplinks

Alternatively you could use the following command to add a port group to a virtual switch. [root@esx1host root]# esxcfg-vswitch --add-pg="Production" vSwitch1

This alternative switch of using --ad-pg I think is clearer for understanding what the command is doing. The --add-pg option can clearly be seen to add a portgroup to a virtual switch, and again is simpler to understand than just -A. The portgroup name in our example is called Production, but it can be what you want. We recommend adoption of a standard across all your virtual infrastructure. I have seen some clients align their portgroup names with the IP subnets, so you could have a portgroup called something like 192.168.1.0 subnet. Although we have now created a new virtual switch and have created a VM port group on it, the virtual switch itself does not have any uplinks. Remember that when we bind a physical network adapter to a virtual switch we are uplinking a vmnic to the switch and the switch then "owns" that adapter, i.e. it is not available to be used by any other virtual switches. We perform the uplink by using the esxcfg-vswitch command with the -L switch for link. [root@esx1host root]# esxcfg-vswitch -L vmnic1 vSwitch1 So in one simple command we have linked the physical network adapter vmnic1 to our new virtual Ethernet switch vSwitch1. If we then realised we had used the wrong physical adapter, we can just as easily unlink with -U. In the next example, we swap the uplinked vmnic1 for an alternative adapter vmnic2 [root@esx1host root]# esxcfg-vswitch -U vmnic1 vSwitch1 [root@esx1host root]# esxcfg-vswitch -L vmnic2 vSwitch1
This changing of vmnic bound to a virtual switch is often required post-installation, as we may select the wrong physical adapter to use for the service console during the install and need to correct our configuration before we can connect to our host with VI client!

VLANs with esxcfg-vswitch If we wish to do VLAN tagging in the virtual switch (VST), then we can assign a VLAN ID to a port group using the -v switch to this command. All traffic passing through this portgroup will now be tagged (IEEE 802.1q) with the VLAN ID specified as a numeric parameter after the -v switch. This must match the VLAN ID of the network defined in the physical switch topology in the range 1 through 4094. The physical switch port that the traffic uplinks through from ESX will also need to be configured to accept q-tagged traffic for that VLAN. In Cisco terminology this is a trunk port, in HP ProCurve terminology this is a tagged port. [root@esx1host root]# esxcfg-vswitch -v 3223 VMPortGroup1 vSwitch1 If you wanted to do VLAN tagging in the guest operating system itself - called Virtual Guest Tagging (VGT), then you can set the VLAN ID of the port group to 4095, which allows tagged traffic from the guest to pass through the portgroup.

Cisco Discovery Protocol with esxcfg-vswitch As of ESX 3.5, VMware added Cisco Discovery Protocol (CDP) support for virtual switches. We can view CDP information of the current neighbour of the physical NIC. In the VI Client, we can see this by clicking on the icon to the right side of the vmnic in the network view of the ESX host.

To display the CDP configuration setting for a virtual switch, we use the lowercase b switch, where we will find which of the four CDP modes it is in: disable, listen, advertise or both. [root@esx1host root]# esxcfg-vswitch -b vSwitch0 listen We can change the CDP mode with the -B (uppercase) option. Here we are changing virtual switch called vSwitch0 to support both advertise and listen. [root@esx1host root]# esxcfg-vswitch -B both vSwitch0 [root@esx1host root]# esxcfg-vswitch -b vSwitch0 both

esxcfg-auth Configures the service console user authentication options including NIS, LDAP, Kerberos and Active Directory. In the following command, we are configuring

authentication for the Active Directory domain called taupoconsulting.com [root@esx1host root]# esxcfg-auth --enablead --addomain=taupoconsulting.com --adddc=dc1.taupoconsulting.com You can also use this tool to set a password policy for service console user accounts. [root@esx1host root]# esxcfg-auth --maxpassdays=90 --minpassdays=30 --passwarnage=75 In the above example, your service console user account password would expire after 90 days, you would get a warning message after 75 and once changed, you would have to keep that password for a minimum period of 30 days.

esxcfg-info Produces an enormous amount of information about the state ESX host, often this tool is the one tool that can tell you what is really going on and not what is in some configuration file. If you run this command with no parameters, then you really need to pipe this to a file for closer examination! Over time as newer releases of ESX server are released, less information will be available in the proc nodes (the /proc/vmware directory structure), so the sooner we can get used to examining the current running configuration of ESX using this command, the better off we will be. In this first example, we will run the command with no switches and pipe the result into a file esxinfo-28-07-2008.txt (we like putting in the date of operation into the filenames of dumped files so we don't lose track!) and we are then viewing the contents with the less command, allowing us to scroll up and down through the file. [root@esx1host root]# esxcfg-info >/tmp/esxinfo-28-07-2008.txt [root@esx1host root]# less /tmp/esxinfo-28-07-2008.txt If you know the area you are looking at, e.g. storage, then we can launch the tool with the appropriate switch. Here are the six switch options: w r s n y o hardware resource storage network system advanced options

If we combine the filtering of the output using the above switches along with a grep filter we can really zoom in on the area we are interested in. An excellent VMware communities post gives an example of using the storage switch whilst looking for Pending reservations on LUNs. We are piping the result of the storage output of esxcfginfo into the input for grep. [root@esx1host]# esxcfg-info -s | grep Pending

Check out the post at http://communities.vmware.com/thread/156778?tstart=0 esxcfg-mpath Manages storage multi-pathing just as the vmkmultipath utility did in previous versions of ESX Server. In the example below we are using the -l switch to list the storage and paths. [root@esx1host tools-isoimages]# esxcfg-mpath -l Disk vmhba0:0:0 /dev/cciss/c0d0 (69459MB) has 1 paths and policy of Fixed Local 2:1.0 vmhba0:0:0 On active preferred Disk vmhba1:0:0 (0MB) has 1 paths and policy of Most Recently Used FC 10:1.0 210000e08b846a72<->5006016930221397 vmhba1:0:0 On active preferred Disk vmhba1:0:6 /dev/sda (9216MB) has 1 paths and policy of Most Recently Used FC 10:1.0 210000e08b846a72<->5006016930221397 vmhba1:0:6 On active preferred Disk vmhba1:0:21 /dev/sdb (10240MB) has 1 paths and policy of Most Recently Used FC 10:1.0 210000e08b846a72<->5006016930221397 vmhba1:0:21 On active preferred

esxcfg-resgrp Used to manage the new ESX feature called resource groups. This command can add, remove or modify existing resource groups.

esxcfg-hbadevs The esxcfg-vmhbadevs command is used to list the equivalent Linux device names for the visible disk devices that the VMkernel references using vmhba notation. [root@esx1host root]# esxcfg-vmhbadevs vmhba0:0:0 /dev/sda vmhba0:0:1 /dev/sdb vmhba0:0:2 /dev/sdc vmhba0:0:3 /dev/sdd vmhba2:0:0 /dev/sde vmhba2:1:0 /dev/sdf If we use this command with the m switch, then we only list the LUNs which contain VMFS partitions. Alongside the Linux device name, a long unique hexadecimal value is listed. This is the VMFS volume signature assigned by the new logical volume manager (LVM). [root@esx1host root]# esxcfg-vmhbadevs -m

vmhba0:0:0:1 /dev/sda1 45407607-fbc43ced-94cb-00145e231ce3 vmhba0:0:2:1 /dev/sdc1 455b08a8-8af7fee3-daa9-00145e231e35 vmhba2:0:0:3 /dev/sde3 4559c75f-831d8f3e-bc81-00145e231e35 You can view these volumes in the directory /vmfs/volumes/

esxcfg-boot Used to configure the GRUB options presented at boot time. One thing to note is that the new esxcfg commands will not run if you boot just into Linux. If you just want to query the boot settings, you can use the -q switch but this must be qualified with the keyword boot or vmkmod. [root@esx1host root]# esxcfg-boot -q boot 272 2:;7:;10:; UUID=847199e4-d3c7-11da-8ef8-930e3d734c03 /vmlinuz-2.4.2137.0.2.ELvmnix /initrd-2.4.21-37.0.2.ELvmnix.img [root@esx1host root]# esxcfg-boot -q vmkmod vmkapimod vmkapimod vmklinux linux cciss.o scsi tg3.o nic qla2300_7xx.o fc This is also used if you making modifications to VMkernel device drivers defaults. For example, if you were modifying the queue depth for a fibre HBA, you would likely be using esxcfg-module. Then to rebuild the boot image you would enter [root@esx1host root]# esxcfg-boot -m After which, you would do a reboot the host to test that the update to the boot image had worked.

esxcfg-init Should not be run manually!

esxcfg-nas

The esxcfg-nas command is used to list, mount and dismount NFS exports for the VMkernel. In the first example we list the NFS datastores which the VMkernel has mounted.
[root@esx1host root]# esxcfg-nas -l NFS01 is /NFS from 100.100.100.253 mounted

In the next example, we add a new VMkernel mount to a remote NFS server. This time we are connecting to the NFS server at IP address 100.100.100.253 and the name of the exported directory on the NFS server is /Test. We are labelling (from the ESX host perspective) this

NFS mount as NFS02. This will appear as the datastore name on the ESX host.
[root@esx1host etc]# esxcfg-nas -a -o 100.100.100.253 -s /Test NFS02 Connecting to NAS volume: NFS02 NFS02 created and connected. Remember that to create a connection to an NFS datastore, the VMkernel needs to have an IP address, as it is the NFS client. We give the VMkernel an IP address by creating a VMkernel port on a virtual Ethernet switch. We can do this at the command line using the command esxcfg-vmknic The command line options for esxcfg-nas are: esxcfg-nas <options> [<label>] -a|--add Add a new NAS filesystem to /vmfs volumes. Requires --host and --share options. -o|--host <host> Set the host name or ip address for a NAS mount. -s|--share <share> Set the name of the NAS share on the remote system. -d|--delete Unmount and delete a filesystem. -l|--list List the currently mounted NAS file systems. -r|--restore Restore all NAS mounts from the configuration file. (FOR INTERNAL USE ONLY). -h|--help Show this message.

esxcfg-route If we add an IP address to the VMkernel by adding a VMkernel port, then we can fully configure that IP stack by also assigning a default gateway. We can view (no parameters) and set (1st parameter) the VMkernel IP default gateway with the esxcfgroute command as shown here. In the following example, we view the current VMkernel gateway (.254) and then change it to a new one (.1) [root@esx1host etc]# esxcfg-route VMkernel default gateway is 100.100.100.254 [root@esx1host etc]# esxcfg-route 100.100.100.1 VMkernel default gateway set to 100.100.100.1 As of ESX 3.5, we have the -a switch which is used to add additional routes to the VMkernel routing table. We also have the -l switch to list the VMkernel routing table. In the following example, we list the routing table, then add a new static route for the 192.168.90.0/24 network and check it has added to the VMkernel routing table correctly. [root@esx1host etc]# esxcfg-route -l VMkernel Routes: Network 100.100.100.0 Netmask 255.255.255.0 Gateway Local Subnet

default

0.0.0.0

100.100.100.1

[root@esx1host etc]# esxcfg-route -a 192.168.90.0/24 100.100.100.165 Adding static route 192.168.90.0/24 to VMkernel [root@esx1host etc]# esxcfg-route -l VMkernel Routes: Network 100.100.100.0 192.168.90.0 default Netmask 255.255.255.0 255.255.255.0 0.0.0.0 Gateway Local Subnet 100.100.100.165 100.100.100.1

If we want to remove an entry from the VMkernel routing we use the -d switch. So in the following example, we are removing the newly added route. [root@esx1host etc]# esxcfg-route -d 192.168.90.0/24 100.100.100.165 Deleting static route 192.168.90.0/24 from VMkernel

esxcfg-vmknic Used to view and set configure the VMkernel ports on virtual Ethernet switches. A VMkernel port is a special type of port group on a virtual Ethernet switch which is used to assign an IP address to the VMkernel. The VMkernel only needs an IP address for VMotion, software-initiated iSCSI or NFS access.

If you need to create a VMkernel port at the command line, then you need to create a port group first and then enable it as a VMkernel port. This tool does not allow you to enable the VMkernel port for VMotion, you must either use vimsh or the VI client for that.
[root@esx1host root]# esxcfg-vswitch -A VMotion vSwitch0 [root@esx1host root]# esxcfg-vmknic -a -i 100.100.100.121 -n 255.255.255.0 VMotion The above commands would result in an additional connection to the virtual Ethernet switch, specifically a VMkernel port. The esxcfg-vmknic command has assigned the VMkernel an IP address & the portgroup called VMotion is now explicitly VMkernel port. Lets now add another VMkernel port, this time for NFS access to our NAS device. [root@esx1host root]# esxcfg-vmknic -a -i 100.100.100.21 -n 255.255.255.0 "NFS Access" The following screenshot displays the new VMkernel port connections on vSwitch0.

In the following example, we list the VMkernel ports, then use esxcfg-vmknic to delete one of them and then list them again. [root@esx1host etc]# esxcfg-vmknic -l Interface Port Group IP Address Netmask Broadcast MAC Address MTU Enabled vmk0 VMotion 100.100.100.121 255.255.255.0 100.100.100.255 00:50:56:6d:7c:7d 1514 true vmk1 NFS access 100.100.100.21 255.255.255.0 100.100.100.255 00:50:56:62:ca:f6 1514 true

[root@esx1host etc]# esxcfg-vmknic -d VMotion [root@esx1host etc]# esxcfg-vmknic -l Interface Port Group IP Address Netmask Broadcast MAC Address MTU Enabled vmk1 NFS access 100.100.100.21 255.255.255.0 100.100.100.255 00:50:56:62:ca:f6 1514 true As of ESX 3.5, we can set the MTU for VMkernel initiated traffic. We should be aware however that currently Jumbo frames is only technically supported for VMotion and not iSCSI or NAS, even though it does work. Anyway, if you decide you want to enable Jumbo Frames for an existing VMkernel port, you are going to have to delete and recreate that VMkernel port. The MTU size for a VMkernel port can only be set at creation time. So, continuing our above example, if we wanted to enable an MTU of 9000 on the port group "NFS Access" we would need to do the following: [root@esx1host etc]# esxcfg-vmknic -d "NFS Access" [root@esx1host etc]# esxcfg-vmknic -a -i 100.100.100.21 -n 255.255.255.0 -m 9000 "NFS Access" [root@esx1host etc]# esxcfg-vmknic -l Interface Broadcast Port Group MAC Address IP Address Netmask MTU Enabled

vmk2 NFS access 100.100.100.21 255.255.255.0 100.100.100.255 00:50:56:62:ca:f6 9000 true Notice that as each VMkernel interface is created, an interface name is created of the form vmkx where x is just an incremental value. So you can see as we recreated the "NFS Access" VMkernel interface, the interface was named a vmk2, where as previously it was vmk1. This shouldn't cause you any problems as this seems to just be an internal reference to the interface. The only time we've needed this number is when using the scary, yet powerful vimsh tool to enable VMotion on a VMkernel port from the command line - something we only tend to do in scripted installs of ESX. One final note on this utility is about the disable function. If you disable the VMkernel port, you cannot delete it while in this state. If you want to delete a VMkernel port, it must be enabled or the call to delete it is ignored. The command line options for esxcfg-vmknic are: esxcfg-vmknic <options> [[<portgroup>]] -a|--add Add a VMkernel NIC to the system, requires IP parameters and portgroup name. -d|--del Delete VMkernel NIC on given portgroup. -e|--enable Enable the given NIC if disabled. -D|--disable Disable the given NIC if enabled. -l|--list List VMkernel NICs. -i|--ip <X.X.X.X> The IP address for this VMkernel NIC. Setting an IP address requires that the --netmask option be given in same command. -n|--netmask <X.X.X.X> The IP netmask for this VMkernel NIC. Setting the IP netmask requires that the --ip option be given in the same command. -r|--restore Restore VMkernel TCP/IP interfaces from Configuration file (FOR INTERNAL USE ONLY). -h|--help Show this message.

esxcfg-dumppart Used to configure the VMkernel crash dump partition. The old ESX 2.x utility for this function (vmkdump) is still present on an ESX 3 server, but appears just to be for extracting dump files. So far, we have only used this utility to interrogate ESX hosts to determine where the dump partition has been created. Here is an example of viewing the dump partition. # esxcfg-dumppart -l VM Kernel Name vmhba0:0:0:7 Console Name /dev/cciss/c0d0p7 Is Active yes Is Configured yes

Remember that the dump partition does not show up when you run the vdf utility.

However it is visible if you run fdisk. In the following example, we are running fdisk to view the partitions. We can see the dump partition as c0d0p7, i.e. partition #7. Notice the Id of that partition is "fc", the custom partition type for VMkernel dump partitions. # fdisk /dev/cciss/c0d0 Disk /dev/cciss/c0d0: 36.3 GB, 36385505280 bytes 64 heads, 32 sectors/track, 34699 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot /dev/cciss/c0d0p1 * /dev/cciss/c0d0p2 /dev/cciss/c0d0p3 /dev/cciss/c0d0p4 (LBA) /dev/cciss/c0d0p5 /dev/cciss/c0d0p6 /dev/cciss/c0d0p7 Start 1 101 5101 7101 7101 7645 34600 End 100 5100 7100 34699 7644 34599 34699 Blocks 102384 5120000 2048000 28261376 557040 27601904 102384 Id 83 83 83 f 82 fb fc System Linux Linux Linux Win95 Ext'd Linux swap Unknown Unknown

The command line options for esxcfg-dumppart are: esxcfg-dumppart <options> [<partition>] -l|--list List the partitions available for Dump Partitions. WARNING: This will scan all LUNs on the system. -t|--get-active Get the active Dump Partition for this system, returns the internal name of the partition vmhbaX:X:X:X) or 'none'. -c|--get-config Get the configured Dump Partition for this system, returns the internal name of the partition vmhbaX:X:X:X) or 'none'. -s|--set Set the Dump Partition for this system and activate it, either vmhbaX:X:X:X or 'none' to deactivate the active dump partition. -f|--find Find usable Dump partitions and list in order of preference. -S|--smart-activate Activate the configured dump partition or find the first appropriate partition and use it(same order as -f). -a|--activate Activate the configured dump partition. -d|--deactivate Deactivate the active dump partition. -h|--help Show this message.

esxcfg-linuxnet There is not normally a command that a virtual infrastructure administrator should need. The tool is automatically used when you start an ESX server in troubleshooting mode; i.e. when you start only the service console Linux kernel and don't start the VMkernel. When you are working in the service console while the VMkernel is loaded, the service console's network interface is not called eth0, but is called vswif0 instead. This is

because the service console network interface is provided via a service console portgroup on a virtual Ethernet switch. If you restart your ESX server without the VMkernel, then standard Linux drivers and network card management is used. Therefore the network interface used in troubleshooting mode is called eth0 - just like any other regular Linux box. This tool is called by starting troubleshooting mode to replicate the IP parameters assigned to vswif0 to eth0. Should you want to investigate this command, the options are: esxcfg-linuxnet --setup --remove -h --help The --setup option cannot be combined with the --remove option.

esxcfg-nics This tool can be used to view and configure the speed and duplex settings of the physical network cards in the ESX Server. This tool can replace the mii-tool and modules.conf for network card management. In the following example, we run the list option to view all physical NICs and their properties. [root@esx1host etc]# esxcfg-nics -l Name PCI Driver Link Speed Duplex Description vmnic2 01:01.00 tg3 Up 1000Mbps Full Broadcom Corporation NetXtreme BCM5703 Gigabit Ethernet vmnic0 01:02.00 tg3 Up 100Mbps Full Broadcom Corporation NC7781 Gigabit Server Adapter (PCI-X, 10,100,1000-T) vmnic1 04:02.00 tg3 Up 1000Mbps Full Broadcom Corporation NC7781 Gigabit Server Adapter (PCI-X, 10,100,1000-T) This command has the following optional parameters: esxcfg-nics <options> [nic] -s|--speed <speed> Set the speed of this NIC to one of 10/100/1000/10000. Requires a NIC parameter. -d|--duplex <duplex> Set the duplex of this NIC to one of 'full' or 'half'. Requires a NIC parameter. -a|--auto Set speed and duplexity automatically. Requires a NIC parameter. -l|--list Print the list of NICs and their settings. -r|--restore Restore the nics configured speed/duplex settings (INTERNAL ONLY) -h|--help Display this message. esxcfg-swiscsi ESX server 3 supports both hardware and software initiated iSCSI. For hardware iSCSI, we can use host bus adapters which perform the TCP offload and so the VMkernel can

just pass SCSI commands to them as normal. The iSCSI hba can then wrap the SCSI command in IP transport and forward them to the iSCSI target. In VI-3, one of the supported iSCSI hardware HBAs is the QLogic 4052. More information about this particular family of adapters can be found at http://support.qlogic.com/support/product_resources.asp?id=964 In software iSCSI initiator, the wrapping of SCSI commands in IP is performed by the VMkernel and a regular physical network card is used to communicate with the iSCSI target. The software iSCSI configuration is exposed in the VI Client as a host bus adapter called vmhba40 in ESX 3.0.x and is called vmhba32 in ESX 3.5. We can use this command line tool esxcfg-swiscsi to configure the software iSCSI initiator. The software iSCSI initiator in the VMkernel has a dependency upon the service console, therefore both the service console and VMkernel must have an IP route to the iSCSI target. The esxcfg-swiscsi command is not used in isolation, we use it in a sequence of commands to fully configure iSCSI from the service console command line. 1. Add a VMkernel port to a vSwitch that has an uplink and route to iSCSI target 2. Ensure service console IP interface has a route to the same iSCSI target 3. Using either the VI Client security profile or the esxcfg-firewall, open a port in the service console firewall for iSCSI (TCP:3260) 4. In the command line, enable iSCSI with the command esxcfg-swiscsi -e 5. Enable a discovery address with the command vmkiscsi-tool -D -a 10.0.0.99 vmhba32 6. List the targets that were discovered with vmkiscsi-tool -T -l vmhba32 7. Perform a rescan with esxcfg-rescan vmhba32 8. List the iSCSI LUNs with vmkiscsi-tool -L -l vmhba32 If you want to ensure the VI client reflects the changes made at command line, it is best to restart the vmware management service with the command service mgmt-vmware restart The full list of command line options for this command are: -e, --enable Enable sw iscsi -d, --disable Disable sw iscsi -q, --query Check if sw iscsi is on/off -s, --scan Scan for disk available through sw iscsi interface -k, --kill Try to forcibly remove iscsi sw stack -r, --restore Restore sw iscsi configuration from file (FOR INTERNAL USE ONLY) -h, --help Show this message

esxcfg-vswif This tool can manage the Ethernet interfaces of the service console. In a big change from previous versions of ESX, the Ethernet interface of the service console is named with the "vswif" prefix and not "eth" prefix as you may be used to in Linux. During installation of ESX server, your service console Ethernet connection should have been created. However, maybe a mistake was made, or we want to add another service

console port for redundancy. In VI Client we can view the network configuration of our ESX host. Here is an example of a typical network configuration.

If we use the esxcfg-vswif tool, we are examining, creating or modifying a service console port. So in the first example here, we are simply listing what ports have been created. # esxcfg-vswif -l Name Port Group Enabled DHCP vswif0 Service Console 192.168.31.255 true IP Address 192.168.31.31 false Netmask 255.255.255.0 Broadcast

So the output is showing the same as the graphical output in VI client. If we wanted to add a 2nd service console port, we could use this command. However, all this command will do is turn a regular portgroup into a service console port and bind an IP address to Linux. So in the following command line example, we create a portgroup first, and then we turn it into a service console port with esxcfg-vswif. # esxcfg-vswitch --add-pg="Service Console Backup" vSwitch1 # esxcfg-vswif -a -i 10.10.1.31 -n 255.255.0.0 -p "Service Console Backup" vswif1 [2007-11-21 11:29:18 'Vnic' warning] Generated New MAC address, 00:50:56:4d:da:97 for vswif1 Nothing to flush. So now if we run esxcfg-vswif to list the service console ports, we will be able to see the original service console port as well as our new one we just created. We've shown you the graphical representation as well from the VI client so you can compare. # esxcfg-vswif -l

Name Port Group IP Address Broadcast Enabled DHCP vswif0 Service Console 192.168.31.31 192.168.31.255 true false vswif1 Service Console Backup 10.10.1.31 10.10.255.255 true false

Netmask 255.255.255.0 255.255.0.0

A new function was added to esxcfg-vswitch when ESX 3.5 was released at the end of 2007. This version of ESX server was the first to support Ethernet Jumbo Frames. This is where the MTU size is increased beyond the default 1500 bytes. In the following example, we are changing the maximum MTU for vSwitch1. # esxcfg-vswitch -m 9000 vSwitch1

Configuration Files
/etc/vmware/esx.conf An all new configuration file for ESX Server 3.x. This file replaces the functionality of the following configuration files found in earlier versions of ESX. /etc/vmware/hwconfig /etc/vmware/devnames.conf /etc/vmware/vmkmodule.conf /etc/vmware/netmap.conf /etc/vmware/vmkconfig This file should not be copied from one ESX host to another in order to duplicate configuration, it is unique to the host. The file groups similar settings by using a notation similar to directories and subdirectories; for example, here is a section of esx.conf

<output>

/etc/nsswitch.conf This is the name service switch configuration file. If you need to modify the order of how names in the service console are resolved, this is the place to make the change. You can view and edit this conf file as usual. There will be a number of lines to this file, but the one you are likely to be interested in will start "hosts:" as shown: hosts: files nisplus dns In the above example, the name service will use the /etc/hosts file, then NIS+ and then the DNS name server specified in the /etc/resolv.conf file. If the application that is trying to use a hostname is using the libc resolver library (by using the gethostbyname function call) the nsswitch.conf file is used. However, an application could use its own resolver library. An example of this is the dig utility for testing DNS lookups - this tool ignores the /etc/nsswitch.conf file.

/usr/bin/vmware-watchdog This process watches over the hostd process and restarts it if it crashes.

hostd This is the daemon that replaces vmware-serverd that was found in the ESX 2.x products. This is the host management agent and is responsible for a number of key management functions on an ESX host. If you are having any "host not responding" type problems, before you even think of an ESX host restart, consider just a restart of the management agent; it's amazing how often a quick restart of hostd gets things going again. We can restart the host management agent with the command service mgmt-vmware restart

/var/log/vmware/hostd.log The log file for the host management agent. /etc/vmware/firewall/services.xml This file contains the definitions for the TCP ports and service names used by the service console firewall. When we use the esxcfg-firewall command to open ports based on friendly service names such as sshServer, that name is a definition in this XML file. A typical service definition in this file looks like <service id='0000'>

<id>sshServer</id> <rule> <direction>inbound</direction> <protocol>tcp</protocol> <port type='dst'>22</port> <flags>-m state --state NEW</flags> </rule> </service> You could modify this XML file to include your own definitions but this is not recommended by VMware. The VMware management agent (hostd) will load everything in this file, whether it is valid or not. Also, we have not tested if such a change would persist through a patching/upgrade, but we suspect not. Duncan Epping over at Yellow Bricks has done some great testing and documentation in this space and at the following link demonstrates how to add your own custom.xml file to the /etc/vmware/firewall directory (using same format as services.xml) to provide custom port definitions. You can read all about it at http://www.yellow-bricks.com/2007/12/31/howto-adding-afirewall-service-on-esx/. Just make sure you use ids in the file that are different than the ones in services.xml.

vpxa This is the name of the VirtualCenter server agent that runs in the service console of ESX 3.x servers (which was called vmware-ccagent in ESX 2.x). This can be stopped, started or restarted with the service command service vmware-vpxa restart

/etc/vmware/vpxa.cfg This is the XML configuration file for the VirtualCenter Server Agent in the service console. Here is a typical vpxa.cfg file. [root@esx1host vmware]# cat vpxa.cfg <config> <log> <outputToConsole>false</outputToConsole> </log> <nfc> <loglevel>error</loglevel> </nfc> <vmacore> <ssl> <doVersionCheck>false</doVersionCheck> </ssl> <threadpool> <TaskMax>10</TaskMax> </threadpool> </vmacore> <vpxa> <datastorePrincipal>root</datastorePrincipal> <hostIp>100.100.100.11</hostIp>

<memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs> <serverIp>100.100.100.172</serverIp> <serverPort>902</serverPort> </vpxa> <workingDir>/var/log/vmware/vpx</workingDir> Notice the <loglevel> tag. If you are trying to troubleshoot an issue, then increasing the logging level is a good idea. We have used the level "verbose", there could be a higher debug level of logging, but we've not tested that. We have also found the loglevel trivia, info, warning and error. /var/log/vmware/vpx/vpxa.log The log file for VirtualCenter agent in the service console.

VMware Command Line Tools

vmkfstools Used to manipulate VMFS and virtual disks at the service console command line. In ESX2.x we used it most often for import and export operations, where a virtual disk is converted from monolithic format to sparse format (previously called COW format). Now we tend to use it in ESX scripted install scripts to automate VMFS configuration. VMFS Manipulation with vmkfstools We can use vmkfstools to create VMFS file system, if we have a partition of type fb already created on it. In the following example, we are creating a VMFS3 datastore on partition 1 on LUN25 accessible via host bus adapter vmhba1. We are specifying a VMFS block size of 2MB and setting a volume label (datastore name) of "fc-lun25-tier1". We like embedding useful information in the datastore name to assist the operator in selecting the appropriate storage when provisioning VMs. vmkfstools -C vmfs3 -b 2m -S fc-lun25-tier1 vmhba1:0:25:1 VMFS volumes can be spanned across LUNs. We are not big fans of this as it tends to indicate storage wasn't planned in the first place and now things have reached crisis! However, they can be useful in certain circumstance and vmkfstools steps up again.

Virtual Disk Manipulation with vmkfstools The -X (case-sensitive) switch is used to extend the size of a virtual disk; e.g. if you had a 10GB virtual disk and wanted to extend it to 20GB, you could use this command. The VM would need to be powered off for this to work.

vmkfstools -X 20GB /vmfs/volumes/storage1/vm.vmdk Note that the -X switch specifies the NEW SIZE of the virtual disk and NOT how much you are extending it by. If you have used the -X switch before in an older version of ESX server (earlier than 3.0) it was possible to specify a small disk size; thereby making the virtual disk smaller. This was dangerous but useful if your partition within the disk did not consume 100% of the disk size. However, this is not possible with vmkfstools command found in ESX Server version 3.x. From ESX 3.5, the size of a virtual disk can now be increased in the VI Client! VMware are implementing more and more in the user interface, less time needed in the service console command line... Previously, the main use of vmkfstools command was to import or export virtual disks. This would be required if you were deploying templates by hand instead of using VirtualCenter. It was also the primary method for moving VMs between the ESX server product and the hosted VMware products such as VMware Workstation or Server. The reason we say "previously" is that moving VMs between servers or between VMware products has become much simpler and cleaner by using the VMware Converter utility. This tool is task oriented and treats the VM as a whole object, not just the virtual disk files as vmkfstools. If you do want to import virtual hard disks that are in 2GB sparse format into monolithic format by hand, then we can use vmkfstools command with the -i switch. vmkfstools -i /importfiles/vm.vmdk /vmfs/volumes/storage1/vm/vm.vmdk Notice that the import option requires two parameters, source and destination. This would not create a VM, but would create the monolithic virtual disk for a VM. You could then create a custom VM in the VI Client and select the option to "use an existing disk". If you want to export a virtual disk you no longer use the -d <type> switch, but just use the same -i switch and specify the virtual disk type at the destination of the import. So if you were exporting a virtual disk from VMFS to a ext3 directory path you could use: vmkfstools -i /vmfs/volumes/storage1/vm/vm.vmdk -d 2gbsparse /exportvm/vm.vmdk Fragmentation of Virtual Disks All being well, our storage is well planned out, disks are thick provisioned at creation and we get no surprises. However, things are not as straightforward as we always want. The business want changes to the virtual disk sizes, they want to save money on storage provisioning etc! So, it is possible that a virtual disk could be fragmented. The vmkfstools command can help us again here. The undocumented -t switch will show us how many contiguous sections a virtual disk has. If it only has 1 section, then it's not

fragmented. vmkfstools -t /vmfs/volumes/storage1/vm/vm.vmdk

vmware-cmd This command has been in ESX for a number of versions and it's functionality has been extended with each major release. We tend to find that the most frequent use of this command is to register or power on VMs from the console command line # vmware-cmd -s register /vmfs/volumes/SharedVMs/vm1/vm1.vmx # vmware-cmd /vmfs/volumes/SharedVMs/vm1/vm1.vmx start If you have a VM that you can't tell if it is powered on or off, you can use the getstate option # vmware-cmd /vmfs/volumes/SharedVMs/vm1/vm1.vmx getstate vm1 is powered on If you need to force the VM to power off, the stop hard function will normally do the trick. This is not very graceful, but can save you time if things are not responding. # vmware-cmd /vmfs/volumes/SharedVMs/vm1/vm1.vmx stop hard If there is limited space in your VMFS volumes, then you will likely want to know if any of your VMs are running in snapshot (where the disk writes are going into a disk delta and not the regular parent virtual disk). It is a nice idea to have a short script to enumerate the VMs on your host and loop through them to check each of them to see if they have a snapshot. The vmware-cmd command again helps us out with this.

vm-support A great built-in tool which collects all configuration files on an ESX host and builds a tar archive that can be sent to VMware support so they can have a complete picture of your system to assist in the troubleshooting effort. A useful function of this tool is to list running VMs using the -x switch. [root@esx1 root]# vm-support -x <output> [root@esx1 root]# Watch out for the creation of empty subdirectories of the name "vm-support.<pid-ofprocess>" in the directory where you run this tool with the -x switch. It is safe to delete these directories. You can't run this command if your current directory is /proc.

A less well-known option of vm-support is the ability to capture host performance data which can be replayed later using esxtop. To invoke the performance capture, we need to specify how frequently a performance "snapshot" is taken and over what period of time. For example, if we wished to capture host performance every 30 seconds for 10 minutes, then we would invoke vm-support with the following options [root@esx1 root]# vm-support -S -i 30 -d 600 The performance snapshots are archived automatically into a tgz file (a tgz file just like a WinZIP (R) archive). The tgz archive file name produced is unique to each time it's run, as the name includes date, time and process id of vm-support. Before we can actually replay the snapshot performance data in esxtop, we need to extract the tgz archive. The tar command is used to "unzip" tgz archive files. [root@esx1 root]# tar -zxvf archive.tgz To replay the data in esxtop, use the "-R" switch to specify replay mode and supply the path to the performance capture file produced by vm-support.

esxupdate This utility is what we use to patch our ESX hosts with updates from VMware. You can use this tool interactively to install individual patches, or use it to scan your ESX host to see which patches are required as well as to do a "what-if" install of a host patch to identify if there will be any problems. The power of the esxupdate command is realised when you use it with a patch repository. A patch repository can be exposed to a host via HTTP, FTP or NFS. esxupdate -d ftp://taupopatchserver/esx35/0710-03 scan - Bundle Name iFlags ESX350-200710049-BG ESX350-200710050-SG ESX350-200710052-BG ESX350-200710053-BG ESX350-200710054-BG ESX350-200710055-BG ESX350-200710058-RG ESX350-200710059-RG AppFlags -------v i------v i------v --------------v --------------v -------v --- Summary --Bugs fixed in some vmkernel. Security bugs fixed in vmkernel module.. Several bugs fixed in vmx module... Provided new PBM for SUSE 11 U2. COS fix for Ooops. More fixes in scsi drivers. This is a roll-up bundle. This is a roll-up security bundle. rmrm-m--rmr-rmrm-

If you choose to use the new VirtualCenter Server 2.5 feature called VMware Update Manager (VUM), then when you perform host scans and remediation, you are in fact just remotely invoking this utility, it's just you don't see it! You can use the --explain switch when scanning to provide a greater level of detail to

your host patch scan operation. If for example, the AppFlags for a patch indicated "c" for conflict, you would probably want to know what exactly the patch was in conflict with.

/var/log/vmware/esxupdate.log The log file for the esxupdate host patch utility. contents.xml Every ESX patch contains a file called contents.xml. This file describes the directory structure of the patch bundle contents.

contents.xml.sig This is a detached PGP signature of the contents.xml file in a ESX patch.

vimsh This is a superb utility that we use on occasion, particularly when we are creating scripted builds for ESX. The industry-recognised experts in the functions of this tool are the folks over at www.xtravirt.com. Where we have found this tool of unique use is in the enabling of a VMkernel port for VMotion. If you are using ESX versions prior to 3.5 then use vimsh -n -e "hostsvc/vmotion/vnic_set portgroupname However, if you are using ESX version 3.5 then we need to use a slightly different syntax for specifying the portgroup to enable. We now need to specify using a vmkx notation. Trouble is, we don't know which portgroup corresponds to which vmkx number. So to first identify the mapping of portgroup name to vmk number, we enter the command vimsh and then enter hostsvc/vmotion/netconfig_get and we'll get a whole pile of output, but buried in there will be the device names in vmkx format that we can then use to enable VMotion on that portgroup with the following: vimsh -n -e "hostsvc/vmotion/vnic_set vmk0 Using the vimsh command for enabling VMotion is just 1% of the functionality of this tool. It's not for the faint hearted and there really is no better source of information about it than the PDF documents that the xtravirt guys have written. Find their article here http://knowledge.xtravirt.com/white-papers/scripting.html .Thanks also to Mike Laverick of RTFM Education (www.rtfm-ed.co.uk) for documenting the changes in vimsh in version 3.5.

vmware-vim-cmd This command is a variation on the vimsh command that allows faster execution as we can invoke this command using the same options we use with vimsh, however this time we don't end up inside the vimsh shell after execution. If you use vimsh, after execution you are in a weird shell with a prompt like the following: [/] $ that you need to type exit to escape from. Using vmware-vim-cmd is straightforward as you just run the command and you are returned to the regular bash shell in the service console. For example vmware-vim-cmd /hostsvc/hostsummary

RPM Utilities
rpm As ESX service console is based on modified Red Hat Enterprise Linux 3, we can use the RPM package installation method to add applications to it. However, we should also point out that it's maybe not the best idea to add software to the service console. It is best to treat the service console as a dedicated console and not add applications to it. If you are unfamiliar with RPMs in Linux, think of them like MSI packages in Windows. The rpm command can be used to list and to install RPM-based applications. In the following example, we are using the command switch (-qa) to list the rpms installed in the service console. # rpm -qa libgcc-3.2.3-53 setup-2.5.27-1 basesystem-8.0-2 tzdata-2005m-1.EL3 glibc-2.3.2-95.37 bzip2-libs-1.0.2-11.EL3.4 etc!..... If we are only interested in the VMware rpms, then we can just pipe the output of rpm -qa command into the grep search tool. rpm -qa |grep VMware

which should yield an output something like VMware-webCenter-esx-2.0.1-32041 VMware-esx-apps-3.0.1-32039 VMware-esx-iscsi-3.0.1-32039 VMware-esx-uwlibs-3.0.1-32039 VMware-esx-vmkernel-3.0.1-32039 VMware-esx-drivers-block-DAC960-2.4.11-32039 VMware-esx-drivers-net-bcm5700-7.3.5-32039 VMware-esx-drivers-net-e100-2.3.40-32039 VMware-esx-drivers-net-pcnet32-1.30c-32039 VMware-esx-drivers-net-tg3-3.43b.1vmw-32039 VMware-esx-drivers-scsi-adp94xx-0.0.5-32039 VMware-esx-drivers-scsi-aic7xxx-6.3.9-32039 VMware-esx-drivers-scsi-lpfcdd-v732-7.3.2.1vmw-32039 VMware-esx-drivers-scsi-megaraid_sas-0.0.2-32039 VMware-esx-drivers-scsi-qla2200-v7.07-7.7.4.1vmw-32039 VMware-esx-drivers-scsi-qla4010-3.24-32039 VMware-esx-drivers-scsi-vmkiscsi-3.4.2-32039 VMware-hostd-esx-3.0.1-32039 VMware-esx-lnxcfg-3.0.1-32039 VMware-esx-perftools-3.0.1-32039 VMware-esx-docs-3.0.1-32039 VMware-esx-tools-3.0.1-32039 VMware-esx-vmkctl-3.0.1-32039 VMware-esx-drivers-block-cciss-2.4.54-32039 VMware-esx-drivers-net-3c90x-1.0.2-32039 VMware-esx-drivers-net-bnx2-1.3.22-32039 VMware-esx-drivers-net-e1000-7.0.33.2vmw-32039 VMware-esx-drivers-net-s2io-1.7.6-32039 VMware-esx-drivers-scsi-aacraid_esx30-1.1.5.1vmw-32039 VMware-esx-drivers-scsi-aic79xx-6.3.9-32039 VMware-esx-drivers-scsi-ips-7.10.17.1vmw-32039 VMware-esx-drivers-scsi-megaraid2-2.10.7-32039 VMware-esx-drivers-scsi-mptscsi_2xx-2.6.34.1vmw-32039 VMware-esx-drivers-scsi-qla2300-v7.07-7.7.4.1vmw-32039 VMware-esx-drivers-scsi-qla4022-3.24-32039 VMware-esx-vmx-3.0.1-32039 VMware-esx-srvrmgmt-3.0.1-32039 VMware-esx-backuptools-3.0.1-32039 VMware-esx-scripts-3.0.1-32039 VMware-esx-3.0.1-32039 VMware-cim-esx-3.0.1-32039 VMware-vpxa-2.0.1-32042

If we then want to find out more information on an individual RPM package, we can use the rpm -qi option to query a package which reports the file version, vendor, license and description. # rpm -qi VMware-hostd-esx-3.0.1-32039 Name Version : VMware-hostd-esx : 3.0.1 Relocations: (not relocatable) Vendor: VMware, Inc.

Release : 32039 Build Date: Tue 26 Sep 2006 01:30:42 AM PDT Install Date: Tue 06 Nov 2007 03:07:02 PM PST Build Host: pabuild43.eng.vmware.com Group : Applications/Emulators Source RPM: VMware-hostd-esx3.0.1-32039.src.rpm Size : 269864433 License: commercial Signature : (none) Summary : VMware Host Agent package. Description : If we then want to know what files are included in the rpm package, we can use query with the list option to see the files inside. For example, to see the files # rpm -ql VMware-hostd-esx-3.0.1-32039 /etc/vmware/hostd/config.xml /etc/vmware/hostd/env/0.xml /etc/vmware/hostd/env/1.xml /etc/vmware/hostd/env/vmconfigoption-esx-2.5.0.xml /etc/vmware/hostd/env/vmconfigoption-esx-3.0.0.xml /etc/vmware/hostd/environments.xml /etc/vmware/hostd/esxinfo.vha ..... If you want to install an RPM, run rpm -ivfh?XXX

rpm2cpio If you are wanting to extract a single file from a RPM package but you don't want to install the RPM, then this is the tool for you. Probably best if you copy the RPM to a temp directory so when you extract the RPM you can then navigate the directory structure created in that temp directory to find the file or files you need. Once you have copied out the file you were after, you can safely delete the contents of that temp directory. In other words, we have used rpm2cpio to extract the RPM archive. Here is an example using the RPM we've used in the previous examples. # rpm2cpio VMware-hostd-esx-3.0.1-32039 | cpio -idmv i = Restore archive d = Create landing directories m = Create previous file modification times

v = verbose

Linux Utilities
/etc/ssh/sshd_config The configuration of SSH client is stored in the text file /etc/ssh/ssh_config The configuration of the SSH server daemon is stored in the text file /etc/ssh/sshd_config. An important setting in this file is PermitRootLogin=No. This is the default setting in ESX 3.x and it is recommended that you keep the setting at "No". This way you have an audit trail and see exactly who is logging in, rather than just "root". You can quickly what the setting is by using a grep operation on the file as shown: # grep Permit /etc/ssh/sshd_config If you do edit the file to change this setting to Yes, then make sure you restart the daemon for the changes to take effect using the command: # service sshd restart It is also possible to explicitly allow or deny specific users to the SSH daemon. The headings in the ssh_config file are DenyUsers and AllowUsers.

su This command is the switch user utility. Think of it as the command line equivalent of Windows Fast User Switching! When it used without parameters, we are specifying to switch to the user root. However, we can use the su command to switch shell to any user account that we know the password of. In the first example, we are logged in as the user kevin and we are switching to user ali. [kevin@esx1host kevin]$ su ali Password: [ali@esx1host kevin] In this second example, we are switching from being logged on as a user called sara to being logged on as root. Notice to switch to root, we don't need to specify a username. [sara@esx1host sara]$ su Password: [root@esx1host root]#

If we restrict the built-in user account root from logging in over the SSH protocol, then we are forcing remote users to authenticate as themselves and then su to run privileged commands if need be, thus leaving a decent audit trail. The downside being that those users would still know the root account password. If you would like to restrict the use of the su command, then we can limit it to the members of a specific group called wheel. This group is defined in the /etc/group file by default and it's membership can be modified by root. In order to limit su to the wheel group members we need to modify a configuration file called /etc/pam.d/su There is a single line in this file that needs to be uncommented to limit the use of su. The line is shown below as it appears it that file, all that is required is the removal of the # symbol at the start of the line. #auth required /lib/security/$ISA/pam_wheel.so user_uid The attempts to switch to the root account are logged in /var/log/messages.

sudo The downside of the su command is that the operators who elevate their privilege to root are now root. They have full privilege, they know the root password, there is no granularity of delegation of privilege. Allows delegation of administration in terms of certain commands that normally only a particular user can execute (usually root). So if the user ali had been given the authority to run vmkfstools, then sudo would be used like: [ali@esx1 ali]$ sudo vmkfstools The vmkfstools command would then run under the security context of the root user. The superb feature of this tool is that the user ali does not need to know or supply the root password to be able to run the delegated command. Further, we keep an audit trail of when sudo was invoked in /var/log/secure. The sudo tool uses the lookup file /etc/sudoers to determine which users can perform which commands. We do not edit this file with a regular text editor like vi or nano, instead we use the tool visudo. visudo This is the vi text editor with extras. When launched, it automatically opens and locks for exclusive edit, the /etc/sudoers file. The point of visudo is to ensure we always edit the right file as the location of the sudoers file differs between nix distributions, but this command is constant and will utilise the right sudoers file for the distribution being used. A great benefit of using visudo over regular vi, is that it performs some basic syntax

checking for us!

/etc/sudoers The text file that contains the sudo users and the rules that apply to them. The first "ALL" relates to all machines (useful if this is a network wide file). Otherwise, this could be the hostname of the one machine we are trying to run the command on. In the following example we are allowing the user "alistair" to run the kill command, all of the commands in the directory /usr/bin and any commands in the directory /usr/sbin/alistair alistair ALL= /bin/kill, /usr/bin/, /usr/sbin/alistair/

In the following line added to the /etc/sudoers file, we are allowing the user sara to run the esxcfg-firewall and esxcfg-vswitch command. sara ESX1= /usr/sbin/esxcfg-firewall, esxcfg-vswitch

You can use aliases within this file to group together users, hosts and commands. User_Alias ESXHOSTADMINS-PROD = john, grant, julie User_Alias ESXHOSTADMINS-TEST = peter Host_Alias PRODESXHOSTS = esxprodsrv01, esxprodsrv02 Host_Alias TESTESXHOSTS = esxtest01, esxtest01 Cmnd_Alias SECURITYCOMMANDS = Cmnd_Alias VMCOMMANDS = Cmnd_Alias NETCOMMANDS = /usr/sbin/esxcfg-vswitch, /usr/sbin/esxcfgnics, /usr/sbin/esxcfg-vmknic Now we can combine these to create rules such as; ESXHOSTADMINS = PRODESXHOSTS NETCOMMANDS Although, rather than maintaining a static configuration file on each ESX host, it would be better to customize the sudoers file during host deployment and include Linux groups. For example, if we wanted to delegate a set of commands to those Linux users who belong to a Linux group, for example, wheel, then we can use the % operator to leverage those group definitions, thus avoiding static user aliases. %wheel = PRODESXHOSTS SECURITYCOMMANDS The best source we've found so far on detailed use and background of sudo can be found at http://aplawrence.com/Basics/sudo.html

w Great for viewing logged on console users. [root@esx1host firewall]# w

12:07:45 up 4 days, 2:16, 3 users, load average: 0.00, 0.00, 0.00 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root tty1 Fri10am 4days 0.02s 0.02s -bash root tty2 Fri 9am 4days 0.06s 0.06s -bash root pts/0 remote7.lab.vmwa 9:29am 0.00s 0.06s 0.00s w

who This command allows use to view who is logged onto the service console either interactively at the console or via an SSH session. The who command without parameters gives us the basics. [root@esx1host firewall]# who root tty1 Jul 25 10:30 root tty2 Jul 25 09:56 root pts/0 Jul 29 09:29 (remote7.lab.b2v.net) If we want to see all the details of users we can use the -a switch to show all data. We tend to combine -a with -H (i.e. -aH) to display column headers making it easier to read and interpret. [root@esx1host firewall]# who -aH NAME LINE TIME IDLE Jul 25 09:51 exit=0 system boot Jul 25 09:51 run-level 3 Jul 25 09:51 Jul 25 09:52 exit=0 root + tty1 Jul 25 10:30 old root + tty2 Jul 25 09:56 old LOGIN tty3 Jul 25 09:52 LOGIN tty4 Jul 25 09:52 LOGIN tty5 Jul 25 09:52 LOGIN tty6 Jul 25 09:52 root + pts/0 Jul 29 09:29 . (remote7.lab.b2v.net)

PID COMMENT 743 id=si last=S 1205 id=l3 2056 2057 2058 2059 2060 2061 18092

EXIT term=0

term=0

id=3 id=4 id=5 id=6

vmkload_mod This command will load and unload VMkernel modules on the fly. The results of this load/unload will happen as you type it and will only be valid for the current booted session. So this command is superb for troubleshooting as we can load and unload modules, e.g. network drivers. In the following example, we are examining the options for the Intel network driver (e1000) with the -s (show parameters) switch and then unloading it using -u (thereby interrupting network operations on that physical interface temporarily) and then loading it again with a new option. Notice to load a VMkernel parameter, we just supply the module name to the vmkload_mod command as a parameter listing any module-specific options as further parameters. [root@esx1host]# vmkload_mod -s e1000

Using /usr/lib/vmware/vmkmod/e1000.o heap_initial int, description "Initial heap size allocated for the driver." heap_max int, description "Maximum attainable heap size for the driver." reset_wait_ms int array (min = 1, max = 32) [root@esx1host]# vmkload_mod -u e1000 [root@esx1host firewall]# vmkload_mod e1000 RxIntDelay=32 Using /usr/lib/vmware/vmkmod/e1000.o Module load of e1000 succeeded. If you were experimenting with a driver setting to determine the right setting for the environment, then you may be loading and unloading a module a number of times. Once we were happy we had found the correct setting, it is likely we would want to make this change persistent, i.e. we would want it to take effect for each time the kernel module is loaded at server boot time. For that, we should use esxcfg-module command.

minicom This is a great utility for talking to serial attached devices; we think of it as HyperTerminal for Linux. Where we have found this particularly useful is for command line administration of your storage array. For example, if you had an HP MSA1000 attached to you ESX host and attached the serial cable to the unit and your host, then you could manage LUN presentation from the service console command line. Minicom uses a configuration file to determine bit rates etc. This configuration file is placed in the /etc directory. We normally create the file with a meaningful name e.g. minirc.com1, so to launch the tool we enter # ./minicom com1 The contents of the minirc.com1 file would typically be: pr pu pu pu pu pu port baudrate bits parity stopbits rtscts /dev/ttyS0 19200 8 N 1 No

Much more detail on minicom can be found at http://www.cisl.ucar.edu/nets/intro/staff/siemsen/tools/minicom.html

vi We can't talk about the command line without talking about vi. This is the simple but powerful text editor in Linux and UNIX. People tend to love it or hate it. Either way, it's nearly always there in any *nix implementation and just by memorising a few commands you can be up and running with it. If you can use Windows Notepad, you can

use vi! vi filename The first thing that throws you is that to enter text into your file, you need to press "i" for Insert mode. You can then enter your text just as any other text editor. When you are done with text entering, just press the Escape (Esc) key to come out of insert mode. If you are happy with your file, then we need to Write & Quit (wq). To enter commands in this command line editor, rather than having menus, we have a command prompt in the application. To reach the vi command prompt, simply enter ":" - the colon character which will automatically place your cursor at the bottom of the session. Here you can enter the "wq" command to write and quit the editor. That's it! Here is a summary of the vi commands i :wq :q! SHIFT ZZ ":wq" Esc key Changes to insert mode where you can edit the text Write the file and quit the editor Quit the editor without saving changes Quit the editor and save any changes made - just a fast way of doing Exits the current mode, e.g. out of insert mode back to view mode.

These commands are just extra if you have the inclination to learn! / search - if you entered /failed then the cursor would move to the first instance of "failed in the text $ jumps to the end of the opened file yy copy - it's y for yank! dd delete a line (cut) if you precede this with a number e.g. 8dd, then it would delete 8 lines p paste %s/old/new/g substitute any occurrences of the world "old" with the world "new" There are some great web sites which document the features of vi in superb depth, one of them is the staff site at University of Washington which helped me. Their site is at http://staff.washington.edu/rells/R110/

nano Another text editor, more friendly than vi but you should use w to avoid word wrap. /etc/ntp.conf [root@esx7 firewall]# cat /etc/ntp.conf # Prohibit general access to this service. restrict default ignore # Permit all access over the loopback interface. This could # be tightened as well, but to do so would effect some of

# the administrative functions. restrict 127.0.0.1 # # # # # # # # # # -- CLIENT NETWORK ------Permit systems on this network to synchronize with this time service. Do not permit those systems to modify the configuration of this service. Also, do not use those systems as peers for synchronization. restrict 192.168.1.0 mask 255.255.255.0 notrust nomodify notrap --- OUR TIMESERVERS ----or remove the default restrict line Permit time synchronization with our time source, but do not permit the source to query or modify the service on this system.

# restrict mytrustedtimeserverip mask 255.255.255.255 nomodify notrap noquery # server mytrustedtimeserverip

# --- NTP MULTICASTCLIENT --#multicastclient # listen on default 224.0.1.1 # restrict 224.0.1.1 mask 255.255.255.255 notrust nomodify notrap # restrict 192.168.1.0 mask 255.255.255.0 notrust nomodify notrap

# --- GENERAL CONFIGURATION --# # Undisciplined Local Clock. This is a fake driver intended for backup # and when no outside source of synchronized time is available. The # default stratum is usually 3, but in this case we elect to use stratum # 0. Since the server line does not have the prefer keyword, this driver # is never used for synchronization, unless no other other # synchronization source is available. In case the local host is # controlled by some external source, such as an external oscillator or # another protocol, the prefer keyword would cause the local host to # disregard all other synchronization sources, unless the kernel # modifications are in use and declare an unsynchronized condition. # server 127.127.1.0 # local clock server 0.vmware.pool.ntp.org server 1.vmware.pool.ntp.org server 2.vmware.pool.ntp.org fudge # # # # # # 127.127.1.0 stratum 10

Drift file. Put this in a directory which the daemon can write to. No symbolic links allowed, either, since the daemon updates the file by creating a temporary in the same directory and then rename()'ing it to the file.

driftfile /var/lib/ntp/drift broadcastdelay 0.008 # # Authentication delay. If you use, or plan to use someday, the # authentication facility you should make the programs in the auth_stuff # directory and figure out what this number should be on your machine. # authenticate yes # # Keys file. If you want to diddle your server at run time, make a # keys file (mode 600 for sure) and define the key number to be # used for making requests. # # PLEASE DO NOT USE THE DEFAULT VALUES HERE. Pick your own, or remote # systems might be able to reset your clock at will. Note also that # ntpd is started with a -A flag, disabling authentication, that # will have to be removed as well. # keys /etc/ntp/keys

/etc/ntp/step-tickers If you have a single time source configured for your service console, then this file will have just 1 line, similar to the following: server 192.168.1.100

ntpdate If you want to synchronise your service console clock with the defined time server, you can use this command with the -u switch. ntpdate -u timeserver.local

ntpq This queries the state of the ntp service. Watch for the back ticks used in the parameters, they are not single quotes!

date If we are checking the time and date of our ESX Service Console, then the date command is very useful. Just entering the "date" command returns what the service console thinks the current date is. If the date is incorrect and you wish to reset it you would enter the command with the -s

switch and specify date in mm/dd/yyyy format. # date -s "12/29/2007 23:48" Once you have set the date, you will want to ensure that the hardware clock matches your newly entered date. We can do this with the hwclock command described below.

hwclock We can use this command to synchronise the server hardware clock with the date we set in the service console. If you enter the command with no parameters then the value of the hardware clock is displayed. # hwclock If we want to synchronise the hardware clock with the service console date and time, we use the following: # hwclock --systohc cal Display calendar for current month or set of months. The following command displays 3 months, current month and the month before and after. # cal -3 March 2006 Su Mo Tu We Th 1 2 5 6 7 8 9 12 13 14 15 16 19 20 21 22 23 26 27 28 29 30

Fr 3 10 17 24 31

Sa 4 11 18 25

April 2006 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Su Mo 1 7 8 14 15 21 22 28 29

May Tu 2 9 16 23 30

2006 We Th 3 4 10 11 17 18 24 25 31

Fr 5 12 19 26

Sa 6 13 20 27

Surprisingly useful!

passwd Used to change the password of the currently logged on user (use the command with no parameters) or for changing the password of a named user account (supply the user name as a parameter). passwd <user> Remember that passwords are not stored in the /etc/passwd file (that's where users are defined) but are actucally stored in the file /etc/shadow If you are ever needing to reset an unknown root account password, then it is this

utility you would run after booting into Linux single user mode.

VMware HA
AAM AAM is the Automated Availability Manager that runs in the service console when you create a VMware High Availability (VMware HA) cluster. The VMware HA feature was previously known as DAS (Distributed Availability Services) but we don't mention that anymore. This software maintains an in-memory database on active nodes in the cluster and uses heartbeats to co-ordinate the active and passive nodes. It is suggested that you configure service console with 2 Ethernet interfaces to remove any single point of failure. This is a piece of licensed Legato software which itself has been renamed to EMC AutoStart. This component has a very high dependency upon fully functional host name resolution. So before you enable VMware HA, check the following files /etc/hosts /etc/FT_HOSTS /etc/resolv.conf /etc/vmware/esx.conf to ensure accuracy. One thing you can do to check the name resolution functionality before enabling HA is run hostname -s to return the short name of the service console. If this fails, then the HA configuration WILL fail. The log file for VMware HA in ESX 3.0.x can be found in the service console in the directory /opt/LGTOaam512/ and for ESX 3.5 can be found in /opt/VMware/ To avoid split brain scenarios, an ESX server can determine if it has become isolated from other servers and we can configure that servers' isolation response. If the AAM component loses contact with the other nodes in the HA cluster, it attempts to contact the configured default gateway for service console using ICMP echo request (PING). If

this fails, then the ESX host is isolated. If your default gateway suppresses ICMP echo requests, then we can configure an alternate IP address called the das.isolationaddress. From ESX 3.5, you can configure multiple isolation addresses so that you can configure a host with more that one address to attempt contact with before declaring itself isolated.

/opt/LGTOaam512/bin/ftcli This utility allows you to view the active nodes in an HA cluster and the managed IP addresses. This utility will help you determine whether the HA agent is in a running state and which IP addresses are visible between those managed hosts.

/etc/FT_HOSTS This file is created when HA is enabled and is a copy of /etc/hosts. If you have problems with name resolution and configuring HA, you can safely delete this file and reconfigure that cluster node for HA again. FT_HOSTS will be re-created.

Networking
hostname This utility displays the service console hostname. There are some useful switches to this command hostname -i displays the IP address and hostname -s displays the short hostname, i.e. without domain name

ifconfig Used to determine what IP address you have, the equivalent of the ipconfig command in Windows. You can use the command without parameters to view all interfaces, or you can be interface specific, e.g. [root@esx1host] # ifconfig vswif0 vswif0 Link encap:Ethernet HWaddr 00:50:56:49:96:03 inet addr:192.168.1.7 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4867312 errors:0 dropped:0 overruns:0 frame:0 TX packets:1980227 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000

RX bytes:1632239875 (1556.6 Mb)

TX bytes:138260324 (131.8 Mb)

Notice this is a quick way of viewing your COS virtual MAC address alongside the IP address. Also note, you don't see the additional optional IP parameters like gateway and DNS servers. The ifconfig command manages the addr part of the more powerful ip command.

ping Our favourite IP connectivity tool; I love the name Packet InterNetwork Groper! This tool uses ICMP to send an "echo request" and looks for an ICMP e"cho reply" . There are a couple of very useful switches we can use with ping, the most common one we use is -c to specify count. The Linux ping command keeps pinging continually by default (Windows needs -t to do that). So if we only want 4 pings, we specify -c4. [root@esx1host root]# ping 192.168.93.200 -c 4 PING 192.168.93.200 (192.168.93.200) 56(84) bytes of data. 64 bytes from 192.168.93.200: icmp_seq=0 ttl=63 time=0.507 ms 64 bytes from 192.168.93.200: icmp_seq=1 ttl=63 time=0.458 ms 64 bytes from 192.168.93.200: icmp_seq=2 ttl=63 time=0.448 ms 64 bytes from 192.168.93.200: icmp_seq=3 ttl=63 time=0.538 ms --- 192.168.93.200 ping statistics --4 packets transmitted, 4 received, 0% packet loss, time 3016ms rtt min/avg/max/mdev = 0.448/0.487/0.538/0.045 ms, pipe 2

Remember that often firewalls block ICMP echo requests, so not getting a reply doesn't mean your host is down! This tool relies on correct ARP functionality. So again, what looks like a ping failure, may in fact be a local ARP issue and unrelated to the destination address of the ping operation performed. If you are testing MTU sizes, you can force ping not to fragment with the -f switch. XXX We can set the ttl with this - poor mans traceroute!

vmkping This ping makes use of IP stack of the VMkernel rather than the Linux network stack in the service console. So if you are trying to troubleshoot VMotion, iSCSI or NAS issues where the VMkernel is directly using its own IP (a VMkernel port). We supply the IP address of the destination as a parameter, just as we do with regular ping. [root@esx1host root]# vmkping 192.168.93.200 64 bytes from 192.168.93.200: icmp_seq=0 ttl=63 time=0.871 ms 64 bytes from 192.168.93.200: icmp_seq=1 ttl=63 time=5.079 ms 64 bytes from 192.168.93.200: icmp_seq=2 ttl=63 time=22.754 ms --- 192.168.93.200 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.871/9.568/22.754 ms Be aware this tool makes use of the service console DNS, so if there are problems there, try vmkping using the IP address of the destination rather than hostname to ensure that any errors you see are unrelated to name resolution problems in the service console. If you use -D it will ping all important stations (own interface, iSCSI and default gateway).

/sbin/arping This is a similar utility to ping, but uses Address Resolution Protocol (ARP) and so the result will only be for local subnet resources, either another host or a gateway. [root@esx1host sbin]# arping -I vswif0 -c 2 192.168.1.1 ARPING 192.168.1.1 from 192.168.1.7 vswif0 Unicast reply from 192.168.1.1 [00:50:56:48:F3:AC] 0.912ms Unicast reply from 192.168.1.1 [00:50:56:48:F3:AC] 0.765ms Sent 2 probes (1 broadcast(s)) Received 2 response(s) Notice in the reply we see the MAC address of the target for the If this fails, it is a layer 2 problem on your local network. arp If you need to view or modify the arp cache in the service console, we can use the arp command. [root@esx1host]# arp -a vc.lab.taupoconsulting.com (192.168.1.200) at 00:0C:29:8D:F3:65 [ether] on vswif0 remote7.lab.taupoconsulting.com (192.168.1.70) at 00:50:56:84:19:56 [ether] on vswif0 remote7.lab.taupoconsulting.com (192.168.1.70) at 00:50:56:84:19:56 [ether] on vswif0 It's unlikely you will need static arp entries, but it can be done using the -s switch.

ethtool This command can be used to view and configure the Ethernet interfaces in your ESX host. We didn't use this tool very often until ESX 3.5, when we started to work with Distributed Power Management (DPM); an experimental feature of DRS clusters. The output of this tool provides a load of information about the network cards, but of particular interest now is the support for Wake-on-LAN (WoL). DPM makes use of this NIC feature and so we need to check that our NICs both support the function AND have

the function enabled. The ethtool allows us to view and set this functionality. # ethtool vmnic1 Settings for vmnic1: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: g Wake-on: g Link detected: yes If we noted that our NIC supported WoL but it was not currently enabled, then we could use this tool to effect the change. # ethtool -s vmnic1 wol g

tcpdump Neat tool for doing network captures at the service console command line. The -i switch is used to specify the interface to be used for the capture. This is important in ESX server, as the default interface for this tool is to use interface eth0, which doesn't exist for us; we have vswif0 as our default Ethernet interface. [root@esx1host]# tcpdump -i vswif0 If you are connected [root@esx1host]# tcpdump -w FILE -i vswif0 The format of the FILE can be read with another popular tool; WireShark! If you need to write a filter for the capture, there are reserved characters that need escape characters before them, e.g. parenthesis. [root@esx1host root]# tcpdump -i vswif0 port 53 ip This is a very powerful command and we don't often need it unless we are network

troubleshooting at the command line.

[root@esx1host root]# ip link show vswif0 6: vswif0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:50:56:49:96:03 brd ff:ff:ff:ff:ff:ff

We can add more IP addresses to your interface, overload your interface. [root@esx1host root]# ip addr add

[root@esx1host root]# ip route show 192.168.90.0/24 dev vswif0 proto kernel scope link src 192.168.90.7 169.254.0.0/16 dev vswif0 scope link default via 192.168.90.254 dev vswif0 We can see neighbours on an interface, which is really another view of the arp cache. [root@esx1host root]# ip neigh show 192.168.90.200 dev vswif0 lladdr 00:0c:29:8d:f3:65 nud stale 192.168.90.70 dev vswif0 lladdr 00:50:56:84:19:56 nud reachable

[root@esx1host root]# ip -help Usage: ip [ OPTIONS ] OBJECT { COMMAND | help } where OBJECT := { link | addr | route | rule | neigh | tunnel | maddr | mroute | monitor } OPTIONS := { -V[ersion] | -s[tatistics] | -r[esolve] | -f[amily] { inet | inet6 | ipx | dnet | link } | -o[neline] }

route This shows and allows editing of the routing table in the service console. If we use the route command with no parameters, the Linux routing table is displayed. If this is taking a long time, this could be due to DNS look ups, so you can use the -n switch to force numeric (no name resolution). [root@esx1host root]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.90.0 0.0.0.0 255.255.255.0 U 0 0 0 vswif0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vswif0 0.0.0.0 192.168.90.254 0.0.0.0 UG 0 0 0 vswif0 If we wanted to add a static route to the routing table, we would use the route

command with the add parameter, specifying the network in CIDR format and listing the local gateway as the last parameter. Here we add a route to the 10.45.0 network, instructing Linux to use 192.168.90.2 as the gateway to that net. We are checking our results after making the change. [root@esx1host root]# route add [root@esx1host root]# route Kernel IP routing table Destination Gateway Iface 192.168.90.0 0.0.0.0 vswif0 169.254.0.0 0.0.0.0 vswif0 10.45.0.0 192.168.90.2 vswif0 0.0.0.0 192.168.90.254 vswif0 -net 10.45.0.0/24 gw 192.168.90.2 Genmask 255.255.255.0 255.255.0.0 255.255.255.0 0.0.0.0 Flags Metric Ref U U U UG 0 0 0 0 0 0 0 0 Use 0 0 0 0

Note, the add option to the route command was not added in ESX until version 3.5.

tracepath As the traceroute command is not present in the ESX service console by default, we should be aware of some alternative tools. This tool traces the path to the specified destination (supplied as a parameter) discovering the Maximum Transmission Unit (MTU) along the path. It uses a random UDP port by default, but can be modified to use a specified port (2nd parameter). [root@esx1host root]# tracepath 192.168.170.201 1: esx1host (192.168.90.7) reached Resume: pmtu 65535 hops 1 back 1 netstat The output of netstat produces more information than just network sockets, so we need to narrow the query to just tcp and udp protocols. [root@esx1host root]# netstat --tcp --udp -a if you use -n to not resolve hostnames and protocol ports to service names The -p switch is extremely useful for determining which processes are using those sockets.

0.173ms

nslookup The nslookup tool is most often used to check forward name resolution. It can be used interactively or in a dedicated shell. If used interactively, then simply supply the name

of the host your are looking up as a parameter. [root@esx1host root]# nslookup mail.example.com The nslookup tool can also be used for reverse resolution by supplying IP address as a parameter [root@esx1host root]# nslookup 10.10.0.4 Proper reverse lookup is recommended for any SSL encrypted connections. dig This tool is a replacement for nslookup in Unix and Linux environments. It is fantastic for displaying exactly what is happening when you are doing a name lookup to DNS. We can see our query, the answer, the authority all in one output. [root@esx1host root]# dig mail.example.com The dig tool can be used for reverse lookup with -x switch. This tool does not use standard libc name service lookup and therefore does not refer to /etc/nsswitch.conf. It goes directly to the DNS servers listed in /etc/resolv.conf. Note, if you have multiple nameserver entries in /etc/resolv.conf we only query the 2nd or 3rd entry if the 1st or 2nd cannot be contacted. If the 1st nameserver responds with an unknown host reply to the query, we stop and don't query the remaining nameservers.

rpcinfo Can be used to verify services at a server. We find this useful for verifying if a server is running NFS v3 over TCP.

Network File System (NFS)

showmount This command is used by a NFS client to see what directories are being exported by a NFS server. [root@esx1host root]# showmount e nfsserver This command can be specified with the hostname name or IP address of the NFS server holding the exported directories. Remember that by default the service console will block nfsClient traffic. You will need to use esxcfg-firewall to open up that port. Also remember when you are accessing NFS servers from the service console you are going out via the Linux network stack; this is not the same operation as adding an NFS

datastore, where the VMkernel connects to NFS via its own stack on its VMkernel port.

portmap If you are wanting to mount a NFS export on a remote system from the service console, you do not need all the nfs server daemons running. All you need is the portmap service. You can start it with [root@esx1host root]# service portmap start And you can ensure this service is launched at boot time using the chkconfig command. Also remember that by default nfsClient is blocked by the service console firewall.

mount It really helps to be able to do simple mounts of remote systems using NFS and the mount command. We can tell mount what type of file access protocol to use with the -t switch, e.g. -t nfs or -t smbfs, however if you are working just with nfs, you can safely omit this [root@esx1host root]# mount server:/export /mnt/

VMware Consolidated Backup (VCB)

vcbVmName If we only know some of the details of a VM, but not all, we can use this query tool to ask VirtualCenter to report back all that it can find about it. For example, lets say you know you want to perform an image-level backup of a VM and the VM has IP address 10.0.0.1. We would [root@esx1host root]# vcbVmName -h vcserver -u vcadminuser -p secret -s ipaddr:10.0.0.1 FoundVM

vcbSnapshot vcbMounter If you want to perform image backups of running virtual machines from the service console command line, then this is the command for you. In a lot of ways this is the replacement of vmsnap.pl found in previous versions of ESX. vcbMounter.exe

This command is the core component of VCB which runs on the VCB Proxy server. vcbExport mountvm.exe This utility is only found on the VCB proxy server. vcbCleanup This command is new to 3.5/2.5 and allows a backup operator to cleanup the VM snapshot state should a VCB backup fail before completion. backuptools.conf If you don't want to specify user credentials on the command line, you can store the credentials in this file in the service console. vcbRestore When virtual infrastructure administrators first start using VCB, questions arise about how they can restore their VCB archives. The vcbMounter.exe command on the backup proxy performs the backup of the live VM, but where is the utility to perform the restore operation? The answer is, you cannot restore directly from the VCB Proxy server into VMFS, the proxy server has read-only access to the VMFS datastores. The vcbRestore command exists in the ESX host service console. Therefore we need to have the VCB VM archive accessible to the vcbRestore command to perform a restore. The archive could be copied to the service console with a tool such as WinSCP or Veeam FastSCP, but often, administrators will simply perform an NFS/CIFS mount from the service console to the location where the VCB archive is. vcbRestore /remotenfs/backups/vm However, as of VirtualCenter 2.5 (with integrated VMware Converter) we have the ability to restore VCB VM archives directly into ESX using the VMware Converter import function. This is a huge step forward in simplifying recovery. vcbUtil This command is only in the service console, not on the backup proxy server.

The /proc Hierarchy (aka proc nodes)

/proc The volatile /proc directory hierarchy that can be treated as a file system but is actually held in RAM. We can interrogate the files and directories in /proc to find out some great information about the running of the service console.

/proc/vmware The volatile /proc/vmware directory hierarchy that can be treated as a file system but is held in RAM. We can interrogate the files and directories in /proc/vmware to find out some great information about the running of the VMkernel.

/proc/vmware/migration/history This is an awesome file to reference when troubleshooting VMotion issues.

/proc/scsi If you want to check which SCSI devices are visible, cat this file.

/proc/vmware/sched/ncpus This is an in-memory file displaying the number of processors (ncpus) in the ESX server. This is a very useful file to inspect when you are unsure how many physical processors you have and if hyper-threading is enabled. # cat /proc/vmware/sched/ncpus 4 logical 2 cores 2 packages You can also get CPU package information from the top three lines of esxtop.

/proc/scsi/qla2x00 If you are using QLogic fibre host bus adapters, this

VirtualCenter Server 2.5 & Update Manager

vpxd.exe This is the process name of the Windows service that is the core service running on the VirtualCenter management server. If there are problems with the VirtualCenter service starting and then stopping almost immediately or a few seconds later, then check your ODBC database string and then the health of the the database server. We have seen this when the database runs out of disk space; check if the log space is full on the DB server, many clients forget about regular backup of this database. When troubleshooting the VirtualCenter service you can try VirtualCenter in stand-alone mode. This is done by invoking the following command at the Windows command line

vpxd -s You will get interactive logging of the start-up activity helping you to pinpoint where the problem is. If all else fails, you can always re-initialize the VirtualCenter database, however we would not recommend this. By re-initializing the VirtualCenter database you are wiping out all VC data!! If you do want this, then use the -b command switch to vpxd.

vpxd.cfg This is the VirtualCenter management server configuration file which the VC service reads at start-up. (Ok, so we are extending this command line guide to cover the VirtualCenter server now as well as the ESX host!) There are a number of configuration changes to VirtualCenter we can make in this file, but as of VC 2.5, one such change you may wish to make is the disabling of "Guided Consolidation". This feature, shown just as a consolidation button in the VI client, is intended to help small customers select which physical Windows hosts are suitable for consolidation and then guide them to perform the physical to virtual migration. If you have already been through the consolidation process, then you don't need this feature. It makes sense to disable the feature if you are not using it as this should improve VC performance. To disable Guided Consolidation, simply edit the vpxd.cfg file on the VC management server and add the lines: <vcp2v> config.vcp2v.dontStartConsolidation = true </vcp2v> You can get rid of the Guided Consolidation feature after VirtualCenter install by executing the following at the Windows command line on the VirtualCenter server. msiexec /qn /x {2A2750C9-E14E-4635-8595-C1CD214445B0} Thanks again to the folks over at Yellow Bricks for that one! vpxd-#.log There will be up to log files for VirtualCenter server. The log data is rotated across 10 log files numbered 0-9. Once a log file reaches 5MB, the next one is used. vpxd-index This file is the index file which indicates which of the numbered vpxd-#.log files is the current active one. This file is found on the VirtualCenter management server. VirtualCenter logs are rotated across 10 log files numbered 0-9. C:\Program Files\VMware\Infrastructure\VirtualCenter Server\tomcat\conf\tomcat-users.xml

There should be no reason for you to look at this file on the VirtualCenter server, however if you want to double check the Apache Tomcat configuration and status using the Tomcat Manager webpage, then you'll need to. Add a line with <user username="admin" password="password" roles="tomcat,manager"/> and then restart the Apache Tomcat service. You can now login to Tomcat Manager.

vum-proxyAuthCfg.exe The Update Manager component of Virtual Infrastructure is new to version 2.5. This component allows the patch management of Windows & Linux guests as well as ESX hosts. When installing the Update Manager component, the Windows installer package prompts the operator if they wish to use a proxy server to connect to the Internet, the only options are proxy IP address and port. If your proxy server requires authentication, then this tool must be run to supply the proxy server credentials.

vci-integrity.xml This is the primary configuration file for the VMware Update Manager (VUM). This file is read at start-up of the UM service and if the XML file is manually edited, then the service should be restarted for the change to take effect. One of the main reasons you may want to edit this file is if you wish to change the directory that patches are downloaded into, i.e. the patch store. The default location is on the C:\ drive which may not be adequately sized; there are a considerable number of patches to download to offer OS and application patch remediation, totally many gigabytes.

vmware-umds.exe This is the VMware Update Manager Download Service. If you don't want the server where Update Manager is installed on to actually connect to the Internet and do the patch downloading, then UMDS is for you. Maybe you don't want the load of update downloads on the UM server or maybe the UM server is on a subnet that can't reach the Internet. Anyway, the UMDS installs on a Windows server (that is not the same server as UM) and doesn't create a start menu program group. To start a download, simply enter the command vmware-umds --download Once the updates are downloaded, we can export them. This means we copy the patches from the download directory to another path. The intended purpose of exporting is to copy all or a subset of the downloaded patches to a location that will then be made

available to the Update Manager server. vmware-umds -E e:\exportedupdates At this time UMDS does not support NFS/CIFS shares for the export operation. This is related to a permissions issue and the SYSTEM account. The export function is intended to be used to copy the downloaded data to CD/DVD or USB stick. That's not to say you can't export to a CIFS share, it's just that's currently unsupported.

vmware-updateDownloadCli.exe This tool is run on the Update Manager server to import the patches made available from the UMDS export. So if you had a DVD burned which had all the updates that was inserted to the UM server and available as drive Z:

Remote Command Line Interface (RCLI)

To perform remote command line operations on an ESX host on versions of ESX up to 3.0.2, required either direct console access or using secure shell, e.g. Putty. As of ESX 3.5.0, there is a new alternative which is called RCLI. There are 3 RCLI options 1. RCLI Appliance (a ready-made downloadable VM appliance built on Debian Linux) 2. RCLI Installer for Windows 3. RCLI Installer for Linux These three options bring the ability to run a subset of the commands available at the service console remotely without having to grant ssh access to the actual console. This RCLI interface provides the ability for users of VMware ESX Server 3i (hardware embedded hypervisor) to run the esxcfg commands. Further, this interface is the only interface that a storage VMotion can be invoked from. To download the RCLI appliance, Windows installer or Linux installer, visit http://www.vmware.com/download/download.do?downloadGroup=VI-RCLI and accept the EULA to reach the download links page. Update July 2008 - the RCLI has been updated for ESX 3.5 Update 2. It is well worth getting the new RCLI tools as VMware have updated them significantly to be more complete as well as adding support for their use with regular ESX 3.5! svmotion This command is run from an RCLI interface to perform a live migrate of a VMs storage from one datastore to another, known as storage VMotion. In a storage VMotion, only the virtual disks of the VM move, unlike a regular VMotion, the VM remains running on

the same host. To perform an interactive storage VMotion, we use the svmotion command with the --interactive switch. In the following example, we see the full text of prompts and responses. To make it easier to read, we've highlighted the user input with yellow. svmotion --interactive Entering interactive mode. All other options and environment variables will be ignored. Enter the VirtualCenter service url you wish to connect to (e.g. https://myvc.mycorp.com/sdk, or just myvc.mycorp.com): 192.168.1.3 Enter your username: Administrator Enter your password: ***** Attempting to connect to https://192.168.1.3/sdk Connected to server. Enter the name of the datacenter: DataCenter-1 Enter the datastore path of the virtual machine (e.g. [datastore1] myvm/myvmx.vmx): [esx1host:storage1] vm1/vm1.vmx Enter the name of the destination datastore: esx1host:SharedVMFS You can also move disks independently of the virtual machine. If you want the disks to stay with the virtual machine, then skip this step.. Would you like to individually place the disks (yes/no)? no Performing Storage VMotion. 0% ################--------------------------------------------------------------------------------------100% Storage VMotion completed successfully. Disconnecting. So, from the above example, you can see that a storage VMotion run interactively is quite straightforward. When mistakes can creep in is when you are prompted for source and destination datastore names. The source datastore name requires square brackets [] around the name, followed by a space character and path to vmx file, whereas the destination prompt only requires the datastore name, this time without square brackets! If you want to script this command, then the inputs can be supplied as parameters to the svmotion command. svmotion --server <virtualcenterserver> --username <user_name> --password <user_password> --vm '[old_datastore] vm/vmx.vmx:new_datastore' If you don't want to include user data in the command, then you can combine this method with an environment variables file called ./visdkrc /etc/.visdkrc This is a hidden file that you can create in the RCLI appliance which stores the

parameters you wish to use when running commands against a host. VI_SERVER = vcserver.taupoconsulting.com VI_USERNAME = Administrator VI_PASSWORD = vmware VI_PROTOCOL = https VI_PORTNUMBER = 443

ESXi 3.5
DCUI Ok, the DCUI isn't a command, but you will hear people talking about it, so we thought it was worth an entry! This is the Direct Console User Interface, in other words, when you go to the physical console of an ESXi host, what do you interact with. The DCUI looks and feels like a BIOS management system with menus navigated using the keyboard.

unsupported To reach a command line interface on an ESXi host, you could enter the command unsupported while viewing virtual terminal 1 (tty1); note that there will be no local echo while you type this command. Be aware that there is no promise that this option will be present in subsequent versions of ESXi and that what you do whilst in this version is exactly as described; i.e. unsupported! dropbear This is a small SSH2 server and client available in the unsupported command line of ESXi. You need to manually enable it by editing /etc/inetd.conf and removing the comment character (#) from the start of the line relating to ssh. This is covered really well in a video over at Richard Garsthagen's site http://www.run-virtual.com/?p=223 as well as on Dave Mishchenko's site http://www.vm-help.com/

busybox When you get into the ESXi shell you will find you have a number of common Linux utilities available to you. You might be forgiven for thinking you have a service console! Actually the ESXi command line is highly limited shell which doesn't let you do much. However, busybox gives you a set of regular tools like vi in a single process. There is more information about busybox at www.busybox.net

vsish The VSI shell is the ESX 3i equivalent of interrogating the service console /proc nodes. You can view the state of running VMkernel worlds and drivers.

Virtual Hardware Intel 440BX-based motherboard with NS338 SIO chip. VMs can now have 64GB RAM, but not for Windows NT guests. VMs can now have "enhanced vmxnet" virtual network adapters supporting TSO & Jumbo Frames (Guest OS restrictions) Build Numbers ESXi ESXi ESXi ESXi ESXi ESXi 4.0 3.5 3.5 3.5 3.5

Installable Installable Installable Installable

Update 3 Build 123629 - 6th November 2008 Update 2 Build 110271 - 13th August 2008 Update 1 Build 82664 - 10th April 2008 Build 70348 - 10th January 2008

ESX4 ESX 4.0.0 Build - 22nd May 2009

ESX3 ESX ESX ESX ESX ESX ESX ESX ESX ESX ESX ESX 3.5.0 3.5.0 3.5.0 3.5.0 3.5.0 3.5.0 3.0.3 3.0.2 3.0.2 3.0.1 3.0.0 Update 4 Build Update 3 Build 123630 - 6th November 2008 Update 2 Build 110268 - 13th August 2008 (corrected build) Update 2 Build 103908 - 25th July 2008 Update 1 Build 82663 - 10th April 2008 Build 64607 - 10th December 2007 Build 104629 - 8th August 2008 Update 1 Build 61618 - 29th October 2007 Build 52542 - 31st July 2007 Build 32039 - 6th October 2006 Build 27701 - 15th June 2006

vCenter Server (Formerly VirtualCenter Server) vCenter 2.5 Update 4 Build 147633 - 23rd February 2009 vCenter 2.5 Update 3 Build 119598 - 3rd October 2008 vCenter 2.5 Update 2 Build 104263 - 25th July 2008 vCenter 2.5 Update 1 Build 84767 - 10th April 2008 VirtualCenter 2.5.0 Build 64201 - 10th December 2007 VirtualCenter 2.0.2 Update 5 Build 104182 - 8th August 2008 VirtualCenter 2.0.2 Update 4 Build 89601 - 30th May 2008 VirtualCenter 2.0.0 Update 3 Build 75762 - 15th February 2008

VirtualCenter VirtualCenter VirtualCenter VirtualCenter VirtualCenter

2.0.2 2.0.2 2.0.2 2.0.1 2.0.0

Update 2 Build 62327 - 8th November 2007 Update 1 Build 61426 - 29th October 2007 Build 50618 - 19th July 2007 Build 32042 - 2nd October 2006 Build 27704 - 15th June 2006

Consolidated Backup Consolidated Consolidated Consolidated Consolidated Consolidated Consolidated Consolidated Converter Converter Converter Converter Converter Converter VMmark VMmark 1.0 Build 20070712 - 23rd July 2007
(C) 2008 B2V - Business to Virtual - provided by Taupo Consulting Ltd, UK

Backup Backup Backup Backup Backup Backup Backup

1.5.0 1.1.0 1.0.3 1.0.3 1.0.2 1.0.1 1.0.0

Build 102898 - 25th July 2008 Build 64559 - 10th December 2007 Update 1 Build 58377 - 31st October 2007 for ESX 3.0.2 Build 51389 - 31st July 2007 for ESX 3.0.1 Build 42090 - 5th April 2007 for ESX 3.0.1 Build 32040 - 2nd October 2006 for ESX 3.0.0 Build 27703 - 15th June 2006

3.0.3 3.0.2 3.0.2 3.0.1 3.0.0

Update 1 (Standalone Enterprise Edition) Build 62456 - 3rd December 2007 (Standalone Enterprise Edition) Build 59994 - 18th October 2007 (Standalone Enterprise Edition) Build 44840 - 26th April 2007 (Standalone Enterprise Edition) Build 36853 - 29th January 2007