Install and Configure ESX Server

Configuration Considerations
Hardware Prepartions
Ensure that: The hardware system appears on the VMware HCL Follow any documented best practices from the vendor for installing ESX Server on the specific server model Server diagnostics, especially RAM tests, are performed Set the BIOS (hardware) clock to the current UTC time Install the latest, supported firmware Preferably, disconnect the Fibre Channel connection, unless booting from SAN Connect the networking cables, decide which Nic to use for Management (Service Console)

Time Settings
VMware recommends using using UTC on the hardware clock Plan to set the ESX Server software clock to local time (or some global customers may choose to use UTC time for this, too.) Ensure that the host synchronize their clocks with the same pool of NTP servers

Disk Partitions
Consider the default partition sizes, which are labeled as Recommended in the installation wizard, to actually be Minimum Recommended partition sizes.. Feel free to add more disc space to any of these partitions or even create additional partitions. When boot disc space is plentiful, the following configurations are often useful: Swap partition: increase to 1600 M (in case you eventually decide to increase the SC RAM up to its maximum of 800 M) For the /var/log, and the root (/), consider at least doubling the recommended size Consider using /var as the mount point, instead of /var/log. Adding other partitions, such as /home and /tmp, can be considered also.

Commands to Correct Initial Configuration Issues

Correcting Timezone and Time Issues after ESX Server Installation
Here are the steps to quickly fix any time or time zone related issues from the service console command line once the ESX 3.0 has been installed. This will save you from having to reinstall or hunt around for a quick fix. 1) To get a list of possible time zone choices, list the contents of the/usr/share/zoneinfo directory using the ls command. For example, the /usr/share/zoneinfo/America/Los_Angeles file would be appropriate to use for setting the time zone to the US Pacific time zone. Knowing the"America/Los_Angeles" part of the path will be important in the next step.

2) follow steps in Knowledge Base article 1436: http://kb.vmware.com/selfservice/viewContent.do?externalId=1436 will set the time zone to the US Pacific time zone. 3) To set the system clock (O/S clock) on the ESX Server to the current local time, use the date command, using this syntax: date MMDDhhmm. Naturally, MMDDhhmm should be replaced with the correct Month, Day, Hour and Minute. Alternatively, we can set the system clock to the value specified by a public ntp server (ntp.ucsd.edu). The first command below enables an opening in the Service Console firewall and the second command sets the time:

esxcfg-firewall -e ntpClient ntpdate -s ntp.ucsd.edu 4) To set the HW clock in the BIOS, the hwclock command should be used. Often, we prefer to set the hardware clock to the current UTC time (although the ESX Server system clock is set to local time). To do this, use the following command: hwclock --systohc --utc. This command will automatically adjust for time zone differences and update the BIOS clock with UTC time. If UTC time was chosen for the HW BIOS clock, then we need to configure ESX Server to understand this, because during reboots, the ESX Server gets its time from the hardware BIOS clock. To accomplish this, verify the following entry: UTC=true" exists in file /etc/sysconfig/clock. This entry might read UTC=false if the "System Clock uses UTC" checkbox was unchecked during ESX installation.

Wrong Nic used for Service Console

Example : - vmnic1 Should Have been used for Service Console connection. - instead, vmnic0 is configured, but it is not cabled into the correct network The following steps can be used to rectify the issue: - unlink vmnic0 from vswitch0 - unlink vmnic1 from vswitch1 - link vmnic1 to vswitch0 The actual commands are: esxcfg-vswitch U vmnic0 vSwitch0 esxcfg-vswitch U vmnic1 vSwitch1

esxcfg-vswitch L vmnic1 vSwitch0 reboot

Wrong IP Address for Service Console

esxcfg-vswif -i <ip-addr> vswif<#> for example: esxcfg-vswif -i 192.168.50.1 vswif0

Network Time Protocol (NTP) Setup

Setting up Network Time Protocol (NTP) on ESX servers is extremely important. ESX servers, by virtue of hosting guest VMs, must maintain accurate time so that the VMs can sync with the host server. Guest VMs do not get the opportunity frequently to utilize 100 percent of a CPU on the ESX server, thus they lose time. On very busy ESX servers where the overall CPU utilization is 70 percent or higher, the amount of time lag inside a VM becomes very noticeable. With the VMware Tools installed and the Time Synchronization check box selected, the Guest VM can maintain near perfect time in-sync with its host ESX server. This of course implies that the ESX server is in-sync with some external time system that the network follows for other reasons. NTP setup on an ESX server is very straightforward and necessary for any virtualization Infrastructure. Listed below are the steps for keeping the host ESX server in-sync with an external time source. Although, the VI Client now provides an option to configure NTP, it still appears a bit buggy and it does not seem to follow the best practices established by VMware in KB article 1339. Rename the default ntp.conf file: mv /etc/ntp.conf /etc/ORIG.ntp.conf Create a new, empty ntp.conf file: touch /etc/ntp.conf Modify ntp.conf: vi /etc/ntp.conf Insert these lines into ntp.conf, if desired replace the hostnames of the ntp servers: restrict default kod nomodify notrap noquery nopeer restrict 127.0.0.1 server 0.us.pool.ntp.org server 1.us.pool.ntp.org server 2.us.pool.ntp.org fudge 127.127.1.0 stratum 10 driftfile /etc/ntp/drift broadcastdelay 0.008 authenticate yes keys /etc/ntp/keys Rename the step-tickers file: mv /etc/ntp/step-tickers /etc/ntp/ORIG.step-tickers Modify step-tickers: vi [or nano] /etc/ntp/step-tickers

Insert these lines into step-tickers, if desired replace the hostnames of the ntp servers: 0.us.pool.ntp.org 1.us.pool.ntp.org 2.us.pool.ntp.org us.pool.ntp.org Open the ntp client port in the Service Console Firewall: esxcfg-firewall e ntpClient Restart the hostd daemon : service mgmt-vmware restart Restart the ntp daemons: service ntpd restart Determine if ntp daemons are set to start automatically at boot chkconfig --list ntpd Manually configure the ntp daemons to start at boot: chkconfig --level 3 ntpd on Run this command periodically over a period of 5-10 minutes. When an * appears at the left of one of the time servers, it indicates that ntpd has successfully synchronized. Do not continue with the next step (hwclock) until your server has synchronized successfully. ntpq -p Set the hardware clock to match the ESX software clock, but adjust for UTC time. Only use --utc if you want the harwdware clock set to the current utc time, rather than the current local time as recommended by VMware hwclock --systohc --utc

Networking
Network Planning
Typically, this planning flow can be followed: Determine the number and purpose of all Physical networks, such as Production, Test, Management, VMotion, iSCSI that an ESX Server must connect. For each physical network, we need at least one physical Nic connected to a vSwitch. For each physical network, if we want redundancy, we must add another physical Nic to the vSwitch. If we use multiple VLans, instead of multiple physical networks, we could use one physical Nic connected to a trunk port and use VLans tagged at VM Port Groups on virtual switches. Naturally, we should add more physical Nics for redundancy VMware recommends dedicating network connections to Service Console, VMotion, and VMs. Many clients also prefer to dedicate network connections for specific categories of VMs, such as production and test. If iSCSI or NFS is used to store VMs, then VMware recommends dedicating network connections for this traffic, also. If Nics and switch ports are plentiful, then dedicate a virtual switch for each traffic type and connect at least two physical connections. If they are scarce, then combine some traffic types, such as Service Console and VMotion to the same virtual switch, with 2 Nics teamed, but use Active and Standby to direct each traffic type to a specific physical Nic in the team, under normal working conditions. If these connections involve two separate VLANs, then VLAN trunking must be used.

Related Network Configuration Commands

esxcfg-vswitch
here are some samples for using the esxcfg-vswitch command: Create a virtual switch esxcfg-vswitch --add vSwitch0 Add a port group named Production esxcfg-vswitch add-pg= Production vSwitch0 Link vmnic2 to vSwitch0: esxcfg-vswitch link=vmnic2 vSwitch0 Create a SC port, named Service Console, with VLan=971, on vSwitch0

esxcfg-vswitch p Service Console

v 971 vSwitch0

Commands for Nic Teaming

An issue exists in ESX version 3.5 where linking two or more Nics to a vswitch results in the all but the first Nic being set to standby instead of active. To work around this issue, use the -M switch to first map the Nics to the port group, then use the L to link the Nics to the switch, such as:
esxcfg-vswitch --add vSwitch1 esxcfg-vswitch --add-pg=Production vSwitch1 esxcfg-vswitch --add-pg-uplink vmnic1 --pg=Production vSwitch1 esxcfg-vswitch --add-pg-uplink vmnic2 --pg=Production vSwitch1 esxcfg-vswitch --link vmnic1 vSwitch1 esxcfg-vswitch --link vmnic2 vSwitch1

where -M could be used as shorthand for add-pg-uplink and L could be as shorthand for --link

Storage / VMFS
Storage Planning
Here are some maximum configuration settings, along with some of my preferred values. The preferred values are certainly subjective, so they are intended merely as a guideline. Number of Luns per Host: o Max = 256 o Preferred max (sweet spot) = about 20 Number of VMs per LUN o Max = ???? o Preferred = 10 to 20 Size of a simple, single VMFS extent: o Max = 2 TB o Preferred = 500 MB Storage Tiers / Classifications often, larger infrastructures implement LUNs with varying characteristics. Commonly, we refer to this as tiers or classfications. Some common properties that may differ among various tiers are: Raid type Size Vm to lun ratio Storage based replication settings

Storage Related Commands

esxcfg-vmhbadevs
The two main storage related commands are fdisk and vmkfstools. Each of these require a different convention for identifying LUNs and partitions. One convention is the vmkernel convention, where a the device name of a partition is in the format vmhbax:y:z:# (x=adapter, y=target= z=lun, #=partition). The other convention is a typical Linux convention in the format /dev/sdx (x=letter assigned to the disk, the first detected disk is a, the next is b, etc.). So, each known lun has two separate device names assigned to it. To learn these device names, use esxcfg-vmhbadevs. esxcfg-vmhbadevs this command returns a row for each known LUN, displaying the device name for each LUN using both conventions (as described above).

fdisk
fdisk is a command that can be used to manipulate the partitions on a disk. It provides a sub-command environment, allowing the user to view, delete, and create partitions. For example, if the goal is to delete a single, existing partition from a disk named /dev/sdc and to create a new partition, filling the entire disk, to hold a vmfs3 file system: fdisk /dev/sdc o p show partitions o d delete partititions o n create a new partition p make it a primary partition 1 assign partition number 1 <enter> start on the first available block

o o

t w

<enter> end on the last available block set type of partition to .. fb .. fb, which is a hex code indicating vmfs3 write the changes to disk

Vmkfstools
vmkfstools is a command that can be used to manipulate VMFS volumes and virtual disks. -C can be used to create a vmfs datastore, for example: vmkfstools -C vmfs3 -S Lun01 vmhba1:0:1:1 where: The name of the vmfs volume is Lun01 The device name of the LUN is vmhba1:0:1 The partition number is 1. For example, the VMFS extend option (-z) can be used to span a VMFS volumes across multiple LUNs. The import option (-i) will import a Cow file to a single monolithic file (convert a legacy template or a VM build on Vmware Workstation into a virtual disk usable by ESX). The virtual disk extend option (-X) can be used to increase the size of a virtual disk, for example: vmkfstools -X 10G /vmfs/volumes/Lun01/vma/vma.vmdk where: 10 GB is the desired new size Lun01 is the name of the vmfs datastore And, vma is the name of the vm.

Storage-based Snapshots
If we take snapshots at the storage level (SAN based snapshots), we may need to enable ReSignaturing on the ESX Server to make it accessible to the ESX Server. In the VI Client, choose the ESX Server, Configuration tab, Advanced Settings. Drill down to LVM and modify the setting for Enable ReSignature. Typically, we make this change only temporarily. After rescanning the SAN, we can disable the ReSignature. If the goal is to present the original Lun and the snapshot Lun to the same host, then ReSignature must be used to force a new VMFS ID to be written to the metadata of the file system. If the goal is to present the snapshot Lun to a new host that does not have access to the original Lun, then use the DisallowSnaphot value instead of the Resignature. This will allow the Lun to be presented without changing the VMFS ID.

System Files (VMFS metadata files)

.fdc.sf - file descriptor system file .sbc.sf - sub-block system file .fbb.sf - file block system file .pbc.sf - pointer block system file .vh.sf - volume header system file

Spannned VMFS3
Spanning (extending) results in overwriting the target. Breaking an extent requires destroying the first extent Is still more fragile than a non-spanned, but is not as fragile as in previous versions

VMFS3
Blocksize: By default, this is 1 M, but can be 2 M, 4M, or 8 M. This can be accomplished by creating the VMFS3 volume with the vmkfstools command. The b switch can be used to set the blocksize. Also, the VI client allows the user to choose the maximum file size and block size for the VMFS3 during its creation.

Proper alignment of VMFS volumes

VMFS partitions should be aligned to 64KByte track boundaries to provide the best performance. Using the VI client in VI3 should result in proper alignements. To verify a VMFS volume is properly aligned: Issue this command esxcfg-vmhbadevs m Sample Results vmhba0:0:17:1 vmhba0:0:26:1 vmhba0:0:5:1 vmhba0:0:18:1 vmhba0:0:27:1 vmhba0:0:28:1 vmhba0:0:29:1 vmhba0:0:30:1 vmhba1:0:0:3

/dev/sda1 /dev/sdc1 /dev/sdh1 /dev/sdb1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdi3

45ca1019-2ebd6c7a-1946-000423a65db6 45ca2efb-0ced1d98-7878-000423a65d90 45880405-f661f9fc-2283-000423a65db6 45cb89f2-a7df656e-3b39-000423a65db6 45ae6401-face6d54-9b37-000423a65d9c 458000c2-8ab48680-5e7d-0050da08aefa 45b78e94-12ed9ec8-c370-000423a66648 4580012a-0baf1888-1031-0050da08aefa 45c89d92-d57bc06c-de03-000423a65db6

So we learned that lun 17 is /dev/sda1, examine /dev/sda this with fdisk: fdisk -lu /dev/sda Sample Results: Disk /dev/sda: 15.7 GB, 15726735360 bytes 255 heads, 63 sectors/track, 1912 cylinders, total 30716280 sectors Units = sectors of 1 * 512 = 512 bytes Device Boot Start End Blocks Id System /dev/sda1 128 30716279 15358076 fb Unknown This indicates the starting block is 128. Look at another level, using /proc/vmware: more /proc/vmware/scsi/vmhba0/0:17 Sample Results: Vendor: COMPAQ Model: MSA1000 VOLUME Rev: 4.24 Type: Direct-Access ANSI SCSI revision: 05 Id: 60 8 5 f3 0 c c 30 0 0 0 0 5f a3 0 7d 4d 53 41 31 30 30 Size: 14998 Mbytes Queue Depth: 32

Block size: 512 Num Blocks: 30716280 Valid Partitions: 2 0: 0 30716280 0x0 1: 128 30716152 0xfb Notice that the VMFS partition is actually partition 1 above, which starts at block number 128. (partition 0 starts at block 0). If the starting block number needs to be adjusted, use: fdisk /dev/sda select x for expert mode select b to adjust the starting block number

Virtual Disk Options

ESX servers use monolithic v-disk files (vmdk files) that are pre-allocated to the configured size. These files do Not grow. Having said that, templates can be stored in various formats and other Vmware products allow additional formats. Here are the options: Thick Single file, preallocated to max size. Zeroredthick single file, prealloacted to max size, but not zeroed out at creation. Instead the VM will zero out any existing data in the virtual disk later. Eagezeroredthick - single file, prealloacted to max size, and zeroed out at creation, which takes longer than zeroedthick. Thin not preallocated, instead the file is expanded and zeroed out on demand later. 2gbsparse split into multiple files of 2G or less, compatible with other VMware platforms. This is required and provided by default for NFS storage.

Note: in some cases, the command vmkfstools can be used to connect a thin provisioned virtual disk file to a VM running on the ESX Server, but it is not recommended except for NFS-based datastores.

iSCSI
iSCSI Design a great document from highly qualified storage experts from EMC, NetApp,
EqualLogic, Lefthand, and VMWare: http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-helpour-mutual-iscsi-customers-using-vmware.html

NFS Design Considerations witten by highly qualified storage experts from EMC and NetApp:
http://virtualgeek.typepad.com/virtual_geek/2009/06/a-multivendor-post-to-help-our-mutual-nfscustomers-using-vmware.html

VC Server / License Server

VC application logs in Event Viewer vpx log files
Some of these messages that appear in the Event Viewer on the Windows server running VC Server appear as unknown. To find the actual text for these entries, look in the log files under this path at log files: C:\windows\temp\vpx

VC Database
To point VC to a database or change the DB user account, using 1.x or 2.x of VC, rerun the installation, choose Repair, modify the ODBC connection and login credentials as needed. For password changes, use the VI Client, Server Settings, click Database. For Virtual Center 2.01, VMware now supports the use of SQL Server 2005 SP1 to store the database. Beginning with VirtualCenter 2.5, VMware now supports the user of SQL 2005 Express for production use in small environments (upto 5 hosts, and 50 VMs)

Audit Trails in VC 2.0

the ability to generate an audit trail does not exist in VC 1.x. However, in VC 2.0 this is possible. The admin must select the "Events" icon in VC 2.0, select "Export Events" then filter the events that were logged. For example: * file to write logged events to * type of events to capture (user events or system events) * severity level of the events to gather (error, info or warning) * time (i.e. last hour, last 2 hours, last day, last 3 days, etc.)

Licenses
Can be host based, if using ESX Server only. Preferably, this should be server based. VC and ESX Server will be pointed to the License Server independently. Replace an existing license file used by License Server: Obtain a new license file. Copy and paste its contents in the vmware.lic located at: C:\Program Files\VMware\VMware License Server\vmware.lic Direct License Server to use new license file without restarting the service by: Click Start Programs VMware VMware License Server License Server Tools Click Start/Stop/Reread tab Click Reread License file View Licenses in License Server Click Start Programs VMware VMware License Server License Server Tools Click Server Diags Tab Click Perform Diagnostics button Review the results to verify all features and quantites appear

View Virtual Center Licenses In VC Client, click Admin Click Licenses Tab Apparently: Customers with full support may upgrade to 3.0 for free, without new features (HA, DRS, VCB). Grace Period when License Server is Down: For 14 days, all VM and ESX Server functionality is still available. The only VC features that are unavailable during the grace period are adding additional ESX Hosts, move ESX Servers between clusters (HA and DRS), change licenses, and upgrading VC. Flexnet The VMware License server is based on FlexNet from Macrovision, which used to be called FlexLM. (for more details: http://en.wikipedia.org/wiki/FLEXlm)

VM Best Practices
After installing O/S, edit settings and uncheck the Connected and Connect at Power-on checkboxes for the virtual CD-ROM Install VMware Tools. Installing the Complete version will typically eliminate the need to reinstall whenever moving a VM across platform types, such as from ESX Server to Workstation. In control panel- vmware tools, check the check box to sync the VMs virtual clock with the ESX host clock OR use another means to synchronize the guest O/S time such as synching with Active Directory Only use multiple v-cpus, when the guest O/S and application support multi-threading and the specific apps performance benefits. Consider using separate v-disk files for O/S, Application, and Data drives. This will provide more options when planning backup and recovery.

Security
ESX Server Based Permission
By passing Virtual Center, we can create user accounts and groups on the ESX Server using the VI Client. If the goal is to create a user account and allow it to be the Administrator of only a few of the VMs, then the following steps are needed: Connect the VI client to ESX Server directly, log in as root On the Users / Groups tab, create a new user account, providing at least a Login name and password In the left pane, select a VM in the right pane click the Permissions Tab Right-click and add a new permission. Choose or type in the user name and click OK. On the right side of the window, select Administrator as the permission Repeat for each VM that the the user should access. Finally, in the left pane, choose the ESX Server. On its permission Tab, set the user account as Read Access. (if the user does not have at least Read to the root level, then it cannot be used to log onto the ESX Server via the VI client) We can create custom roles by using the Admin button.

Ports
TCP Ports 22, 80, 902 for VI Client to ESX Server 3 communication TCP Ports 80, 902, 905 for VI Client to VC Server communication (905 appears to only be needed if VC Server was upgraded from 1.x) TCP port 903 for VI client to ESX Server to allow VM console access TCP Port 443 for Web Access to ESX Server and VC Server TCP Ports 27000, 27010 for ESX Server 3 licensing TCP 902 for ESX Server 3 communication for DRS and HA UDP Port 902 for heartbeat between ESX Server 3 and VC Server Port 32808 is a source port on ESX Server 3, whose destination is UDP Port 902 Port 2049 for NAS (NFS) communication Port 3260 for iSCSI communication Ports 2050-5000 and 8042-8045 for ESX Server 3 traffic for HA

NFS Security
NFS ver 3 does not provide password authentication, but it does provide IP filtering: The NFS server controls the who can mount what. For example, the /etc/exports file may contain the following line : /bfd 192.168.0.0/24(rw,no_root_squash,sync)

That allows anything on the 192.168.0.0/24 network to mount /bfd with the options of rw,no_root_squash,sync. Decent generic HOW-TO document : http://tldp.org/HOWTO/NFS-HOWTO/index.html

Root account via ssh

Root account is not allowed to login via ssh directly. Instead, login as a non-root account and use the su command to change to root. However, if we Upgrade Directly from ESX Server 2.5x to 3.0, then the root account login IS allowed. To enable direct logins using the root account via ssh, the /etc/ssh/sshd_config file could be modified (although it is not typically recommended): Change PermitRootLogin no To PermitRootLogin yes Then use this command to restart the sshd service: service sshd restart

Configure AD Authentication
Perform the following steps to configure the ESX Server to allow Active Directory to authenticate its logins: Enable AD authentication with this command:

esxcfg-auth --enablead --addomain domainname.org --addc domainname.org Some documentation may suggest using this format:

esxcfg-auth --enablead --addomain=domainname.org -addc=server1.nodata.org, server2.dom.org Which identifies each acceptable domain controller by name. but the first syntax, which uses the fully qualified domain name (instead of fully qualified domain controller names), allows the DNS server to supply the name of all known domain controllers in the domain. So, the first syntax is preferred for scalability and ease of management. Create user accounts matching AD accounts, but do Not specify passwords on the ESX Server To create user accounts and add the user account to the group named wheel, use this command: useradd -G wheel username (the gui will not let us add accounts with no passwords) Esxcfg-auth will make all appropriate config file changes and open the appropriate SC firewall ports

Other User Account Settings:

Allow all members of the wheel group to execute all commands, except change the root account password:

o visudo /etc/sudoers o uncommenting this line: o %wheel ALL=(ALL) ALL, !/usr/bin/passwd root o Save the file o Issue this command at the command prompt: o PATH=$PATH:/usr/sbin Establish a login banner: o Modify /etc/ssh/sshd_config o Locate the line: #Banner /some/path and change it to Banner /etc/issue o Create the file /etc/issue and key in the desired text for the login warning: This is a private computer facility, protected by a security system. Access to and use requires explicit written, current authorization and is limited to purposes of the organization's business. Unauthorized access or attempts to use, alter, destroy, or damage data, programs, or equipment may violate applicable law and could result in criminal prosecution, civil liability, or both. o service sshd restart Restrict which users can use su by modifying the wheel group and only allowing this group to use su o On the ESX Server console, logon as root and edit the su file as follows o vi /etc/pam.d/su o uncomment the following line, which will require a user to be the member of the Wheel group in order to use the su command o auth required /lib/security/$ISA/pam_wheel.so use_uid o save the file Modify the path of each of these individual accounts: o vi .bashrc o add the following statement to the end of the file o PATH=$PATH:/usr/sbin o Save the file

Security Certificates for VirtualCenter

See page 17 of the Security Hardening Document: http://www.vmware.com/pdf/vi3_security_hardening_wp.pdf

3rd Party Tool for checking VMotion CPU Compatibility:

http://www.run-virtual.com/?page_id=155

Vmotion CPU Compatibility Matrix for Dell systems:

http://support.dell.com/support/edocs/software/eslvmwre/AdditionalDocs/VMotion_Matrix/48704A01.pdf

VMotion CPU Compatibility Matrix for HP Systems:

http://h18004.www1.hp.com/products/servers/software/vmware/hpvmotion-compatibility-matrix.html

Service Console
Linux Service Console is a limited distribution of Linux based on Red Hate Enterprise Linux 3, Update 6 (RHEL 3 U6). Configure Service Console Port with a VLAN ID
If the SC port should be configured to supply a VLAN ID and is not currently allowing IP Connectivity, then the following commands can be used to correct the problem: Query the virtual switches to obtain the switch and port names:

esxcfg-vswitch l Typically, the Service Console port is labeled as Service Console. Locate the port and identify its name and its virtual switch name. Reconfigure the Service Console port with the correct VLAN number: esxcfg-vswitch --pg=Service Console -v=971 vSwitch0 In the example above, the port label is in quotes (Service Console), which should be replaced if the Service Console port uses another name. Also, in the example, the VLan number is 971, which should be replaced with the desired VLan number. If the goal is to remover the VLan tag, then specifiy 0 for the VLan number.

Service Console RAM

By default, SC is allocated 272 MB of RAM. This is sufficient, regardless of the amount of VMs that will run on the server. If any plans exist to install additional, 3rd party agents in SC, such as IBM Directory, Insight Manager, or backup agents, then the SC RAM may need to be increased. To change this after ESX Server installation, use the VI client to connect directly to the ESX Server (not virtual center), go to the configuration tab and look for Service Console RAM. We have to reboot. SC swap VMware recommends that the size of the SC swap partition (544 MB by default) should be at least twice the size of the allocated SC RAM. Assuming that plenty of local disc space is available, it may be best to allocated 1600 MB of swap space to SC. (The maximum amount of RAM typically allocated to SC would be 800 MB, so this would be adequate swap to support such an increase).

Is Service Console a VM? No, but it is more like a VM than ever before. It does not have direct access to storage or network, instead the I/O for these is handled by the vmkernel just like it does for a VM. But, SC does have dedicated RAM. Its RAM is not shared by any means with VMs or vice versa. SC can still only execute on CPU 0, but it does not own exclusive rights. VMs are often migrated dynamically to CPU0. Service Console IP The preferred way to change SC IP is now via the VI client or with the esxcfg-vswif command. But, here is lower level information on making the change, just in case. If the wrong IP address was assigned to SC during the installation, resulting in no IP connectivity, the IP can be changed by interacting with the ESX Server Console:

Logon as root Use a text editor, like vi, to modify the ifcfg-vswif0 file, as follows:
vi /etc/sysconfig/network-scripts/ifcfg-vswif0 press I key move to the line with the IP address modify the IP address press ESC key press SHIFT and : keys at colon prompt, enter wq (to write and quit)

Issue the service network restart command

SC IP if we change this address, be sure to change the /etc/hosts file also, it is critical to VMWare HA

Miscellaneous
Issue attaching a console to a VM Ensure that DNS resolution works everywhere. This is a repeated theme for VMware HA, DRS, etc. It also relates to attaching consoles. After installing the VC Server 2.01 patch (build 33643), we could no longer attach consoles to VMs via VI Client logged into VC Server. We could log the VI Client directly into ESX Server, then attach consoles to VMs. Everything else appeared to work well, including VMotion. We discovered on the client PC we could not successfully resolve the ESX Server by IP. By correcting this DNS issue, we corrected to console problem

Problem Stopping a VM hangs at 95%

when shutting down a virtual machine and it hangs at 95%. Attempts to attach a console or use the VI client connected directly to the ESX Host also fail.

Connect via SSH to the ESX Server. Use this command to determine the process ID of the VM: ps -efwww | grep "VMNAME.vmx" Use this command to kill the VM: kill -9 PID (where, PID = process ID) Example: ps -efwww | grep "VM1.vmx" which results are: root 25240 1 0 Mar02 ? 00:00:28 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user -@ pipe=/tmp/vmhsdaemon0/vmxd399ac93d25a2ebc;vm=d399ac9d425a2ebc /vmfs/volumes/448960cd-68bc05c3-825800110a77cd51/VM1/VM1.vmx so, we kill the 25240 process: kill -9 25240

Web Access
When browsing with IE to the Virtual Center or ESX Server, the Web Access link using http, rather than https. Typically, we have to modify the URL to insert the s. So, if http://192.168.1.1/ui automatically appears and fails, we have to modify to https://192.168.1.1/ui Troubleshooting: On the Virtual Center Server, a service named VMware Virtual Infrastructure Web Access should be examined and restarted if web access is lost. On the ESX Server, the web services can be restarted with the following command: service vmware-webAccess restart

Guest Customization
The customization specifications are no longer stored in xml files. Instead, they are stored in the database. In the VI client, choose Edit Edit Customization Specifications to view and modify using a wizard. When deploying VMs from templates and using the guest customization wizard, an option appears to supply a password for the Administrator account. For Win 2003 VMs, this option will actually only succeed if no password was set on the Administrator account in the template. If a password was set, then the Delete all Users option can be chosen in the wizard to force the customization to enforce the new password.

Typical fix
Whenever issues arise with certificates, attaching consoles, VC Server connections to ESX hosts, VI client connections, etc. these commands may fix the issue. They restart hostd, vmkauthd, and vpxa: service mgmt-vmware restart service vmware-vmkauthd restart service vmware-vpxa restart

Issue upgrading VC 2.01

We installed the patch (build 33643) for VC Server 2.01. It appeared to install successfully, but one (out of four) esx hosts would not reconnect. The Tasks / Events show: Failed to upgrade VirtualCenter agent on <hostname>. The System Logs (vpx-2.log) shows: 'App' 2176 error][VpxdHostUpgraderLRO]Failed to upgrade host <hostname>; vim.fault.AgentInstallFailed. To fix this: On the ESX Server we issued these commands: o service mgmt-vmware restart service vmware-vmkauthd restart o service vmware-vpxa restart In VC Server: o Right-click the ESX Server, choose Connect o Supplied root password

Issue attaching a console to a VM Ensure that DNS resolution works everywhere. This is a repeated theme for VMware HA, DRS, etc. It also relates to attaching consoles. After installing the VC Server 2.01 patch (build 33643), we could no longer attach consoles to VMs via VI Client logged into VC Server. We could log the VI Client directly into ESX Server, then attach consoles to VMs. Everything else appeared to work well, including VMotion. We discovered on the client PC we could not successfully resolve the ESX Server by IP. By correcting this DNS issue, we corrected to console problem

vmware-cmd
At the command level, vmware-cmd can be used to manipulate VMs. Among other options, it provides switches to start, stop, register, and modify VMs. It is best to fully qualify the path to VM config files using the actual folder name (a long numeric number), rather than the vmfs label. (use the number instead of Local). Apparently, the label works in most scenerios, but not all.

Files types in VMFS3 volumes

Virtual Machine files vmx - virtual machine configuration file .vmx.lck - A lock file created when vm is powered on or is the active tab in the UI. .nvram The BIOS of virtual machine .vmdk The virtual disk - this does Not contain the data instead it contains metadata *flat.vmdk the Data file for the virtual disk .vmsd a dictionary file for snapshots and associated disk .vswp Virtual machine memory mapped to a file (only present when the vm is powered on or has a powered on snapshot) .vmss virtual machine suspend file .vmsn - Virtual machine configuration of a snapshot . flat-vmdk - Single preallocated disk file that contains the data

Vlance morphs into vmxnet explanation

(from pang cheng) The morphing really means that installing VMware Tools seamlessly replaces the default AMD PCI virtual network adapter in the Guest to an Accelerated AMD virtual network adapter. This is done without: - having to power off the VM and having to set the network adapter type from vlance to vmxnet (there is now no selection; all default to vlance) - having the nuisance of creating a second adapter inside the Guest-- in Windows the network adapter stays the same and no new LAN connection is created. The new virtual network adapter runs at the faster vmxnet speed. Note that in the case of an upgrade, the new adapter comes from a combination of upgrading the virtual hardware and installing VMware Tools. Installing VMware Tools before upgrading the virtual hardware will not install the accelerated adapter.

Balloon Driver
ESX Server provides a Ballooning Mechanism to borrow RAM from a rich VM and give to a poor VM. Here is a link to a good, detailed article that includes an explanation of ballooning and other memory related information:

http://www.vmware.com/pdf/usenix_resource_mgmt.pdf

Issues with CD writers on ESX Server 3.0

When you install ESX Server 3.0 on a box with a CD-RW drive, the installer helpfully adds hda=ide-scsi to /etc/grub.conf for you. Unfortunately, this causes the CD-RW drive to not work at all after the install is done and you boot into ESX Server. Why? Remember that, in ESX Server 3, the VMkernel owns all the devices; this includes the CD-ROM drive, which is managed by a userworld. And the VMkernel does not do the SCSI-over-IDE emulation that hda=ide-scsi calls for.\ The workaround is to remove every occurrence of hda=ide-scsi from /etc/grub.conf and reboot.

esxcfg-cmd commands
ESX Server 3.x provides a new standard set of commands for modifying the ESX Server. These offer alternatives to making changes interactively via the VI Client. The commands are of the form esxcfg-___. Troubleshooting Service Console Networking If certain parts of the service consoles networking are misconfigured, you will lose your ability to access your ESX Server host with the VI Client. In the event that this happens, you can reconfigure networking by connecting directly to service console and using the following service console commands: esxcfg-vswif -l Provides a list of the service consoles current network interfaces. Check that vswif0 is present and that the current IP address and Netmask are correct. esxcfg-vswitch -l Provides a list of current virtual switch configurations. Check that the uplink adapter configured for the service console is connected to the appropriate physical network. exscfg-nics -l Provides a list of current network adapters. Check that the uplink adapter configured for the service console is up and that the speed and duplex are both correct. esxcfg-nics -s <speed> <nic> Changes the speed of a network adapter. esxcfg-nics -d <duplex> <nic> Changes the duplex of a network adapter. esxcfg-vswif -i <new ip address> vswifX Changes the service consoles IP address. esxcfg-vswif -n <new netmask> vswifX Changes the service consoles netmask. esxcfg-vswitch -U <old vmnic> <service console vswitch> Removes the uplink for the service console esxcfg-vswitch -L <new vmnic> <service console vswitch> Changes the uplink for the service console. If you encounter long waits when using esxcfg-* commands, it is possible that DNS is misconfigured. The esxcfg-* commands require that DNS be configured so that localhost name resolution works properly. This requires that the /etc/hosts file contain an entry for the configured IP address and the 127.0.0.1 localhost address. Use the less or more command to look at the results of the esxcfg-info -n (-n limits output to networking information) command and look at the Network Hint line. A few lines above this line, we can find information about how the VMkernel snooped the v-switch to listen to determine what

IP ranges the vswitch hears. (the same information seen in the VI client in the properties of the virtual switch)

Default number of ports on a v-switch

It appears that the v-switch created by the ESX Server Installer has 24 ports enabled by default, but switches created by the VI client have 56 ports enabled by default.

Issues with copying files during Migration, Clones, and Templates

If a network copy error, such as network copy failed for file nvram, appears during a migration, clone, or template process, then it is likely a name resolution issue. One way to solve may be to add each ESX Servers hostname and IP to the /etc/hosts file of each ESX Server.

VM Management
Snapshots o Can be taken while vms are running, or suspended, or powered off Cold Migrations o Can be performed while vm is powered off or suspended Clone and Template creation o Can only be performed while vm is powered off Auto stop and start o Per esx server still, configurable via mui o We can set global delay per start and per stop, we can set unique delays per vm, per start vs stop, we can change the order at start and the order at stop

VMware HA
Troubleshooting If HA does not work, ensure each ESX server can resolve the name of the other esx servers, using: nslookup <fqn>. Each ESX host must be able to resolve the short name and the fully qualified name of every other ESX Server in the HA Cluster. By default, if an ESX Server can longer communicate with another ESX Server in the HA cluster, then it will try to ping the SC gateway, to verify that it is not simply isolated from the network. For example, if we simply pull the Service Console network cable, it may think that all other ESX servers are down, but then realizes that it simply has no IP connectivity itself when it fails to receive ping replies from its own SC gateway. In some demos, we had a working gateway that was not configured to respond to pings, so HA would fail. To correct this, in VC, edit settings for the HA cluster and choose Advanced Options, set das.isolationaddress 192.168.50.208 Technically, if the das.isolationaddress variable is assigned a value, VMware HA will attempt to ping that address And the SC gateway. To instruct VMware HA Not to ping the SC Gateway, set this advanced variable value:
das.usedefaultisolationaddress = false

To specify multiple alternative addresses to ping to determine if a server is isolated, use the varialbles;

das.isolationaddress1 das.isolationaddress2 etc. Whenever multiple isolation addresses are specified, a host may ping each address whenever it loses connections to its HA Cluster partners. A reply from any of these addresses would indicate that the host is Not isolated. If HA fails to auto start VMs, then we reboot the ESX that we deliberately crashed, its VMs may not be able to start, instead VC shows insufficient resources to satisfy configured failover for HA. To fix this, we simply disable HA, temporarily. Unable to add an ESX Server to HA Cluster: If we changed the SC IP address of an ESX Server, after the initial install, the /etc/hosts file may still have the old IP address, meaning that the ESX server cannot resolve itself. We need to correct this. Steps: here is a sample of steps to take when having trouble to initially setup HA: Disable the HA cluster edit settings on the cluster and uncheck the HA box Watch the associated tasks and ensure that they complete Enable the HA cluster, again watch to see that the steps complete Check the summary tab of the cluster and each esx host to verify that no messages appear indicating a problem with HA. If one esx indicates that ha is misconfigured, try right-click and choose reconfigure for ha. If this does not fix HA, we can be more aggressive (ensure that each step fully completes before launching the next step) o Disable HA cluster o Shutdown vms o Put hosts in maintenance mode o Drag each ESX out of the cluster o Verify each ESX host is referenced with its fully qualified host name, not an IP. (to correct, remove and re-add the host) o Remove the cluster o Readd a cluster, only check HA (do not check DRS) o Do not make any resource pools o Drag the esx servers back into the cluster, one at a time. Allows the task to fully complete before dragging in the next ESX server. o Start only a couple of VMs server 1. Be sure to choose VMs that are on shared storage. o Reboot server 1 and verify that HA now starts the VMs that were running on server 2.

Timing 15 seconds will elapse from the moment that an ESX Server fails (or becomes unreachable via IP) and the moment that others servers detect this as a Host Failure. If an ESX host becomes isolated from IP (say, we unplug the SC cat-5 cable), twelve seconds will elapse before it declares it is isolated from the network (because it releases it cannot ping its own SC gateway or the address specified by das.isolationaddress). Once an ESX server determines it is isolated (12 seconds after actual IP failure), it will then begin powering off its VMs automatically. If IP is restored between 12 and 14 seconds after failure, the other hosts will not detect a host failure. In this case, some or all VMs may be powered down but not restarted anywhere.

More Details on VMware HA: http://www.yellow-bricks.com/vmware-high-availability-deepdiv/

How to determine which cpu a VM is currently running

Use esxtop to determine the VMs ID number, then use the cat command to display the VMs cpu file under the proc path: cat /proc/vmware/vm/<vmid>/cpu/status Where, <vmid> is the id number of the VM. In the results, look for the cpu column, which indicates the logical CPU, where the VM is currently mapped. It also shows cpu affinity, uptime, status, etc.

Upgrade to ESX 3.0 Issues

Upgrade fails reporting error that initrd is full. This may be due to /tmp being full.

Patching ESX Server

Beginning with VI 3.5, a new product called the VMware Update Manager has become the preferred means to patch ESX Servers. The underlying technology, which has remained the same, is described below. This information may become useful to understanding how the Update Manager functions, in case troubleshooting is needed. As of Feb 10th 2008, fourteen patches are already available for ver 3.01. Some are classified as Critical or Security, which should be installed on all servers. Other patches are classified as General, which are not as important, but again, these should probably be installed. Each patch comes with specific instructions on how they should be installed, but most now involve the use of the esxupdate command and rebooting the ESX Server. Patch Management Guide: http://www.vmware.com/pdf/esx3_esxupdate.pdf Basically, patching requires: Download each patch Extract (decompress) the files from the tar ball (tar xvzf ) Change default folder to location of the patch (cd command) Install the patch with esxupdate command (esxupdate update) The noreboot option can be used to delay the reboot. Here is a link to an alternative (3rd party ) approach: http://www.vmts.net/VMTSPatchManager.htm NOTES about Update Manager: It does patch ESX Servers It does patch VMs guest operating systems including Windows and Linux Guest O/S patches com from shavlik It does Not patch VirtulCenter It does Not include updates for VMware Tools Updating VMware Tools using VI Toolkit: http://halr9000.com/article/642

Accessing ESX Server directly via VI Client

Why? If we use Virtual Center, then we should typically connect the VI Client to Virtual Center for all of our activities. The only functions that cannot be performed while connected to the Virtual Center, but instead must directly connect to an ESX Server are: To create ESX User and Group accounts To apply ESX Server Level permissions

WINSCP issue copying files

By default, we cannot use root to connect to ESX Server via WinSCP, so we typically connect as a nonroot user. By default, non-root users, created via the VI Client, can copy files to their /home folders without issue. They can also copy files to vmfs volumes, but an error will appear as it attempts to remove the .filepart extention. To correct this, in WinSCP, Choose Option-Preferences. In the left pane, select Transfer-Resume. In the right pane, disable the Resume button. The downside is that IP issues during a file transfer can only be fixed by restarting the transfer. (see VMware Knowledge Base article 2082)

Restart hostd on ESX Server

service mgmt-vmware restart

Commands for using VCB to Backup a VMDK

vcbVmName can be used ask VirtualCenter to search for a VM vcbMounter can be used to prepare a VM for backup to tape. This command will snapshot and either mount or copy Every virtual disk attached to a specific VM Alternatively, the sequence of steps below allow you to backup a Single virtual disk, instead of all the virtual disks attached to a VM. vcbExport is the command that will create the backup by exporting the desired virtual disk. Here is the sequence:

1. 2. 3. 4. 5.

Use vcbVmName to get the VMs ID. Use vcbSnapshot to create the snapshot of the VM. Use vcbSnapshot to get a list of the disks in the snapshot. Use vcbExport to export the desired disk(s). Use vcbSnapshot to remove the snapshot.

Alarms New Features

Now include disk and network usage triggers Now, we can specify the number of minutes to wait before allowing the trigger to re-trigger. Now, we can specify a percentage in which the usage must drop below following a trigger, before it can re-trigger

Specifications
These change constanly, so be certain to double-check the latest settings on VMware.com. Here is a link to a document containing maximums for VI 3.5 update 2: http://vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_config_max.pdf

Maximum ESX Server specs:

Storage o 16 HBAs o 128 LUNs per storage array o 256 LUNs per ESX server o 32 paths to each LUN o 1024 paths per ESX Serve o 255 Max Lun ID o 2 TB per VMFS extent o 32 physical extents per spanned VMFS volume o 64 TB max VMFS size (spanned) Processor o 32 threads (logical CPUs) o 192 VCPUs RAM o 256 G per ESX Server I/O Adapters o 64 max total (all types) o 20 Nics (GigE or 100 Mbps) 1016 ports per virtual ethernet switch (1024 including hidden ports) Maximum number of powered on VMs per ESX server =170

Maximum VM specs:
64 G RAM 4 virtual Nics Six PCI slots (one is for video, 5 total for other types) 4 serial ports 3 parallel ports

VirtualCenter Server Specifications

Virtual Center Database Size:
Storage requirements can be larger if your database runs on the same hardware as the VirtualCenter server machine. The size of the database varies with the number hosts and virtual machines you manage. Using default settings for a year with 25 hosts and 816 virtual machines each, the total database size can consume up to 2.2GB (SQL) or 1.0GB (Oracle). DBAs still need to periodically truncate transaction logs in VC 2.0 database, as they did in VC 1.x. Estimate VirtualCenter Database size the best way is to use the calculator in the VC menu: In the VI client connected to VirtualCenter, open this menu: VirtualCenter Management Server Configuration Click on the Statistics link

Enter the number of hosts and VMs and view the DB size.
The size of the database will vary depending on the number of hosts & VMs managed, frequency of performance data collection and type of database. Each stat sample collected is about 60 bytes for SQL, 100 bytes for Oracle, and each event stored is 1600 bytes for SQL, 600 bytes for Oracle. Using default settings, the statistical data for 25 hosts running 8-16 VMs per host will plateau around 40-60 MB in a year (80-140 MB if set to full). Each month, the average number of events generated will also consume about 190 MB in SQL, and 70 MB in Oracle. Total DB size after a year is expected to be around 2.20 Gb in SQL, and 1.0 Gb in Oracle. Using default settings, the statistical data for 75 hosts running 8-16 VMs per host will plateau around 90-150 MB in a year (200-330 MB if set to full). Each month, the average number of events generated will also consume about 190 MB in SQL, and 70 MB in Oracle. Total DB size after a year is expected to be around 2.40 Gb in SQL, and 1.2 Gb in Oracle. To extract data from the VC database VMware has provided a mechnism called Database Views. Have the student read the white paper on our website at http://www.vmware.com/pdf/vc_dbviews_11.pdf

More Details:

VirtualCenter Server Minimum specs:

Assuming that the database resides on a separate server, then the minimum specifications for VC Server follows, which can support up to 20 concurrent client connections, 50 ESX hosts, and 1000 VMs One 2.0 GHz CPU 2 GB RAM 560 MB available for program and %temp% (2 GB recommended) 10 Mbs Nic (GigE Nic preferred) Windows 2000 SP4, Windows 2003 (non-64 bit versions)

Clustering VirtualCenter
Beginning with version 2.01 patch 2, VC Server is now clusterable in MSCS. See this link: http://www.vmware.com/pdf/VC_MSCS.pdf

VI Client 2.0 connection issues

VI client cannot connect to VC Server using a blank password. Expect to see a bad login password error By Default, ESX Server only allows a grand total of 8 concurrent connections including VI Clients, Remote Consoles, and VCB connections. If this is exceeded, additional attempts to connect VI Clients directly to an ESX Server will result with connection or exception messages.

VC Concurrent Connections
The maximum concurrent VC Client connections to a VC Server appears to be 20.

ESX Server Connections

For ESX Server, the default number of concurrent connection is limited to eight. This limit prevents the possibility of jeopardizing the performance or stability of the ESX Server service console by overloading it with too many instances. These eight connections can be used for virtual machine remote consoles, VCB connections (each mounted virtual disk is a connection), and VI Clients that connect directly to the ESX Server. .

If you do not use a VirtualCenter Server (standalone ESX Server host) or if you need to concurrently mount more drives for VCB backups, then you can increase the total number of concurrent connections that hostd allows by editing /etc/vmware/hostd/config.xml. For example, the following line could be added to the section of this configuration file to increase the total number of concurrent connections allowed to 100: <vmdb><maxconnectioncount>100</maxconnectioncount></vmdb>

Virtual Center users and groups

Virtual Center 2.0 (just like 1.x) interfaces with Active Directory (and local user and group accounts on the Windows server running VC Server), but it still does not interface with other LDAP directory services.

Boot ESX 3 into Single User Mode

boot to SC single user mode: - reboot server - at console, at boot menu, choose 2nd item (SC boot) and press e for edit - select the 3rd line, beginning with kernel, press e for edit - type " 1" at the end of the statement, press enter - press "b" to boot

VMDesched
VMware Tools offers a component called VMDesched, which is not installed or enabled by default. This appears to be an experiment by VMware, so far, but it is intended to help the VM guest OS improve its accountability for time caught up. To install this in a Windows VM, re-run the VMware Tools install and choose Custom. Afterwards, use the Services option in Administrator Tools to set the service to automatically start.

Virtual Machine Log File Rotation / Size Options

The simplest way to disable logging for a specific VM is to use the VI client to: Locate and right-click on VM Edit Settings Options Advanced Configuration Parameters Click the Add button to add a new value: o Name = isolation.tools.log.disable o Value = true

We can disable logging, change the rotation, the log files max size, etc. We can adjust these four variables in the VMs log files:

logging log.rotateSize log.keepOld log.fileName

See Knowledge Base Doc ID: 8182749

Static MAC Address for VMs

The VI client does not appear to allow a direct means of assigning a static Mac addresss. Here are the steps to accomplish this by modifying the VMs configuration file (vmx file). Remove the two original ethernet entries in the .vmx file and add these: ethernetN.address = "00:50:56:00-3F:NN:NN" ethernetN.addresstype = "static" Be sure to use an address with the 00:50:56 prefix and the fourth octet somewhere in the 00-3F range or you will get an error in a popup window when you try to boot the VM.

VMware Tools
This should be installed in all VMs, when possible. It may not be possible in unsupported VMs, such as Win NT 4.0 below the SP6 level. However, many of the drivers can still be loaded manually in these scenerios. Choose to install VMware Tools, which attaches the proper ISO to the virtual CD, then browse the CD looking for drivers, such as disc, scsi, video. Typically, whenever major upgrades are made to ESX Server, such as changes to virtual hardware, a new version of VMware Tools is included. Installing the Complete version of VMware Tools (instead merely choosing Typical) is good for environments having a mixture of platforms, such as ESX Server, VMware Server, and Workstation. It simplifies the migration of VMs from one platform type to another.

Syslog
To ensure that important system log files are recoverable, we can modify syslog.conf so specific log files will be copied to another server. The method to this is the same as it was in ESX 2.5, and the same in standard Red Hat. For example, to copy the maillog file to an active server called server1.company.com, modify the following line
mail.* /var/log/maillog

by appending the server name, as follows:

mail.* /var/log/maillog, server1.company.com

Note: the mail.* and /var/log/ above are separated by a Tab, not by blank spaces.

Locate a VMs Swap File in a Separate Location

sched.swap.dir is the advanced setting to accomplish selecting where swap files can live, whether with the vm or in another vmfs. See: http://www.vmware.com/pdf/vi3_esx_resource_mgmt.pdf page 134 for reference.

VC Keeps Old VMFS Names

Sometimes, when trying to assign a VMFS label, VC appends a number to the end of it, as if the name was already used.

VC keeps track of previously used labels. If you change the inventory view to datastores you can see the old name. Select the object and look at the display on the right. You should see that no servers are accessing it. Then right click on it and select either remove/delete. Once it is removed you can change the current datastore label back to what you want.

VCP on VI3 Exam

Is now available. The exam number is VCP-310. The cost is $175. Register at www.vue.com The exam is rather fair. It mostly only ask questions from the Install and Configure course. It does not ask questions about commands but, still learn any commands mentioned in the Install and Configure course It does not ask questions about Upgrading Rely on your course books as your main study reference. Pay attention to details. Preferably, practice performing the steps from your labs until you are comfortable that you can accomplish the main goals without having to follow the book step by step. I recommend reading the following guides at least once, quickly. You do not have to really learn additional material from these, but they should help reinforce what you know: o VI 3 admin guide o VI3 resource management guide o VI3 server configuration guide o Look for these at: http://www.vmware.com/support/pubs/vi_pubs.html Expect to see a few confusing questions. Out of 75 questions, I (and others from my office) agreed that about 10 questions were either poorly written or some assumptions need to be made or something. But each of us had success on the first crack, scoring at least a 89. So, we were either lucky or we managed to make the right assumptions at the right time. Here is a link to a hand reference card, that several students recommended: http://www.vmreference.com/downloads/vmreferenceVI3card1.3for3.5U3.pdf Here is a link to the official VMware VCP Study Blueprint: http://mylearn1.vmware.com/lcms/mL_faq/528/VCP%20on%20VI3%20Blueprint.pdf

64 Bit Support
With VI 3.01, 64 bit is now fully supported for several guest O/S. Check the latest product information and release notes on vmware.com to see the latest support levels for each guest o/s. Supported Guests O/Ss: Microsoft Windows Server 2003 (Standard and Enterprise Server R2) Red Hat Enterprise Linux 3 64-bit (UP7, UP8) Red Hat Enterprise Linux 4 64-bit (UP2, UP3) SuSE Linux Server (SLES) 10 64-bit Sun Solaris 10 (U2) Required hardware: There are specific hardware requirements for 64-bit guest operating system support. For AMDbased systems, the processors must be Athlon64 or Opteron Rev E or later. For Intel-based systems, the processors must include support for Intel's Virtualization Technology (VT). Note that servers that include CPUs with VT support might ship with it disabled by default. You'll have to enable it in the BIOS setup screen (or possible need a new BIOS version). We have a CPU compatibility tool included on the product CD-ROM to check this for you.

Copying files to and from ESX Servers

Persons accustomed to using Putty, WinSCP, and other common tools to copy files to their ESX Servers, may have some issues. Here are some changes: SSH we cannot logon directly as root. Instead, log on as a non-root account, then use su to change to the root account.

Likewise, we cannot log onto WinSCP as root. Use another account (see WinSCP copy files issue above) scp command: if we first SSH into our ESX Server, then attempt to use the SCP command to connect to another ESX Server, we will see a connection refused command, because SSH client is blocked by SC Firewall, by default. Use the VI client, under Configuration Security, modify the SC Firewall and open the SSH client.

VCB
Prepare a hardware system, installing an HBA and attaching a tape drive. Install Windows 2003 on this server (proxy server) Install the Backup Software. Test the backup software functionality by backing up some local files to tape. Establish IP connectivity between the proxy server and VC Server [ or individual ESX host (Service Console port)]. Port 902 is used. Disable automatic drive letter assignments by: o Verify that the proxy server is Not connected to the SAN. (if so, disconnect and reboot) o diskpart o automount disable o automount scrub o exit Use SAN administration tools to ensure that the proxy server is zoned, masked, and permitted to the see the necessary SAN LUNs Ensure that the LUN numbers that appear to the proxy server directly match the LUN numbers that appear to the ESX Servers. (If ESX sees LUN 7, then the proxy server should recognize the same LUN as LUN 7, not as some other number) Install the VCB Framework, which can be launched using the setup command from the VCB installation cd or download. Either install VCB at the default location or choose a different location. If necessary, obtain and install the Integration Modules (Zip files provided by the vendor or vmware allowing the backup software to integrated with VCB framework) Configure VCB by modifying the config.js file in the config subfolder at the location where VCB was installed.

Upgrade VI 2 to VI 3
Methodology
For upgrading to VI 3, NAT typically prefers to use a fresh install plus well paced VM migrations, rather than in-place upgrades. This methodology is typically preferred for the various reasons: It mirrors the preferred methodology that NAT uses when upgrading other technologies, such as Windows NT domains to Active Directory. VMware has reported various issues when upgrading ESX Servers and Virtual Center in place. (They have reported fixes and work arounds for these.) The recommended disk partitions for ESX Server 3 are different than ESX Server 2. In-place upgrades do not provide a means of adjusting the existing partitions.

For these reasons, the following Work Breakout Structure assumes that fresh installs and VM migrations will be used during the project.

Work Breakout Structure

The following is the a sample of a proposed workbreakout structure for upgrading VI 2 to VI 3: Planning o Select the order in which ESX Hosts will be upgraded. o Select the order in which VMs will be upgraded. o Select a schedule for the migrations, allowing ample time to test each migrated VM. o Select an amount of time (days) to leave the original VM on its original ESX Server, but powered off, for backup. o Obtain any necessary VI licenses. o Decide if VC should be upgraded in-place or freshly installed. (The steps below assume a fresh install. The steps also assume that VC will be installed on a physical Windows server, not a VM) Virtual Center o Create a new, empty database on the same SQL Server where the current VC database resides o Create a SQL login to the new database and provide dbowner permissions o On a dedicated system, install Windows 2003, make it a member of the Active Directory domain o Install VC Server o Create an ODBC connection to the SQL Server database created above o Install the VMware License Server o Install the VI License file o Using the VI client, create a single Datacenter object in VC o Install the sysprep program to allow customizations. ESX Server o Migrate all VMs from the first selected ESX Host to be migrated to other ESX Hosts. o Temporarily disconnect the fiber connections on the selected ESX Host. o Freshly install ESX 3 on the host, using default partitioning and networking. o Configure the virtual switches and VM port groups. VMFS Volume o Create a new SAN LUN o Format it with the VMFS3 Add the new ESX Server to VC Server o Using the VI client, add the ESX host to the datacenter in VC server. o Configure the ESX Host to obtain licenses from the license server. Migrate ESX 2 Servers to new VC 2.0 server o Shutdown the old VC 1.x server, but keep handy for backout o In the VC 2.0 Server, add each of the old ESX Servers. Migrate VMs when scheduled. The pace should be deliberate to allow ample testing. Each VM will be shutdown when it is migrated. When migrating a VM, follow these steps o Shutdown the VM, which is running on the ESX 2 servers o In the VI client, Clone the VM from the ESX 2 / VMFS2 locations to the ESX 3 / VMFS3 location. o Plan to keep the original VM intact for backout. o Upgrade the virtual hardware of the new VM o Upgrade VMWare Tools in the new VM o Have end-users test and validate the new VM After a set of VMs have been migrated and tested, typically at least a few days, then delete the original VM from the ESX 2 server and VMFS 2 volume. Once a VMFS2 volume has been emptied, delete the volume and reformat with VMFS3 Likewise, once an ESX 2 server has been emptied, re-install (fresh install) ESX 3.

Determine which port number each VM is connected:

esxtop press n to see network information

Some Basic Commands

to use vi text editor: vi filename press I key make your changes - arrow keys, Del, BKSP, all work the way you expect press ESC key press SHIFT and : keys at colon prompt, enter wq (to write and quit), or enter q! to quit Now.

For Example: Use vi to Modify ifcfg-vswif0 file to change SC IP address vi /etc/sysconfig/network-scripts/ifcfg-vswif0 press I key move to the line with the IP address modify the IP address press ESC key press SHIFT and : keys at colon prompt, enter wq (to write and quit) Note: if we do change the SC IP address, we should make the same change to: /etc/hosts

To copy files from a Windows Share to the ESX Server: smbclient //192.168.28.11/vmimages -U student cd Day1 ls lcd /vmimages get Classfiles.iso Note: we will need to use the VI Client to open the SMB client port in the SC Firewall.

Use scp to copy files from one ESX to another:

scp /vmfs/volumes/vmfs1/VM1/VM1.vmdk username@192.168.28.52:/vmfs/volumes/vmfs2/VM1/VM1.vmdk

Notes: replace "username" with a valid non-root user account. Be sure to use the VI client to open the ssh client port in the SC firewall To register / unregister a VM: vmware-cmd -s register /vmfs/volumes/<vmfsname>/<vmname>/<vmname>.vmx example, vm name = a and vmfs name = storage1 vmware-cmd -s register /vmfs/volumes/storage1/a/a.vmx unregister: vmware-cmd -s unregister /vmfs/volumes/<vmfsname>/<vmname>/<vmname>.vmx

To Import a virtual disc from VMware Workstation:

vmkfstools -i /vmimages/nodea.vmdk /vmfs/Local/nodea.vmdk assuming the file was already visible to the ESX Server in the /vmimages folder

Basic Linux Commands: cd - change default directory ls - list files cp - copy files mv - move file (also use to rename) cat - display a text file less scroll thru a text file rm - remove a file pwd - show present working directory ps -ef - show every process, full information vi - text editor Problem with Repeating Characters In the Remote Conosle, if keys are wrongly being repeated, modify the vmx file and insert the following line: keyboard.typematicMinDelay = 2000000 NOTE: this same setting can now be made in the VI Client, in the Edit Settings window Options tab Configuration Parameters tab. Register a VM To register a VM, use the VI Client to browse the datastore, right click on the VMs .vmx file and choose Add to Inventory. Or use the follwing command vmware-cmd -s register /vmfs/volumes/vmfs1/server1/server1.vmx File Permissions The following command will show a list of files stored in the current default directory. It includes details, such as file size, user (owner) and group (associated with the file). ls -l here is a sample of the prtial results for the folder named /etc.
-rw-r--r-drwxr-xr-x -rw-r--r-1 2 1 root root root root root root 44 4096 1497 Apr 9 Oct 17 Aug 29 11:45 adjtime 2006 alternatives 2002 bashrc

The 3rd column (containing root in the first row of the sample) is the user account that owns the file. The 4th column (the 2nd column containing root in the first row of the sample) is the group associtated with the file. The first column (containing rw-rr in the first row) shows the configured file permissions and attributes. The first character indicates the file type, the next three indicate the permissions assigned to the user (owner), the next three indicate permissions assigned to the group (associated with the file), and the last three indicate the permissions assigned to everyone else (other). Permissions: r = read, w = write, x = execute (allowed to use) The first column is an attribute file that has possible values, such as d for directory. Example:

drwxr-xr-x

root

root

4096

Oct 17

2006 alternatives

1st character = d: indicates that alternatives is actually a directory. Characters 2-4 = rwx: indicates that the owning user (root) has read, write, and execute permissions on the directory. Characters 5-7 = r-x: indicates that the associated group (root) has read and execute permissions (but not write) on the directory. Characters 8-10 = r-x: indicates that the everyone else (other) has read, write, and execute permissions on the directory.

esxtop used to monitor the cpu, memory, network, and disk utilization of the ESX Server. It is interactive and updates every few seconds. Press c, m, n, or d: c cpu utilization, including a row detailing each VM, Service console, drivers, and other items running within the vmkernel. Press e to expand a specific VM to see the utilization of each of its worlds. m memory utilization, including fields related to memory controller (ballooning) and swapping. Press f to change the list of fields. d disk utilization, press u to see information per unit, or use e, c, t, l, to expand the utilization breakdown by the channel, target, and lun numbers. The fields include number of VMs, I/O statistics, and Queue statistics (queue statistics are not currently available in the Performance Graphs of the VI Client) n network statistics, including which VMs or physical Nics are connected to which virtual ports.

URL for esxcfg- and other ESX commands: http://www.penguinpunk.net/blog/?p=7 Reference for VIMSH for Ver 3.5 http://knowledge.xtravirt.com/whitepapers/index.php?option=com_remository&func=download&id=9&chk=c87dd71e82212b156d972829a3bf c97f&no_html=1 URL for basic Linux Commands http://www.ss64.com/bash/index.html Linux tutorial plus labs: http://tldp.org/LDP/intro-linux/html/index.html Other O/S commands: http://www.ss64.com/index.html To automate basically anything in VI 3, the VI ToolKit can be used. The Windows version of this works with powershell. VI ToolKit Download: http://www.vmware.com/sdk/vitk_win/index.html

Sample scripts to run in Service Console

Script to determine if any registered VMs on the server have a snapshot: for i in $(vmware-cmd -l); do ls $i; vmware-cmd $i hassnapshot; done

Script to register to locate and all vmx files located in a VMFS volume named SharedVMs and register as VMs: for i in $(find /vmfs/volumes/SharedVMs/ -name *.vmx); do vmware-cmd -s register $i; done Script to start VMs on an ESX Server in the order specified in a text file:
for i in $(cat servlist);do for j in $(find /vmfs/volumes/ -name $i.vmx);do vmware-cmd $j getuptime;done;done

Without VC Server, these steps are required to create a Windows template

Shutdown the original VM in VI Client, create a new target VM choose Custom provide new name and answer questions much like before Disk - for now create a new virtual disk, but make it small, ( perhaps 1 M) we plan to delete this later (or, connect an existing virtual disk, that again, we will remove later). Unfortunately, the GUI does not allow us to create a VM with No initial virtual disk. Identify the virtual disk files in the original VM by: in the left pane, select the ESX Server in the right pane, select the Configuration tab in the Config tab, click Storage in the Storage pane, right-click on the vmfs volume where the the VM is stored and choose browse. in the browser, locate the folder for the original VM and double-click to open in the list of files, locate the file(s) that have the extension of vmdk make a note of these file names Copy the identified vmdk files from the original VM to the new VM by: on the physical console of the ESX Server, logon as root use the command below to copy vmdk files from the source to the target folder. In this example, I assume the source VM is named SourceVM and the target is named TargetVM. I also assume the vmfs volume is named vmfs1. cp /vmfs/volumes/vmfs1/SourceVM/*.vmdk /vmfs/volumes/vmfs1/TargetVM/ Modify the target vm to remove the small virtual disk and add the new virtual disk by: In the VI client, in the left pane right-click the target VM and choose Edit Settings choose the Hard Drive and click the Remove Button. Check the Box that says Delete from VM and delete files from Disk, then click OK button to add the copied disk, Edit Settings again and Click the Add button, choose Hard Drive, choose existing drive, browse to the name of the virtual disk file as previously identified Now we have an identical clone. We should start the VM and apply Sysprep to strip out its identity (let me know if you do are not familiar with Sysprep)

After sysprep, we can either keep this VM indefinitely and use it as a template, where we simply copy from it (using similar steps as above) each time we want a new VM. We would no longer have to use Sysprep, though.

Smart Cards
In some Active Directory environments, users are authenticated using smart cards, rather than using password authentication. In some cases, the users do not even know their AD password. A question arises, what is the best means for configuring Virtual Center in these cases. No direct means appears to exist to allow users to log into VC Server using AD accounts in this case. Passthrough authentication is not possible on VI client as is, you need to authenticate with a user name and password. ..also there is no known module for smart card authentication. Here are some responses from the VMware Instructor / Professional Services Discussion Board, related to 3rd party options: You might be able to use the SSO Tools/Program with the SmartCard to input the username/password into the VI3 client login window. But this is very dependant of which SmartCard solution you use. This might require user training where to insert the password and the username. Have a look at Citrix Password Manager. More and more Citrix products are less tied to Presentation Server. http://www.citrix.com/English/ps2/products/feature.asp?contentID=21008 This works as a SSO with Smart Card support. Here is another one, but it may need it's own card type. http://www.cylink.com/solutions/sso.asp

USB Dongle
Here is a thread to a good discussion on using USB dongles on VMs running on ESX hosts: http://www.vmware.com/community/thread.jspa?messageID=545615&#545615

Storage VMotion
Storage VMotion is a feature that allows VMs to hot-migrated from one datastore to another, but remain on the same host. This is a slick feature, but apparently VMware is not ready to place it in prime-time. They did not make it available in the VI Client or Service Console. Instead, we need to install the VMware Remote CLI onto a Windows or Linux Workstion and run the svmotion command from there. 3rd party (non-supported) VI Client Plug-in for Storage VMotion A 3rd party created a plug-in for the VI Client to run Storage Vmotion. NOTE: this is not supported by VMware: http://sourceforge.net/project/showfiles.php?group_id=228535

SVMotion command via the Vmware Remote CLI: Choose a PC, Windows server or VM to install vmware remote cli (Linux version is also available) Download the VMware Remote CLI install program. Double-click to install the CLI, accept all defaults. modify the Windows path variable to include: C:\Program Files\VMware\VMware VI Remote CLI\Perl\bin Ensure the path variable change is applied by: right-click on My computer - properties Advanced tab Enironment Variables button

edit the variable named "Path" do not make any changes, but click OK to ensure that the path variable value is applied

In a command prompt, enter: cd C:\Program Files\VMware\VMware VI Remote CLI\bin svmotion.pl --interactive supply the IP address and login credentials for VirtualCenter it will prompt for datastore. Enter the name of the datastore where the ESX server is attached. when prompted for datastore / vm, use this syntax: [datastorename] vmname/vmname.vmx for example, for a vm named server1 stored in SharedVMs; [SharedVMs] server1/server1.vmx It will prompt for the target datastore. Enter the name of the datastore where the VM should be moved.

Extend the size of a virtual disk

To effectively increase the size of an existing virtual disk, some work is required in the VI Client and in the Guest O/S. For example, if a Windows VM has an E: drive that is 2 GB and stored in dedicated VMDK files and if the goal is to extend its E: drive to 4 GB, then do the following: Shutdown the VM In the VI Client Edit Settings choose the correct virtual disk modify its size from 2 GB to 4 GB. Open a console to the VM and power on In Disk Administrator, the second disk appears as 4 GB containing a 2 GB E: partition and an unused 2 GB partition Use the diskpart command to extend the E: partition to use the enitre disk: In a command prompt, enter diskpart Select disk 1 to select the appropriate disk Select partition 1 to select the appropriate partition Extend to extend the partition to fill the remainder of the disk If the drive to be extended includes the C: drive, then temporarily connect it as a second disk to another VM and extend it from there.

Site Recovery Manager

Here is a link to some videos that demonstrate the configuration and usage of SRM: http://www.getyournerdon.com/chris/Lists/Posts/Post.aspx?ID=14