Documente Academic
Documente Profesional
Documente Cultură
Wiki
Blog
Lists
User Voice
Downloads
Xen Planet
Contents
1 Comprehensive Debian Wheezy Xen Tutorial
1.1 Introduction
1.1.1 Objectives Overview
1.2 UEFI Configuration
1.3 Wheezy Installation
1.3.1 References
1.4 Wheezy Configuration
1.4.1 References
1.5 Compiling a Custom Linux Kernel
1.5.1 References:
1.6 Compiling and Installing Xen
1.6.1 References:
1.7 Xen Configuration
1.7.1 References
1.8 Installing Windows HVM
1.8.1 References
1.9 Installing PFSense HVM
1.10 Managerial Activities
1.10.1 Drive Partitioning
1.10.2 LVM
1.10.3 Using mount With LVM Virtual Partitions
1.10.4 dd
Objectives Overview
I wanted a single physical computer, that could handle three computers worth of separated activities at all times, including these four specifics:
Router/Firewall (PFSense)
DNS & Web Development (Debian Squeeze)
Application Development (Debian Wheezy)
Multimedia & Gaming (Windows 7)
I investigated alternative software, including VMWare's ESXi and Citrix's XenServer.
I had come from a VMWare platform having used both VMware Server 2 and VMWare Workstation 8 for the same key objectives previously but with a Windows Host, subject to
Windows Updates which let to my investigation.
ESXi was easy to install, but missing numerous drivers for hardware components. I quickly ruled it out as I wanted a home-use system, and there was no convenient or
well-documented method to installing or even launching a virtual machine from their on-server interface.
Citrix's XenServer was easily my favorite of the options, it's design made for a great user experience. Everything was easy to find and do. However it came with a minimum price tag of
$1000, and if I wanted PCI passthrough for graphics cards $2500. This was beyond my reach as a college student.
I chose Xen since it had support for my hardware, was a free open-source project, and had a great community with lots of activity.
Things you Will Need:
Compatible Hardware
ATI Graphics Card
Motherboard with UEFI & VT-d & Onboard Graphics
Latest Ubuntu Live DVD
Debian Wheezy Beta1 (Or Newer) Installer
nVidia Cards can be made to work, with extensive patching, in Windows XP and supposedly Windows 8 Preview, for more information visit David Techer's blog (http://www.davidgis.fr
/blog/) .
If you plan to pass your graphics card to a virtual machine, you will need either a second graphics card, onboard graphics, or a second computer to manage your Dom0 system and
install virtual machines over VNC.
UEFI compatible boot DVD will save you an undocumented step for setting up a UEFI boot loader.
Xen is picky, different hardware may yield different results, both at compile time and runtime. If you want to save yourself some hassle, here is my hardware list and some suggestions
to avoid:
Motherboard:
ASRock Z68 Extreme4 Gen3
CPU:
Intel Core i2600
RAM:
12GB 1333Mhz Corsair XMS (2x2G 2x4G)
Boot Disk:
240GB OCZ Vertex 3
GPU:
Do not install a boot loader, select the Continue without a bootloader option and be sure to jott down the information it provides us for later.
Complete the installation and restart our computer.
Insert an Ubuntu Live DVD (or live USB created from the DVD iso) and press whatever key opens your motherboard's boot menu.
Select the UEFI boot option from the boot menu.
When presented with the Grub menu screen, press c on the keyboard to access the Emergency Grub Console.
Emergency Grub allows us to manually boot our Linux, since we did not install a boot loader.
Type ls to list the disks, we should have an (hd0) and three gpt partitions. If booting off a Live USB these will appear under (hd1) instead as the USB drive is (hd0).
Assuming the same partitioning as mine, in the below example tab indicates hit the tab key to auto-complete, and we can either use the data provided when we selected Continue
without a bootloader or just substitute xen for the name we gave our Volume Group name, and linux for our root (/)Logical Volume:
grub>
grub>
grub>
grub>
set root=(hd0,gpt2)
linux /vmlinu_tab_ ro root=/dev/mapper/xen-linux
initrd /initrd_tab_
boot
Gibberish may be displayed when tabbing to auto-complete, this is normal and can be ignored. After entering boot it will begin starting Linux.
If done successfully a terminal login will be presented, which concludes this segment of the guide.
In the next segment we will start by enter our credentials and we can start installing some key packages to get UEFI working.
References
EFI Grub on Wheezy (http://blog.garyhawkins.me.uk/?p=185)
To do this I created bold headers with line breaks for special sub-sections, whose subsequent contents you can skip if said bold text does not apply to your situation.
Assuming you have just installed Debian Wheezy and are at the login prompt, you want to login as root so you can install packages (alternatively you can use su to change users if you
prefer).
UEFI STEPS
Assuming you came from my previous guide and have just manually booted the Debian system in EFI mode. Your first objective should be getting an efi bootloader installed:
aptitude install grub-efi-amd64
Once installed you will want to run the install and update scripts:
grub-install
update-grub
Successful installation of the boot manager will result in a list of the boot order with an entry "debian" listed. If this didn't work then you may not have booted in EFI mode.
END UEFI STEPS
Now let's install a couple more basic packages:
aptitude install ssh sudo
With ssh installed we can, optionlly, access remotely for the remainder of the guide. With sudo we can add our user to the sudoers group like so (cdelorme is my username):
usermod -aG sudo cdelorme
If you omit the a you will replace all existing groups your user belongs to, so be careful.
I recommend rebooting now to get ssh working, and also so you can login as your user to use the sudo command (if you used su you may need to logout and log back in for sudo to
work).
Next we want to prime our server with some general utility packages:
parted
samba
ntp
screen
p7zip-full
Delicious Copy Pasta:
sudo aptitude install parted samba ntp screen p7zip-full
Note: I used to install ntfsprogs but the package is currently bugged and eats CPU cycles, so I would avoid it if possible.
Configuration of samba and ntp will vary by system, I recommend using other guides online for this since as long as they work they will be fine when we start using Xen. If you want my
configurations specifically, ask and I may consider adding them to a future edition of this guide.
Development Environment Steps
Since I intend to use Debian Wheezy as a development environment, I also installed these packages:
openjdk-7-jdk
ant
g
libboost-dev
gnustep
python
python-dev
Delicious Copy Pasta:
sudo aptitude install openjdk-7-jdk ant g
If you intend to use Xen with SDL later on, I recommend replacing xserver-xorg-core with xorg and xorg-dev packages, as these are what we will install later, and they contain
the xserver-xorg-core.
As a bare-bones install of Gnome3, it should take roughly 5 minutes as opposed to 30.
At this stage you can run startx to launch the GUI environment, but I generally reboot to test the login screen.
End GUI Environment Steps
Optional GUI Additions
Optional additions include enabling audio, installing some basic multimedia, utilities, and important software like Guake terminal, Google Chrome, and Sublime Text.
Starting with Audio:
sudo aptitude install alsa-base
sudo alsactl init
font-manager
gnome-disk-utility
gnome-screenshot
guake
Delicious Copy Pasta:
sudo aptitude install python-xdg font-manager gnome-disk-utility gnome-screenshot guake
Note: Guake has a bug with Gnome3 that prevents it from starting at boot, here is how you can fix it:
This is actually a very simple edit, in this specific version of guake (located at /usr/bin/guake) line 653 contains a notification.show(), we just have to wrap it in a simple
try-catch so it doesn't crash when it is run before the GUI is ready to accept a pynotify event:
try:
notification.show()
except Exception:
pass
Next we want to set Guake to start at login. You can make an auto-start file via:
mkdir ~/.config/autostart
cd ~/.config/autostart
echo "[Desktop Entry]" >> guake.desktop
echo "Name=Guake Terminal" >> guake.desktop
echo "Name[pt]=Guake Terminal" >> guake.desktop
echo "Name[pt_BR]=Guake Terminal" >> guake.desktop
echo "Comment=Use the command line in a Quake-like terminal" >> guake.desktop
echo "TryExec=guake" >> guake.desktop
echo "Exec=guake" >> guake.desktop
echo "Icon=/usr/share/pixmaps/guake/guake.png" >> guake.desktop
echo "Type=Application" >> guake.desktop
echo "Categories=GNOME;GTK;Utility;TerminalEmulator;" >> guake.desktop
For google chrome we need to add a new debian source to /etc/apt/sources.list, and download their package key:
su root
Note: I am aware that google source is for Squeeze, I have had no problems running it in Wheezy.
After google-chrome has been installed, it will create its own sources.list file entry, so we want to go back in and remove the line we added from /etc/apt/sources.list, and re-run
sudo aptitude update to make sure it works.
Finally, I am in love with Sublime Text 2, so I went to their website for the download, and went through these steps for a complete installation.
Once downloaded, let's extract it and move it to /usr/local, and create a new command to run it from terminal:
tar xf sublime-text.tar.bz2
sudo mv sublime-text /usr/local/
cd /usr/local/sublime-text
sudo ln -s /usr/local/sublime-text/sublime_text /usr/bin/subl
Next let's create a desktop file so we can run it as an application and set default mimetypes from the GUI. Let's navigate to /usr/share/applications/ and create a new file
subl.desktop with these contents:
[Desktop Entry]
Name=Sublime Text 2
Comment=The Best Text Editor in the World!
TryExec=subl
Exec=subl
Icon=/usr/local/sublime-text/Icon/256x256/sublime_text.png
Type=Application
There are user-account alternatives that may seem more appropriate, but I wanted these to apply system-wide.
End Optional GUI Additions
Thus concludes our Debian Wheezy configuration, and we are now ready to move onto compiling a Custom Linux Kernel.
References
Missing ia32-libs-gtk for Gnome3 (http://lunduke.com/forum/discussion/126/ubuntu-12-04-64bit-error-while-loading-shared-libraries-libgtk-x11-2-0-so-0/p1)
Samba Config (http://www.samba.org/samba/docs/man/manpages-3/smb.conf.5.html)
cd ~/src/linux
To improve compiling speed we can set the concurrency to double the number of physical CPU cores by running these commands:
su root
echo "CONCURRENCY_LEVEL=8" >> /etc/kernel-pkg.conf
exit
IMPORTANT NOTE: Kernel 3.5 and 3.5.2 are bugged[1] (http://wiki.xenproject.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial#endnote_3.5-bug) and passthrough with USB
Controllers fails (though VGA Passthrough works for some odd reason). That is why this guide is using 3.4.9, however 3.3.8 has also been tested and works.
Our next step is downloading, extracting, and inserting the current kernel configuration before adjusting the kernel we are about to compile:
wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.4.9.tar.bz2
tar jxf linux-3.4.9.tar.bz2 && rm linux-3.4.9.tar.bz2
cd linux-3.4.9/
cp /boot/config-3.2.0-3-amd64 .config
make menuconfig
Configuring a kernel is not as complex as it seems, the menu is just a hierarchy of options, and thankfully you can search for settings with / and get details on individual settings with h.
Here is a tree-map list of all the important settings we need to change:
Note that in later versions of gcc the revision must begin with a number.
On my machine it takes roughly 20 minutes to compile a kernel, so feel free to take a break.
Once the compilation is complete, we can back up a directory where we should now find a portable .deb installation package. I highly recommend making a backup of this file if you
want to reinstall that kernel without all this effort in the future:
cd ..
sudo dpkg -i linux-image-3.4.9_3.4.9.custom_amd64.deb
At the end of the installation, it should automatically run update-grub, at which point you can reboot your system.
Once the machine has rebooted and you are logged in, type uname -r and if the value it spits out is 3.4.9 (or whatever kernel version you compiled) then you are successful!
Strap in, because the next segment will cover compiling the latest Xen from source.
References:
1. ^ Bug 46341 - Xen PCI passthrough does not work on kernel 3.5.2 (https://bugzilla.kernel.org/show_bug.cgi?id=46341)
sudo aptitude install bridge-utils build-essential libncurses-dev python-dev uuid uuid-dev libglib2.0-dev libyajl-dev bcc gcc-multilib iasl libpci-dev mercurial fle
Note: libpci-dev replaces pciutils-dev (which routes to said package), just an FYI.
These packages appear to be on the requirements list, but were already on my system at this stage. If you have not been following from the previous segment you may want to run this
for good measure;
sudo aptitude install bcc patch libncurses5-dev python python-dev libglib2.0-dev bin86 bzip2 module-init-tools make gcc libc6-dev libcurl3 iproute xz-utils
Obviously you have the choice to use only one, you do not need both, but if you intend to use the GUI environment in Debian I highly recommend you give SDL a spin, it's much faster
than VNC, though VNC does offer remote access.
End GUI Xen Packages
Complete The Dependencies
The list I already provided is a minimalists list, containing only exactly what was required to compile and run Xen for my own purposes, which includes PVHVM systems of Windows,
FreeBSD, and Linux, as well as PCI Passthrough, and both VNC and SDL. It has worked for me since I started tinkering back in March, with minor modifications since.
For those who want the whole package and have a spare gigabyte of OS Disk space available, I have put together this sub-section.
I should also note that some of these packages will change flags marked as no from ./configure to a yes, but that did not stop me from compiling and running the system:
zlib1g-dev
libbz2-dev
liblzo2-dev
e2fslibs-dev
ocaml
ocaml-nox
ocaml-findlib
lzma
lzma-dev
liblzma-dev
markdown
libcurl4-openssl-dev
transfig
tgif
libvncserver-dev
python-twisted
libjpeg62-dev
gawk
git-core
texinfo
texlive-latex-base
texlive-latex-recommended
texlive-fonts-extra
texlive-fonts-recommended
If anyone knows exactly what any of these dependencies do or are used for and can shed some light on them I would greatly appreciate it.
End Complete The Dependencies
Next we can visit xenbits to get the latest revision. In our case we will use revision 25824[2] (http://wiki.xenproject.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial#endnote_rev25824)
[3] (http://wiki.xenproject.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial#endnote_rev25784) , and use mercurial to download the source:
mkdir -p ~/src/xen && cd ~/src/xen
rev=25824
hg clone -r $rev http://xenbits.xen.org/hg/xen-unstable.hg/ xen-unstable.hg-rev-${rev}
cd xen-unstable.hg-rev-${rev}
2. ^ Original guide suggests using Xen 4.2-rc4-pre, revision 25777. Xen 4.2-rc4, revision 25824 has also worked for another user, User:V
3. ^ Xen 4.2-rc4, revision 25784 compiled with an error for a potentially uninstantiated variable, which may suggest a gcc compatibility bug
EFI Source Modification
Given that I encountered no make errors, I was shocked to find that they still hadn't addressed the grub efi compatibility bug where it fails to recognize available system memory.
The fix for this is a source modification, though they have supposedly added a build process for xen.efi, which would replace the debian.efi generated by grub. I have yet to find
adequate instructions to make that work, and even if I did may break adding xen-pciback.hide to the grub configuration, which we need if we are planning to do the easy method of
PCI Passthrough.
So, let's open up xen/arch/x86/setup.c with our favorite text editor, and find the line containing ( e820_raw_nr != 0 ), and make that area look like this:
#if 0
else if ( e820_raw_nr != 0 )
{
memmap_type = "Xen-e820";
}
else if ( bootsym(lowmem_kb) )
{
memmap_type = "Xen-e801";
e820_raw[0].addr = 0;
e820_raw[0].size = bootsym(lowmem_kb) << 10;
e820_raw[0].type = E820_RAM;
e820_raw[1].addr = 0x100000;
e820_raw[1].size = bootsym(highmem_kb) << 10;
e820_raw[1].type = E820_RAM;
e820_raw_nr = 2;
}
#endif
else if ( mbi->flags & MBI_MEMMAP )
Notice we are commenting out a section of code using "if 0", which always evaluates to false, to eliminate the section of e801 mapping which breaks our ram recognition.
I have no idea what kind of adverse effects this has on other systems, but it has worked for me since Xen 4.1.2.
End EFI Source Modification
SSH Users
Those of us using SSH to follow along should now run screen so we don't loose our footing while compiling.
End SSH Users
We can now run the configuration script and compile the source:
./configure && make -j4 world && make -j4 deb
Notice the -j4 flag which sets concurrency for us, with that setting I have compiled Xen ins just under 7 minutes. It will consume roughly 1GB of space once fully compiled.
Instead of make install I used make deb to create a debian package which we can backup for future installations, and to simplify removal. This file will be placed in the dist/
folder.
At this stage if you don't see any make errors and the debian file is in the dist folder, you have successfully compiled Xen and are ready to move to the next phase of installing and
configuring Xen!
References:
Xenbits Repositories (http://xenbits.xensource.com/)
Latest Instructions (http://wiki.xen.org/wiki/Xen_4.2_RC2_test_instructions)
4.1 Compilation Instructions (http://wiki.xensource.com/xenwiki/Xen4.1)
4.0 Compilation Instructions (http://wiki.xensource.com/xenwiki/Xen4.0)
David Techer's Blog (http://www.davidgis.fr/blog/)
EFI Memory Error (http://serverfault.com/questions/342109/xen-only-sees-512mb-of-system-ram-should-be-8gb-uefi-boot)
Our work is not done yet, we have a lot of post-install configuration to take care of before we can reboot and verify that Xen is working.
Let's start with a trip to /boot to delete extra symbolic links and related files:
cd /boot
sudo rm xen.gz xen-4.gz xen-4.2.gz xen-syms-4.2.0-rc4
Next we want to have Xen boot before the Linux Dom0 kernel, Xen will load that after. To do this we need to move the grub configuration file forward and update grub via:
sudo mv /etc/grub.d/20_linux_xen /etc/grub.d/09_linux_xen
sudo update-grub
Finally, before we reboot the system we need to make sure we initialize the Xen toolstack and related features at boot time via:
At this stage we reboot our system, and if all goes well the first option in grub will be Xen, which will bring us to our linux login.
Our next step is logging in and testing the xl toolstack with sudo xl dmesg, if the output you receive is a not an error, then you have successfully installed Xen.
Next we need to setup a virtual bridge for our virtual machines going forward. Here is a simple recommended configuration you can use to replace the contents of /etc/network
/interfaces (See my PFSense Segment for a more complex multi-bridge configuration):
auto lo xenbr0
iface lo inet loopback
iface eth0 inet manual
iface xenbr0 inet dhcp
bridge_ports eth0
To make this change take effect you can restart the network service, ifconfig, or take my approach and reboot.
Making Things Easier
As many are aware, security comes at the cost of usability.
So this section is a way to make using Xen easier at the slight cost of some security.
Add one of the following segments of code to the end of /etc/sudoers to omit entering a password when running the xl command:
Option 1:
# Add no-password xl access
%sudo ALL=(ALL:ALL) ALL, !/usr/sbin/xl, NOPASSWD: /usr/sbin/xl
Option 2:
# Cmnd alias specification
Cmnd_Alias XL = /usr/sbin/xl
# Allow members of group sudo to execute any command
%sudo
ALL=(ALL:ALL) ALL, !XL, NOPASSWD: XL </nowiki>
If using vi you may have to use :wq! to force write quit when saving.
Next let's add an alias so whenever we type xl it assumes sudo xl by adding an alias to /etc/bash.bashrc:
# XL Alias
alias xl='sudo xl'
Now when we login next instead of using sudo xl and entering a password, we can just use xl and no password is necessary!
End Making Things Easier
This concludes Xen Configuration, we are ready to move onto setting up HVM Virtual Machines!
I recommend checking the table of contents for details on managerial activities, which includes backing up and restoring, LVM and partitioning, and a few other helpful related activities.
References
Xen Man Pages (http://wiki.xen.org/wiki/Xen_Man_Pages)
Outdated Document Mentions Xen Syms File (http://xen.xensource.com/files/xen_user_manual.pdf)
For now we are using xenbr0 for our network bring, but when we set the vif in our configuration it must have a mac address with a prefix of 00:16:3e. For this HVM I will be using the
emulated Intel e1000 gigabit ethernet adapter.
Here is my Windows 7 configuration file for installation:
name='windows'
builder='hvm'
vcpus=4
memory=6144
disk=[
'/dev/mapper/xen-win2,,hda,w',
'/media/samba/docs/win7.iso,,hdc,r,devtype=cdrom'
]
vif=[
'bridge=xenbr0,model=e1000,mac=00:16:3e:14:b1:1c'
]
boot='dc'
pae=1
nx=1
videoram=16
stdvga=1
sdl=1
vnc=0
usb=1
usbdevice="tablet"
localtime=1
Once Windows has been installed we can change boot='dc' to boot='c', and remove the iso from the drives.
Changes to the configuration will not take effect until we have shut down the HVM and run the xl create command again.
Passthrough
Passthrough is an important and moderately complex topic.
First we should cover the basics of PCI devices in Linux.
Linux recognizes devices using a BDF code, which stands for Bus, Device, Function. You will commonly see it in the format of BB:DD.f, and it is how we point to devices when we
want Linux to do something with them.
To check your systems PCI device identifiers you can use lspci. This will print them in BDF format. You can get more details by adding the verbosity flag -v, you can add up to triple
verbosity, which is several pages of in formation.
We usually just need basic names to identify our devices, here is an example lspci from my ASRock Z68 Extreme4 Gen3 Motherboard:
00:00.0
00:01.0
00:02.0
00:16.0
00:1a.0
00:1b.0
00:1c.0
00:1c.4
00:1c.5
00:1c.6
00:1c.7
00:1d.0
00:1f.0
00:1f.2
00:1f.3
01:00.0
01:00.1
03:00.0
04:00.0
05:00.0
06:00.0
07:01.0
07:04.0
07:05.0
07:06.0
07:07.0
07:08.0
07:09.0
08:00.0
09:00.0
0b:00.0
0c:00.0
Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)
Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)
PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5)
PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
ISA bridge: Intel Corporation Z68 Express Chipset Family LPC Controller (rev 05)
SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)
SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Barts XT [Radeon HD 6800 Series]
Audio device: Advanced Micro Devices [AMD] nee ATI Barts HDMI Audio [Radeon HD 6800 Series]
SATA controller: Marvell Technology Group Ltd. 88SE9120 SATA 6Gb/s Controller (rev 12)
USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)
FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire Controller (rev 01)
Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10)
Unfortunately USB devices are not labeled on the board as they are seen in the lspci printout, but I have 2 of each type of controller, and which items are connected to which
controller is unknown. I had to spend some time testing which controller was which by passing one at a time to a virtual machine. When passed I could not use them in Dom0, so be
sure you have some form of remote control or a PS/2 keyboard and mouse as a backup.
I tested my USB 3.0 ports in linux and had zero gains in speed, so I decided to pass both USB 3.0 controllers and the larger USB 2.0 controller to the pciback module for passthrough.
I ended up with these five items as shown in BDF format:
The line we want to modify starts with module, and here is what we add:
module
quiet xen-pciback.hide=(00:1d.0)(01:00.0)(01:00.1)(04:00.0)(05:00.0)
We will have to force write-quit to save the file (:wq!), then when we reboot the system our devices will be available for passthrough!
Next we add this to our HVM Configuration File:
pci = [
'00:1d.0',
'04:00.0',
'05:00.0',
'01:00.0',
'01:00.1'
]
It is important to note that you may NOT want to pass through devices until after you have installed and updated the virtual machine. I have encountered many problems with passing
devices during installation.
Important Note for Graphics Passthrough
nVidia cards do not work out of the box, but ATI does.
However, most cards do not support FLR (Function Level Reset), which is important because it prevents the card from operating properly without the right steps.
Essentially, unlike a physical computer, when the virtual machine is shut down or restarted, it does not change the power supplied to the graphics card. This means the graphics card is
never re-initialized, so FLR was invented. However, because most graphics cards do not support FLR you have to mimic this effect by ejecting the card after every reboot, otherwise
you will see a severe performance degradation.
In addition, if you do not reset the card, and it is not fresh when you attempt to install or uninstall drivers, the process may fail leaving your system crippled, plus a BSOD is likely to
appear.
So, my recommendation when dealing with GPU drivers and passthrough, always reboot the entire machine, or take extra care to reset the card before making any changes.
After the installation, if you reboot the HVM you can use the Eject USB Media icon in the taskbar at the lower right hand corner of the screen to eject the card which will attempt a
manual reset. You will loose video for a few seconds as the card re-initializes. This should fix performance on reboot.
PV on HVM Drivers are awesome
They can improve network and hard drive performance by a substantial amount. In Windows these are referred to as GPLPV Drivers.
To my understanding they bypass the emulated hardware layers reducing the latency between operations.
My Windows Experience Index with a SSD went from 6.8 for hard drive to 7.8 by installing these drivers.
Supposedly to enable PV on HVM you have to add this line to your HVM Configuration:
xen_platform_pci=1
The final configuration after my devices are passed and PVHVM Drivers are installed looks like this:
name='windows'
builder='hvm'
vcpus=4
memory=6144
disk=[
'/dev/mapper/xen-win2,,hda,w'
]
vif=[
'bridge=xenbr0,model=e1000,mac=00:16:3e:14:b1:1c'
]
pci = [
'00:1d.0',
'04:00.0',
'05:00.0',
'01:00.0',
'01:00.1'
]
boot='c'
pae=1
nx=1
videoram=16
stdvga=1
#sdl=1
vnc=1
vnclisten="0.0.0.0:10"
vncpasswd='password'
usb=1
usbdevice="tablet"
localtime=1
xen_platform_pci=1
If you do not have ntp setup on the host machine (Dom0) do so immediately. Setting localtime=1 should use the correct host time, without it Windows clock will drift at boot and
during runtime by a substantial amount.
Note that as unimportant as clock time seems, if it drifts far enough your cookies, sessions and SSL certifications will disappear, become invalid, and the operating system itself
becomes absolutely unstable. For example all of my taskbar icons disappeared and were repalced with blank spaces, including my eject USB Media icon. It was a scary experience.
Since I use this system for recreational purposes, I have posted a performance demonstration video that shows HDD, SSD, and USB data transfer speeds, as well as graphics
performance with video games.
References
Xen Man Pages (http://wiki.xen.org/wiki/Xen_Man_Pages)
Managerial Activities
There are a few tricks I can share with regards to managing Xen virtual machines.
However, having an understanding of the fundamentals is also helpful.
Here are the key areas:
Drive Partitioning
LVM
Using mount With LVM Virtual Partitions
dd
Drive Partitioning
This is a fundamental, you are welcome to skip ahead if you want the how and not the why.
I find a lot of emails floating around the xen-users email list due to misunderstandings in how Xen recognizes and managed partitions.
It is important to understand how Xen Virtual Machines treat partitions.
It sees them as entire hard drives, and will create a brand new partition table and partitions inside of it.
The result is sub-partitioning, and this causes an offset between the start of the actual partition and the start of the sub-partition containing all of the data on the virtual machine.
In more detail
Computers typically segment hard drives into three components.
Partition Tables
Partitions
File Systems
A partition table tells the disk where the beginning and end of a partition exists and usually the file system.
The Partitions are simple blocks of storage.
The File System tells the operating system how to access the partition.
A normal installation of Windows for example, will look like:
Drive Block 0
Partition Table:
Starts at Block 256 File System NTFS
Drive Block 256
Partition
Data
LVM
Linux has LVM handle all the tough parts of managing the positions of data blocks, and makes everything else easier for the user.
Hence it is called the Logical Volume Manager, because it manages logical volumes.
What this means is when we create a LVM Partition, we are actually using an LVM "File System", which looks like this:
Drive Block 0
Partition Table:
Starts at Block 256 File System LVM
Drive Block 256
Partition
Data
We can then create partitions using LVM which automatically handles the location of the data inside the LVM Partition.
For example, we can create two LVM Partitions for a Windows and Linux Virtual Machine, which would look like this:
Drive Block 0
Partition Table:
Starts at Block 256 File System LVM
Drive Block 256
Partition
Data
LV Windows
LV Linux
Only an operating system that recognizes LVM can see "LV Windows" or "LV Linux" though, so if you plug that drive into a basic Windows machine, it won't be able to do anything with
the data.
Finally we have Xen
With Xen we give one of those partitions to our virtual machine, but as stated it sees that as a hard drive.
If we use the previous drive configuration and just add a Windows Virtual Machine to LV Windows our configuration looks like this:
Drive Block 0
Partition Table:
Starts at Block 256 File System LVM
Drive Block 256
Partition
Data
LV Windows
LV Block 0
Partition Table
Partition at Block 256 File System NTFS
LV Block 256
Partition
Data
LV Linux
At this point, we cannot simple access the data on our Xen partition because it has been sub-partitioned.
If you try simply using mount it will fail, because it requires the offset to the start of the NTFS sub-partition!
Using unit B we indicate bytes which will be used by the mount command, and we get this:
Number Start
1
1048576B
2
105906176B
End
105906175B
171796594687B
Size
104857600B
171690688512B
Type
primary
primary
Important to note that Windows 7 creates a 100MB Boot Partition, so the first item with the boot flag is actually the boot partition, we want the Start value of the second, which is
105906176B.
We can now mount the partition using:
sudo mount -o offset=105906176 /dev/mapper/xen-windows /path/to/mount/dir
dd
The utility dd is probably one of my favorites among the tools that come with most linux/unix systems.
It can be used to create an exact copy of data, and is not subject to a number of limitations with higher level software.
It is not the fastest utility, but supplemented with tools like gzip you can compress a complete backup into a size similar to that you might see with professional software.
Here is a backup Example that compressed an 80GB Windows LV with 70GB of Data into a 43GB GZipped Image File:
sudo dd if=/dev/mapper/xen-windows of=/path/to/backup/windows.img bs=1M
sudo gzip -9 windows.img
Alternative one-liner (Takes a lot longer since it has to compress "on the fly"):
sudo dd if=/dev/mapper/xen-windows bs=1M | sudo gzip -9 > /path/to/backup/windows.img.gz
I can also use this command to restore my system from that compressed image:
sudo gzip -d /path/to/backup/windows.img.gz
sudo dd if=/path/to/backup/windows.img of=/dev/mapper/xen-windows bs=1M
There are plenty of other uses I have found for the dd command, but backups and restoration make Linux amazingly easy to restore after terrible failure, and believe me in writing this
guide I encountered that plenty of times.
You've reached the end of my guide!
Thanks for taking your time to read through my documentation, if you found it at all helpful please feel free to shoot me an email on the Xen Users email list.
Any feedback or corrections are also welcomed.
I am still working on the PFSense segment and will have that up as soon as I can.
Categories Xen
Users
Tutorial
Debian
VGA
2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.