STICKING IT

TO THE MAN

BUILDING A
PRODUCTION
VDI WITH
FOSS
CHETAN VENKATESH BRIFORUM 2012
CTO & FOUNDER CHICAGO
ATLANTIS COMPUTING
VIVA LA REVOLUCION
Tweet me at
@chetan
Tweet from this session
#BRIFORUM-STIM
This presentation is Licensed under Creative
Commons
The Creative Commons copyright licenses and tools forge a balance inside the
traditional all rights reserved setting that copyright law creates. Our tools give
everyone from individual creators to large companies and institutions a simple,
standardized way to grant copyright permissions to their creative work. The
combination of our tools and our users is a vast and growing digital commons, a
pool of content that can be copied, distributed, edited, remixed, and built upon, all
within the boundaries of copyright law.

2012 Chetan Venkatesh. All Rights Reserved.

AGENDA
IN NO PARTICULAR ORDER
Danger VDI ahead
The open source conundrum
Anatomy of FOSS VDI
The Storage Platform
The Hypervisor
The Windows Layer
The Thin Client Platform

Putting it all together

ACHTUNG! ACHTUNG!

VDI
VDI AHEAD
AHEAD
TOP 3 VDI MYTHS
VDI is a form of STD that originated in IBM
VDI is transmitted through toilet seats
VDI is the accidental love child of a Vmware engineer and
Microsoft Bob (and or Windows 98 ME)

- Source of all these myths

The VDI Delusion by Brian Madden & Gabe
Knuth

Audio tape of book played backwards

MANDATORY REDDIT
STYLE MEME
TOP 3 VDI REALITIES
VDI is complex
VDI is expensive
VDI is politics

But VDI does work

Large Global bank 80,000 VDI desktops out of 300,000 total
Large Japanese Financial 200,000 VDI desktops
VDI seats growth in deployment between 2011-2012
Averages 1000%
VDI is among top 3 initiatives at majority of Fortune 500 CIOs
The VDI stack is finally mature (enough) for at scale, across
enterprise deployments
WHY OSS VDI
Because we can the stack is ready for commoditization
Components for every piece of the stack exist if you care
to look for them and make them work
Its a great way to learn cloud computing
Its CAPEX friendly
It can be made to scale to 1000s of users with time,
patience and effort
Its a massive commercial opportunity
Imagine a Redhat like VDI distro thats Free but
commercially supported
THE OPEN SOURCE
CONUNDRUM
This is NOT easy stuff
Heavy Duty Skills required
Serious Linux Skills more than ls, pwd and chown
C / C++ to hack code because stuff wont work
Make files because stuff needs to be compiled
Bash/Perl/Python because OSS is full of script kiddies
A decent understanding of Linux internals
Hypervisors are kernel modules/extensions
Storage and I/O are complex
Plethora of Filesystems some good, mostly bad
Tons and Tons and Tons of projects/code that you need to
sift through to figure all this out
COMMERCIAL VDI
% Time spent
70
60
60
50
40
30 % Time spent
20
10 10 10
10 5 5
0
ROLL YOUR OWN VDI
% Time spent
50 45
45
40
35
30
25 19 20
20 % Time spent
15 10
10 5
5 1
0
GOALS OF THIS
SESSION
Overview of a production grade VDI using nothing but
FOSS
Except Windows of course
Provide a detailed architecture of a soup to nuts VDI OSS
stack for 500 Windows 7 users
Provide enough detail to put together a turnkey VDI
Appliance like the ones for N***nix etc
Provide a website where you can download all of the code
from and howtos to put it together (watch on twitter)
Eventually a complete packaged one click installer that
will do it all
DEFY YOUR OWN GROUP. REBEL AGAINST
YOURSELF

ANATOMY OF
VDI
WAN Optimization

Thin
Thin Client
Client Platform
Platform

Windows
Windows &
& App
App Virtualization
Virtualization

Broker
Broker

Hypervisor

Network

Storage
DESIGN GOALS
Must be practical to deploy
Cost effective
Perform within expectations
Equivalent Analogs to commercial components
Support both Persistent & Non Persistent Desktops
Must support HA for persistent desktops at storage level
Must support thin provisioning for cloning VMs (ala linked
clones/MCS)
Nice to have - a vBlock like POD architecture for 500 users
STORAGE
VMs use a virtualized Hard drive that
needs storage that is
Consistent
What you write is what you read
back
Highly Available
Should be able to read and write all
the time, any time
Adequate Performance
Should be able to drive IOPS hungry
VMs and Windows Operations
Windows Desktop VMs are very I/O
hungry & need very high performance
storage
CAP THEOREM &
STORAGE

Consistency

Pick Any Two

Partition/
Availability
Performance
WINDOWS DESKTOPS &
VIRTUALIZATION
Write intensive
70%-80% of I/O is write worse in the case of Linked Cl one/Thin Provisioned VMs
Small I/O
Majority of I/O averages at 4KB.
Lots of small direct I/O writes issued by workloads
Mostly Random
Hypervisors perform very poor I/O scheduling
Hypervisors unaware of underlying Storage Layout
Blend different types of I/O from different VMs into a variable and highly random access
pattern
High Bursts
Will read and write at queue depth of Virtual Disk driver not physical hardware
Cumulative queue depth is order of magnitude higher than the physical path
10X more VMs per host than with server virtualization
Interactive Writes cause Read starvation challenges
Reads are synchronous and block application/UI on interactive workloads
Stateless/Stateful separation challenges
WINDOWS IS OIL / STORAGE IS WATER
Windows is built for real not virtual
Windows has many Optimizations for Physical PCs that run on spinning media (HDD)
Optimizations eat I/O capacity because Windows is unaware that its storage is virtualized

Windows Storage System optimizations that hurt VDI

Windows Moves data to outer lying sectors to reduce access latency
Windows re-orders file on disk layout to make files contiguous on disk
Windows does not zero deleted blocks only marks as deleted (breaks TP/linked clones)
Windows Blocks Applications/GUI if outstanding I/O depth reaches 12 (the hourglass)
Read ahead is implemented as block ranges and not file-system inode traversals

Virtual Memory is a big overhead

Interactive workloads have erratic virtual memory utilization
Windows will page memory/disk regardless of the amount of RAM given to the OS
Page faults decrease application performance by 1330X compared to if data was in memory
HOW IOPS ARE USED IN A WINDOWS
DESKTOP
User Data
Paging & Virtual
Windows OS Memory I/O
Core

A/V Access
Scanning Sector latency
optimization
File Layout
optimization
WINDOWS & CAP

Consistency

Partition/
Availability
Performance
STORAGE HARDWARE
Hardware for SAN/NAS
4U Rack Server Chasis with dual redundant power supply
Single Socket Dual Core 2Ghz Processor 48GB DDR3 RAM
Battery backed caching RAID card
24 Drive Bays with 10K SAS drives 300GB Each
RAW Capacity of 6.7 TB
Usable Capacity of 4.4 TB RAID 50
Average Persistent VM size 40GB = 102 VMs
Average NP VM size 5GB = 819 VMs
Usable IOPS capacity 1100 4KB IOPS
Average VM requires 30 IOPS = 36 VMs*
So for 500 VMs I would need a 333 Drives or add 13 More
Storage Servers like this one.
STORAGE OPTIONS
SAN/NAS Local Disk

Consistent Yes Yes

Available Yes No

IOPS No No

Cost Effective No Yes

Snaphot Features Yes No

Persistent VMs Yes No

Non Persistent Yes Yes

VMs
SO IS THERE A
BETTER WAY?
What if you could use local storage but get all the desirable
features of SAN

Centralized Storage
Snapshots/Clones
Highly Available
Using Local Disk?
INTRODUCING
SHEEPDOG
Stupid name great technology
One of the most exciting Open Source projects today
A Virtualization Specific Storage Infrastructure
A distributed Block Level Storage system
Runs on local storage
Aggregates a server/nodes local storage into a collective
pool.
Scales to Hundreds of Nodes
Supports advanced volume management features
Snapshots, cloning, thin privisioning
STORAGE OPTIONS
SAN/NAS Local Disk Local Disk with
Sheepdog

Consistent Yes Yes Yes

Available Yes No Yes
IOPS No No Yes
Cost No Yes Yes
Effective
Snaphot Yes No Yes
Features
Persistent Yes No Yes
VMs
Non Yes Yes Yes
Persistent
VMs
HOW DOES SD WORK?

Windows 7 VMs 2GB RAM

4 VMs per core = 32 VMs
Slight memory overcommit

Sheepdog Sheepdog node with Corosync

Linux 3.0 with KVM enabled

Linux
Linux KVM
KVM KSM enabled

Dual Socket Quad Core 2Ghz or better

64 GB DDR3 RAM
4 X 10K SAS Drives R0 = 1.2 TB / 600
IOPS
4 X 1Gbe - 3 X 1Gbe LACP links for
RAID 0 Stripe Storage
HOW DOES SD WORK?

Automatic VM
Replication
HOW SD WORKS ..
C-A OF CAP
HOW SD WORKS ..
ADD NODES DYNAMICALLY
SD PERFORMANCE
3 HOSTS & 1 VM
SD PERFORMANCE 64
HOSTS & 256 VMS
Linear Scalability in-terms of performance
C.A.P NOW AGREES

Consistency

Performance
Availability
*sort off
SD FEATURES OF
INTEREST
Create a VM
$ qemu-img create sheepdog:<vmname> 256G
Enable Local node based I/O caching
$ qemu-system-x86_64 -drive
file=sheepdog:<vmname>,cache=writeback
Boot a VM
qemu-system-x86_64 sheepdog:<vmname>
Create a Snapshot or linked clone
$ qemu-img snapshot -c <snap-name> sheepdog:<vmname>
How to manage sheepdog
Bash scripting
Write your own code using libvirt
Use Openstack (bindings in Alpha Stage)
WHERE TO GET
SHEEPDOG
Google Sheepdog KVM (github.com/collie/sheepdog/wiki)
Requirements
2 or more Linux Machines (or VMs)
Linux 2.6.27 or better
Corosync and Corosync lib
Qemu 0.13 or later
Recommend using Ubuntu or Debian Distributions since the packages
already exist

Caution
Might have to recompile Corosync to make it work
If you have a problem check the corosync conf file and make sure IPs
are correct
SD BASED VDI BOM
Node Configuration
Dual Socket Quad Core 2Ghz or better
64 GB DDR3 RAM
4 X 10K SAS Drives R0 = 1.2 TB / 600 IOPS
4 X 1Gbe - 3 X 1Gbe LACP links for Storage

Node Density is 32 VMs

Node IOPS is 600 IOPS = 18 IOPS RAW
with local I/O caching = 24 IOPS (striking distance of 30 IOPS)

500 VMs need 15 Nodes + 2 Nodes for Standby (HA)

Cost of each node = $5500 = Cost Per VM $200
HYPERVISOR
Choice of Hypervisors is limited by Sheepdog Storage
Today Sheepdog only supports Linux KVM
Future support promised for Xen and Vmware

But thats OK KVM is a fantastic technology

Super scalable
Fast 95% efficiency virtual is faster on some workloads
per Redhat
KVM HIGHLIGHTS
KSM kernel Same Page merging (memory dedupe)
KSM is a memory-saving feature that merges anonymous (private) pages.
CPU/PCI hotplug Add (p) CPUs & PCI devices to a running
KVM server
Virtio high performance low latency networking
SCSI disk high performance storage driver
Balloon Memory driver that doesnt kill CPU
Boot from SAN pxe boot
VM Migration

Note you need a processor that supports VT extensions

Run egrep -c '(vmx|svm)' /proc/cpuinfo
If it returns 1 you have support for VT extensions
INSTALLATION &
USAGE
Relatively simple for someone versed in linux, xen.org,
ESX 2.0 (cant say the same for Virtual Center jockeys)
Most distributions ship with KVM support
I recommend debian/ubuntu or centos for ease of use
Easy installation
Takes about 30 mins to deploy a node manually
Can be fully scripted to install in 3-5 mins using Proxmox

Yak shaving Hint - Install KVM and then install sheepdog

WINDOWS
DEPLOYMENT
Use Unattended (unattended.sourceforge.net)
Provides automated unattended installs of Windows
Servers and Desktops
What you will need
A Windows CIFS share (use a windows server easier)
Or use linux & samba a bit more involved
Download Unattended and install (3 commands to install)
Configure DNS
Prepare the Unattended server (windows example)
c:\> net share install=c:\path\to\unattended\install
c:\> cacls c:\path\to\unattended\install /T /E /G guest:R
c:\> net user guest /ACTIVE:YES /DOMAIN
WINDOWS
DEPLOYMENT ..
Prepare the Operating System Image
Configure the target VM to boot from floppy (djgpp utility)
Place unattended components on the CIFS share
Provide a copy of the Windows ISO
Provide any service packs
Unattended will do all of the following for you
Boot the VM from floppy image
Mount Z: on your CIFS share
Partition/Format your c:
Start installation of OS
Insert Product Key
Set local admin password
Customize computer name and join domain
Optionally even install applications for you i.e MS Office etc
APP VIRTUALIZATION
Very rudimentary support for App Virtualization
The best approach is to build portable Apps (portable
apps)
Includes a utility called Portable apps launcher which
sort of prepares an MSI of an application
Minimal coding
Will install and run the application automatically on every
launch
Will clean up on exit
This is NOT App Virtualization just a way to share a
common app payload between Windows Instances
Your mileage will vary on this one significantly
USER PROFILES
My Recommendation stick to using Windows infrastructure
for all of this.

For the hardcore

Setup a Linux Samba server for CIFS serving
Designate the Linux server as a domain controller
Download and install Script Start Community
Point SSC to the linux domain controller
Use SSC console to map drives, printers, setup MAPI etc.
See http://www.wikihow.com/Create-and-Manage-Microsoft-
Windows-User-Profiles-With-a-Linux-Server
THIN CLIENT
PLATFORM
Linux has a very mature thin client platform - ltsp
Allows a variety of hardware to be repurposed as a thin
client
Support boot from network so the thin client is diskless
Supports ICA/RDP protocols
Use the edubuntu distribution for this build a dedicated
VM with edubuntu to serve out thinclient images
You could even use the live ltsp simply boot each device
from USB (like iGel)
LTSP ON LOCAL
QUICK HOWTO
Download and burn an LTSP live (edubuntu live) iso onto
USB (lots of free utils)

Boot from USB load edubuntu

Launch LTSP installer
LTSP ON LOCAL
QUICK HOWTO
Select a network interface (defaults to eth0)
And watch ltsp install to your local drive

Quick & Simple

BROKER
No open source broker exists currently*
Ulteo claims to have released one but I have not investigated

Best option is to use cross roads load balancer and have

it load balance RDP ports
http://geekcubo.com/2011/09/connection-broker-cross-roads-
load-balancer
/
Just works use the sample xml file in the page

Simple to setup and configure 10 minute install

Works great for non persistent desktop

PUTTING IT ALL TOGETHER

Repurposed PCs and

notebooks as thin clients

Infrastructure Services
Windows Cross Roads LTSP Network Crossroads broker
AD Broker Server LTSP diskless thin client
Windows for AD

Linux KVM
SheepDog for storage
VM cloning

Unattended for Windows

Installation
Portable Apps for app virt
BRIFORUM 2013
How to make a Low earth orbit satellites and your
own secure mobile telecom infrastructure using
Party Balloons
Chewing Gum
Beagle Board computers
String
Tin Cans
And of course lots of open source projects

Stay tuned!!
 Balloons
Open
Source Of
course

Beagle
Board
Satellite

String

Super
Uber
Magnificient
cloud
You are very welcome