Documente Academic
Documente Profesional
Documente Cultură
Setup a private cloud using open source eucalyptus on Ubuntu 9.04 jaunty OS. I used 'xen 3.0' instead of
default package 'kvm' for Ubuntu 9.04.
This Private cloud setup uses 'STATIC' mode and list of IPs and machine address are provided to the
Eucalyptus installation to be used, because I was using the existing subnet and pick the available IPs from
the subnet.
Contents
more
Link
CitationEmailPrint FavoriteCollect this page
My interest in cloud computing started when I worked on a project, to deploy application on Amazon cloud and
setup the clusters for MySql, Apache and other technologies to load balance. While working on Amazon cloud I
got some insights of the cloud computing and various underlying technologies.
Here I am going to illustrate all the steps that I have followed to setup cloud. I will be describing the cloud
packages, how to setup front end and back end setup, cloud implementation, ec2 or euca2ools setup, creating
xen-supported guest machine, creating EMIs for eucalyptus, run/stop instances and some xen/eucalyptus/ec2
useful command.
Technical background
When I start on cloud setup, I was sure I will go for XEN rather than KVM. Then came the Ubuntu jaunty as
this version was built for the virtualization and designed with built-in support. I used eucalyptus, an opensource
tool that provide Amazon like features and is EC2 tools compliance. So I starts researching the contents on all
these technologies. The more I study, the more I got confident to use these technologies for cloud setup.
Though, Ubuntu prefers KVM for the visualization support and has added to the default packages. But
limitation of KVM to run only virtualization extensions (Intel VT or AMD-V) processors
only(see http://www.linux-kvm.org/page/Main_Page), makes my decision to use XEN for virtualization.
Xen, is the underlaying technology used by eucalyptus. Xen hypervisor allows several guest operating systems
to be executed on the same computer hardware concurrently. Xen partitions a single physical machine into
multiple virtual machines, to provide server consolidation and utility computing. Existing applications and
binaries run unmodified. The hypervisor controls the MMU, CPU scheduling, and interrupt controller,
presenting a virtual machine to guests.
The first guest operating system, called in Xen terminology "domain 0" (dom0), is booted automatically when
the hypervisor boots and given special management privileges and direct access to all physical hardware by
default. The system administrator can log into dom0 in order to manage any further guest operating systems,
called "domain U" (domU) in Xen terminology.
The basic architecture how client, cloud controller, cluster controller and node controllers communicate with
each other in a eucalyptus setup.
----> eucalyptus-nc
|
|
Client ---> eucalyptus-cloud ----> eucalyptus-cc ------> eucalyptus-nc
|
|
----> eucalyptus-nc
In my private cloud setup, the front end runs both eucalyptus-cloud and eucalyptus-cc and 3 machines act as
node controller and run eucalyptus-nc.
I have setup a Dell machine as front end. Front end runs both eucalyptus-cloud and eucalyptus-cc. So I install
both packages on this machine.
Install Ubuntu 9.04 on a machine. Select ssh server to be installed on the machine by default, so We can
connect to machine remotely.
Postfix is a mail server. Eucalyptus send mails to users that are added by admin on eucalyptus web
interface. So we install postfix that eucalyptus will use as mail server.
Steps to install postfix and to setup configurations and testing successful Postfix installation
command# apt-get install postfix
Set configurations
command# sudo dpkg-reconfigure postfix
Install the packages provided by Ubuntu, all the dependencies will be installed automatically by package
manager.
Install the DHCP server which is used by eucalyptus for assigning the IPs to instances. DHCP server
should not be in running state and set the DHCP server not to run when system is booted up again.
Set the DHCP server path in the eucalyptus config file '/etc/eucalyptus/eucalyptus.conf'
VNET_DHCPDAEMON="/usr/sbin/dhcpd3"
Set the static mode for eucalyptus so that eucalyptus uses IP's from the pool of IP's provided in the config
file and use the machine addresses provided along with IPs.
Configuration file for eucalyptus is '/etc/eucalyptus/eucalyptus.conf'
open /etc/eucalyptus/eucalyptus.conf and set the following parameters
VNET_INTERFACE="peth0"
VNET_BRIDGE="eth0"
Add machine and IPs for instances to be used and change the netowork setting to current network settings
VNET_MODE="STATIC"
VNET_SUBNET="192.168.145.0"
VNET_NETMASK="255.255.255.0"
VNET_BROADCAST="192.168.145.255"
VNET_ROUTER="192.168.145.1"
VNET_DNS="192.168.150.42"
VNET_MACMAP="AA:DD:11:CE:FF:ED=192.168.145.73 AA:DD:11:CE:FF:EE=192.168.145.75
AA:DD:11:CE:FF:EF=192.168.145.77"
Edit : /usr/share/eucalyptus/gen_libvirt_xml
Locate all 'sda' to 'xvda'
Edit : /etc/xen/xend-config.sxp
(xend-http-server yes)
gateway = 192.168.145.1
netmask = 255.255.255.0
broadcast = 192.168.145.255
Once eucalyptus front end has been installed, we can use the eucalyptus web interface and setup cluster and
set other parameters.
Eucalyptus cloud web URL: https://<front end ip>:8443/
Create kernel and ramdisk images and set in eucalyptus site. (Procedure to create images will be described
later in the page)
On the systems where we want to run instances, we will install node controller and XEN 3.0. Since I used the
XEN as virtualization package, I install the node controller package eucalyptus-nc manually. I installed
required dependencies using the package manager apt-get, but the eucalyptus-nc was installed by
donwloading and manually installing bypassing dependencies list.
eucalyptus-nc requires 'libvirt-bin' package. Libvirt is a C toolkit to interact with the virtualization
capabilities of recent versions of Linux (and other OSes). More info can be found at: http://libvirt.org/
To install 'libvirt-bin' package, certain flags should be supported by CPU.
I am using the XEN virtualization package instead of Ubuntu's default KVM, I install the eucalyptus-nc
dependencies manually.
command# apt-get install iproute iptables module-init-tools python2.5 python2.6
command# apt-get install xen-utils
command# apt-get install ubuntu-xen-server
eucalyptus-nc requires 'libvirt-bin' package. Libvirt is a C toolkit to interact with the virtualization
capabilities of recent versions of Linux (and other OSes). More info can be found at: http://libvirt.org/
command# apt-get install adduser bridge-utils dnsmasq-base iptables libsasl2-2 libxen3 logrotate
command# apt-get install netcat-openbsd libavahi-client3 libavahi-common3 libc6 libdbus-1-3
command# apt-get install libgcrypt11 libgnutls26 libhal1 libpolkit-dbus2 libpolkit2 libreadline5
libselinux1
command# apt-get install libtasn1-3 libvirt0 libxml2 zlib1g policykit
Install lib-virt
command# dpkg -i libvirt-bin_0.6.1-0ubuntu5_i386.deb
command# sudo adduser $USER libvirtd
Setup grub menus and load xen images to boot into XEN kernel.
Download the modules and image from URLs:
wget http://ftp.debian.org/debian/pool/main/l/linux-2.6/linux-modules-2.6.26-2-xen-686_2.6.26-
15_i386.deb
wget http://ftp.debian.org/debian/pool/main/l/linux-2.6/linux-image-2.6.26-2-xen-686_2.6.26-15_i386.deb
Stop the eucalyptus daemon and set the configuration depending upon your installation.
Edit : /etc/sysctl.conf
(uncomment net.ipv4.ip_forward=1)
Edit : /etc/eucalyptus/eucalyptus.conf
(set)
VNET_BRIDGE="eth0"
VNET_INTERFACE="peth0"
HYPERVISOR="xen"
VNET_MODE="STATIC"
Edit : /usr/share/eucalyptus/gen_libvirt_xml
locate 'sda' s and replace to 'xvda' s
(xend-http-server yes)
(xend-unix-server yes)
Cloud integration
Now, we have front controller and node controllers ready. We have to setup now the communication
between cluster controller and node controllers. We will add the nodes list to the cloud, so cloud know about the
available node.
command# sudo euca_conf -addnode <node ip> <node ip> <node ip>
This will add nodes to cloud config file and replicates the permission and certificate files to the nodes to be
able to have password-less communication between nodes and cloud.
To check the cloud installation, and node communication, you can run 'euca-describe-availability-zones
verbose' command and check if all the nodes are listed in output.
Euca2ools setup
Install dependencies
command# apt-get install curl libopenssl-ruby ruby
Steps
Install dependencies
Install jdk1.6.0_12 java version.
Steps
Download ec2-ami-tools-1.3-26357 and ec2-api-tools-1.3-30349.
unzip these archives
Copy the 'lib' and 'bin' folders under ~/.euca folder.
Export parameters
export EC2_HOME=~/.euca
export JAVA_HOME=<directory>/jdk1.6.0_12
export PATH=$PATH:<directory>/jdk1.6.0_12/bin
export PATH=$PATH:~/.euca/bin
Unzip this archive under ~/.euca folder and export the eucarc file.
Steps
Create 'firstboot' file, which is run first time, instance boots up.
apt-get -y install openssh-server
It creates the image file that i used for creating emi. You can found the image under 'ubuntu-xen' folder.
This image can be tested, if its compatible to xen. Use xm create command.
command# xm create /usr/akash/cloud/ubuntu-xen/xen.cfg
If you see your emi ID in the output, your image can be run with XEN.
Creating EMIs
Now, cloud is setup, we should create images that can be run using eucalyptus. We will create emis for
kernel, ramdisk and instance.
kernel emi
Steps
command# mkdir kernel
command# ec2-bundle-image -i vmlinuz-2.6.28-13-server -d ./kernel --kernel true
command# ec2-upload-bundle -b eucaKernel -m ./kernel/vmlinuz-2.6.28-13-server.manifest.xml
command# EKI=`ec2-register eucaKernel/vmlinuz-2.6.28-13-server.manifest.xml | awk '{print $2}'`
command# echo $EKI
ramdisk emi
Steps
command# mkdir ramdisk
command# ec2-bundle-image -i initrd.img-2.6.28-13-server -d ./ramdisk --ramdisk true
command# ec2-upload-bundle -b eucaRamdisk -m ./ramdisk/initrd.img-2.6.28-13-server.manifest.xml
command# ERI=`ec2-register eucaRamdisk/initrd.img-2.6.28-13-server.manifest.xml | awk '{print $2}'`
command# echo $ERI
instance emi
Steps
command# mkdir image
command# ec2-bundle-image -i root.img -d ./image
command# ec2-upload-bundle -b plabImage -m ./image/root.img.manifest.xml
command# EMI=`ec2-register plabImage/root.img.manifest.xml | awk '{print $2}'`
command# echo $EMI
Run instance
Troubleshoot
Once I started installing eucalyptus, I encounter many issues and face problems because of no/less help
available. To resolve the issues I have to google or search http://open.eucalyptus.com forums or post my issues
in forums to get the solution. I will try to list some of the issues thats I face, it might be of some help if you got
similar issues.
Eucalyptus Logs
When I got any issue, my first area of interest was the logs generated by eucalyptus and its
dependencies. Eucalyptus maintain all the logs under pre-defined directory '/var/log/eucalyptus'.
euca_test_nc.log
On node controllers, if there is any issue with starting eucalytus-nc command, check euca_test_nc.log
file for any errors. euca_test_nc.log file contains all the errors eucalytus-nc encounter while starting.
nc.log
nc.log file contains all the logs generated in cluster-node communication.
While starting a instance, initially I wasnt aware of buggy log handling of eucalyptus and it takes a lot
time to understand this. While starting instance, and keep on adding the logs for non-existing network while
instance is currently in booting up. Because of this issue, I got following in the nc.log file
[EUCAINFO ] doDescribeResource() invoked
[EUCAINFO ] doDescribeResource() invoked
[EUCAERROR ] libvirt: Domain not found (code=42)
which indicates, domain can not be started, but after some time this error is resolved automatically
when network is started for the instance. So we can omit this error message in nc.log file.
Cloud resources
Some times when you install the eucalyptus and cluster, cloud and nodes are running successfully on
systems. Still you can not run instances, this may be issue with the node resource registration with the cluster.
Or all the resources has been consumed by the instances. You can check the resource status for cluster using
'ec2-describe-availability-zones verbose' command.
command# ec2-describe-availability-zones verbose
From the output, we can clearly visualize the resource management, free/used resources and
number of instance we can run. Output also lists the nodes registered/available currently to cluster.
Instance issues
Some times, I got stuck with instance related issues. When I start instance using ec2 tools, instance will
die and nc.log or any other log file doesnt provide useful info. So, best way to investigate issue and save time to
run the instance using xm command provided by xen.
Common steps are,
- copy the image file used to create emi to any of the node.
- create xen conf file, check following xen config file content
kernel = '/boot/vmlinuz-2.6.28-13-server'
ramdisk = '/boot/initrd.img-2.6.28-13-server'
memory = 128
name = 'Ubuntu'
dhcp = 'dhcp'
vif = ['']
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
- while xm create is starting instance, use console command to check the console output.
command# xm console Ubuntu
Ubuntu - is the name option provided in xen.cfg file
Useful commands
Xen commands
command# xm list
list all the available Xen instances you're allowed to control
command# xm status
current status of your Xen guest
EC2 command
Delete image
command# ec2-deregister emi-55150E20
command# euca-delete-bundle -a $EC2_ACCESS_KEY -s $EC2_SECRET_KEY --url $S3_URL -b
eucaubuntu -p image --clear
Check cloud status, returns all the available nodes and resource available
command# euca-describe-availability-zones verbose
EUCALYPTUS Commands
Delete node
command# euca_conf -delnode 192.168.145.71
References