Complete guide to setting up an OpenStack

development environment: devstack in kvm,

with kvm-in-kvm for guests

Devstack is a really nice tool made by members of the OpenStack community that sets up a single-node OpenStack
deployment. I've used it in the past, but I really didn't want to install it directly into my Ubuntu desktop OS running on
my new machine, because it tends to stomp all over a lot of things. I want to be able to create an isolated test
environment so I can blow it away if I ever need to start from scratch.
Generally I love LXC for this kind of stuff, but unfortunately Devstack doesn't play well with it, because it needs
access to a number of physical-ish interfaces on the host. I haven't tried too hard here, but usually there are issues
with networking, block storage, and permissions. So I just went with KVM, since I knew my new machine should
support nested virtualization for fast OpenStack guests.
I'd like to thank Sam Corbett, who wrote a blog post titled Getting Started with Heat, DevStack & Vagrant. I learned
a lot about setting up a dev environment from that, but I wanted the speed of nested KVMs instead of
Vagrant/VirtualBox.
As an aside, there seems to be a lot of confusion in the community about whether it's possible to have OpenStack
spawn KVM guests when it's running in a KVM. Many say it's impossible, and that you must use software-based
QEMU virtualization, but I'd like to be clear that it is possible to do KVM in KVM. If you have nested virtualization
enabled as I describe below, devstack will Just Work and spawn your instances with KVM. I recommend checking
with "ps ax |grep kvm" after you spin up a guest to see if it's using kvm or not.

Nested virtualization with KVM in KVM
The first thing I needed to do was enable nested virtualization -- I'm not sure why it wasn't already, but oh well.
host$ cat /sys/module/kvm_intel/parameters/nested
N
(Actually, the first way I discovered I didn't have nested virtualization available was that I created a kvm instance
and ran "kvm-ok" inside of it, which reported that kvm wouldn't work).
So I'm still a little bit confused about this part, since most things I read said I'd need to reconfigure grub or change
stuff in /etc/modprobe.d, but all I had to do was this:
host$ sudo modprobe -r kvm-intel
host$ sudo modprobe kvm-intel nested=1
Of course this will only work if your kvm-intel module isn't currently in use, which basically means you've shut down
all your guests and turned off libvirtd. I didn't expect this to survive a reboot, but it did. YMMV!
By the way, if you have an AMD machine instead of Intel, I think everything's the same except that the module is
called "kvm-amd".

Create the OpenStack host
Next step: create an Ubuntu 12.04 KVM guest on my physical host. Devstack only officially supports 12.04 right now
(not, for example, 13.04), and I didn't feel like trying to figure out what would break, so I just went straight for the
standard.
I used virt-manager, which provides a really nice GUI for managing your virtual machines with libvirtd. I did a pretty
standard install, with 30G of RAM and 20G of disk space. I used
http://archive.ubuntu.com/ubuntu/dists/precise/main/installer-amd64/ as the source for a network-based install. I
didn't customize anything about anything else (not even the network!).
Word to the wise: if you're doing this headless without virt-manager, I know from past experience that it's a bit of a
pain to get access to the console on an Ubuntu installer; I think I had to use the "mini" installer last time I tried that,
and pass some extra arguments to virt-install to get it to use the console instead of a GUI.
So, that's pretty much it. You should ssh into the KVM guest that you just created.

Enable NAT for the OpenStack guests
Run This Command:
guest$ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
(ideally, set it up so that it's run on every boot of your OpenStack host, but I haven't gotten around to that yet).

Set up Devstack
Devstack is almost as easy to use as they try to make you believe on the front page of devstack.org. In my
experience, though, networking will not be configured correctly unless you edit your localrc ahead of time and
specify some stuff. Unfortunately, I am super ignorant about all things networking, so it took me a while to figure out
exactly what to put in.
KVM happened to allocate the IP 192.168.122.14 for my new guest, bound to its virtual eth0 interface (this is
hooked up to a new virbr0 interface on my host, with IP 192.168.122.1).
Here's the localrc I came up with, given that information:
HOST_IP=192.168.122.214 # /24
HOST_IP_IFACE=eth0
FLAT_INTERFACE=br100
PUBLIC_INTERFACE=eth0
FLOATING_RANGE=192.168.122.224/27
There are a couple of interesting things here. The easy things are HOST_IP, HOST_IP_IFACE, and
PUBLIC_INTERFACE (well, actually, I wasn't sure if PUBLIC_INTERFACE should be the same as
HOST_IP_IFACE at first, but it turns out to work fine). FLAT_INTERFACE I pretty much just copied out of someone
else's example.
FLOATING_RANGE gave me the most pause; this is where I had to relearn about netmasks. My main goal was to
allow floating IPs to be allocated on the same network that the OpenStack host was on, to allow my physical host
machine to be able to access them (so I could easily use my web browser, or whatever, to connect to web apps in
the OpenStack guests). Now, I don't really know how libvirt or virt-manager allocates guest IPs in my virbr0 network,
but I wanted to make sure that I at least chose a floating IP range that wouldn't stomp on my existing IPs
(192.168.122.1 and 192.168.122.214). After tinkering with ****an online IP address calculator for a while, I
eventually came to the decision of using 192.168.122.224/27. This is the area I'm least comfortable with, so I'm not
sure if it's a good way to set it up in the long run; it works for me for now!
At this point you should be able to just run ./stack.sh and everything should work.

Do the crap that you're going to have to do after every run of ./stack.sh
Add your SSH key to nova (you should know how to generate or copy your SSH key into place already)
guest$ nova keypair-add --pub-key ~/.ssh/id_rsa.pub mykey
Allow all traffic into the OpenStack guests:
guest$ nova secgroup-add-rule default tcp 1 65535 0.0.0.0/0

Boot an instance!
guest$ nova boot --flavor m1.small --image cirros-0.3.1-x86_64-uec --key-name mykey testinstance
Now you should be able to ssh to your instance and make sure you have Internet access:
guest$ ssh cirros@10.0.0.2 # this is *probably* the IP that was assigned
# ping google.com
Now let's make sure our floating IPs work. Exit back out to your OpenStack host and run:
guest$ nova floating-ip-create
+-----------------+-------------+----------+--------+
| Ip | Instance Id | Fixed Ip | Pool |
+-----------------+-------------+----------+--------+
| 192.168.122.225 | None | None | public |
+-----------------+-------------+----------+--------+
guest$ nova add-floating-ip teststack-WikiDatabase 192.168.122.225
guest$ ssh 192.168.122.225
If you're in, then everything works! You should also be able to connect to that floating IP address from your host-
host, if everything's right.

Extra Credit: setting up Heat
This was super easy, with a bit of copy-n-paste. You'll need to add the heat services to your localrc file:
ENABLED_SERVICES+=,heat,h-api,h-api-cfn,h-api-cw,h-eng
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F17-i386-cfntools.qcow2,htt
p://fedorapeople.org/groups/heat/prebuilt-jeos-images/F17-x86_64-cfntools.qcow2"
And then run ./stack.sh again. Enabling Heat on your devstack will cause it to download some images from the
specified URL and configure a Heat instance to talk to the rest of Devstack (don't worry, the URLs will only be
downloaded once and cached in your files/ directory. It will disable the cirros instances so they won't be available
after your ./stack.sh call.
Then create a stack:
guest$ wget https://raw.github.com/openstack/heat-templates/master/cfn/WordPress_Single_Instance.t
emplate
guest$ heat stack-create teststack -f WordPress_Single_Instance.template -P "InstanceType=m1.large
;DBUsername=wp;DBPassword=verybaddpassword;KeyName=mykey;LinuxDistribution=F17"
Use "heat stack-list" and "nova list" to figure out when your instance and stack is ready. I then sshed to
ec2user@10.0.0.2 and watched the /var/log/cloud-init-output.log logfile to see how the provisioning was going.
When it was all done, I associated a floating IP with the instance (as described above) and was able to hit it with the
web browser on my desktop. Yay!

Extra Credit: sharing directories with NFS
Since I wanted to do Heat hacking, I wanted an easy way to share source code between my physical machine and
the OpenStack host. At first I tried sshfs, but that's incredibly slow, so I went with NFS. I was scared at first, but it
was pretty trivial to set up.
All I had to do was install the nfs-kernel-server package and then edit /etc/exports to add the following line:
/home/radix/Projects *(rw,async,root_squash,no_subtree_check)
Then, in my OpenStack guest, I installed nfs-common and added the following line to /etc/fstab:
host:/home/radix/Projects /home/radix/Projects nfs
Don't forget to mkdir /home/radix/Projects in the guest. You can then reboot the OpenStack host or just mount
/home/radix/Projects.