! ! "#$%&! '(!&)*%+*$#,(!,-!./&(0$*12!-,3!$4&!5"6! '+$4,3708! 93!:$&)&!"4,3(! 9*$&! ;<=<=;>??! ! ! ! ! ! ! ! ! ! ! ! ! ! ! "4#0!3&/,3$!@*0!-+(A&A!$43,+B4!$4&!0+//,3$!,-!$4&!5C:DE!+(A&3!B3*($! 5C=6>FGFH<=?I!$4&!JK!&L:1#&(1&!5(B#(&&3#(B!"*02!6,31&!M&$@,32N!

An Evaluation of OpenStack for the Engineering Task Force

Steve Thorn, University of Edinburgh September 2011

Abstract
OpenStack is a set of components for implementing an Infrastructure as a Service Cloud. It was formed from Rackspace Hosting and NASA in July 2010 and is supported by many other companies. It allows a user to manage a large number of virtual machines including associated storage and networking. In addition, an object store is provided for long-term storage of static or rarely changing data. It is at least partly compatible with Amazon's Elastic Compute Cloud (EC2) and Simple Storage Service (S3). A modest deployment of the major components was done for this evaluation. OpenStack provides a rich set of features for implementing an Infrastructure as a Service cloud. Not all documented features work as advertised and it would be challenging to deploy and maintain a production quality system at present. However, this is likely to change in the near future given that the development of OpenStack is progressing at a rapid pace.

Introduction
The UK's Engineering Task Force (ETF) is evaluating several Grid middleware solutions in order to better understand their deployability on the resources within the National Grid Service (NGS) and those of the wider UK e-Science community. This report presents the results of the evaluation of the OpenStack system. The main purpose of this evaluation is to examine the strengths and weaknesses of the OpenStack system and identify the issues that would need to be considered before deployment in a production environment.

General Information
Provider
OpenStack was formed in July 2010 from Rackspace Hosting and NASA. Over 25 companies, including Citrix Systems, Dell, AMD, Intel, Canonical, HP, and Cisco, support OpenStack by contributing code and providing design input. It seems likely given the level of industrial support that OpenStack will continue to exist for the next few years. The install base has been difficult to determine.

Components
OpenStack is made up of a number of components: Compute (code-named Nova) allows a user to manage a large number of virtual machines including associated storage and networking. It supports seven major hypervisors including Xen, KVM, Hyper-V and VMWare ESXi. It is equivalent to Amazon's Elastic Compute Cloud (EC2) and implements a subset of that API. Object Storage (code-named Swift) creates a redundant, scalable object storage using clusters of standard servers intended to be used for long-term storage of static or rarely changing data. It is equivalent to Amazon's Simple Storage Service (S3) and implements a subset of that API. Image Service (code-named Glance) provides a repository for storing virtual disk images. It can use a number of different back-end stores including raw disk and the Object Storage. Identity Service (code-named Keystone) provides unified authentication across all OpenStack components and integrates with existing authentication systems. Dashboard enables administrators and users to control resources via a web portal.

The Identity Service and Dashboard components are described as "incubated" and will be incorporated into the core release in the next version. For this reason they were not considered in the evaluation.

Licensing
Apache 2.0

Supported platforms
Ubuntu is the reference platform for OpenStack. Components are available for Ubuntu, RHEL, SUSE, Debian, and Fedora. Not all components are available on all platforms. Being open source it is quite likely that the OpenStack components can be built on most Linux flavours, but this has not been attempted here. A number of repositories maintained by community members provide packages for OpenStack. The following table shows the availability for each OpenStack component: Component Compute Object Store Image Service Identity Service Dashboard Ubuntu RHEL SUSE Debian Fedora yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes

Note: this information is not likely to be complete as only repositories referred to in the OpenStack documentation and website are given here. This report is based on an evaluation using Ubuntu 11.04 installing the latest (as of October 2011) 2011.3 "Diabolo" release of OpenStack from packages. A modest testbed of four servers running 3GHz Intel Core2 Duo CPUs with 2 GB RAM was used.

Support
Commercial support is not available from OpenStack. Community support is available through a wiki, a mailing list, an IRC channel and a bug tracker based on Launchpad.

Systems Management
Documentation for System Managers
A large amount of documentation is available on the OpenStack website, some of which is available as PDF. Unfortunately the online documentation is in two places. Administration guides are available at http://docs.openstack.com, but further documentation is available for each component, for example, at http://nova.openstack.com, http://swift.openstack.com, http://glance.openstack.com and http://keystone.openstack.com. Neither source is comprehensive and the reader is forced to consult both. Overall the documentation seems reasonably comprehensive if somewhat disjointed and inconsistent, perhaps due to its partly community contributed philosophy. Only limited quickstart and troubleshooting sections are provided. The online documentation is searchable. However, the search on the component pages is not sophisticated and, for example, failed to return any content for "novamanage" on http://nova.openstack.com, whereas Google returns multiple pages. Searching http://docs.openstack.com seems better, but its navigation pane suffers from a bug where it jumps to a different section from that viewed. The installation is fairly well documented and straightforward to achieve. Configuration is less well described. For example, configuration options often have less than a sentence description and examples are rare. Please see the "Server Deployment" section below for more observations on configuration.

Server Deployment
Installation of packages from the Ubuntu maintained OpenStack repository was straightforward. Although the installation was done with Ubuntu 11.04 it is not required to use the latest Ubuntu version. Java is not required. OpenStack is a very complex software with many intercommunicating components distributed across the network. System requirements are not clearly documented and were found to be different in at least two places. However, this could be because OpenStack components are designed to be run on commodity hardware and stringent system requirements are not necessary. The number of software dependencies and required libraries is large. As an illustration of this the following is a list of additional packages installed on a Compute "head" node:
ajaxterm, apache2, apache2.2-bin, apache2.2-common, apache2-mpm-worker, apache2-utils, bridge-utils, cloud-utils, cpu-checker, curl, dnsmasq-base, dnsmasq-utils, ebtables, erlang-base, erlang-crypto, erlang-inets, erlang-mnesia, erlang-os-mon, erlang-publickey, erlang-runtime-tools, erlang-snmp, erlang-ssl, erlang-syntax-tools, euca2ools, expect, gawk, iputils-arping, iscsitarget, kpartx, libaio1, libapr1, libaprutil1, libaprutil1-dbd-sqlite3, libaprutil1-ldap, libasound2, libavahi-client3, libavahicommon3, libavahi-common-data, libcurl3, libdbd-mysql-perl, libdbi-perl, libdevmapper-

event1.02.1, libflac8, libhtml-template-perl, libice6, libmysqlclient16, libnet-daemonperl, libnetfilter-conntrack3, libogg0, libopts25, libpciaccess0, libplrpc-perl, libpulse0, libpython2.7, libreadline5, libsctp1, libsdl1.2debian, libsdl1.2debian-alsa, libsm6, libsndfile1, libsysfs2, libvirt0, libvirt-bin, libvorbis0a, libvorbisenc2, libx11-xcb1, libxcb-atom1, libxen3, libxml2-utils, libxslt1.1, libxtst6, libyaml-0-2, lksctp-tools, lvm2, msr-tools, mysql-client-5.1, mysql-client-core-5.1, mysql-common, mysql-server, mysql-server-5.1, mysql-server-core-5.1, ntp, open-iscsi, open-iscsi-utils, python-amqplib, python-anyjson, python-argparse, python-boto, python-carrot, pythoncheetah, python-daemon, python-decorator, python-dingus, python-eventlet, pythonformencode, python-gflags, python-greenlet, python-kombu, python-ldap, python-libvirt, python-libxml2, python-lockfile, python-lxml, python-m2crypto, python-migrate, pythonmysqldb, python-netaddr, python-nose, python-openid, python-paramiko, python-paste, python-pastedeploy, python-pastescript, python-prettytable, python-routes, python-scgi, python-setuptools, python-software-properties, python-sqlalchemy, python-sqlalchemy-ext, python-stompy, python-tempita, python-webob, python-xattr, python-yaml, qemu-common, qemu-kvm, rabbitmq-server, seabios, socat, ssl-cert, tcl8.5, unattended-upgrades, unzip, vgabios, vlan, watershed, x11-common

On a Linux system with an advanced package manager such as yum or apt this is not much of a problem assuming the packages are present in the repositories. Configuring OpenStack is a challenging task. Unfortunately, this is where the documentation is most lacking. A system administrator typically wants to know: Where the configuration data are stored? For example, text files or a database Are there any tools to manipulate the configuration data? If there are multiple sources of configuration data, which software components use which source? When software components read configuration data? How to trigger a re-loading of configuration data in a consistent way? Is it even sensible to re-load configuration data during operation? How to return configuration data to a previous or just-installed state?

The location of configuration data is well documented and exists in two places: text configuration files under /etc/nova, /etc/swift and /etc/glance for the Image Service and Compute components, a relational database (MySQL, PostgreSQL or SQLite)

A database manipulation tool exists for the Compute component (nova-manage). Direct manipulation of the databases is also required for some configuration steps. Answers to the remaining questions are either badly documented or missing. Some configuration data end up being duplicated between text file and database, but it is not at all clear where each component gets its configuration data. This makes changing a misconfigured system most frustrating. Many of the OpenStack components communicate over the network. The ports that they listen on are not particularly well documented. The best source of information is the documentation of the text configuration file format. Changing the listening ports is possible via these files and the default ports are given. Which components communicate with which is not explicitly documented making the creation of firewall

rules a challenge. An additional complexity arises because OpenStack Compute uses extensive iptables manipulation to route traffic to and from the virtual machines. A scripted installation and configuration is provided but this was not tested here as it was felt it would not reveal enough of the installation process.

Client Deployment
There are a number of client tools for communicating with the OpenStack components. Some are provided by OpenStack and some are third-party, which interact with the Amazon Web Service compatible APIs (EC2 and S3). Other Amazon compatible clients not tested here are likely to work to some extent. All the clients used here communicate with OpenStack on one server port and there is no incoming traffic to the clients apart from related and/or established packets. Firewall rules would be simple and only necessary if outbound traffic is filtered from the client machine.

nova client
The nova client can be used to interact with OpenStack Compute over the native OpenStack API on tcp port 8774 by default. Installation under Ubuntu is straightforward and lightweight. Other Linux distributions were not tested. The configuration of this and euca2ools (below) is done with environment variables and is simplified using a file, obtained from the OpenStack installation, which sets the variables directly.

euca2ools
The euca2ools client from Eucalyptus is relatively easy to install. Packages are available for Ubuntu, RHEL, SUSE, Debian. It communicates with the OpenStack's implementation of Amazon's EC2 API. Outgoing network requirements are limited to one tcp port (8773 by default).

curl
The curl command is used to transfer data to or from a server, using one of a number of protocols. It is used to communicate with the OpenStack Object Storage (Swift) API, which is a RESTful web service, on tcp port 8080. In this case curl uses the http and https protocols. It is easy to install and is often already present in Linux distributions. It is also possible to use curl to interact with the OpenStack's implementation of Amazon's S3 API, but this was not tried.

swift
The swift client is easy to install under Ubuntu although not particularly lightweight. Other Linux distributions were not tested. It communicates with the native Object Storage API on tcp port 8080 by default.

Account Management
It is simple to add new Compute users with the nova-manage command. It is also possible to create projects to allow collaboration between users. Managing Object Storage accounts is possible with the swauth-add-user command, but is poorly documented. It is suspected that this is because the Identity Service will replace authentication and authorisation in the next release. There is no web interface for a user to apply for access unlike with Eucalyptus. Compatibility with Grid authentication and authorisation mechanisms such as VOMS is not supported.

Reliability
Only one issue has been observed with this test. When requesting a large number of instances such as 10 on a system that only has two physical CPUs results in only a fraction of the 10 running successfully. The remaining instances stay pending indefinitely. Curiously, it is possible to achieve a large number of running instances by asking for a smaller number, such as two, repeatedly. A number of high availability features exist especially in the Object Storage, which is designed to be massively scalable. Attempts have been made to alleviate the problem that external traffic to the virtual machines is routed through the node running the nova-network service. This was not attempted here as the code is not yet in the release version. A number of strategies for improving availability are described in the documentation.

Distributed management
For the Compute component, the configuration files are designed to be identical across all nodes and only one of the services (nova-compute) needs to run on each compute node. The nodes also query the central database for configuration data. The Object Storage is straightforward to scale across nodes.

Scalability
See above section "Reliability".

User Experience
Documentation for Users
Very little documentation exists that is aimed directly at users. Documentation of the OpenStack provided command-line tools is limited to a few examples and brief man pages. A user can interact with OpenStack via its implementation of the Amazon Web Services API using a compatible tool such as euca2ools or Hybridfox. Documentation for euca2ools is known to be quite good.

Joining the Infrastructure

At present the only way for a user to request access to an OpenStack installation is to contact the administration staff directly. The administrators then have to create accounts manually. The user will then have to install a suitable client on their local machine or perhaps login to a "User Interface" node, which already has the relevant command-line tools, provided by the administrators of an OpenStack deployment. See "Client Deployment" for further information.

Legacy Application Integration

This has not been considered in detail apart from stating that IaaS clouds provide a natural location for running legacy codes.

Migration between platforms

This has not been considered apart from stating that it should be straightforward to run virtual machine images developed for OpenStack on Eucalyptus or Amazon clouds.

Usability
Users will find interacting with OpenStack no more difficult than interacting with Eucalyptus. However, experience has shown that many users find it difficult to adapt their requirements to IaaS clouds.

Security from the User's Perspective

The user is given an "access key" and "secret key", which are alphanumeric strings, that act in a similar way to a username and password. An X.509 certificate is also issued to the user. It is not transparent which credentials are used where. This is similar to the mechanism to the one Eucalyptus uses.

The authentication and authorisation mechanisms are in a state of flux. Existing mechanisms will be removed and replaced by Keystone, OpenStack's centralised identity service.

Verification
Users have access to some diagnostic information including viewing the virtual machine console logs. However, it can still be difficult to determine why a virtual image has failed to run.

Technical
Architecture
All the OpenStack components except Dashboard are implemented as RESTful web services.

Standards & Specifications

All the OpenStack components except Dashboard are implemented as RESTful web services.

Security
Security in OpenStack is in a state of flux. The Identity Service, which provides centralised authentication, authorization, and a service catalogue is being developed. This is planned to replace security in individual components. A number of issues with the current authorisation has been observed. Users have been able to do things outside what their roles should allow. For example, a user has been able to change the public IP address of another user's virtual machine. Further investigation into this was not deemed fruitful as it is likely to change with the introduction of the Identity Service.

Capability & Functionality

Purely in terms of implementing an Infrastructure as a Service cloud, OpenStack goes a long way to provide the services needed to implement a production system. The usefulness of IaaS clouds for UK e-Science applications has increasingly been demonstrated. See, for example, the All Hands Meeting 2011.

Conclusions
This evaluation has found that OpenStack is a mature, well-backed software for implementing an Infrastructure as a Service Cloud. The set of features and multicomponent architecture allows many different deployment scenarios to be developed addressing differing needs for scale, availability and reliability. Development of OpenStack is clearly progressing at a fast pace. Documentation is reasonably good, but perhaps has to catch up with the rapid development. The weakest area of the documentation is on configuration and this remains the biggest issue. It is compounded by the storage of configuration data in more than one place and multiple ways to modify it. Not everything works as expected or as described, and combined with the rapid development pace, this does not make a stable base for deploying and maintaining a production infrastructure at present.