HA in OpenStack

September 24th 2013

1
Agenda
• What is HA

• Compute HA

• Controller HA
• Corosync, Pacemaker and DRBD
• Galera
• HAProxy, keepalived, VRRP

• Resources and Summary

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

2
What is HA?

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

3
High Availability

Minimize data loss

Minimize system downtime

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

4
High Availability Concepts
• Stateless services
– There is no dependency between requests
– For example: Nova API, Nova Scheduler, etc.

• Stateful services
– An action typically comprises multiple requests
– For example: MySQL, RabbitMQ, etc.

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

5
High Availability Concepts
• Active/Passive
– Redundant instances of stateless services are load balanced
– For Stateful services a replacement resource can be brought online.

• Active/Active
– Redundant instances of stateless services are load balanced
– Stateful services are managed in such a way that services are redundant, and that all
instances have an identical state.
– Updates to one instance of a database would also update all other instances.

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

6
Compute HA

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

7
No vSphere Style HA with KVM

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

8
Server Evacuation

• Without Shared Storage

– The instance will be booted from a
new disk, but will preserve the
configuration, e.g. id, name, uid,
ip...etc.

• With Shared Storage

– The instance will be booted from
same disk and data will be preserved

Evacuation

Evacuation

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

9
Instance Migration

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

10
 11

Virtualization vs. Cloud

• Virtualization needs care and feeding
– Name the VM
– Tune and groom regularly
– Feed it with good food and supplements
– Take to the vet when sick

• Cloud servers are disposable

– VMs are not unique
– Tune and groom apps not the cows
– Keep the cow upright
– Shoot the cow when it is sick

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Scale Up vs. Scale Out

Traditional Cloud

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

12
Application HA

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

13
Controller HA Options

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

14
Pacemaker, Corosync and DRBD
• Pacemaker
– high availability and load balancing stack for
the Linux platform.
– Interacts with applications through Resource
Agents (RA)
• Corosync
– Totem single-ring ordering and membership
protocol
– UDP and InfiniBand based messaging,
quorum, and cluster membership to
Pacemaker.
• DRBD (Distributed Replication Block
Device)
– Synchronizes Data at the block device
– Uses a journaling system (such as ext3 or
ext4)

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

15
Galera
• Synchronous multi-master cluster
technology for MySQL/InnoDB
– MySQL patched for wsrep (Write Set
REPlication)
– Active/active multi-master topology
– Read and write to any cluster node
– True parallel replication, in row level
– No slave lag or integrity issues

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

16
HA with Keepalived and HAProxy

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

17
Keepalived, HAProxy and VRRP

• HAProxy
– Load Balancing and Proxying for HTTP and TCP Applications
– Works over multiple connections
– Used to load balance API services

• VRRP (Virtual Router Redundancy Protocol)

– Eliminates SPOF in a static default routed environment

• Keepalived
– Based on Linux Virtual Server (IPVS) kernel module to provide layer 4 Load Balancing
– Implements a set of checkers to check service status and to maintain health
– Leverage the VRRP Protocol to remap VIPS in event of failure

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

18
Sample OpenStack HA Architecture
Fixed Network (private)
Inside LB VLAN
Dedicated Firewalls Load Balancers Storage Network (private)

Redundant Network Switches

BOND

BOND

BOND

BOND

BOND

BOND

BOND

BOND

BOND

BOND

BOND

BOND
0

1
Controller Controller
Compute 1 Compute N Compute 1 Compute N
API Services API Services
G1 G11
API & Horizon API & Horizon G5 G15
G2 G12 G1
Cinder API Cinder API G6
6
Nova Scheduler Nova Scheduler G3 G7 G13 G17
Keystone Keystone G4 G14
Glance Glance
RabbitMQ RabbitMQ
KVM KVM KVM KVM
MYSQL MYSQL .… .…
Chef Chef Availability Availability
Server Server Zone 1 Zone 2
Recipes Recipes

BOND
BOND

BOND

BOND
BOND

BOND

2
2

2
2

Redundant Network Switches

Storage
EMC, NetApp, or
Vols
Solidfire

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

19
HA on OpenStack

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

20
HA on OpenStack

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

21
Comparison

Database Replication Strengths Weakness/Limita

method tions
Keepalived/HAPro Works on MySQL Simple to Master-master
xy/VRRP master-master implement and replication does
replication understand. not work beyond 2
Works for any nodes.
storage system.
Pacemaker/Coros Mirroring on Block Well tested More complex to
ync/DRBD Devices setup. Split Brain
possibility
Galera Based on write- No Slave lag Needs at least 3
set Replication nodes. Relatively
(wsrep) new.
Others MySQL Cluster, Well tested More complex
RHCS with setup.
DAS/SAN storage RACKSPACE® HOSTING | WWW.RACKSPACE.COM

22
Resources and Contact

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

23
Resources
• OpenStack
– openstack.org
– launchpad.net/openstack
– #openstack
– #openstack on webchat.freenode.net
• OpenStack HA
– http://docs.openstack.org/trunk/openstack-ha/openstack-ha-guide-trunk.pdf
– https://github.com/rcbops-cookbooks/
• MySQL HA
– http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration-
deployment-guide/
– http://dev.mysql.com/doc/refman/5.7/en/ha-overview.html
– https://www.hastexo.com/
– http://www.drbd.org/

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

For More Information

You can reach me at:

Kenneth Hui
Open Cloud Architect
Rackspace

E-mail: ken.hui@rackspace.com
Twitter: @hui_kenneth
Blog: http://cloudarchitectmusings.com

RACKSPACE® HOSTING | | WWW.RACKSPACE.COM

WWW.RACKSPACE.COM
2
RACKSPACE® HOSTING
5
 RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM
26