Apache High Availability Howto, using LVS and Heartbeat

Assalam-u-alaikum, This howto is also a bit old now. But I thought of uploading it in the howtos section, as it still works. Created : Mid 2007 Last updated: Mid 2007 The following link is very nice tutorial :

http://www.howtoforge.com/high_availability_loadbalanced_apache_cluster but uses Debian!

I have used CentOS 4.3 and all heartbeat and ipvs packages from centos43 extras. Nothing needs to be downloaded from ANYWHERE. No need to go to Ultra monkey site, etc.

Make sure that example.com has proper Zone entries on your DNS server for:

www1.example.com. www2.example.com. lb1.example.com. lb2.example.com. www.example.com. IN IN

IN IN A A

A A

192.168.0.101 192.168.0.102 192.168.0.103 192.168.0.104

IN

192.168.0.105

192.168.0.103

lb1.example.com

lb1

HOSTNAME=lb1.example.com GATEWAY=192.168.0.254

IPADDR=192.168.0.103 NETMASK=255.255.225.0

DEVICE=eth0 BOOTPROTO=static HWADDR=00:40:26:5F:5C:60 ONBOOT=yes TYPE=Ethernet IPADDR=192.168.0.102

NETMASK=255.255.255.0

-------------------------------------both Webservers

mkdir /media/nfs ; mkdir /media/extras mount -t nfs 192.168.0.254:/data/cdimages/centos43nfs /media/nfs echo "mount -t nfs 192.168.0.254:/data/cdimages/centos43nfs /media/nfs" >> /etc/rc.local mount -t nfs 192.168.0.254:/data/cdimages/centos43extrasnfs /media/extras/ echo "mount -t nfs 192.168.0.254:/data/cdimages/centos43extrasnfs /media/extras/" >> /etc/rc.local -------

rpm -ivh httpd-2.0.52-22.ent.centos4.i386.rpm apr-0.9.4-24.5.c4.1.i386.rpm httpd-suexec-2.0.5222.ent.centos4.i386.rpm apr-util-0.9.4-21.i386.rpm

-----------------------------------------------Load Balancers: /etc/modules does not exist on RedHAT and clones.

On both load balancers (also placed the same in /etc/rc.local) /etc/modprobe.conf can be used but I do not know how to use it, so /etc/rc.local is the easy way out.

modprobe ip_vs_dh modprobe ip_vs_ftp modprobe ip_vs modprobe ip_vs_lblc modprobe ip_vs_lblcr modprobe ip_vs_lc modprobe ip_vs_nq modprobe ip_vs_rr modprobe ip_vs_sed modprobe ip_vs_sh modprobe ip_vs_wlc modprobe ip_vs_wrr

-------------------both LBs

rpm -ivh curl-7.12.1-8.rhel4.i386.rpm perl-Crypt-SSLeay-0.51-5.i386.rpm perl-HTML-Parser-3.35-6.i386.rpm perlLDAP-0.31-5.noarch.rpm perl-Net-DNS-0.48-1.i386.rpm perl-libwww-perl-5.79-5.noarch.rpm libidn-0.5.6-1.i386.rpm perl-Convert-ASN1-0.18-3.noarch.rpm perl-Digest-HMAC-1.01-13.noarch.rpm perl-HTML-Tagset-3.03-30.noarch.rpm perl-URI-1.30-4.noarch.rpm perl-XML-SAX-0.12-7.noarch.rpm perl-Digest-SHA1-2.07-5.i386.rpm perl-XMLNamespaceSupport-1.08-6.noarch.rpm

rpm -ivh perl-Mail-IMAPClient-2.2.9-1.rf.noarch.rpm heartbeat-* ipvsadm-1.24-6.i386.rpm

------------------

Install Heartbeat packages

-------------------

vi /etc/sysctl.conf

# Enables packet forwarding net.ipv4.ip_forward = 1

sysctl -p

-------------------

USE "UNAME -n" for node names vi /etc/ha.d/ha.cf

logfacility bcast

local0 eth0 # Linux

mcast eth0 225.0.0.1 694 1 0 auto_failback off node node lb1.example.com lb2.example.com

respawn hacluster /usr/lib/heartbeat/ipfail apiauth ipfail gid=haclient uid=hacluster

------------------USE "UNAME -n" for node names

LB1:

vi /etc/ha.d/haresources

lb1.example.com

ldirectord::ldirectord.cf \ LVSSyncDaemonSwap::master \ IPaddr2::192.168.0.105/24/eth0/192.168.0.255

LB2:

lb2.example.com

ldirectord::ldirectord.cf \ LVSSyncDaemonSwap::master \ IPaddr2::192.168.0.105/24/eth0/192.168.0.255

-----------both LBs

vi /etc/ha.d/authkeys

auth 3 3 md5 redhat

---------------------both LBs

chmod 600 /etc/ha.d/authkeys

----------both LBs

vi /etc/ha.d/ldirectord.cf

checktimeout=10 checkinterval=2 autoreload=no logfile="local0" quiescent=yes

virtual=192.168.0.105:80 real=192.168.0.101:80 gate

real=192.168.0.102:80 gate fallback=127.0.0.1:80 gate service=http request="ldirector.html" receive="Test Page" scheduler=rr protocol=tcp checktype=negotiate

--------Both LBs

chkconfig --level 35 ldirectord off chkconfig --level 35 heartbeat on

service ldirectord stop service heartbeat start +++++++++++++++++++++++++

[root@lb1 /]# service ldirectord stop Stopping ldirectord [root@lb1 /]# service heartbeat start Starting High-Availability services: ldirectord is stopped for /etc/ha.d/ldirectord.cf [FAILED] heartbeat: 2007/02/16_04:12:30 ERROR: Current node [lb1.example.com] not in configuration! heartbeat: 2007/02/16_04:12:30 info: By default, cluster nodes are named by `uname -n` and must be declared with a 'node' directive in the ha.cf file. heartbeat: 2007/02/16_04:12:30 ERROR: Configuration error, heartbeat not started. [ OK ]

[root@lb1 /]# uname -n lb1.example.com [root@lb1 /]#

OHHHHH

Files corrected with uname -n output . i.e. instead of lb1, I have now changed that to lb1.example.com

[root@lb1 /]# service heartbeat start Starting High-Availability services: ldirectord is stopped for /etc/ha.d/ldirectord.cf [ OK ] [root@lb1 /]#

[root@lb2 /]# service ldirectord stop Stopping ldirectord [root@lb2 /]# service heartbeat start Starting High-Availability services: ldirectord is stopped for /etc/ha.d/ldirectord.cf [ OK ] [root@lb2 /]# [ OK ]

Good. Alhumdulilah.

LB1:

[root@lb1 /]# ip addr sh eth0 2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:08:c7:69:13:9d brd ff:ff:ff:ff:ff:ff inet 192.168.0.103/24 brd 192.168.0.255 scope global eth0 inet 192.168.0.105/24 brd 192.168.0.255 scope global secondary eth0 inet6 fe80::208:c7ff:fe69:139d/64 scope link valid_lft forever preferred_lft forever [root@lb1 /]#

Notice LB1 has two IPs , which DO NOT SHOW in ifconfig -a output.

LB2: [root@lb2 /]# ip addr sh eth0 2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:50:8b:60:c8:f8 brd ff:ff:ff:ff:ff:ff inet 192.168.0.104/24 brd 192.168.0.255 scope global eth0 inet6 fe80::250:8bff:fe60:c8f8/64 scope link valid_lft forever preferred_lft forever [root@lb2 /]#

----------------------------

[root@lb1 /]# ldirectord ldirectord.cf status ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 3978 [root@lb1 /]#

NOTE: The above output (running) will be displayed on ACTIVE LB

[root@lb2 /]# ldirectord ldirectord.cf status ldirectord is stopped for /etc/ha.d/ldirectord.cf [root@lb2 /]#

-------------------

[root@lb1 /]# ipvsadm -L -n IP Virtual Server version 1.2.0 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port [root@lb1 /]# Forward Weight ActiveConn InActConn

[root@lb2 /]# ipvsadm -L -n IP Virtual Server version 1.2.0 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port [root@lb2 /]# Forward Weight ActiveConn InActConn

NOTE: If LB1 does not show as below, try restarting heartbeat on LB1 and LB2. In the same order I have written here.

[root@lb1 /]# ipvsadm -L -n IP Virtual Server version 1.2.0 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port TCP 192.168.0.105:80 rr -> 192.168.0.101:80 -> 192.168.0.102:80 -> 127.0.0.1:80 [root@lb1 /]# Route 0 Route 0 Local 1 0 0 0 0 0 0 Forward Weight ActiveConn InActConn

--------------

[root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master running (ipvs_syncmaster pid: 4704) [root@lb1 /]#

NOTE: Master running on LB1

[root@lb2 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master stopped (ipvs_syncbackup pid: 4488) [root@lb2 /]#

--------------------

WWW1 and WWW2

vi /etc/sysctl.conf

# When an arp request is received on eth0, only respond if that address is # configured on eth0. In particular, do not respond if the address is # configured on lo net.ipv4.conf.eth0.arp_ignore = 1

# Ditto for eth1, add for all ARPing interfaces #net.ipv4.conf.eth1.arp_ignore = 1

# Enable configuration of arp_announce option net.ipv4.conf.all.arp_announce = 2

# When making an ARP request sent through eth0 Always use an address that # is configured on eth0 as the source address of the ARP request. If this # is not set, and packets are being sent out eth0 for an address that is on # lo, and an arp request is required, then the address on lo will be used. # As the source IP address of arp requests is entered into the ARP cache on # the destination, it has the effect of announcing this address. This is

# not desirable in this case as adresses on lo on the real-servers should # be announced only by the linux-director. net.ipv4.conf.eth0.arp_announce = 2

# Ditto for eth1, add for all ARPing interfaces #net.ipv4.conf.eth1.arp_announce = 2

sysctl -p on BOTH WWWs

---------------------

Note : /etc/network/interfaces exists in debian only it does not exist in RedHat. For that you need to use the /etc/sysconfig/network-scripts/ifcfg-* files.

DEBIAN

vi /etc/network/interfaces

auto lo:0 iface lo:0 inet static address 192.168.0.105 netmask 255.255.255.255 pre-up sysctl -p > /dev/null

------------------

WWW1 and WWW2

REDHAT / CentOS / Fedora

cd /etc/sysconfig/network-scripts/

cp ifcfg-lo ifcfg-lo:0

vi ifcfg-lo\:0

DEVICE=lo:0 BOOTPROTO=static IPADDR=192.168.0.105

NETMASK=255.255.255.255 ONBOOT=yes NAME=loopback

----------------both WWWs

[root@www1 network-scripts]# ifconfig eth0 Link encap:Ethernet HWaddr 00:0A:5E:05:97:B4 inet addr:192.168.0.101 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::20a:5eff:fe05:97b4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:7914 errors:472 dropped:0 overruns:0 frame:472 TX packets:2875 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2257584 (2.1 MiB) TX bytes:293073 (286.2 KiB) Interrupt:5 Base address:0x7080

lo

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:26 errors:0 dropped:0 overruns:0 frame:0 TX packets:26 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1820 (1.7 KiB) TX bytes:1820 (1.7 KiB)

[root@www1 network-scripts]# ifup lo:0

[root@www1 network-scripts]# ifconfig eth0 Link encap:Ethernet HWaddr 00:0A:5E:05:97:B4 inet addr:192.168.0.101 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::20a:5eff:fe05:97b4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8001 errors:472 dropped:0 overruns:0 frame:472 TX packets:2923 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2266978 (2.1 MiB) TX bytes:299149 (292.1 KiB) Interrupt:5 Base address:0x7080

lo

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:26 errors:0 dropped:0 overruns:0 frame:0 TX packets:26 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1820 (1.7 KiB) TX bytes:1820 (1.7 KiB)

lo:0

Link encap:Local Loopback inet addr:192.168.0.105 Mask:255.255.255.255 UP LOOPBACK RUNNING MTU:16436 Metric:1

[root@www1 network-scripts]#

[root@www2 network-scripts]# ifconfig eth0 Link encap:Ethernet HWaddr 00:40:26:5F:5C:60 inet addr:192.168.0.102 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::240:26ff:fe5f:5c60/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8349 errors:0 dropped:0 overruns:0 frame:0 TX packets:4236 errors:0 dropped:0 overruns:0 carrier:0 collisions:132 txqueuelen:1000 RX bytes:2187763 (2.0 MiB) TX bytes:450715 (440.1 KiB) Interrupt:5 Base address:0x7080

lo

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:53 errors:0 dropped:0 overruns:0 frame:0 TX packets:53 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5347 (5.2 KiB) TX bytes:5347 (5.2 KiB)

[root@www2 network-scripts]# ifup lo:0 [root@www2 network-scripts]# ifconfig eth0 Link encap:Ethernet HWaddr 00:40:26:5F:5C:60

inet addr:192.168.0.102 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::240:26ff:fe5f:5c60/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8412 errors:0 dropped:0 overruns:0 frame:0 TX packets:4275 errors:0 dropped:0 overruns:0 carrier:0 collisions:132 txqueuelen:1000 RX bytes:2193922 (2.0 MiB) TX bytes:455943 (445.2 KiB) Interrupt:5 Base address:0x7080

lo

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:53 errors:0 dropped:0 overruns:0 frame:0 TX packets:53 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5347 (5.2 KiB) TX bytes:5347 (5.2 KiB)

lo:0

Link encap:Local Loopback inet addr:192.168.0.105 Mask:255.255.255.255 UP LOOPBACK RUNNING MTU:16436 Metric:1

[root@www2 network-scripts]#

----------------------Both WWWs

vi /var/www/html/ldirector.html Test Page

-------------

Both WWWs

Configure the ServerName Directive repectively for each node:

WWW1 ServerName www1.example.com

WWW2

ServerName www2.example.com

Both WWWs: cd /etc/httpd/conf.d/ mv welcome.conf welcome.conf.orig

service httpd start chkconfig --level 35 httpd on

WWW1 (to represent some web content):

[root@www1 conf.d]# cd /var/www/html/ [root@www1 html]# vi index.html WWW1 Website page

WWW2 (to represent some web content):

[root@www2 conf.d]# cd /var/www/html/ [root@www2 html]# vi index.html WWW2 Website page

NOTE: I have used different webpage contents on both servers to actually notice the difference if one node fails. In real life, of-course the web content will be same.

Now try from some webclient / browser (192.168.0.22) the following:

http://www.example.com

OR

http://192.168.0.105

You will see WWW1 one time you press GO or ENTER, and will see WWW2 the second time. And it will keep switching like this proving that the load balancer is working properly and is sending connections to the web server which has lesser load. Since there are only two web servers abnd one client to check both servers one by one, the

Load Balancer takes you to www1.example.com and to www2.example.com turn by turn.

Great Alhumdulillah.

Try stopping apache on WWW1, you would see the WWW2 page on your browser and if you keep clicking GO/ refresh, you will keep seeing WWW2 page. As soon as you turn up WWW1 server, and press refresh on client. You will see WWW1 page. Same goes for WWW2 server. Great.

-----------------------------

Lets Fail the LB now. First check which one is active?

[root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master running (ipvs_syncmaster pid: 4704) [root@lb1 /]#

So LB1 is ACTIVE LB. Let's fail that and see if heartbeat takes over or not. So we still see our web page on 192.168.0.105 (www.example.com) or not?

[root@lb1 /]# service heartbeat stop Stopping High-Availability services: [ OK ] [root@lb1 /]#

Check status again:

[root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master stopped (ipvs_syncbackup pid: 5125) [root@lb1 /]#

Check to make sure if LB2 has taken over as master:

[root@lb2 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master running (ipvs_syncmaster pid: 4743)

[root@lb2 /]#

NOTICE the master has stopped on LB1. Check browser on client, Working perfectly. We still get our page.

If I start heartbeat on LB1 again, LB1 must remain inactive and LB2 must still remain ACTIVE. Lets see:

[root@lb1 /]# service heartbeat start Starting High-Availability services: ldirectord is stopped for /etc/ha.d/ldirectord.cf [ OK ] [root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status master stopped (ipvs_syncbackup pid: 5125) [root@lb1 /]#

Notice that LB2 is still ACTIVE.

Alhumdulillah.

----------------------------------------------------------------================================================================

Lets connect null modem wire to serial ports of the two LBs and change appropriate settings in various config files on both LBs , so the too much broadcast for heartbeat signals on the switch is removed / minimized.

Both LBs :

vi /etc/ha.d/ha.cf

logfacility

local0

serial /dev/ttyS0 baud 19200

#bcast

eth0

# Linux

#mcast eth0 225.0.0.1 694 1 0 auto_failback off node lb1.example.com

node

lb2.example.com

respawn hacluster /usr/lib/heartbeat/ipfail apiauth ipfail gid=haclient uid=hacluster

[root@lb2 ~]# service heartbeat stop Stopping High-Availability services: [ OK ]

[root@lb1 ~]# service heartbeat stop Stopping High-Availability services: [ OK ]

Checked through browser. Found everything working. Success. Alhumdulillah