Failsafe System – Cluster Linux con Heartbeat

Heartbeat consente di configurare facilmente un cluster Linux, ovvero un sistema che incrementa l’affidabilità assicurando che in caso di malfunzionamento/spegnimento di un server, i servizi vengano automaticamente presi in carico ed erogati da uno dei servers “secondari”, con un downtime del servizio quasi impercettibile.

In pratica Heartbeat si occupa di spostare l’erogazione del servizio (nell’esempio di seguito il servizio http) da un server ad un altro al verificarsi di particolari condizioni configurabili. Per far ciò ovviamente non gestisce solamente il servizio, ma anche l’indirizzo ip secondario virtuale, qui definito VIP, attraverso il quale il servizio stesso viene erogato.

1. SCENARIO

2 X OS Ubuntu 12.04 server 64bit LAMP, full updated, root enable.
Both servers are configured as web-servers (apache2), up and running:
web-1: eth0 192.168.252.129/24 gw 192.168.252.2 (note: hostname must be web-1, check using the command “hostname”) -> NODE 1
web-2: eth0 192.168.252.130/24 gw 192.168.252.2 (note: hostname must be web-2) -> NODE 2
During configuration we need to check which server replies to our request (or which server is working as “master”). For this reason we configure differently two /var/www/index.html page:
/var/www/index.html page on web-1 contains : “Ciao, sono WEB-1 WEB-1 WEB-1”
/var/www/index.html page on web-2 contains : “Ciao, sono WEB-2 WEB-2 WEB-2”

At the end of this procedure node 1 (configured as default “master”) will have the subinterface eth0:0 (VIP) with ip address 192.168.252.135/24 and provide web services. In the event of node 1 failure, node 2 (configured as default “slave”) will become “master” and it will start to provide web services from subinterface eth0:0 with same ip address 192.168.252.135/24. If the fault is fixed web-1 become master again (failback).

2. NETWORK CONFIGURATION

2.1 check hostname and name resolving of two nodes (very important for heartbeat)
# NODE 1
# check the hostname:
root@web-1:~# hostname
web-1 # -> OK

Edit file hosts:

# NODE 1
# Edit hosts file:
root@web-1:~# vim /etc/hosts
127.0.0.1 localhost.localdomain localhost
127.0.1.1 web-1 
192.168.252.129 web-1
192.168.252.130 web-2
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
# NODE 2
# check the hostname:
root@web-2:~# hostname
web-2 -> OK
# NODE 2
# Edit file hosts
root@web-2:~# vim /etc/hosts
127.0.0.1 localhost.localdomain localhost
127.0.1.1 web-2
192.168.252.130 web-2
192.168.252.129 web-1
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

From each node check if you can successfully ping the hostname of the other node:

root@web-1:~# ping web-2
PING web-2 (192.168.252.130) 56(84) bytes of data.
64 bytes from web-2 (192.168.252.130): icmp_req=1 ttl=64 time=0.940 ms
64 bytes from web-2 (192.168.252.130): icmp_req=2 ttl=64 time=0.271 ms

And viceversa:

root@web-2:~# ping web-1
PING web-1 (192.168.252.129) 56(84) bytes of data.
64 bytes from web-1 (192.168.252.129): icmp_req=1 ttl=64 time=0.314 ms
64 bytes from web-1 (192.168.252.129): icmp_req=2 ttl=64 time=0.246 ms
2.2 add a NIC heartbeat dedicated

Now we add a dedicated NIC to heartbeat, so the connection will be more reliable.
Two NICs may be connected between web-1 and web-2 with a cross cable.
This is network configuration:

NODE 1
root@web-1:~# cat /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface INTERFACE FOR COSTUMER SERVICES
auto eth0
iface eth0 inet static
 address 192.168.252.129
 netmask 255.255.255.0
 network 192.168.252.0
 broadcast 192.168.252.255
 gateway 192.168.252.2
auto eth1
iface eth1 inet static
 address 10.10.50.90
 netmask 255.255.255.0
 network 10.10.50.0
 broadcast 10.10.50.255
NODE 2
root@web-2:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface INTERFACE FOR COSTUMER SERVICES
auto eth0
iface eth0 inet static
 address 192.168.252.130
 netmask 255.255.255.0
 network 192.168.252.0
 broadcast 192.168.252.255
 gateway 192.168.252.2
auto eth1
iface eth1 inet static
 address 10.10.50.91
 netmask 255.255.255.0
 network 10.10.50.0
 broadcast 10.10.50.255

On web-1 we have the NIC eth1 with ip 10.10.50.90
On web-2 we have the NIC eth1 with ip 10.10.50.91
Connect with a cross cable the two NICs and check connectivity pinging the new NIC from a node to the other.

3. HEARTBEAT INSTALLATION AND CONFIGURATION

# NODE 1
# Execute:
root@web-1:/etc/ha.d# apt-get install heartbeat
# Go to the heartbeat configuration directory:
root@web-1:/etc/ha.d# cd /etc/ha.d
# Create and edit file “authkeys” for authentication key:
root@web-1:/etc/ha.d# vim authkeys
auth 1 ### use key 1:
1 sha1 zoobe1234! ### key 1 has the sha1 encryption key “zoobe1234!”

# NB This file must be readable only by root:
root@web-1:# chmod 0600 /etc/ha.d/authkeys
# NODE 1
# Create and edit “ha.cf” config file :
root@web-1:/etc/ha.d# vim ha.cf
logfacility daemon ### facility to use for logging
keepalive 1 #### heartbeat packets frequency
deadtime 5 ### after 5 lost packets the other server became "master"
warntime 3 ### after 3 lost packets a warn log appears
initdead 60 ### after a reboot heartbeat wait 60 sec to start running
ping 192.168.252.2 ### ping the default gateway to check if all the network is dead
#ucast eth1 10.10.50.130 ### heartbeat keepalive destination
udpport 694 ### listening port for heartbeat broadcast
bcast eth1 ### broadcast outgoing interface
auto_failback on ### failback is active
node web-1 ### node-1 hostname
node web-2 ### node-2 hostname
# NODE 1
# Create and edit resurces file “haresources”:
root@web-1:/etc/ha.d# vim haresources
web-1 IPaddr::192.168.252.135/24/eth0 apache2 
## web-1 is the "master",
## 192.168.252.135/24 on eth0 is the "VIP"
## apache2 is the clustered service
# NODE 2
# Execute
root@web-2:# apt-get install heartbeat
# Go to heartbeat configuration directory:
root@web-2:# cd /etc/ha.d
# Create and edit file “authkeys” for authentication key:
root@web-2:/etc/ha.d# vim authkeys
auth 1 ### use key 1
1 sha1 zoobe1234! ### key 1 has the sha1 encryption key “zoobe1234!”
# This file must be readable only by root:
root@web-2:# chmod 0600 /etc/ha.d/authkeys
# NODE 2
# Create and edit “ha.cf” config file :
root@web-2:/etc/ha.d# vim ha.cf
logfacility daemon ### facility to use for logging
keepalive 1 #### heartbeat packets frequency
deadtime 5 ### after 5 lost packets the other server became "master"
warntime 3 ### after 3 lost packets a warn log appears
initdead 60 ### after a reboot heartbeat wait 60 sec to start running
ping 192.168.252.2 ### ping he default gateway to check if all the network is dead
#ucast eth1 10.10.50.129 ### heartbeat keepalive destination
udpport 694 ### listening port for heartbeat broadcast
bcast eth1 ### broadcats outgoing interface
auto_failback on ### failback is active
node web-1 ### node-1 hostname
node web-2 ### node-2 hostname
# NODE 2
# Create and edit resurces file “haresources”:
root@web-2:/etc/ha.d# vim haresources
web-1 IPaddr::192.168.252.135/24/eth0 apache2 ## web-1 is the "master",
## 192.168.252.135/24 on eth0 is the "VIP"
## apache2 is the clustered service

4. HEARTBEAT STARTS

Start heartbeat on both servers:

/etc/init.d/heartbeat start

4. CLUSTER AND FAILOVER TEST

From a browser: http://192.168.252.135. You display the page “Ciao, sono WEB-1 WEB-1 WEB-1” (web-1 is replying to your requests) -> OK, web-1 is now “master”;
If you try http://192.168.252.130 you don’t display anything: apache2 on web-2 is down -> OK
From a client in the network try to permanent ping (ping –t) 192.168.252.135.
Shutdown web-1.
After few seconds you loose only one packet and then the ip starts again to respond correctly: VIP is switched from web-1 to web-2.
From a browser: http://192.168.252.135. You display the page “Ciao, sono WEB-2 WEB-2 WEB-2” (web-2 is replying to your requests) -> OK, web-2 is now “master”;
Power on web-1. After few seconds web-1 become “master” again: failover feature works correctly.

I file /etc/hosts e /etc/resolv.conf

Spunti da: good-linux-tips.com.

Il file /etc/hosts ha il compito di eseguire la risoluzione dei nomi: contiene entry dns statiche. Il sistema legge questo file prima di avviare le query dns ai server dns esterni. E’ quindi usato primariamente per definire alcuni hostname di macchine interne (locali) attestate sulla rete LAN.
Può anche essere usato per definire una sorta di brutale black-list per bloccare l’accesso della macchina a siti indesiderati: è sufficiente associare l’URL del sito indesiderato all’IP 127.0.0.1 (localhost) oppure allo 0.0.0.0.
Quando il sistema deve risolvere un nome, prima guarda le entry contenute in /etc/hosts. Se non trova alcun match invia la query dns al primo dns server disponibile così come listato nel file /etc/resolv.conf.

Sintassi file /etc/hosts
La prima colonna a sinistra contiene gli indirizzi ip. La seconda l’hostname.domainname (fqdn). La terza è un breve alias, generalmente corrispondente al solo hostname.
Le colonne sono separate da spazi o <tab>.

Esempio di un file /etc/hosts:

root@UbuOnMac:~# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 UbuOnMac.enterprise.it UbuOnMac
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Il file /etc/resolv.conf specifica i server dns esterni a cui il sistema si rivolge se non riesce a risolvere i nomi utilizzando /etc/hosts.
Un esempio di file /etc/resolv.conf è il seguente:

root@UbuOnMac:~# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 8.8.4.4
nameserver 151.99.125.2

In Ubuntu versione desktop (quindi con interfaccia grafica) la gestione della rete è interamente demandata all’applicativo “Network Manager”, quindi il contenuto di /etc/resolv.conf sarà un semplice puntamento al localhost.
Il file /etc/hosts viene invece letto in modo prioritario anche dalla versione desktop.

In ubuntu versione server (senza interfaccia grafica) il contenuto del file /etc/resolv.conf può essere modificato direttamente da qualsiasi editor di testo e le modifiche diverranno operative. Al riavvio della macchine, però, le modifiche andranno perdute, a meno che non si proceda come indicato di seguito:

– modificare il file /etc/resolvconf/resolv.conf.d/head aggiungendo i server dns desiderati:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 123.123.123.123
nameserver 321.321.321.321

– Aggiornare il file resolv.conf mediante il comando:

root@UbuOnMac:~# resolv.conf -u

Se si legge il file /etc/resolv.conf si troveranno inserite le nuove entry, confermate ad ogni riavvio.

Un altro modo molto utilizzato per settare i server dns esterni in modo permanente è quello di inserire delle entry “dns-nameservers” nel file principale di configurazione delle interfacce di rete /etc/network/interfaces. Ecco un esempio di tali entry:

# The primary network interface
auto eth0
iface eth0 inet static
 address 192.168.201.136
 netmask 255.255.255.0
 gateway 192.168.201.20
 dns-nameservers 192.168.201.136 151.99.125.2

Il file hosts di Linux è del tutto simile a quello di Windows.
Linux e Windows hanno anche una cache locale che stora le precedenti risoluzioni mantenendole in memoria fino alla scadenza del TTL del record dns.
Alcuni comandi Windows:

ipconfig /displaydns
ipconfig /flushdns

Per cancellare la cache dns in Ubuntu:

# /etc/init.d/dns-clean restart