Simple Apache failover cluster on Ubuntu with config synchronization

In my previous post, I was explaining how to setup a simple failover cluster using Ucarp on Ubuntu.

In this article we are going to cover the same topic but this time we’ll have an Apache server running and we, of course, want to synchronize its configuration.

This time again, we are going to keep things simple. We are going to use Ucarp to provide the IP failover mechanism (read my previous article if it is not clear), and a little script to synchronize Apache configuration and content (including httpd.conf, sites, ssl certificates, web sites, …).

Here is an overall picture of what we want to achieve :

Step 1 : Setup your two Apache servers

You’ll need two Apache servers, it can be virtual servers. In my case, I’m using Ubuntu 10.04 LTS, one machine is physical, and the second one is virtual (but you can do whatever you want). Simply install Apache on both servers, and make sure you configure at least the master server (Apache #1) so that Apache runs correctly. Keep in mind that you need to run the same version of Apache on both servers (I even recommend the same version of Operating System) :

sudo apt-get install apache2

Then, install Ucarp on both servers as well :

sudo apt-get install ucarp

Step 2 : Configure Ucarp

Ucarp is our IP failover mechanism. It will bascially allow us to have a “virtual IP” address that will “point” to Apache #1 if the server is up, and to Apache #2 in case the first server is down (again, see my previous article if you want to know more).

Here is the network configuration for Apache #1 :

sudo nano /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        ################################
        # standard network configuration
        ################################
        address 172.20.1.26
        netmask 255.255.240.0
        network 172.20.0.0
        broadcast 172.20.15.255
        gateway 172.20.1.250

        ################################
        # ucarp configuration
        ################################
        # vid : The ID of the virtual server [1-255]
        ucarp-vid 1
        # vip : The virtual address
        ucarp-vip 172.20.1.16
        # password : A password used to encrypt Carp communications
        ucarp-password secret
        # advskew : Advertisement skew [1-255]
        ucarp-advskew 1
        # advbase : Interval in seconds that advertisements will occur
        ucarp-advbase 1
        # master : determine if this server is the master
        ucarp-master yes

# The carp network interface, on top of eth0
iface eth0:ucarp inet static
        address 172.20.1.16
        netmask 255.255.240.0

And here is the network configuration for Apache #2 :

sudo nano /etc/network/interfaces

The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        ################################
        # standard network configuration
        ################################
        address 172.20.1.36
        netmask 255.255.240.0
        network 172.20.0.0
        broadcast 172.20.15.255
        gateway 172.20.1.250

        ################################
        # ucarp configuration
        ################################
        # vid : The ID of the virtual server [1-255]
        ucarp-vid 1
        # vip : The virtual address
        ucarp-vip 172.20.1.16
        # password : A password used to encrypt Carp communications
        ucarp-password secret
        # advskew : Advertisement skew [1-255]
        ucarp-advskew 100
        # advbase : Interval in seconds that advertisements will occur
        ucarp-advbase 1
        # master : determine if this server is the master
        ucarp-master no

# The carp network interface, on top of eth0
iface eth0:ucarp inet static
        address 172.20.1.16
        netmask 255.255.240.0

Once you have finished, restart the network interfaces on both servers :

sudo /etc/init.d/networking restart

You’ll then be able to see that if both servers are up, the virtual IP (172.20.1.16) is assigned to the Apache #1 server. If you shutdown the Apache #1 server, the Apache #2 server will get the virtual IP. To test, simply look at the result of the following command on each servers :

sudo ifconfig

eth0      Link encap:Ethernet  HWaddr b8:ac:6f:90:31:19
          inet addr:172.20.1.26  Bcast:172.20.15.255  Mask:255.255.240.0
          inet6 addr: fe80::baac:6fff:fe90:3119/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:739943 errors:0 dropped:0 overruns:0 frame:0
          TX packets:593742 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:668890196 (668.8 MB)  TX bytes:717991771 (717.9 MB)
          Interrupt:16 Memory:da000000-da012800

eth0:ucarp Link encap:Ethernet  HWaddr b8:ac:6f:90:31:19
          inet addr:172.20.1.16  Bcast:172.20.15.255  Mask:255.255.240.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:16 Memory:da000000-da012800

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:312 (312.0 B)  TX bytes:312 (312.0 B)

Step 3 : Synchronizing the two servers

The server Apache #1 is called our master server. We will basically replicate any Apache related changes made on that server to the second server Apache #2. This means if you want to modify the apache config of your failover farm, you’ll have to perform those changes on the server Apache #1.

Create a new file :

sudo echo "dummy content" > /root/sync-apache-conf.sh

Give the file execution permissions :

sudo chmod 744 /root/sync-apache-conf.sh

Open it with nano, and copy the following content into it

sudo nano /root/sync-apache-conf.sh

#!/usr/bin/env bash

#############################################################
#
# Synchronizes the apache configuration and related stuff
# accross the network using rsync. The master server is
# bepscnet26, we grab everything from there...
#
# H@cked by Laurent Bel in April 2012
#
# Credits :
# - http://tech.tomgoren.com/archives/214
# - http://laurentbel.com/?p=230
############################################################

# Variables (change it accordingly)
master_server="172.20.1.26"
current_date=$(date)
log_file="/var/log/sync-apache-conf.log"

# We sync the apache config, the ssl certificates, and the hosts file, www folder
result=$(rsync -ai --delete --force ${master_server}:/etc/apache2/ /etc/apache2/)
rsync -ai --delete --force ${master_server}:/etc/ssl/ /etc/ssl/
rsync -ai --delete --force ${master_server}:/etc/hosts /etc/hosts
rsync -ai --delete --force ${master_server}:/var/www/ /var/www/

# if nothing to do, the result variable will be empty, in which case the
# changes variable will be equal to 0
changes=$(echo -ne "$result" | wc -m)

# We some changes were performed, we need to reload the apache config
if [[ $changes > 0 ]]; then
        # We log
        echo "$current_date - Apache Conf Sync : Changes were found, we reload apache" >> $log_file
        # We reload apache
        /etc/init.d/apache2 reload
else
        #  We log
        echo "$current_date - Apache Conf Sync : No changes were found, we don't do anything" >> $log_file
fi

The script is quite well documented and self explanatory. Long story short, we are using rsync to copy files (apache config, ssl certificates, hosts file, and www folder) from Apache #1 to Apache #2. If some changes are detected in the apache config, we then perform a “reload” of the apache server. You’ll note the variables at the beginning of the script that you can change according to your needs.

We now can run the synchronization manually. You can test the script by changing something on Apache #1 and then run :

sudo /root/sync-apache-conf.sh

This should update the files on Apache #2 and perform if needed an Apache reload.

You can view the logs by typing

sudo cat /var/log/sync-apache-conf.log

Important note : If the script does not work, or if rsync keeps prompting for a password (it will !), just follow those steps to get rid of the password prompting.

Step 4 : Automate the synchronization to run every 5 minutes

Now we are going to automate this synchronization (we are lazy…) using crontab. The synchronization will run every 5 minutes.

Let’s start crontab and insert a new scheduled tasks :

sudo crontab -e

Append the following lines to the file

# We synchronize the apache conf every 5 minutes
*/5 * * * * /root/sync-apache-conf.sh

Step 5 : Test !

Now let’s test a little. First of all, check in the log file if your scheduled task is running correctly every 5 minutes :

sudo cat /var/log/sync-apache-conf.log

Try to shutdown the server Apache #1 and see if the second one is taking over correctly.

Try to modify the config (anything, apache, or a web site) on Server Apache #1 and wait 5 minutes to see if it gets synchronized on Apache #2.

Once you are confident that it works fine, just release your work to production. And you are done !

Conclusion :

In roughly half a day (overall, including testing), you can setup a simple Apache failover cluster with automatic synchronization of the config (you might have to adjust the sync script to perform any other action you want). It is simple, efficient… everything you need.

Note that in a more complex environments, you might want to achieve something similar using Puppet. It allows you to synchronize configuration across several servers and it is of course much more powerful than our little script, but it is also more complex and time consuming to setup.