Server Backup and Disaster Recovery

I'd been backing up the web/mail server (crimson) in a rather ad-hoc manner, by archiving the key data files to a compressed tar file in /tmp and then scp'ing the file to gold. However, I knew I was not backing all the files I should, so I needed a slightly better solution.

I stumbled upon netcat, a neat tool for network communications. Basically, netcat allows you to pipe data from one machine to another, on arbitrary port numbers. This ability meant that I could run an archive process (either tar or cpio) on a file hierarchy bigger than the free disk space on crimson, and pipe the output to gold (which has scads of free disk space).

On FreeBSD, I built the netcat port /usr/ports/net/netcat. Gold, a Debian machine, already had it installed (not sure I remember doing that!).

To use netcat (the binary is called nc) I used the following process:

First, on gold, issue the command:

  nc -l -p 1234 >home.tar.gz

nc then awaits data on port 1234. On crimson:

  cd /home;tar czPf - | nc -w 3 gold 1234

The tar output is sent to gold on port 1234.

For just the root directory, I used cpio, since I could ensure, via find, that the other filesystems were not traversed:

  find -x / -print | cpio -o --quiet | gzip | nc -w 3 gold 1234

Naturally, this requires a slightly different receive netcat setup on the target machine:

  nc -l -p 1234 >root.cpio.gz

This still doesn't provide a very satisfactory remedy if I lose crimson's hard drive. I will still need to rebuild FreeBSD on a new drive, and restore the key data files via netcat from gold.

After some thought, I had a cunning plan...

The machine I used as a Samba file and printer server, chrome, had plenty of disk space, and could certainly support the additional files and load of the Apache web server. Agreed, chrome only has a 100MHz processor, but the demands on the website and mail server on crimson were not great, so chrome as a temporary solution should work. All I need to do was to install Apache, upgrade the box to the latest Debian patch revision, get exim working and I then had a hot standby for crimson.

Installation of Apache was easy, using apt-get. However, I had a little more difficulty in upgrading Debian to the latest Woody revision levels.

I first updated the package indices, using apt-get update, and then installed the latest packages by apt-get dist-upgrade. I noticed that no new kernel was installed. This was a little disturbing, as I knew that there had been a number of patches issued, via the debian-security mailing list.

I was using the 2.4.18-bf2.4 kernel (for the ext3 support). If I tried apt-get upgrade kernel-image-2.4.18-bf2.4, I was told that all the packages were up to date. I then reasoned that I'd never actually installed the kernel-image package, as the kernel had arrived as part of the original installation. Based on this, I tried apt-get install kernel-image-2.4.18-bf2.4. Ah, that's better, now it was downloaded and installed. I also obtained the matching kernel headers (apt-get install kernel-headers-2.4.18-bf2.4), and was fully up-to-date.

Apache was relatively easy to configure, as I was used to the configuration files from crimson. I did have to installed the python2.2 package though, as the Woody standard python was 2.1, a little too old for my scripts. The python binary location was different in Debian, being /usr/bin/python2.2 rather than /usr/local/bin/python. This meant the shebang lines in the python scripts needed to be changed.

Getting exim to do as wanted was a little more interesting.

As soon as this was all set, I had yet another, even more cunning plan. What if I rebuilt chrome using FreeBSD? Then I could have a ready made disk to swap in if the crimson hard drive failed. More on this later...