Do you run Linux servers on your network? A boot CD might not be enough when it comes time to repair your servers. Something will go wrong, and the Boot CD might not have all the kernel modules and tools required to fix your server. Make a custom boot hard drive. Here's how!
Linux Recovery Drive: Why a Hard Drive and not a Boot CD?
If you run a number of servers based on the Linux operating system, you are eventually going to run into a problem that a standard Linux boot CD (like the Trinity Rescue Kit) will not be able to fix. Having ran a network of over a hundred Linux based servers, I can say without a shadow of a doubt that using dedicated hard drives for server recovery and repair is generally better.
Boot CD's limit your repair options. Perhaps the kernel that is installed doesn't support the hardware that your server is running. RAID controllers, network controllers, and other specialized hardware may not be supported by the average boot CD. Sure, you could make your own boot CD but that would require more time than it might be worth.
A hard drive, or even USB flash drive would be a far better option. With it, you can install your own distribution. If your servers run Ubuntu, Debian, CentOS, or some other distribution, you can install the exact same distribution and version on your hard drive or USB drive. You can also install all the kernel modules that you could possibly need and compile the kernel initrd accordingly. This allows the flexibility you will need for repairing any of your servers. Let us look at some basic steps.
Basic Boot Disk considerations
For this small introduction, I'm going to focus on the Debian distribution of Linux since it is what I am currently running on my own server. Ubuntu is similar, and the methods translate to Red Hat based distributions just fine. Some of the specifics might be different, but the principals are all the same.
Hardware Inventory and Kernel Modules
Imagine that you have 10 servers. They all run Debian, and the hardware varies. They have different hardware RAID controllers, various network controller chipsets, but all run the 2.6 kernel.
Look on each machine in the /etc/modules file and see which modules are installed on each server. Make a list of all of them, as you will need to make sure all of these modules are installed on your hard drive or USB flash drive. Also, make sure that the hard drive you use is bootable on all of your servers. Sometimes an old IDE hard drive is the best. Put it in an external case that is USB powered (usually 2.5" hard drives in portables cases) and you won't have to tear into the servers to install the drive. Of course with a sufficiently large USB drive, this would not be an issue.
Making the Boot Disk
Set up a test PC, configure the hard drive or USB drive, and install your Linux distribution on it. Now it is time to install all of the kernel modules that you use in your servers. Methods vary, but in general make sure that the modules are installed using apt-get, yum, or whatever your Linux distro uses.
Once they are installed and enabled in /etc/modules, you need to create a new initrd.img with the command
mkinitrd /boot/initrd.img-<version> <version>
where <version> is your current kernel version. Now you have a drive that should boot on all of your servers. You'll need to consider specifics for your server and test the drive on each server to make sure it works. You'll also want to install all the recovery utilities available for Linux.
We hope this has informed you on the basics and the reasons for using a bootable hard disk or USB drive for your Linux recovery drives.