This is the third post about Amanda, an open source backup system for UNIX-based computers. The previous two posts were a general introduction to Amanda inner workings, and instructions for configuring a Mac OS X amanda client.
In this post I’ll explain how to recover from a catastrophic failure, like when a hard drive dies. Although much of the steps are identical, this post is focused on how to recover the entire file system and not a small set of files that an user accidentally deleted. To recover something like that, you can simply run amrecover on the server, recover the files you need and transfer them using SFTP or any other protocol to the client machine.
Before going any further, let me just point out an important detail: if you have a downtime on a machine that lasts for a few days, and your tape rotation cycle is not long enough, you may loose your backups at some point (assuming you backup frequently enough). To avoid this, you have several options. The easiest one is to simply disable whatever mechanism triggers your backups (launchd, crontab, etc). Of course, this only makes sense if you have only one machine being backed up on a given Amanda configuration. If stopping the backups is not possible, you can copy the tape directories or, better yet, the tar files that match the affected machine’s volumes to a safe place. Remember you have to copy the correct files, from level 0 to whatever level you are on. If Amanda overwrites the tapes after doing this, you can always recover directly from the tar files.
To recover from an Amanda backup, you should be prepared with a small hardware setup that will make your life easier. I recommend you prepare the following before a problem occurs, so you can avoid doing this under stress. What you’ll need is an external hard drive with Mac OS X installed and the Amanda client installed on it. This way, you can boot the Mac with the freshly installed empty drive and restore the backed-up contents directly to that drive. Note that if you handle PowerPC and Intel Macs, you’ll need to be ready for both situations. The easiest way is to have two drives, although I believe it’s possible to have two bootable partitions on the same drive using a few tricks. Also, I recommend you install the latest OS for both processor families (Leopard on PowerPC, Snow Leopard on Intel).
When recovering, the Amanda client needs to contact the server, which means (assuming you’re using ssh security) you need to generate a public/private key pair on the client and install the public key on your server. I recommend you remove the public key from the server as soon as your recovery operation ends, for security reasons.
So, let’s assume you have a client made out of a Mac with a new, formatted and empty internal hard drive, and running Mac OS X installed from an external drive, with Amanda client installed and a public/private ssh key pair whose public key is already on the server. Let’s also assume the volume list of this machine is the one I recommended on my previous post:
/ /Applications /Library /Users
Here are the steps to recover everything from the backups to the empty drive:
1. Launch amrecover on the client using the following command. This will start the recovering console.
/usr/local/sbin/amrecover -C configname -s my.server.com -t my.server.com -oauth="ssh"
Replace configname by the name of the Amanda configuration you want to use for recovering, and my.server.com for your backup server. Note that you must refer to your backup server using a canonical name, not an IP. If you don’t have a canonical name, you can edit /etc/hosts and create one.
2. Run the sethost command to set the host you want to recover. You can get a list of hosts managed by the current Amanda configuration using listhost. To recover from a host called andromeda, you would use:
sethost andromeda
3. Use the setdisk command to define what volume you want to recover. Let’s start with the root volume, so type the following:
setdisk /
Like in the hosts case, you can use the disklist command to list the backed-up volumes.
4. Use the lcd and lpwd commands to move to the root of the empty disk. This will tell Amanda where you want to recover stuff to. If your empty disk is called NewDisk, the command would be:
lcd /Volumes/NewDisk/
You can use the lpwd command to verify you defined the local directory to the correct path. It’s extremely important you don’t forget to define the local directory! Doing so results in Amanda recovering the data to whatever local directory is starts on, which is almost certainly not what you want to happen.
5. Add the list of items to recover.
We want to recover everything, so let’s just add the root directory:
add /
You can use the usual UNIX commands (cd, pwd) to navigate trough the backed-up hierarchy and select exactly what you want to recover. Note the paths are relative to the root of the current volume, not the root of the original client.
6. Initiate the recovering procedure. To do that, simply use the extract command:
extract
Amanda asks you to confirm the operation, and will ask you if you want to load the needed tapes during operation. For unattended operation, I recommend you to hit the return key several times to answer “Yes” to all the tape change questions (assuming you’re using virtual tapes, and not real tapes, of course). This will take a while.
Repeat the following 3 steps for the remaining volumes (I’ll use /Library as an example):
7. Use Finder or another terminal window to create the root folder of the volume:
cd /Volumes/NewDisk mkdir Library
8. Back on Amanda, change disk and local directory to match the new volume:
setdisk /Library lcd /Volumes/NewDisk/Library
9. Add all the files in the current volume, and initiate extraction:
add / extract
Again, repeat the previous steps for all the volumes you want to recover.
10. Use Disk Utility to Repair Disk Permissions on the new disk. This will make sure the proper ownership and permission settings are assigned to the directories you created (and everything else).
You should now be able to boot from the recovered drive. You’re done. :)