OS X Server First Things First: Backup and Recovery and Other Preliminaries

Even before you fire up Server.app to get your Mac Mini server delivering websites and other services, it’s important to plan for backup and recovery of the machine itself, should anything go wrong. This is part 1 of an ongoing series on setting up your OS X Server as if your livelihood depended on it.

One of the first things to do — probably before even logging into your shiny new server — is to check the IP address or addresses you’ve been assigned by your colocation facility. The last thing you want is to get all set up on a new server only to discover later that — oops — one of your IPs recently belonged to a spammer and wound up getting blacklisted. So, do yourself a favour and stop by the major blacklisting sites (spamhaus.org, spamcop.net, etc.), and be sure your IPs are ‘clean’.

Next up, before setting up anything, I think it’s worth immediately checking for and installing any available software updates. I’ve personally run into all kinds of hassles trying to set up an outdated version of Server.app, so save some time and update it (and anything else) before setting up anything.

With software updated, next up is the obligatory clean software restore image: let’s make a clean image of the startup disk exactly as it is right now. We want a separate backup to a disk image because it is never practical to do a complete fresh reinstall of OS X remotely — that unfortunately requires local “hands on” to get through the installation and restore the necessary settings to enable networking and remote management. So if a clean install is ever needed, you can rely instead on restoring a clean backup image to a fresh drive and then using Migration Assistant to grab settings, accounts, etc. If you’re doing this with the original volume before anything is set up, make sure to keep a note of your original username and login details as provided by your host or colocation facility; you’ll want to change these shortly, but you’ll have to have the originals if the restore image is ever used.

First verify that all is well with your startup drive using “Verify Disk” and “Repair Disk Permissions” in Disk Utility, then select “New > Disk Image from Folder”.

In the subsequent dialog box, select the startup volume itself, enter a filename for your saved image, and set the preferences for a compressed image without encryption. Save the image to your second hard drive, and when it’s done, it’s worth going back to “Images > Scan Image for Restore”; this will save a significant chunk of time should you ever need to use it. This whole process — creating the image and then scanning it — is going to take awhile, regardless of how fast your machine, disks or SSDs might be. It’s best to sit back and let it run; resist temptation to skip ahead and start configuring something!

(NOTE: The steps above should work fine on a Mountain Lion Mac Mini, but in my limited experience with a new 2012 Mac Mini running 10.8.2, Disk Utility absolutely could not be persuaded to finish creating an image of the startup drive. On every occasion, it would go fine until near the end and then would hang indefinitely. The log showed that it got up to the “Copy” stage and then nothing — no error, nada. Early on, the log showed an “Authentication Error” on the root level of the drive, despite the fact that I was prompted for and successfully entered an administrator password prior to commencing the procedure, but that was it. In any case, I found that SuperDuper! could do the same job just fine, which led me to conclude that it was overwhelmingly likely to be a problem with the Apple-provided Disk Utility — which has notoriously been the source of much hand-wringing over the years — rather than a nasty drive problem of some sort.)

You’ll want to keep this restore image in a safe place, because again, this is going to be a substitute for doing a clean reinstall of the whole system, should that ever be needed. For the moment, let’s copy it back to the main startup volume and move on to the next step: repartitioning the secondary drive. (This sequence of steps assumes that your primary and secondary drives are of different capacities, as would usually be the case if your primary drive is an SSD and the secondary drive a spinning disk. If the two are the same, then you’ll need to modify this strategy accordingly.)

Our aim with partitioning the secondary drive is to provide space for a two-pronged approach to backup and recovery:

1. a complete clone of our startup volume, and
2. a Time Machine volume.

I personally also like to keep a third volume on that secondary drive — an extra ‘scratch space’ where I can store additional backups, create another bootable volume should I need it, etc. Whether you go with two volumes, three, or even more, the partitioning order is important. Why? Because the partitions are physically laid out on the disk from the outside (where the disk spins fastest in terms of the linear distance of physical platter covered per unit of time) to the inside (where it spins the slowest), earlier partitions will, other things being equal, offer better performance than later partitions. Therefore, I would suggest placing boot volume mirror first, the Time Machine volume second, and any ‘extra’ partition third.

With this in mind, the first partition on the secondary drive should be the same size as your boot volume, and the second should be the drive’s total size minus the size of the boot volume. If you’re using a third volume, then the second partition should be the drive’s total size minus the size of both the boot volume and the extra volume, while the extra volume should be whatever size you’d like it to be.

With the drive partitioned, we’re now in a position to enable part 1 of the backup and recovery strategy by cloning the startup disk to the partition intended for that use. First, though, be sure to copy that clean install image to your new Time Machine volume or your extra volume, and erase it from your from boot drive. Then clone the startup drive to the new partition you just set up. Either SuperDuper! or Carbon Copy Cloner will do the trick, and the latter can also be set to run an incremental update on the clone on a scheduled basis — once per day, say. When scheduling regular updates of the clone, it’s important to set the destination partition to be ejected once the update completes.

Unfortunately, SuperDuper! has not been updated to make use of the modern launchd method for scheduling tasks that was introduced in OS X 10.4, and it cannot run backups without a user being logged in. Since it is normal practice not to leave a user logged in on a server, this fact alone makes SuperDuper! unsuitable for regularly scheduled backups on a server. However, each package has its merits and its drawbacks; both may have a place in your arsenal of backup tools.

It’s a great idea to actually set your clone as the startup disk (via System Preferences) and boot from it, to be sure it works, then set the startup disk back to your original boot drive and restart again. This exercise should run perfectly smoothly, with no problems of any kind — but if it doesn’t, far better to discover that now than to wait until later when you’re serving live websites from the machine.

One last note on cloning the startup drive and then restarting from the clone: this can be used as a substitute for some occasions where you would otherwise boot into safe mode, which for obvious reasons would present a problem for a remotely managed Mac. Why? When cloning the drive with a tool like SuperDuper! or CCC, a large set of files which Apple recommends against copying will actually be excluded from the clone (making it not quite a clone after all); among those are the cache files which otherwise get cleared out as a result of booting into safe mode. Of course it’s possible just to manually delete caches and then restart, but this does not always have the same effect as booting into safe mode or as booting from a freshly cloned drive. So, if you run into a situation where a recommended troubleshooting step involves booting into safe mode just for the sake of its side effects (as distinct from booting into safe mode to accomplish some specific task on the command line), it’s worth trying cloning, booting from the clone, cloning again back to the original drive, and booting again from the original. For me, this has been of more than theoretical interest, since both of two new Mac Minis (one 2011, one 2012) which I set up recently arrived fresh from the colo providers in such a state that they were spewing hundreds of error messages per hour into the logs. The newer one, for example, was repeatedly issuing the following:

Now I have no idea what the real underlying cause of this problem was — that’s not the point. The point is that after much googling and fiddling and manually deleting caches and such, I eventually arrived at the next recommendation in the list, which was to boot into safe mode. I used the clone-boot-clone-boot approach instead, and all was well. (At least, it was until a few weeks later, after a few reboots for various other unrelated purposes, when the coresymbolicationd error reared its ugly head again… As far as I can tell, these errors are pretty widespread with Mountain Lion and have nothing to do with Server.app itself.)

Returning to the main setup for backup and recovery, you can enable part 2 of the strategy by setting Time Machine to use the partition set up for that purpose. (If you’d like to save a bit of space at this point by deleting bundled applications like GarageBand, iMovie, or iPhoto, now would be a good time to do so, before they get backed up. Alternatively, wait until after they’re backed up just in case you later get the urge to fire up GarageBand and start playing loud guitar compositions through the colo facility…) Note that when enabling Time Machine on a drive with more than one partition, it will immediately warn you that you should not choose a backup volume which is on the same disk as other volumes being protected. That’s OK, because we don’t want to back up those other volumes anyway; instead, set Time Machine to exclude your boot mirror and any extra partition(s) you’ve set up on that same drive. (After setting Time Machine to exclude your boot mirror, it’s a good idea to close System Preferences and then open it again to double check that Time Machine has not excluded your boot drive itself. This confusion over which of two cloned volumes has been excluded has been known to happen under many different conditions, and essential to be sure that Time Machine is not excluding your main boot volume!) In other words, we’re only using Time Machine to back up the primary boot drive, and we’re setting Time Machine to use a different drive for the backup. Presto! We now have an immediately bootable backup, should something go wrong, and we also have all the benefits of Time Machine’s historical, ongoing backup system.

We now have a fallback comprising:

1. a software restore image for use when and if all hell breaks loose,
2. a bootable current clone, and
3. an ongoing Time Machine backup.

In the following articles, I’ll revisit this backup strategy and add three more elements:

1. MySQL database backup (which I’ll delay until running through how to install and configure MySQL),
2. backing up to a remote server, and
3. cloning to a fast removable device, to help your colo partners get you up and running quickly again if something breaks.

But for now, the two-pronged strategy outlined above should be considered the absolute minimum for running any Mac OS X-based server.

Last but not least, if you’re using a SSD or other startup drive installed by your colo provider, there’s likely to be a tiny fly in the ointment which will create some extra bumpf in your logs; unfortunately there’s not a lot you can do about it. If your machine is running from a startup drive installed by your colo provider, it’s unlikely the colo provider will have installed Mountain Lion on the drive in such a way as to create a recovery partition. In one sense, this is not a big deal: you’re never going to use the recovery partition remotely anyway. On the other hand, though, the fact that it’s missing is going to cause a warning along the following lines to be written to your logs every time Time Machine runs:

This is one of those warnings which is best just ignored.

• backups
• macOS