Low Stress Server Upgrades on OS X

Version 5 of the Server app for OS X has much to recommend it, but if you host websites, you’ll need to be ready with several fixes for problems both big and small. Here’s one approach to handling upgrades which has worked well for the last several versions, including 5.

Architectural Changes in Server 5

With the previous upgrade, from Mavericks and Server 3 to Yosemite and Server 4 (which updated Apache to 2.4), I didn’t even mention the experience here on the site — in part because I waited many months before taking the plunge on anything other than a test server, and in part because once I did, there wasn’t all that much to report. In my experience, it turned out to be easier than the previous upgrade. Sure, Yosemite made it necessary to roll back the TRIM support I’d mentioned in “Enabling OS X TRIM Support for Non-Apple SSDs”, and yes, the new PHP opcache replaced APC (“Boost PHP Performance on OS X by Installing APC”). It was also necessary to create a simple MySQL launcher since it lost the ability to start up automatically (“Installing, Configuring and Performance Tweaking MySQL for OS X Server”), and including the following little gem turned out to be rather important due to problems with AllowOverride taking on its default value:

<Directory />
	AllowOverride NONE
</Directory>

But while it turned out there were several things that wound up being important for a smooth upgrade experience and minimal downtime, it was nothing at all like Server 5.

With Server 5, Apple has inserted a proxy layer between websites and the outside world, with a separate dedicated Apache instance now providing a reverse proxy, ferrying communications between the outside world and the web service Apache instance. You can read some of the rationale behind this decision in the documentation included in the installation:

/Library/Server/Web/Config/apache2/ReadMe.txt

Many of the problems you may encounter getting your server back to full working order are directly a result of these architectural changes.

But while the new architecture has certainly created its share of problems, I haven’t yet encountered any which weren’t fixable, and in fact I’ve seen noticeably better performance with Server 5 than with its predecessors. I’m also hopeful that this new way of doing things will pave the way for Apple to update its archaic TLS support in the proxy layer; thoughts on that and other matters HTTPS are included in later articles.

Before an Upgrade, and the Immediate Aftermath

Focusing solely on ensuring that website downtime can be kept to a minimum, here’s the general list of things I run through before a Server app upgrade, adapted and expanded specifically for the case of Server 5, which may introduce some problems I hadn’t experienced on previous upgrades:

• Clone the startup drive with Carbon Copy Cloner.
• Run a Time Machine backup.
• Run remote backups — i.e., from my server to another server — using my usual suite of remote backup tools.
• Make a local copy of selected parts of my backups.
• Make an additional backup copy, on the server itself, of the whole /Library/Server/Web/Config directory — and a second backup, just in case.
• Take screenshots of the Server app’s list of sites, to make it quicker and easier to see at a glance where I am in the process of rebuilding sites, should that turn out to be necessary.

Next up is the update itself, followed by:

• Start up the Server app and make a backup copy, on the server itself, of the fresh /Library/Server/Web/Config directory before making any changes to it.
• Verify immediately after starting up Server again that PHP is enabled; when this very occasionally gets switched off by an upgrade, it means raw PHP code gets dumped to the user’s browser. That’s not pretty.
• Verify whether web apps have been switched off and whether .htaccess support has been switched off for that small number of sites where I use .htaccess rather than web apps.
• Verify whether MySQL is up and running, and if the OS was updated at the same time, recreate MySQL’s log directory at /var/log/mysql with appropriate permissions.

At this stage — the immediate aftermath of the upgrade — it’s fairly likely in the case of Server 5 that you will find that none of your sites works at all, and there may well be more than one reason why that is the case. When making the move to Server 5, it may be not only that none of the sites works; it may also be that none of the sites even appears in the Server app interface. But not to worry: they’re not gone, just (temporarily) forgotten.

Immediate Fixes After the Upgrade

Restore Essential Include Files

In earlier articles, I described a few different tweaks which, to be effective, needed to be made directly in the main web service configuration file:

/Library/Server/Web/Config/apache2/httpd_server_app.conf

Since that file will be overwritten with a fresh copy during the upgrade, one of the first steps is to restore anything that the server may have been relying on via these types of changes. For example, as I mentioned above, when Server 4 arrived and updated Apache to 2.4, it became necessary to start including something like the following:

<Directory />
	AllowOverride NONE
</Directory>

This might be part of a global performance tweaks file added to httpd_server_app.conf as an include, as I described in “Essential Performance Tweaks for Your New OS X Server”. Restoring a performance tweaks include file is a simple matter of adding that single line back to httpd_server_app.conf. If your server relies on other essentials which are provided via includes, now is the time to restore them as well and restart Apache.

Note that as of Server 5, restarting the web service Apache instance (and the one which uses this configuration file) now requires the following:

This differs from the command used to restart the Apache instance providing the proxy:

If you’ve implemented OCSP stapling or made changes to the supported cipher suite, these changes will also need to be reapplied. Separate articles cover those topics for Server 5 users:

• “Set Up OCSP Stapling on OS X Server”
• “Tune Cipher Suites and Support Forward Secrecy on OS X Server”

Restore Missing Sites and/or Re-Enable Essential Web Apps

A much more time consuming scenario arises when it’s not just a few include files which need to be re-connected with the main httpd_server_app.conf, but rather one or more sites which are either gone entirely or need to have web apps switched on again. In my own case, I found during one upgrade to Server 5 that several sites were missing entirely from the Server app interface, and in another I found that all virtual hosts had had all web apps shut off. In the case of missing sites, it turned out to be my own fault — I’d left a botched web app active on a test server. In the case of web apps being shut off, by contrast, I have not been able to identify any reason at all why it would happen, and restoring the exact same configuration of various web apps across a couple of dozen sites returned everything to working exactly as it had before.

If you’re very lucky, you may find that quitting the Server app, dropping the backup copy of servermgr_web_apache2_config.plist left over from the previous version into /Library/Server/Web/Config/apache2, and starting the app again may nudge missing sites and their settings back into the Server app’s consciousness. In my experience, however, it appears that the probability of that method succeeding decreases rapidly with the complexity of your setup in terms of number of sites and their individual settings.

If you should wind up with entirely missing sites, they can be restored to the Server app’s awareness simply by adding them again using the Server app interface; since no site files are actually removed, adding them again means you can simply hook them up with their existing files, exactly as they were before. Re-enabling web apps, unfortunately, requires repeated trips through the ludicrously tiny window which the Server app provides for scrolling through available web apps and toggling checkboxes. (As someone who uses dozens of different web apps across dozens of different sites, that tiny little window is just a bad joke.)

Additional Fixes Specific to Server 5

The new proxy can benefit from several adjustments, ranging from merely convenient in almost all cases to downright essential, in the case of serving sites over HTTPS. I’ve covered these separately:

• “Adjust the Apache Proxy in Server 5 for Higher Performance”
• “Tweaking the Server 5 Apache Proxy to Handle HTTPS Sites Correctly”
• “Making WordPress Happy With HTTPS Behind the OS X Server Proxy”

Speaking of WordPress, if you’re planning to run Server 5 under El Capitan, as distinct from under Yosemite, setting up WordPress to work with SSH2 upgrades as I described in “Enabling WordPress to Work with SSH2 Upgrades — Without Extravagant Permissions” is no longer quite as straightforward for the simple reason that SSH2 itself is less straightforward to install. It can be done, of course, but it’s simpler just to use the plugin that replaces SSH2 with phpseclib (a library which has quite a bit going for it anyway).

In part due to the architectural shift to include a reverse proxy and in part because Apple has finally included Apache log rotation by default, several log-related settings also need to be tweaked after the Server 5 upgrade to make them usable again. Most significantly, if you’re accustomed to viewing logs through the Console app, you’ll no longer be able to see server logs at all with the default setup. (You won’t notice this right away: the problem arises only after the logs have been rotated once.) I’ve now added some notes on log rotation and on getting the log format to work as it was intended:

• “Adjusting the Apache Log Format on Server 5 and Updating Bot Exclusions”
• “Making Apache Log Rotation Usable Again on Server 5”

• macOS
• Server 5