Upgrading Your OS X Server to Mavericks: Pitfalls and Fixes

You know you’ve upgraded your OS X Server to Mavericks when websites disappear, MySQL won’t start up, your PHP includes have been erased, htaccess-based security is gone, and Apache performance bogs down. Fortunately, it’s not too difficult to put everything back together again.

For those using OS X Server to host just a single site (or none at all), and who haven’t added much in the way of customisations or performance enhancements, it’s entirely possible that the upgrade to OS X 10.9 and Server 3.0.1 — the versions current as of this writing — could be as smooth as silk. However, for those running multiple virtual hosts, or relying on MySQL databases, or using custom logging, or using htaccess-based security, or caching PHP in RAM, or even relying on global PHP includes, quite a few things may need tweaking after the update.

This article is not at all intended as a step-by-step guide, but rather as a hodge-podge of my own experiences with the update that may be specific to my own particular setup. Therefore, for readers contemplating the update, I’d suggest skimming through the whole article first to check what, if anything, might apply. In addition, I’m not offering much in the way of diagnosis — i.e., in many cases I have no idea why OS X Server might have done things like, say, switching off htaccess-based overrides, I’ve only noticed that it has, and marked the behaviour down as something that a server administrator may need to fix once the upgrade is complete.

Preliminaries: Things to Back Up (Again) Before Mavericks

After the update, I found it useful to have backup copies handy of the PHP includes from /usr/lib/php, the PHP configuration file from /etc, and the primary Apache config file httpd_server_app.conf from .../Web/Config/apache2. Since, after the update, PHP will need to be reconfigured and will need to have any extensions re-installed, it may also be useful to grab copies of the archives relevant to your installation in advance, just to save a few minutes after the update.

Since Xcode’s command line tools will be needed for installing those extensions, it’s also worth noting that the “Install Command Line Tools” option has disappeared from the latest version of Xcode, and the notice which Xcode displays when first starting up about downloading extra components is not referring to the command line tools. Instead, getting the essential command line tools once the latest version of Xcode has been installed requires the following command, which will pop up an ordinary alert asking for confirmation before downloading the tools:

For servers which use any of several methods for altering the SSH port (e.g., “Changing the Default SSH Port on Mac OS X Server”), it may be worth reviewing what has been done and maybe even considering changing back to the normal port on a temporary basis, along with double-checking that remote connections still work as expected. The last thing anybody wants is to upgrade a server only to lose SSH connectivity.

Also on the security front, servers which rely on htaccess-based security on the default site may be left without any security immediately after the update. For that reason, it may be preferable to add OS X Server’s own access restrictions to the site before the upgrade takes place.

Finally, for servers which provide the wiki service on more than one virtual host (see “How to Enable Wiki Services on Multiple Virtual Hosts with Mountain Lion Server”), it may be helpful to switch off the wiki enabler web app prior the upgrade, as it will need to be replaced with an updated version designed to work with Server 3.

Basics Right After the Upgrade to Mavericks

After the upgrade to 10.9, a whole slew of alerts can be shut off via the Notifications preference pane, and many of the standard server-friendly preferences should be manually restored if the upgrade has altered them — such as setting the machine to start itself after a power failure, scheduling a nightly startup (just in case), not putting hard disks to sleep, and disabling bonjour broadcasting. The “Displays have separate spaces” option in Mission Control is also probably best switched off, and for machines living in co-location facilities or otherwise on certain network topographies, an apparent unicast bug may need to be addressed in order to stop the server losing packets on a frequent basis. (The relevant facility should be able to provide details.) For more on these sorts of basic environmental settings, see the earlier article “Basic Configuration and Preparing the Way for Extra Software OS X Server Needs”.

For servers with a solid state drive not made by Apple, the upgrade to Mavericks will also necessitate the re-application of whatever method had previously been used to enable TRIM support. My understanding is that the method which I suggested previously (“Enabling OS X TRIM Support for Non-Apple SSDs”) still works on the new version of the relevant file under Mavericks, but not having tried it myself yet, I cannot say with certainty.

MySQL Won’t Start Up

If you’ve configured MySQL to write logs to a standard location like /var/log/mysql, you’ll likely find that MySQL will not start up after the Mavericks upgrade, because the necessary directory will no longer exist. (See “Installing, Configuring and Performance Tweaking MySQL for OS X Server”.)

Fortunately, all it takes is a quick fix to get MySQL back to its former happiness:

Now a quick press of the button in the MySQL preference pane or the usual command line utterance should fire it right up.

If it does not fire right up, and instead returns an error about the /usr/local/mysql path being unavailable, your machine may simply need a new symbolic link at /usr/local/mysql to point to the version-specific MySQL directory in /usr/local. In my own experience, this odd error cropped up only when upgrading straight to 10.9.2 from Mountain Lion, but not when upgrading to the older 10.9.

Disappearing Sites after the Mavericks Upgrade, Disappearing Security

Even on my test platform, it was more than a little disconcerting to complete the Mavericks and OS X Server 3.0.1 update and immediately find that all my websites except for the default one had disappeared — and the default one no longer had any htaccess-based security in place. The second problem is quick and easy to fix: the option to “Allow overrides using .htaccess files” was switched off after the upgrade, so if that happens to your machine, all that’s needed is to switch it back on for both the default site and the SSL version of the default site, each of which is configured separately in the Server interface. (For a stronger way of setting up security in the first place, especially when using the wiki service, see the earlier article “Password Protecting Directories, Files or Whole Domains via a Web App”.)

The complete disappearance of all the virtual hosts configured for the server is a bit more puzzling. After the upgrade, I found that the .../apache2 directory had been copied to .../apache2.previous but not before several changes had already been made to it. (In other words, it’s not really a “previous” version at all!) In addition, within both of those directories, a new one appeared, sites-previous-unusable-12345, where ‘12345’ represents some five-digit number, which I would guess may differ from one installation to the next.

In my experience, it’s possible simply to copy the individual vhosts’s *.conf files (but not those for the default site or the global configuration file) from the sites-previous-unusable-12345 directory into the normal sites directory, stop the web service, stop and start Apache from the command line, and restart the web service to restore all the lost sites. The additional step of stopping and starting Apache from the command line is required because simply turning off the web service does not shut down Apache:

For some unknown reason, it may be necessary to perform this sequence a couple of times before the missing sites finally appear in the Server app interface.

However, simply restoring the sites is not quite the end of things, since the upgrade will also overwrite the server’s Apache configuration and wreak a little havoc with PHP.

Mavericks Server Update Overwrites Apache Configuration

As noted above, it’s worth keeping a copy of the server’s original Mountain Lion httpd_server_app.conf. The update to Mavericks and Server 3.0.1 will overwrite the configuration file (and a copy will not be saved in .../apache2.previous), so it’s helpful to use a diff tool such as BBEdit’s “Compare Two Front Windows” utility to merge any relevant customisations from the old configuration file to the new one. Generally speaking, this file shouldn’t be home to too many significant changes, but in a few cases achieving a particular effect requires modifying the file directly. For example, I’ve mentioned previously how to improve logging, tweak specific Apache performance settings, fix delivery of content to Internet Explorer, and implement a global ban list, all of which require modification of the file:

• “Essential Performance Tweaks for Your New OS X Server”
• “Improve OS X Server Performance for IE Browsers”
• “Improve OS X Performance by Tweaking Apache Logging, And Options for Rotating Logs”
• “Improve OS X Server Performance With a Server-Wide Ban List”

After merging in any relevant customisations to this main Apache configuration file, of course it’s necessary to restart Apache for the changes to take effect:

Mavericks Server Update Upgrades PHP to 5.4.x and Deletes PHP Includes

Server 3.0.1 for Mavericks comes with PHP version 5.4.17, and this causes a few obvious configuration changes as well as some more subtle changes in terms of the operating environment provided by 5.4.x.

As a result of the latter, this change isn’t something to be undertaken without a full awareness of what, if any, impact the upgrade may have on any custom code the server is running. For example, code which passes variables by reference when calling a function will elicit the new “Fatal Error: Call-time pass-by-reference has been removed”. WordPress developers who use WP_DEBUG and WP_DEBUG_DISPLAY will find far more errors being reported under 5.4.x than under 5.3.x because error reporting for STRICT is now included in error reporting ALL; since WordPress does not respect the specific error reporting set in the server’s php.ini file, expect to see a slew of new (non-)errors being reported for code using any but the latest coding standards. (For example, including two class constructors for the sake of backward compatibility will now generate a STRICT error when using WordPress debugging regardless of whether the php.ini specifies not to report them.)

As for the configuration changes themselves, relevant portions of the server’s previous /etc/php.ini file can be merged with the new one for 5.4.x using the same approach as for the httpd_server_app.conf file mentioned above.

In my case, I found that not only was the include path specified in the new php.ini blank, but in fact the entire set of includes previous stored in /usr/lib/php had been nuked by the upgrade. Thank goodness for those backups. There is also a new php.ini file which will appear after the update in .../Web/Config/php, but this is only for a few Server-specific settings which will override those in the main /etc/php.ini.

In addition, the fresh PHP 5.4.x will be missing any extensions which might previously have been added, such as those enabling SSH2 functionality for WordPress, or APC. It will be necessary to re-install those extensions as they were originally:

• “Enabling WordPress to Work with SSH2 Upgrades — Without Extravagant Permissions”
• “Boost PHP Performance on OS X by Installing APC”

With PHP’s include path set correctly, any deleted includes restored from backup, the php.ini file updated appropriately, and missing extensions such a APC re-installed, the PHP side of things should be good to go.

When re-installing APC version 3.1.13, it is essential to enter “no” when asked about enabling internal debugging. Just accepting the default “no”, or entering “n” instead of “no”, may wind up saddling your machine with APC internal debugging enabled anyway, which can cause no end of problems with PHP.

Re-Enabling the Adaptive Firewall

You may find after the upgrade that using afctl to interact with the adaptive firewall fails silently; if so, just use Apple’s instructions for enabling the adaptive firewall.

Using the Mavericks Server Wiki Service on Multiple Virtual Hosts

Finally, if the server had been using the wiki service on more than one website (see “How to Enable Wiki Services on Multiple Virtual Hosts with Mountain Lion Server”), the original method for enabling the service on multiple sites no longer works under Mavericks. The general method and approach does work, however, so the wiki enabler I described earlier can easily be reconstructed. I’ll cover that in a separate article.

• macOS