Improve OS X Server Performance via Newsyslog Automatic Log Rotation, Without Piping

The first of two built-in options for rotating Apache access logs and error logs on OS X Server is newsyslog. You can set it up with just a couple of lines dropped into a configuration file.

As I mentioned in earlier articles, the default OS X setup for logging Apache activity and errors leaves quite a bit to be desired, both in terms of usability and in terms of performance. (See “Improve OS X Performance by Tweaking Apache Logging, And Options for Rotating Logs” and “Server.app’s Apache Log Viewer is Fairly Useless”.)

To prevent Apache log files from growing too large individually, and from taking up too much space collectively, it’s straightforward to tap into Mac OS X’s newsyslog utility, which is responsible for rotating and compressing many of the logs you see in Console. Its configuration files live at /etc/newsyslog.conf and /etc/newsyslog.d/*. As explained on its man page, all .conf files in newsyslog.d will by default get loaded when newsyslog runs, so it’s easy enough to drop in a new one that will specifically let newsyslog know to take care of our Apache access and error log rotation.

Something like this, stored in a separate file dropped into /etc/newsyslog.d, will rotate both access and error logs every day at midnight and will keep three weeks of logs before deleting them:

# logfilename                   [owner:group]  mode  count  size  when  flags [/pid_file] [sig_num]
/private/var/log/apache2/access_log             644   21     *  $D0     B     /var/run/httpd.pid  30
/private/var/log/apache2/error_log              644   21     *  $D0     B     /var/run/httpd.pid  30

The first portion each conf line provides the path to the relevant log file, the second (which we’ve left blank) indicates an optional change of owner and group, while the third specifies the rotated files’ permissions. The ‘count’ portion specifies how many logs to retain — in this example, 21 days’ worth — while the ‘size’ portion of the configuration indicates a maximum size for the logs before they are automatically rotated. In this example, I’ve entered ‘*’ to indicate that the logs shouldn’t be rotated on the basis of size. The cryptic ‘$D0’ says to rotate every day at midnight^*, while the ‘B’ flag tells newsyslog not to write a header to the log indicating that it performed the rotation. You’ll find this header in most of the log files which newsyslog rotates, and the ‘B’ flag is normally used to avoid corrupting logs that are binary. Although Apache logs are plain text rather than binary, we can use that same flag to prevent newsyslog from introducing a header that doesn’t fit the usual pattern of an Apache log line and which would, if present, cause problems for log analysis software, should you ever want to use some. (If you’re certain you won’t care whether your logs adhere strictly to the standard log format, there’s no harm in letting newsyslog leave its stamp.)

Note that we’re specifically not including a header like ‘J’ which would cause the rotated logs to be compressed. The reason is that Apache logs are buffered, which means newsyslog doesn’t know for certain when Apache will finish writing to the file; start compressing the previous log file too quickly, and you risk losing data still sitting in the buffer. If you’d like to write a separate shell script to compress old rotated logs, it’s fine to do so, but you should introduce a wait of at least a few minutes — or whatever might be appropriate for your own server’s traffic levels — to ensure that whatever you’re compressing isn’t still in use by Apache.

The next portion of the configuration line tells newsyslog where to find the ID of the process that needs to be informed when its logs have been rotated — in this case, Apache — and the final numerical code indicates the signal that should be sent to that process when logs are rotated, in this case 30, which is defined in signal.h as SIGUSR1. This signal (not SIGHUP) gracefully restarts Apache, which is how newsyslog persuades Apache to stop writing to the old file and begin writing to the new file.

You can check the man page for additional configuration options if you’d like to rotate the logs at different times, keep different numbers of old logs, etc., and you can find several other examples in Apple’s own conf files stored in the same directory, as well as the main newsyslog.conf in /etc.

* Update (23 December 2013): Originally, I’d mentioned being puzzled that on my own server, adding log rotations for Apache or anything else and scheduling them for midnight seemed to result in rotation actually occurring at 12:30. Recently Bill Bushnell wrote in with the solution to this mystery: by default, newsyslog itself only runs every hour on the half hour. Since the extra configuration files are only checked when it actually runs, this explains why a task scheduled for midnight wouldn’t get picked up until half an hour later. Many thanks to Bill for taking the time to clear this up!

• macOS
• page speed