Glen Turner (vk5tu) wrote,
Glen Turner

Software which should exist #234: user-space ring-buffer log

I am running Linux from a flash memory SD Card à la Raspberry Pi. Which the system logger is writing to. The typical configuration of the system logger is to force the disk to write the more important messages immediately. That means it might do 50 writes per 4 kilobyte disk block. If something goes wrong and the system logger is hammered then the writes can lead to failure of the SD Card in short order (which makes things worse, as the logger then tries to write the I/O error messages at high priority).

A workaround

The system logger can be told not to sync files. Traditional system loggers do that using a hyphen before the log file name. So go through /etc/syslog.conf or equivalent and put a hyphen before every filename on the SD Card.

*.*;auth,authpriv.none  /var/log/syslog
*.*;auth,authpriv.none  -/var/log/syslog
A hack

The kernel has a ring-buffer logger. This is displayed using dmesg. I could use that for user-space messages too. The device /dev/kmsg will place messages into the dmesg ring buffer. I could alter /etc/rsyslog.conf to make /dev/kmsg the destination for all log messages:

*.info /dev/kmsg

That will loop, because the system logger is logging messages from dmesg to the system log, and the system log is dmesg, and .... So let's save some of the Raspberry Pi's 700Mhz for real work by setting /etc/syslog.conf to not listen to kernel messages by deleting the line:

$ModLoad imklog

But anyone can type dmesg. Allowing anyone to look at the super-secret log messages isn't the best idea. There is a sysctl to restrict looking at the kernel's ring buffer to root. Edit /etc/sysctl.d/local.conf to set that:

kernel.dmesg_restrict = 1

That's more like it:

$ dmesg
dmesg: klogctl failed: Operation not permitted

This works, but it is not really optimal. The kernel is very inefficient at writing messages: for some issues the only tool the authors have is the log messages, so they trade off efficiency for increasing the probability the log message will be seen. Kernel facilities also lack the resource allocation and control of user space: the message will be logged, whatever other more useful work is pending.

Doing it right

The needed programs are:

  • A daemon holding a ring-buffer of log messages and their access permissions.

  • A module for rsyslog which sends log messages to that daemon's ring-buffer. By hanging off rsyslog we get logging of journald output thrown in for free.

  • A command which requests log messages from that daemon's ring-buffer and displays them

This can be efficient (we're not absolutely desperate to see all messages come hell or high water) and fair (if the system is busy then logging can get a fair share). It would be useful for large embedded systems and embedded-like systems such as the RPi.

Tags: linux
  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded