cyclog 1 user commands nosh
cyclog safely log standard input to status files and log directories
`cyclog` --max-file-size max-file-size --max-total-size max-total-size --margin margin directories

## Description

`cyclog` is a simple logging utility that logs everything that it receives on its standard input to a set of log file directories named as its command arguments. It automatically timestamps logs, produces time-sortable log file names, rotates log files, and caps log directory sizes.

At startup, `cyclog` attempts to open all of the directories. If they are directories (or symbolic links to the same), it treats them as log directories. Otherwise, if they don't exist or are other types of file such as regular files, device files, or FIFOs, it aborts startup without reading from standard input.

It keeps a file descriptor open to each log directory for its lifetime, and accesses files relative to that open file descriptor. Log directories can thus be renamed as it is running, and it will continue to use the original directory, wherever it is renamed to.

For convenience, both max-file-size and max-total-size can comprise a number followed by a suffix `Gi`, `G`, `Mi`, `M`, `Ki`, or `k` denoting multipliers of 1073741824, 1000000000, 1048576, 1000000, 1024, and 1000 respectively.

### Log directory

A log directory is compatible with the log directories employed by daemontools' and daemontools-encore's multilog 1, s6's s6-log 1, and runit's svlog 1. It contains:

- a `lock` file (compatible with setlock 1) that prevents two logging processes from sharing an active log directory;

- a set of old log files, named as `@` followed by a TAI64N timestamp in external form (16 hexadecimal digits of seconds and 8 hexadecimal digits of nanoseconds) and either `.s` (safely written to disc) or `.u` (aborted whilst writing to disc and possibly incomplete); and

- a `current` file where incoming log data are currently being appended.

An active `current` file, or an old log file or a `current` file that was not written to disc by `cyclog` properly shutting down, has permissions 0644. A `current` file or an old log file where `cyclog` properly shut down has permissions 0744. (See the Compatibility section.)

`cyclog` shuts down when either it sees EOF on its standard input, or it receives one of the "good" termination signals (`SIGTERM`, `SIGPIPE`, `SIGINT`, or `SIGHUP`). At shut down, `cyclog` writes all pending data that it has already read from standard input, flushes `current` to disc, and changes `current`'s permissions. At start up, it checks `current`'s permissions, creating the file with permissions 0744 if it does not already exist. If they indicate proper shutdown, as they will also do if the file was created because it did not exist, it carries on. If they indicate that the file was not properly written at shutdown, it performs automatic log rotation to prevent appending to a potentially corrupt file (that may be sparse or contain ranges of zeroed blocks).

### Reliable piped input

In the normal case, the standard input of `cyclog` will be a pipe; and both ends of the the pipe will be open in a long-lived supervisor process such as service-manager 1. `cyclog`, as aforementioned, ensures that it writes all data read from its standard input, even if it is terminated by a (good) signal.

Thus no log data are lost, even if `cyclog` is shut down and restarted.

### Automatic log rotation

`cyclog` performs automatic log rotation when it writes a linefeed within margin bytes of the maximum size (max-file-size) of the `current` file, when it is about to exceed that size, when it receives a `SIGALRM`, or when it finds an improperly written to disc `current` file at startup. When recovering from an improperly finalized `current`, it simply renames it to a timestamped `.u` name. Otherwise, it renames it to a timestamped `.u` name, flushes it to disc, changes its permissions, and then renames it to a timestamped `.s` name. In both cases, it then creates a new `current` file. The TAI64N timestamp of an old log file is the timestamp of when `cyclog` rotated `current` to that file.

At log rotation, and also at startup, it checks to ensure that the total size of all log files in the directory does not exceed the maximum total size (max-total-size). (It only totals the sizes of `current` and old log files. Other files, not managed or created by `cyclog`, are ignored.) If the total exceeds that maximum, it deletes each old log file with the numerically lowest name until either the total is less than the maximum or there is only the `current` file left.

Thus the maximum size of all log files at any time is max-total-size (the total size after the last rotation) plus max-file-size (the data written since that rotation). The default max-file-size is 16MiB and the default max-total-size is 1GiB.

`cyclog` totals file sizes rather than space allocated on disc. The amount of space allocated to all files may be, depending from the filesystem type and the maxima chosen, higher or lower than the space usage calculated by `cyclog`.

### Timestamps

`cyclog` writes a timestamp at the beginning of every line written to `current`, which is the time when it processes the beginning of a line. This is slightly after it has read the beginning of that line. The timestamp is in TAI64N external form (16 hexadecimal digits of seconds and 8 hexadecimal digits of nanoseconds), which can be converted to human-readable form using tai64nlocal 1.

`cyclog` uses the `CLOCK_REALTIME` clock of the clock_gettime 2 system call. On many systems this has nanosecond resolution.

`cyclog` attempts to generate real TAI timestamps. On a Linux system where it detects an Olson "right" timezone currently in use, it knows that the system clock is TAI seconds since the Epoch and performs a simple conversion. On other Linux systems, and on BSDs, it assumes that the system clock is UTC seconds since the Epoch and attempts to correct for (known) UTC leap seconds in order to determine TAI.

## Compatibility

A cyclog 1 utility was part of daemontools version 0.53. This utility is a workalike, with some minor additions to automatic log rotation and changes to log directory semantics.

daemontools later replaced cyclog 1 with multilog 1, which treats its command arguments as a script and has the abilities to do pattern matching and run external commands. `cyclog` is intended for the commonest, simple, use cases of logging services where there is just one plain logging directory with no bells and whistles. As such, it does not implement a script syntax, does not have pattern matching, and does not invoke other programs at all.

The daemontools version 0.53 cyclog 1 used permissions 0644 and 0444 to flag files that had not been written to disc. `cyclog` adopts the convention employed by the later multilog 1, and thence s6-log 1 and svlog 1. This convention changed to using the owner execute permission bit rather than the owner write permission bit as the flag. `cyclog` sets and clears the execution bit as appropriate for interoperability with those tools, and uses it itself for the same purpose. multilog 1 altered the convention in order to cope with scenarios where it would end up trying to open a read-only file for writing.

The daemontools version 0.53 cyclog 1 used the TAI64 timestamp of when a log file was started as its timestamp. Again `cyclog` adopts the convention employed by later tools.

daemontools' and daemontools-encore's multilog 1 uses a timer with only microsecond resolution, and for the same events will thus produce different timestamps to `cyclog`.

See the timestamps section of the nosh Guide for detailed information on the differences from daemontools, daemontools-encore, and other toolsets in how `cyclog` handles TAI.

## Author

Jonathan de Boyne Pollard