service-bundle 5 file formats nosh
service-bundle service bundle directories

## Description

The service bundle mechanism is built from a fundamental mechanism that is upwardly compatible with daemontools and Bruce Guenter's daemontools-encore.

### Fundamental daemontools-compatible mechanisms

service-manager 1 manages services, which are defined by two directories somewhere in the filesystem, a supervise directory and a service directory, which do not in themselves require a connection to each other. The service manager itself neither knows nor cares where in the filesystem these directories are. That is the province of the utilities that feed control requests to it.

It is not necessary for the relationship between service directories and supervise directories to be one-to-one. One service directory can be shared amongst multiple services, as long as they each have an individual supervise directory.

The service-control 1 and service-status 1 commands directly operate upon services, and share a convention with service-dt-scanner 1 that the service directory is the parent directory of the supervise directory. However, it is not necessary for supervise directories to be subdirectories of service directories. Within a service bundle, supervise and service directories are siblings under the bundle's root directory, and these tools also understand that structure.

### Service bundles

The system-control 1 command takes those two subdirectories and places them together into a subtree, forming a service bundle. The contents of the bundle define dependencies and relationships amongst services, controlling both what services system-control 1 takes action upon when it is instructed to do a job and how it tells the service manager to plumb the standard inputs and outputs of services together.

It is not necessary for the I/O plumbing relationship between services to be exactly one "main" service feeding its output into one subordinate "log" service. The service manager permits arbitrary-length pipelines of services, as well as fan-in. (However, fan-in should be used sparingly as it generally causes more administrative headaches than it solves.)

### Flavours of service bundle

system-control 1 makes a distinction between services that are "services" and services that are "targets". Although there is no constraint enforced, and `system-control` simply follows symbolic links, by convention service bundle directories generally live under `/etc/service-bundles/services`, `/run/service-bundles/services`, or `/var/service-bundles/services`; and target bundle directories generally live under `/run/service-bundles/targets` or `/etc/service-bundles/targets`. There is a potential for "services" and "targets" to have the same names. system-control 1 allows the bundle name of a service to be suffixed with `.service` or `.target` if it is necessary to distinguish between two such conflicting service bundles.

As far as basic service management is concerned, there is no difference between "services" and "targets". A service generally has a fully-fledged dæmon process that runs, whereas a target often runs just a dummy placeholder program. A target generally has active dependencies, whereas a service generally only has passive dependencies. There are several conventional "standard targets" used in system administration.

## Service directories

A service directory is the current directory in which a service process is run. It contains:

- a `run` file, which is the executable file for the service process itself;

- a `start` file, which is the executable file to be run when a service is first brought up (but not when it is automatically restarted);

- a `restart` file, which is the executable file to be run when a service has ended (to determine whether it should automatically be restarted);

- a `stop` file, which is the executable file to be run when a service is finally taken down (but not when it is automatically restarted);

Although there is nothing to stop them from being binaries, the executable files are usually scripts interpreted by nosh 1, execlineb 1, or a shell. They set up various parts of the process state (using commands such as softlimit 1, setenv 1, setuidgid 1, and open-controlling-terminal 1) and then chain to the service program proper.

A service directory can also contain:

- ancillary files required by the service itself, varying from service to service. For examples:

  - A tcp-socket-accept 1 service could have an access-control database managed by ucspi-socket-rules-check 1.

  - Many services have `env` subdirectories read by envdir 1 in order to control dæmon process environment variables.

- further files used by other tools. For examples:

  - A `down` file indicates to system-control 1, service-is-enabled 1, and service-dt-scanner 1 that a service should not be auto-started at bootstrap.

  - A `remain` file indicates to system-control 1 that a service should be marked as "run on empty", so that it is considered still running even if it has no processes.

  - A `ready_after_run` file indicates to system-control 1 that a service should be considered "ready" after it has finished running, and has either remained in the running state with no processes or transitioned to the stopped state with a prior run recorded in its status.

  - A `use_hangup` file indicates to system-control 1 that a service should (additionally) be sent the `SIGHUP` signal when shutting it down.

  - A `no_kill_signal` file indicates to system-control 1 that a service should not be sent the `SIGKILL` signal when shutting it down.

These files are ignored by `service-manager`.

The service manager does not need write access to the service directory or to any of the executables within it. This permits service directories (as long as the services themselves do not require write access to their service directories) to reside on read-only volumes.

### Restart

Automatic restart is tailorable to individual services. If the `restart` program does not exist, or does not exit with a success (i.e. zero) status when run, the service `run` program is not restarted.

For the simplest cases `restart` can just be a (symbolic) link to `/bin/true` or `/bin/false`, to provide always-restart and never-restart services, respectively. (If using the nosh flavours of true 1 and false 1 do not use links to them. They will see themselves invoked under the unknown (to them) name `restart` and complain. Instead, write a short nosh 1 script.)

However, `restart` is invoked with two pieces of information, which together represent the most recent exit status of the `run` program, that allow finer control over the restart decision, if desired. The two pieces of information are its three command line arguments.

The first is a code, one of `exit`, `term`, `kill`, `abort`, or `crash`. This categorizes how `run` exited. Everything apart from `exit` denotes being terminated by an uncaught signal. `term` denotes the "good" termination signals `SIGTERM`, `SIGPIPE`, `SIGHUP`, and `SIGINT`. `kill` denotes `SIGKILL`. `abort` denotes `SIGABRT`, `SIGALRM`, or `SIGQUIT`. And `crash` is everything else.

The second is either (for `exit`) the decimal exit status of the process or (for everything else) a symbolic designation (falling back to a decimal code) of the specific signal, if the first argument is not specific enough to make a decision.

For convenience, the third is (for other than `exit`) always the decimal code of the specific signal.

## Supervise directories

A supervise directory provides the control/status API for the service supervisor. It contains:

- an `ok` FIFO that does nothing more than signify that the service manager has loaded the service;

- a `control` FIFO through which commands to control the individual service process (for which see service-control 1) are sent;

- a `status` file that contains a record of the service process ID, start time, and control state; and

- a `lock` file (compatible with setlock 1) that prevents the service manager from re-using an active supervise directory.

Other tools may use further files in a supervise directory. As with extraneous files in service directories, these files are ignored by `service-manager`.

The service manager requires read-write access to these files, and write access to the supervise directory itself, as it creates the files if they do not exist to start with. However, it does not require write access to the supervise directory once the files have been created. (The supervise 1 program in daemontools repeatedly re-creates the `status` file, in contrast.)

Control of services and access to service status is thus subject to ordinary permissions and ACLs on these files.

### The status file

Bernstein's daemontools employs an 18-byte `status` file. daemontools has no notion of "starting", "failing", or "stopping" states for services, and its status file provides only simple binary "up" or "down" state information. Guenter's daemontools-encore employs a 19-byte `status` that includes extra state information for the aforementioned states. `service-manager` employs an 87-byte `status` that adds exit status and timestamp information for the start, run, restart, and stop programs. The `status` file contents are:

1. 12-byte TAI64N timestamp of last service status change event.

2. 4-byte current main process ID in host byte order, `0` meaning no process and `-1` meaning process #0.

3. 1-byte paused flag, if the dæmon process is in the stopped state.

4. 1-byte pending command flag, which is an ASCII-encoded character:

   `d`

     down

   `u`

     up

   `o`

     once

   `O`

     at most once

5. 1-byte daemontools-encore status (for details see later) which is a binary number:

   `0`

     stopped

   `1`

     starting

   `2`

     started

   `3`

     running

   `4`

     stopping

   `5`

     failed

6. 68 bytes comprising 4 groups of status information for the last `start`, `run`, `restart`, and `stop` programs to terminate:

   1. 1-byte code for termination status, which is a binary number:

      `0`

        Not yet terminated, other fields should not be considered valid.

      `1`

        Terminated normally with an exit code.

      `2`

        Terminated by a signal.

      `3`

        Terminated by a signal, and core dumped.

   2. 4-byte exit code or signal number, in host byte order.

   3. 12-byte TAI64N timestamp.

### Service states

If a service is not known to the service manager, it is in an unloaded state, and none of the information in the `status` file is valid. Otherwise, service states in that file follow the daemontools-encore paradigm:

stopped

  No service process is executing.

starting

  The service's `start` program is currently executing.

started

  This state is not used.

running

  The service's `run` program is currently executing, or no program is currently executing in a run-on-empty service that was running the `run` program immedately prior.

failed

  The service's `restart` program is currently executing.

stopping

  The service's `stop` program is currently executing.

## Bundle directories

A bundle directory contains the command information, dependency information, relationship information, and control/status API for a service. It contains various subdirectories (which can be symbolic links to directories elsewhere):

- The `service/` and `supervise/` subdirectories are (respectively) the service and supervise directories for the service, comprising the command information and the control/status API. (service-dt-scanner 1 implements a slight extension here, for compatibility.)

  When bundles are on a read-only filesystem, such as a CD-ROM, `supervise` must be a symbolic link pointing to a directory on a read-write filesystem, since supervise directories must be writable. One might choose a subdirectory of `/run/service-bundles/early-supervise/` for example. `system-control` attempts to create the target of such a symbolic link before attempting to load the service, to avoid a chicken-and-egg situation with "sysinit" services that are set up this way.

- The `wants/`, `requires/`, `conflicts/`, `wanted-by/`, `stopped-by/`, and `required-by/` subdirectories comprise dependency or dependency installation information. Each contains a set of symbolic links to other bundle directories for other services.

  - `wants/` contains links to services that the current service "wants". Starting the current service implies starting the linked-to service as well.

  - `requires/` contains links to services that should be "required-by" the current service when it is enabled. Enabling the service places it in the targets' "required-by" lists, and disabling the service removes it.

  - `conflicts/` contains links to services that "conflict" with the current service. Starting the current service implies stopping the linked-to services.

  - `required-by/` contains links to services that "require" the current service. Stopping the current service implies stopping the linked-to services as well.

  - `wanted-by/` contains links to (standard) targets that should "want" the current service when it is enabled. Enabling the service places it in the targets' `wants/` lists, and disabling the service removes it.

  - `stopped-by/` contains links to (standard) targets that should "conflict" with the current service when it is enabled. Enabling the service places it in the targets' "conflicts" lists, and disabling the service removes it.

- The `after/` and `before/` subdirectories comprise relationship information. Relationships govern the ordering between the individual actions in a start/stop job.

  - `after/` contains links to services that should be started after/stopped before the current service when executing a start/stop job.

  - `before/` contains links to services that should be started before/stopped after the current service when executing a start/stop job.

Relationships are not the same as dependencies. A service can "want" another service without having any ordering against it with respect to start/stop actions. Indeed, this is a common case for system targets. The `normal` target, for example, `wants/` the `server`, `workstation`, `multi-user`, `static-networking` , and `users` targets (and by extension everything that they in their turn `wants/`), but all of the wanted services are started in parallel with the `normal` target itself.

Similarly, a service can be ordered with respect to another service without having a dependency from it. For example, both "log" and "main" services are "wanted" by the `server` and `workstation` targets, explicitly. The individual "main" services, therefore, only need to be ordered after their corresponding "log" services, and do not `wants/` them. ("log" services are first-class citizens in this respect.)

There are no implicit relationships or dependencies in a bundle. The relationships and dependencies are exactly what is explicitly in the filesystem. `system-control` will fail if the recorded relationships and dependencies are self-contradictory or impossible (such as a service that conflicts with itself, for example).

## Bundle search paths and conventional locations

Several system-control 1 subcommands take a list of bundle names. Each name can be the full absolute pathname of a bundle directory or a relative pathname with a directory prefix, in which case it is used as-is, or a bundle directory name without any directory prefix, in which case a set of standard locations is searched for a bundle directory by that name. The search algorithm is as follows:

- A name that ends in `.target` is searched for (sans the `.target` suffix) in the standard locations for target bundles:

  - System-wide target directories:

    1. `/usr/local/etc/service-bundles/targets/`

    2. `/etc/service-bundles/targets/`

    3. `/var/local/service-bundles/targets/`

    4. `/run/service-bundles/targets/`

    5. `/var/service-bundles/targets/`

  - Per-user target directories:

    1. ``$XDG_RUNTIME_DIR`/service-bundles/targets/`

    2. ``$HOME`/.config/service-bundles/targets/`

- A name that ends in `.service` or `.socket` is searched for (sans those suffixes) in the standard locations for service bundles:

  - System-wide service directories:

    1. `/usr/local/etc/service-bundles/services/`

    2. `/etc/service-bundles/services/`

    3. `/var/local/service-bundles/services/`

    4. `/var/local/sv/` (‡)

    5. `/run/service-bundles/services/`

    6. `/var/service-bundles/services/`

    7. `/var/sv/`

       (‡)

    8. `/var/svc.d/`

       (‡)

    9. `/service/`

       (‡)
    Items marked ‡ are searched for backwards compatibility with earlier versions of the nosh toolset, with Wayne Marshall's service directory convention, or with Daniel J. Bernstein's service directory convention. Do not place modern service bundles in these locations unless coexisting with those systems.

  - Per-user service directories:

    1. ``$XDG_RUNTIME_DIR`/service-bundles/services/`

    2. ``$HOME`/.config/service-bundles/services/`

- Any other name is searched for (as-is) in the standard locations for service and for target bundles.

If the path search fails, the name is simply assumed to name a bundle directory in the current directory.

## Standard targets

Several standard targets are defined. They conventionally live in the `/etc/service-bundles/targets/` directory. Various subcommands of system-control 1 employ them for system management. The subcommands send a signal to the system-manager 1, which in turn spawns another system-control 1 directly commanding the start or stop of one of these standard targets.

### startup

These standard targets are (indirectly) started by the `init` subcommand.

`normal`

  This target is the normal target started by the `init` subcommand. This should bring up with it all of the services involved in the normal operation of the system, and an administrator-defined set of multiple login user interfaces that may include both TUI and GUI login prompts.

  It `wants/` the `workstation`, `server`, and `multi-user` targets. Thus starting it starts the union of all services wanted by those targets. It is not ordered with respect to those targets. So in the absence of orderings in between the individual services wanted by those targets, defined in their bundles, all wanted workstation, server, and multi-user services are brought up in parallel.

`emergency`

  This target is the target started by the `init` subcommand with the -b option. Starting it is intended to bring up the system in "emergency mode", bringing up no services, mounting no (non-system) volumes, and bringing up just "emergency mode" services, such as a secure login on the default terminal.

  It `wants/` no other targets or services. So in "emergency mode" (if nothing else has been started) only the system filesystems are available, and not necessarily even read-write.

`rescue`

  This target is the target started by the `init` subcommand with the -s option. Starting it is intended to bring up the system in "rescue mode", bringing up with it the fundamental system initialization services and a secure login user interface.

  It `wants/` the `sysinit` target, but not the `basic` target.

`update`

  This target is the target started by the `init` subcommand with the --update option. Starting it is intended to bring up the system in "update mode", bringing up with it the fundamental system initialization services and the system update utility.

  It `wants/` the `sysinit` target, but not the `basic` target.

### shutdown

The `poweroff`, `powercycle`, `halt`, and `reboot` subcommands invoke several standard targets indirectly.

`poweroff`
`powercycle`
`halt`
`reboot`

  These four targets both `wants/` the `shutdown` target and order themselves to start after it. Thus starting them shuts down all services and then, once this has been done, starts these targets. They are usually started by the System Manager, in response to system control commands requesting system reboot, halt, or power off. When they (finally) start, they invoke the `reboot`, `halt`, `powercycle`, and `poweroff` subcommands of `system-control` with the -f command-line option. This signals the System Manager to enact the actual reboot, halt, power-cycle, or power-off action.

Do not start these targets directly with the `start`, `reset`, or `activate`. They will end up terminating the very `system-control` process that is directing the state change, mid-change. Always use the `poweroff`, `powercycle`, `halt`, and `reboot` subcommands.

For example: When system-manager 8 is process #1, system halt operates as follows:

1. The superuser runs `system-control` halt. This sends `SIGUSR1` (on BSD) or `SIGRTMIN + 3` (on Linux) to the system manager process.

2. In response to that signal, the system manager runs `system-control` activate halt.

3. `system-control` does everything necessary for activating the `halt` target, including stopping all other (conflicting) services and targets. In the service control ordering starting the `halt` target comes after processing every other service, because of the ordering between `halt` and `shutdown` and between `shutdown` and everything else.

4. When the `halt` target is finally started it runs `system-control` halt -f. This sends `SIGRTMIN + 13` to the system manager process.

5. In response to that signal, the system manager performs the finalization action of actually telling the operating system kernel to halt the system, after various standard system shutdown actions such as flushing the filesystem cache.

Other standard targets are started by telinit 1.

### service collections

User-supplied services and targets that are auto-started at system bootstrap should list themselves in the `wants/` and `conflicts/` lists of subordinate targets (via the `enable` and `disable` subcommands). These targets are usually not started explicitly.

`workstation`

  All services that normally form part of bringing the system up as a "workstation" should be listed in this target's `wants/` list. Such services will include things such as the cron 1 dæmon. This target `wants/` the `basic` target, so starting it (directly or indirectly) will bring the basic system up. It is a collection that is used to group services, but is not an ordering milestone and has no `before/` or `after/` relationships.

`server`

  All services that normally form part of bringing the system up as a "server" should be listed in this target's `wants/` list. Such services will include things such as HTTP/FTP/SMTP dæmons, and dæmons that provide services over the network. This target `wants/` the `basic` target, so starting it (directly or indirectly) will bring the basic system up. It is a collection that is used to group services, but is not an ordering milestone and has no `before/` or `after/` relationships.

`multi-user`

  All services that normally form part of bringing up multi-user logins should be listed in this target's `wants/` list. Such services will include things such as the login services for terminals (real and virtual) and X. This target `wants/` the `basic` target, so starting it (directly or indirectly) will bring the basic system up. It is a collection that is used to group services, but is not an ordering milestone and has no `before/` or `after/` relationships.

`shutdown`

  Starting this target is intended to shut down all running normal services. All normal services should be in this target's `conflicts/` list. It is also an ordering milestone, in that all services listed in its `conflicts/` list should also be listed in its `after/` list. `shutdown` is then listed in the `after/` list of the `poweroff`, `powercycle`, `halt`, and `reboot` target to ensure that they are only started after the system has entered "shutdown mode".

`unmount`

  Starting this target is intended to unmount all filesystems, both local and remote. All `mount@*` services should be in this target's `conflicts/` list. `unmount` is then listed in the `after/` list of the `poweroff`, `powercycle`, `halt`, and `reboot` target to ensure that they are only started after the system has unmounted all filesystems.

### core subordinate targets

Core services and targets are incorporated into a set of further subordinate targets.

#### Pure collections

These core targets are solely collections. They are a mechanism for listing a `wants/` dependency on a group of services without explicitly enumerating every single member of the group. They are not ordering milestones and have no `before/` or `after/` relationships.

`basic`

  This target implies the "basic" system, excluding all workstation, server, and multi-user services. It `wants/` the `sysinit` target.

`static-networking`

  This target implies the static setup/teardown of network interfaces and routing. It `wants/` all of the generated `static_protocol@id` services.

`sysinit`

  All fundamental system initialization services and steps should be listed in this target's `wants/` list. Such services will include things such as the `hostname` service that runs set-dynamic-hostname 1, the `machine-id` service that runs setup-machine-id 1, the `local-fs` target, and the `remote-fs` target.

`swapauto`

  This target is wanted by "sysinit". All "auto" "swap" services should be listed in its `wants/` lists.

`swaplate`

  This target is wanted by "sysinit". All "late" "swap" services should be listed in its `wants/` lists.

`dumpauto`

  This target is wanted by "sysinit". All "dump" services should be listed in its `wants/` lists.

`finish-install`
`finish-update`

  These targets `wants/` the various services that finish a system install or a system upgrade.

#### Pure ordering milestones

These core targets are not collections; their `wants/` lists should be empty. They are solely ordering milestones, listed in the `before/` and `after/` lists of other services. They exist because services that need to occur before them do not have another way to hold back the services that need to occur after they become ready.

Conventionally, they are brought in only if the services that want to order `before/` them are themselves brought in, and it is those services that list these milestones in their `wants/` lists. This is only a convention, however. There is no technical reason that it could not be all of the services that want to order `after/` the milestones that list them in their `wants/` lists. It is simply the case that something must bring the target in, for the orderings to actually take effect.

`local-fs-pre`

  This "local filesystems precursor" target exists because system initialization services that need to occur before any filesystem checks or mounts, usually special services that initialize an interactive UI for fsck 1, do not have another way to hold the checks/mounts from commencing until they are done. Such services should list this target in their `wants/` and `before/` lists. Local filesystem checks, decryptions, and mounts should list this target in their `after/` lists, so that they order relative to it if the target is brought in but do not themselves bring it in.

`fs-servers`

  This "filesystems servers" target exists in order to ensure that remote filesystem mounts of volumes on the local machine work, by ordering them after the startup of any filesystem servers (e.g. SAMBA or NFS server dæmons). Such server services should list this target in their `wants/` and `before/` lists. Remote filesystem mounts should list this target in their `after/` lists, so that they order relative to it if the target is brought in but do not themselves bring it in.

`multi-user-pre`

  This "multi-user precursor" target exists for services that need to start/execute before any multi-user login. Such services should list this target in their `wants/` and `before/` lists. Login services should list this target in their `after/` lists.

`name-services`

  This target exists because some name/address services lookup clients cannot be caused via the client-server communications mechanism itself to block indefinitely until the server is ready; and so have to have an explicit ordering to order them after any respective servers. One lists `name-services` in the `wants/` and `after/` list of a service or target to ensure that it is only brought up after such name/address lookup services are available. All services that are involved in providing local name/address lookup services should be listed in its `after/`.

`databases`

  This "database servers" target exists because some database clients cannot be caused via the client-server communications mechanism itself to block indefinitely until the server is ready; and so have to have an explicit ordering to order them after any respective servers. One lists `databases` in the `wants/` and `after/` lists of a service or target to ensure that it is only brought up after such database services are available. All services that are involved in providing database services should list it in their `before/` lists.

`machine-id`

  One lists `machine-id` in the `wants/` and `after/` lists of a service or target to ensure that it is only brought up after the machine ID is available. All services that are involved in setting up the machine ID should list it in their `before/` lists.

`hostname`

  One lists `hostname` in the `wants/` and `after/` lists of a service or target to ensure that it is only brought up after the dynamic hostname has been initialized. All services that are involved in initializing up the dynamic hostname should list it in their `before/` lists.

`persistent-clock-loaded`

  This target exists so that time synchronization services can wait until the system clock has been initialized from some internal persistent clock that keeps, or at least statically saves the latest runtime value of, the current time over a powerdown or a restart. All services that are involved in loading the time from internal persistent clocks should list it in their `wants/` and `before/` lists, and only alter the system clock before they enter the running/ready state, so that they cease controlling the system clock before time synchronization services take over. All services that are involved in time synchronization should list it in just their `after/` lists, so that they do not begin controlling the system clock until the hardware clock has been read, and so that they order relative to the target if it is brought in but do not themselves bring it in.

`system-time-synchronized`

  One lists `system-time-synchronized` in the `wants/` and `after/` lists of a service or target to ensure that it is only brought up after the point at which time synchronization services using dynamic external clocks are running. All services that are involved in such time synchronization should list it in their `before/` lists.

  Note that because of the dependencies of the synchronization services themselves, this can potentially involve an ordering after an arbitrary number of other milestones, including the local file system mounts, networking initialization, and virtual machine guest services.

#### Both collections and ordering milestones

Generally, with some exceptions, all services listed in these targets' `wants/` lists should also be listed in their `after/` lists.

`local-fs`

  This "local filesystems" target lists all system initialization services and steps for checking and mounting local filesystems, including remounting the root filesystem in read-write mode. One can thus list `local-fs` in the `after/` list of a service or target to ensure that it is only brought up after local filesystems have been mounted and checked.

  Services encompassed by this target may not rely upon filesystems being checked and mounted, nor upon the root filesystem being mounted read-write. To handle the latter, in particular, their `supervise/` subdirectories should be symbolic links to a suitable subdirectory of `/run/service-bundles/early-supervise`.
`remote-fs`

  This "remote filesystems" target lists all system initialization services and steps for checking and mounting remote filesystems. One can thus list `remote-fs` in the `after/` list of a service or target to ensure that it is only brought up after remote filesystems have been mounted and checked.

`reboot-after-install`
`reboot-after-update`

  These target `wants/` the `reboot` target. They are ordering milestones, with all `finish-install`/`finish-update` services expected to order themselves `before/` these targets.

## Author

Jonathan de Boyne Pollard