# Troubleshooting

## A service is continually switching between the "running" and "failed" states, and shows an uptime of 0 seconds (or 1 second).

This service is failing to start properly; investigate its log and check its run script.

## A service is stuck in the "starting", "failed", or "stopping" states.

The `service-control` command causes the service manager to send signals to the service process(es). Those processes include the ones that run in the "starting", "failed", and "stopping" states — namely the `start`, `restart`, and `stop` programs and anything that they in their turn run. So just terminate the process(es) with `service-control --terminate`.

Bear in mind that some "sysinit" services do things in their `start` programs that may take a long time. The "fsck@" service `fsck`s a volume in its "starting" state, for example. These long-running initialization actions are not necessarily stuck just because they take a long time.

## The system bootstrap never fully completes.

Troubleshoot this by:

- … looking at the logs, if possible. The system manager's own log output can be seen in `/run/system-manager/log/`.

- … watching the bootstrap process interactively. Start the system in rescue mode, then bring up the "normal" target interactively with:

  >     system-control start --verbose normal

Watch for things that are "BLOCKED" and note how far through the boot process `system-control` gets.

### `system-control` experiences a conflict

The most common conflict is one between kernel virtual terminals and user-mode virtual terminals — the `ttylogin@ttyv1` service conflicting with the `console-kvt-realizer@ttyv1` service. This is hard, but not impossible, to cause. Simply disable the one or the other service.

### Volumes are not mounted, and services needing those volumes to be mounted cannot start

The `mount@` services that mount volumes are not enabled in the base service bundles package.

Specifically, if you have your `/usr` and `/var` filesystems on separate volumes and you haven't enabled the `mount@-usr` and `mount@-var` services then many other services will fail not only to start but even to be found by the "normal" target when `system-control` goes looking for its dependencies.

You must enable these services. This can be done explicitly, or by the autoconfiguration system of `redo` scripts, which will create and enable mount services based upon the contents of `/etc/fstab`.

### Services remain blocked

In this scenario service A specifies that it is to be started after service B (or service B has specified that it is to start before service A), one or both services are directly or indirectly wanted by the "normal" target, and service B has failed to start.

Look at service B's own logs, and the output of

>     system-control status B

### The service manager reports errors when attempting to load a service

The most common cause of this is that you have an enabled service whose `supervise` directory is on a read-only filesystem. The service manager is thus unable to create/update the control/status API files in that directory.

Notice that "sysinit" services locate their `supervise` directories under `/run/service-bundles/early-supervise/` for this very reason. The point where they are loaded by the service manager is well before the point that the `/var` or `/` volumes are mounted read-write.

### `fsck` ran and then nothing else started

The initial non-interactive `fsck` of mounted filesystems usually runs in "preen mode", which proceeds relatively quickly. If, however filesystem tunable options cause a full `fsck` to run, `system-control` can sometimes time out waiting for the filesystem check services to finish. This is because there's a time limit to how long `system-control` will attempt to bring the system up.

The limit is there so that a system that fails to fully come up can be brought down. Without the time limit, the on-going attempts to bring the system up would defeat the process of trying to bring the system down, if an administrator chose to reboot. And this is the remedial action to take here, in fact. Simply reboot the system after all `fsck`s have successfully run, and the `monitor-fsck-progress` service has removed its user interface.

## An `/etc/fstab` entry is ignored.

The conversion utility ignores entries that are not marked as `rw`, `rq`, `ro`, or `sw`. Ensure that the entry is marked with a correct mount type.

The conversion utility uses the conventional C library functions `getfsent()` et al. to parse the `/etc/fstab` file. It does not use its own private custom parser. These C library functions do not behave as do the custom parsers in some other utilities.

In particular, note that `getfsent()` does not determine the `sw` status from the filesystem type field as some custom parsers do. The BSDs explicitly require, as stated in their `/etc/fstab` manuals, that swap entries be marked as `sw`. The same requirement actually exists in the GNU C library, but its manual does not mention this and some custom parsers deviate from the behaviour of the GNU C library and thereby make it erroneously appear that this is not necessary.

## Stuck modifier keys or mouse buttons in a user-space virtual terminal

The currently pressed set of modifier keys is shared amongst HID realizers (both keyboard and mouce) according to the user-space virtual terminal configuration that has been set up. Normally, realizers clear out their device's individual contributions to the overall aggregate keyboard state when they terminate. But if a realizer is forcefully terminated, or ABENDs, this will not happen.

Resetting keyboard modifier state is a simple matter of zeroing out the state file, however. Any modifiers currently physically pressed at that moment will appear released. But normal function will resume as soon as the human being stops holding down the keys. The counts of keyboards currently pressing a key are not permitted to go negative. A key release when the count is zero will just be ignored, and will not underflow.

This can be done quite effectively with the simple `truncate` command:

> `truncate -c -s 0 keyboards-aggregate/${FILENAME} mice/aggregate/${FILENAME}`

## GNOME Terminal refuses to start.

The GNOME Terminal command does little more than send its environment and arguments via the Desktop Bus to the `org.GNOME.Terminal` service. Unfortunately, this service is excessively fussy and easy to break.

- Despite the fact that the GNOME Terminal server is supposed to be capable of talking to multiple X displays, and so should start up in a "not connected to any displays" mode initially, it actually attempts to connect to an X display at startup. So not only must it have the `DISPLAY` environment variable set in its per-service environment, that must denote a currently active X display that will accept cookie-less connections from the GNOME Terminal service.

  This is just an outright bug.

  You can transfer the environment variables from another, already running, interactive session in another terminal emulator:

      system-control --user set-service-env org.gnome.Terminal DISPLAY "$DISPLAY"
      system-control --user set-service-env org.gnome.Terminal XAUTHORITY "$XAUTHORITY"

- The GNOME Terminal server requires that a UTF-8 locale be configured in its `LANG` environment variable, and refuses to start if it is not. It cannot be just the `C` locale. This is configured in the environment of the service definition itself, in the usual way:

      system-control --user set-service-env org.gnome.Terminal LANG en_GB.UTF-8

  An important proviso is that the named locale must exist as a locale definition on the system; the GNOME Terminal server requiring this, too. Some systems have a `C.UTF-8` locale definition (including those that have adopted the GNU C library project); some do not.

- The GNOME Terminal client must be able to talk to the correct Desktop Bus broker for the user, and so the `gnome-terminal` command will fail if invoked under the aegis of another user account, via `su`, `sudo`, or similar. This is because it uses the effective user ID that it is run under to construct the conventional location of the per-user runtime directory, and it (of course) accesses that runtime directory with the access rights of that effective user ID.

- The GNOME Terminal client requires that the Desktop Bus broker for the user be running in the first place. This is of course the per-user user service named `dbus`, which must be enabled:

      system-control --user enable dbus

  The system-level service that runs per-user service management in the first place must also be running.

- The GNOME Terminal server requires the ability to talk to a whole bunch of other per-user Desktop Bus servers. These must be enabled so that the per-user Desktop Bus broker can trigger the per-user service manager to auto-start them on demand.

  The services will include the GVFS Dæmon, GVFS HAL Monitor, GVFS Photo Volume Monitor (sic! For a terminal emulator.), GNOME Configuration Dæmon, and Assistive Technology Service Provider Accessibility Services Bus. You can usually see which ones GNOME Terminal server is attempting to auto-start by inspecting the log of the `dbus` service:

      system-control --user status --log-lines 40 dbus