Learning systemd

software

linux

Origin

systemd is a replacement for Linux previous init components, i.e SysVInit, BSD init, service and chkconfig.

The fundamental purpose of any init systsem is to initialize the components that must be started after the Linux kernel is booted (traditionally known as the “userland” components). The init system should also be able to manage services and daemon on the server at any given time when the system is running.

Red Hat is the inventor and primary booster of systemd.

Basic Unit Management

The most basic object that systemd manages and acts upon is a unit. Units can be of 12 different types, with the most common one being “service” i.e files that end with .service.

.service: describes how to manage a service or application on the server i.e how to start or stop the service, under which circumstances it should be automatically started, dependency information.
.device: kernel device names, which is a unit that has been designated as needing systemd management by udev or sysfs filesystem (not all devices will have .device files).
.mount: filesystem mountpoints. Entries within /etc/fstab can have units created automatically.
.automount unit files configure mountpoints that will be automatically mounted. They are named after their associated and matching .mount unit file to define the specifics of the mount.
.path unit files define a path that can be used for path-based activation. By default, a .service file with the same base name will be started when the path reaches the specified state. It uses inotify to monitor the path for changes.
Scope: external processes not started by systemd
.slice: a management grouping of multiple unit files. The name reflects its hierarchical position within the cgroup tree.
.snapshot: systemd saved state that is created automatically by systemctl snapshot, which will allow us to reconstruct the current state of the system after making changes. They should be used only temporarily and do not persist across sessions.
.socket: describes a network or IPC socket, or FIFO buffer that systemd uses for socket-based activation. These always have an associated .service file that will be started when the activity defined in the .socket file is seen.
.target A target unit is used to provide synchronization points for other units when booting up or changing states. They also can be used to bring the system to a new state. Other units specify their relation to targets to become tied to the target’s operations.
.swap: a swap unit file describes swap space on the system. It must defines the device or file path of the space.
.timer: systemd timer, similarly to a cron job for delayed and scheduled activation. There should be a matching unit which would start when the timer is reached.
.scope units are created automaticallty by systemd from information received from its bus interfaces. They are used to manage sets of system processes that are created externally.

To list all units of a specific types, we can do:

systemctl list-units --type <unit type name>

Basic Unit controls include:

Starting a service
```
sudo systemctl start something.service
```
Stopping a service
```
sudo systemctl stop something.service
```

Restart a service

sudo systemctl restart something.service

Reload a service without interrupting normal functionality:
```
systemctl reload something.service
```

To see an overview of the current state of a specific unit:

systemctl status something.service

systemd knows to look for .*service files for all service management commands, so we can type the shortcutted version which is systemctl reload something.

Checking

There are also methods for checking specific states:

systemctl is-active something

systemctl is-enabled something

systemctl is-failed something

Enabling and Disabling Units

Enabling and Disabling are to do with starting a unit at boot (or not). What both of these commands do is to hook the unit up (or remove the hook) to a certain boot “target”. Hooking a unit to a target triggers it to be run when that target is started.

sudo systemctl enable something.service

sudo systemctl disable something.service

Getting an Overview of the system state

Get all unit files that systemd has listed as “active”:

 systemctl list-units

List all units that systemd has loaded, or attempted to load into memory, including those that aren’t active:

systemctl list-units --all

List all of the units installed on the system, including those that systemd has not attempted to load into memory.

systemctl list-unit-files

`journalctl`

journalctl is a component of systemd that collects and manages journal entries from all part of the system. This is basically log information from applications and kernel.

The following command will list all entries from current and previous boots. That depends on how journald is configured and whether it has been configured to save logs from previous boot. Some distributions enable this by default, whilst others do not.

journalctl

adding -b to see logs from only current boot. -k to see only kernel messages (dmesg for example)

We can also use journalctl to see all journal entries for a specific unit by passing the -u option with the unit name.

journalctl -u something.service

Inspecting Units

To see low-level details of the unit’s setting on the system:

systemctl show something.service

To see the full content of a unit file:

journalctl cat something.service

To see the dependency tree of a unit:

systemctl list-dependencies something.service

(We can add the --all flag to ssee all dependent units recursively.)

Anatomy of a Unit File

The internal structure of unit files are organized within sections. Individual sections are denoted by a a pair of square brackets [ and ] with the section name enclosed within. Each section extends until the next section starts.

Section names are well defined and case-sensitive. If we need to add a non-standard section to be parsed by applications other than systemd, add X- prefix.

[Section]
Directive1=value
Directive2=value
...

We can override default system unit directives. For example if we have this in the system’s copy of a unit file:

Directive1=default_value

.. we can override that in something.service.d:

Directive1=

.. For more information about each section directive, see: https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files

Modifying Units

We can modify a unit files by:

appending to the file, which will override default settings: sudo systemctl edit something.service.
modifying the full content of the file, sudo systemctl edit --ful something.service.

After we do our changes, we should reaload systemd process itself so it can pick up the changes:

sudo systemctl daemon-reload

Targets

Targets in systemd are basically synchronization points that the server can use to bring itself into a specific state. What this allows is for the server to transition itself between different states.

Service and unit files can be tied to a target. Multiple targets can be active at the same time.

To see all targets on a system:

systemctl list-unit-files --type=target

To see the default target that systemd tries to reach at boot (which will starts all the units that have been tied to the target, along with all other units that make up the dependencies tree):

systemctl get-default

We can use set-default to change the default target that will be used at boot.

To see all units that are tied to a specific target: systemctl list-dependencies something.target.

We can do systemctl isolate something.target to stop all units that are not tied to the specified target. For example, if we are using a graphical environment with graphical.target active, we can shut down the whole graphical system and put it into a multi-user command line state by isolating the multi-user.target. Since graphical.target depends on multi-user.target, but not the other way around, all of the GUI units will be stopped.

Stopping or rebooting the server

The major states that a system can transition to can be shortcutted:

sudo systemctl poweroff

sudo systemctl reboot

sudo systemctl rescue

..most systems have aliases for these, which are..

sudo poweroff

sudo reboot

sudo rescue

Where are `systemd` files found?

The files that define how systemd will handle a unit can be found in various locations, each of which has a different priorities and implications.

The system’s copy of unit files are kept in /lib/systemd/system/. When software install unit files on the system, this is the location where they are placed by default. These files can be started and stopped on-demand during a session. These are the generic vanilla files that are maintained by upstream project’s maintainer that should work on any system with systemd. We should not edit files here. Instead we should override them if necessary, using another unit file location which will supersede the file in this location.

If we want to modify the way that a unit functions, the best location to do so is within /etc/systemd/system directory. Unit files that are found here take precedence over any of the other locations in the filesystem. This is the best and safest location to override a system’s copy of a unit file.

If we want to override specific directives from the system’s unit file only, what we can do is to provide unit file snippets within a subdirectory. These will append or modify the directives of the system’s copy of the unit file. This allows us to change the behaviour of only the specific things.

The correct wayt to do this is to create a directory with the unit file name, adding a .d at the end. For example, if we want to override something.service, create a dir something.service.d. Within this directory, a file ending with .conf can be used to override, or extend the attributes of the system’s unit file.

There is another location for run-time unit definitions at /run/systemd/system/. Files in this location take precedence over /lib.. but not /etc/... systemd itself uses this location for dynamically created unit files created at runtime. It can be used to change the system’s unit behaviour for the duration of the session. All changes in this dir will be lost when the server is rebooted.

Priority: /etc/systemd/system/ -> /run/systemd/system/ -> /lib/systemd/system/.