Systemd

Cheat sheet

Task	Command
list all failed units on the system	systemctl list-units --failed
Return code denotes if a service is down	systemctl is-failed nginx.service
show the status of a service/unit/timer	systemctl status ssh.service
show a service in detail	systemctl show vboxweb.service
kernel messages	journalctl -kjournalctl -ab
follow messages	journalctl -f
logs for a specific unit	journalctl -u ssh.service
Analyze why things are slow	systemd-analyze blame systemd-analyze critical-chain

Runlevels vs Targets

Unlike the old init system with different runlevels, Systemd defines sets of services to run inside targets.

Instead of using init 5:

# systemctl isolate graphical.target

or init 3:

# systemctl isolate multi-user.target

Journal

Systemd keeps tracks of service logs.

Clearing Journal

Clear all the entries by running:

# journalctl --vacuum-time=1seconds

Or

# journalctl --vacuum-size=1M

Limiting Log Sizes

Edit /etc/systemd/journald.conf and define a value for SystemMaxUse. Eg. SystemMaxUse=100M.

Tasks

Creating a Timer

You can use systemd to create a timer which can periodically run something, similar to Cronjobs.

A systemd timer can trigger a service. To make use of a systemd timer, you will need to create the service definition (that tells systemd how to run what you want to run) and then the timer definition itself (which tells Systemd when to run it, and how often).

For our service component, we would want a one-shot service that just executes a script. Create a service at /etc/systemd/system/gather-metrics.service:

[Unit]
Description=Logs system statistics to the systemd journal
Wants=gather-metrics.timer

[Service]
Type=oneshot
ExecStart=/bin/gather-script.sh

[Install]
WantedBy=multi-user.target

For our timer component, we want to run this every 10 minutes. Create a timer at /etc/systemd/system/gather-metrics.timer:

[Unit]
Description=Gathers metrics from this server
Requires=gather-metrics.service

[Timer]
Unit=gather-metrics.service
OnCalendar=*-*-* *:0/10:00

[Install]
WantedBy=timers.target

The OnCalendar specification takes the form of "DOW YYYY-MM-DD HH:MM:SS". An asterisk is a wildcard and matches on any value. The timer will trigger on the next date that matches this format.

Other useful specifications are:

OnCalendar	The job will run...
*-*-* *:*:00	Every minute
*-*-* *:0/10:00	Every 10 minutes
*-*-* 00:15:30	Every day at 12:15:30 AM
Weekly, or Mon *-*-* 00:00:00	Every Monday at 00:00:00
Mon *-05~03	On the next Monday which is 3 days from the end of May
Mon..Fri *-08~04	On the next weekday which is 4 days from the end of August

Creating a Service

Creating a service in systemd requires writing a .service file. The service file defines the service, any dependencies, and what to actually run. Service files are typically placed in /etc/systemd/system/. The most basic service file will look something like this:

[Unit]
Description=Run a script that does not fork

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
ExecStart=/root/scripts/foreground-script.sh
Restart=always

We are using Type=simple because the bash script we're executing doesn't fork or daemonize itself. For programs that do daemonize or run in the background, use Type=oneshot.

Other options that may be of interest are listed below.

options	behavior
Restart=always	restart when the program exits, regardless of the exit code.
User=bob Group=bob	Run the service with a particular user / group

Once the file has been created, reload systemd and enable the service:

## Reloads the service files so systemd is aware of the new one
# systemctl daemon-reload
# systemctl enable mag-read

SystemV-like Init Scripts

If you have custom packages that make use of the old SystemV startup scripts, you can use the following files to make use of such scripts.

In /usr/lib/systemd/system/startup.service

[Unit]
Description=Custom SystemV-like Startup
After=network.target
ConditionFileIsExecutable=/usr/local/sbin/custom-startup-initd

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/custom-startup-initd
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
SysVStartPriority=99

[Install]
WantedBy=multi-user.target

A few things of interest.

• The ConditionFileIsExecutable will only run the script if it's executable. It basically does a test -x on the file before this continues.
• RemainAfterExit will cause systemd to show the service as active even after the script has finished execution.

In /usr/local/sbin/custom-startup-initd:

#!/bin/sh
StartupDir="/usr/local/boot/startup"

# Run all scripts starting with 'S'
for i in `ls $StartupDir/S*` ; do
        ScriptName=`basename $i`
        sh $StartupDir/$ScriptName
done

# Success exit code.
exit 0

Make the directory:

# mkdir -p /usr/local/boot/startup/

Then place your scripts in the startup directory starting with the letter 'S'.

Enable the new service by running:

# systemctl enable startup
# systemctl start startup

Verify this:

# systemctl status startup.service

Editing a service

You can create overrides to an existing service by running systemctl edit something.service.

For example, to make Docker start after ZFS is ready, add the following overrides:

# systemctl edit docker.service
After=zfs-mount.service 
Requires=zfs-mount.service 
Wants=zfs-mount.service 
BindsTo=zfs-mount.service

This will create an override.conf file under /etc/systemd/system/docker.service.d/override.conf.