Systemd

Cheat sheet

Task	Command
list all failed units on the system	systemctl list-units --failed
Return code denotes if a service is down	systemctl is-failed nginx.service
show the status of a service/unit/timer	systemctl status ssh.service
show a service in detail	systemctl show vboxweb.service
kernel messages	journalctl -kjournalctl -ab
follow messages	journalctl -f
logs for a specific unit. Eg. SSH	journalctl -u ssh.service
logs for a specific syslog facility. Eg. authpriv and auth logs:	journalctl -q SYSLOG_FACILITY=10 SYSLOG_FACILITY=4
Analyze why things are slow	systemd-analyze blame systemd-analyze critical-chain

Switch and change targets (runlevel)

In the traditional init system such as SystemV, whether a system boots into a graphical interface or to a text-based console is determined by the default runlevel. In systemd, the idea of runlevels has been replaced by targets. A target is simply a collection of units (services) that should start. Now, to change a system's 'runlevel', we need to use tell systemd to change its target.

Instead of using init 5 to start a graphical system:

# systemctl isolate graphical.target

or init 3 for a text-based system:

# systemctl isolate multi-user.target

You can see the default target:

# systemctl get-default
graphical.target

You can change the default target:

# systemctl set-default multi-user.target
# systemctl set-default graphical.target

Journal

Systemd keeps tracks of service logs.

Clearing Journal

Clear all the entries by running:

# journalctl --vacuum-time=1seconds

Or

# journalctl --vacuum-size=1M

Limiting Log Sizes

Edit /etc/systemd/journald.conf and define a value for SystemMaxUse. Eg. SystemMaxUse=100M.

Tasks

Creating a Timer

You can use systemd to create a timer which can periodically run something, similar to Cronjobs.

A systemd timer can trigger a service. To make use of a systemd timer, you will need to create the service definition (that tells systemd how to run what you want to run) and then the timer definition itself (which tells Systemd when to run it, and how often).

For our service component, we would want a one-shot service that just executes a script. Create a service at /etc/systemd/system/gather-metrics.service:

[Unit]
Description=Logs system statistics to the systemd journal
Wants=gather-metrics.timer

[Service]
Type=oneshot
ExecStart=/bin/gather-script.sh

[Install]
WantedBy=multi-user.target

For our timer component, we want to run this every 10 minutes. Create a timer at /etc/systemd/system/gather-metrics.timer:

[Unit]
Description=Gathers metrics from this server
Requires=gather-metrics.service

[Timer]
Unit=gather-metrics.service
OnCalendar=*-*-* *:0/10:00

[Install]
WantedBy=timers.target

The OnCalendar specification takes the form of "DOW YYYY-MM-DD HH:MM:SS". An asterisk is a wildcard and matches on any value. The timer will trigger on the next date that matches this format.

Other useful specifications are:

OnCalendar	The job will run...
*-*-* *:*:00	Every minute
*-*-* *:0/10:00	Every 10 minutes
*-*-* *:00:00	Every hour
*-*-* */6:00:00	Every 6 hours
*-*-* 00:15:30	Every day at 12:15:30 AM
*-*-* 9-17:00:00	Every day between 9 AM and 5 PM each hour, on the hour
* *-01,04,07,10-01 00:00:00	Every quarter (on the first of January, April, July, October) at 00:00:00
* *-01-01 00:00:00	Every year (on the first of January) at 00:00:00
Weekly, or Mon *-*-* 00:00:00	Every Monday at 00:00:00
Mon...Fri *-*-* 00:00:00	Every weekday at 00:00:00
Mon *-05~03	On the next Monday which is 3 days from the end of May
Mon..Fri *-08~04	On the next weekday which is 4 days from the end of August

Creating a Service

Creating a service in systemd requires writing a .service file. The service file defines the service, any dependencies, and what to actually run. Service files are typically placed in /etc/systemd/system/. The most basic service file will look something like this:

[Unit]
Description=Run a script that does not fork

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
ExecStart=/root/scripts/foreground-script.sh
Restart=always

We are using Type=simple because the bash script we're executing doesn't fork or daemonize itself. For programs that do daemonize or run in the background, use Type=oneshot.

Other options that may be of interest are listed below.

options	behavior
Restart=always	restart when the program exits, regardless of the exit code.
User=bob Group=bob	Run the service with a particular user / group

Once the file has been created, reload systemd and enable the service:

## Reloads the service files so systemd is aware of the new one
# systemctl daemon-reload
# systemctl enable mag-read

SystemV-like Init Scripts

Systemd supports the old SystemV startup scripts that are placed in /etc/rc.local and the /etc/rc.d directories using the rc-local.service.

If you need to quickly run something at startup without wanting to deal with Systemd services, just edit the /etc/rc.local file and enable the service with systemctl enable rc-local.service.

Custom service script

Alternatively, create a custom systemd service that runs your startup scripts. For example, create a file at /usr/lib/systemd/system/startup.service with the following:

[Unit]
Description=Custom SystemV-like Startup
After=network.target
ConditionFileIsExecutable=/usr/local/sbin/custom-startup-initd

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/custom-startup-initd
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
SysVStartPriority=99

[Install]
WantedBy=multi-user.target

A few things of interest in this service file:

• The ConditionFileIsExecutable will only run the script if it's executable. It basically does a test -x on the file before this continues.
• RemainAfterExit will cause systemd to show the service as active even after the script has finished execution.

In /usr/local/sbin/custom-startup-initd:

#!/bin/sh
StartupDir="/usr/local/boot/startup"

# Run all scripts starting with 'S'
for i in `ls $StartupDir/S*` ; do
        ScriptName=`basename $i`
        sh $StartupDir/$ScriptName
done

# Success exit code.
exit 0

Make the directory:

# mkdir -p /usr/local/boot/startup/

Then place your scripts in the startup directory starting with the letter 'S'.

Enable the new service by running:

# systemctl enable startup
# systemctl start startup

Verify this:

# systemctl status startup.service

Editing a service

You can create overrides to an existing service by running systemctl edit something.service.

For example, to make Docker start after ZFS is ready, add the following overrides:

# systemctl edit docker.service
After=zfs-mount.service 
Requires=zfs-mount.service 
Wants=zfs-mount.service 
BindsTo=zfs-mount.service

This will create an override.conf file under /etc/systemd/system/docker.service.d/override.conf.

Run something in a cgroup

Systemd makes it easy to run something within a new cgroup. For example:

# Use up to 2GB memory and begin swapping if it exceeds this limit
$ systemd-run --user --pty --property MemoryHigh=2G firefox

# Use up to 2GB memory and kill with OOM if it exceeds this limit
$ systemd-run --user --pty --property MemoryMax=2G firefox