System Management with systemd

Feel free to suggest a talk or even volunteer to give one yourself! Send mail to operations@club.cc.cmu.edu.

We do lots of cutting edge system administration, so join!

Green Hills make the world's highest performing compilers, most secure real-time operating systems, revolutionary debuggers, and virtualization solutions for embedded systems.

The systemd project develops several core components of GNU/Linux, including the init system, the eponymous "systemd", but also udev, logind, and some others. systemd v1 was released 30 March 2010, and v216 was released 19 August 2014. The init system component has now been adopted by all the major GNU/Linux distributions. Fedora, Arch, RHEL, OpenSUSE, and Mageia have shipped systemd in their most recent releases. Debian and Ubuntu plan to ship systemd in their next releases, Debian Jessie and Ubuntu 15.04, which are both about 6 months away. It is a simple, elegant piece of software that is easy to integrate with existing systems and shell-scripts.

It's well documented, check out the man pages! Check the man page systemd.index(7) for an index of all the documenation.

Let me briefly clarify a small detail about systemd's architecture that I believe has been misunderstood. The systemd project provides a binary that runs as pid 1, and it's called "systemd", but the vast majority of functionality provided by the systemd project is run outside pid 1.

systemd's pid 1 start daemons in a clean context and wait() on them, restarting them if they fail, throughout early boot, normal execution, and late shutdown. It also provides an API for the rest of the system to request immediate daemon stops, starts or restarts.

So let's get started! To start manipulating the service state of some systemd services, you must first write an XML administration descriptor and load it into your virtual LDAP cache, then instantiate a ServiceManipulatorFactory object to product ServiceManipulator objects.

Just kidding!

The main tool for manipulating the system. Check out the manpage, systemctl(1).

# see a list of all services 
systemctl -t service

# check the status of the sshd service
systemctl status sshd

# As root...
# start the sshd service
# this blocks until the service finishes startup 
# (if you want it to be async, background it or pass --no-block)
systemctl start sshd

# stop the sshd service
# again, blocks until the service is actually fully stopped 
systemctl stop sshd

# restart the sshd service
# again, blocks until the service is actually fully restarted 
systemctl restart sshd

# enable the sshd service so it starts at system boot
systemctl enable sshd

# disable the sshd service so it does not start at system boot
systemctl disable sshd

But how does systemd know when sshd finishes startup? And how does it know how to start sshd? The .service file!

The systemd dependency tree includes a wide variety of nodes, since services can depend on all kinds of things. A "unit file" is a description of something in this dependency tree, like a device or a socket or something, which can depend on other things in the tree or be depended on in turn.

A "service file" is a "unit file" that specifically describes a daemon. Basically the direct equivalent of an init script.

For more information, read the manpage systemd.unit(5). Specifically for service files, read systemd.service(5).

Happily, unit files are way, way, way easier to write than init scripts. And they're standardized across distros! You can use them across any distro! Here's an excellent tutorial.

Here is an example of a sysvinit script which I borrowed from that tutorial. Be horrified! Thankfully what we are about to write is much more concise.

I encourage you to just read that tutorial instead of this section, as it is much more articulate. I'll only be giving an abbreviated description of the components of a systemd service file.

A common section for all unit files, including a description and dependency information. Read systemd.unit(7) for more information.

[Unit]
# A description of the service 
Description=OpenSSH Daemon
# A suggestion about what order to start the service in
After=network.target

After= is a suggestion about what order to start the service in After= should be used if your service doesn't have a hard dependency, and can run (maybe with reduced functionality) even if, say, the network isn't fully up yet. If you need full dependency, use Requires=, but usually you really do not need full dependency, thanks to socket activation of other serivces and other tricks.

To figure out the names of services you want to depend on, run "systemctl". Or just read the service files of similar services. It's really hard to mess it up so feel free to copy from other service files in /lib/systemd/system.

Units ending in ".target" are targets, which are kind of like advanced run-levels.

A section specific to service files, detailing how to start and stop the service, and how systemd can tell whether it's started successfully. Read systemd.service(7) for more information.

[Service]
# the command to run to start the service
ExecStart=/usr/bin/sshd -D

By default systemd assumes the service is fully initialized as soon as ExecStart is run, and that that process will not exit until the service fails. That is "Type=simple". You can set Type= to something else to change this; check the man page systemd.service(7).

An optional section for any unit file which describes what "systemctl enable" and "systemctl disable" should do. Read systemd.unit(7) for more information.

[Install]
# This section is usually just this same line.
WantedBy=multi-user.target

The WantedBy= setting causes "systemctl enable" to create a symlink in the /etc/systemd/system/multi-user.target.wants directory.

When systemd starts multi-user.target, it will start everything in /etc/systemd/system/multi-user.target.wants. Or, as a general rule, when systemd starts foo.unit, it will start everything in /etc/systemd/system/foo.unit.wants. See systemd.unit(5) for more info, particularly the section mentioning the "Wants=" key.

The target multi-user.target is the equivalent of runlevel 2 in Debian sysvinit. That is, it's the default target for a normal multi-user system.

[Unit]
Description=OpenSSH Daemon
After=network.target

[Service]
ExecStart=/usr/bin/sshd -D

[Install]
WantedBy=multi-user.target

Much simpler than a sysvinit script, eh? Which, as a reminder, looks like this.

System unit files can go in multiple places. In decreasing order of precedence:

1. /etc/systemd/system

   Here go all of the local administrator's unit files, overriding any units with the same names in /lib/systemd/system.

   You should put any unit files you write in this directory.

2. /lib/systemd/system

   Here go all of the unit files installed by packages/the distro. As a general rule, unrelated to systemd, any configuration files provided by the distro should go in /lib or /usr/share.

3. /run/systemd/system, /run/systemd/generator

   Here go runtime auto-generated unit files, which are generated from (for example) /etc/fstab, so that systemd knows about filesystem mounts and can express dependencies on them. These can actually have any priority (their priority is a parameter written into the file at generation time), but actually messing with these or writing generators is pretty rare, so don't worry about them.

Say a service has root, or is otherwise able to read things in /home (there's lots of globally readable things in there). Like, say, this bash script:

#!/bin/bash
#/usr/bin/thiefd
echo "I'm gonna steal your banking passwords!!"
cat /home/sbaugh/browser-data
echo "Did I get them?"
sleep infinity

Let's start it up with unit file:

[Unit]
Description=Thief Daemon

[Service]
ExecStart=/usr/bin/thiefd
ProtectHome=true

[Install]
WantedBy=multi-user.target

No, thiefd! You get nothing! Good day sir! Read systemd.exec(5) for more on that option.

NoNewPrivilieges= stops a service from being able to run su, sudo, or otherwise change UID.

#!/bin/bash
#/usr/bin/skidd
echo "I'm gonna install my PHP r00t shell!"
# Oh no, I turned on passwordless sudo for all users!
sudo touch /root/0wned
echo "I am da best!"
sleep infinity

Let's start it up with unit file:

[Unit]
Description=Super-Class-A Hacker Daemon

[Service]
ExecStart=/usr/bin/skidd
# Traditional user-based dropping of privileges
User=nobody
Group=nobody
# For additional, useful new security functions
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Root shell? Sorry, skidd! Not today! Read systemd.exec(5) for more on that option.

Services go awry and insane and start forking like wild!

#!/bin/bash
#/usr/bin/forkd
echo "I'm gonna forkbomb!"
fork() { fork | fork & };
fork

Let's start it up with unit file:

[Unit]
Description=Fork Daemon

[Service]
ExecStart=/usr/bin/forkd

# Ha ha! No special settings necessary! We get this one for free! Thanks cgroups!

# But for the sake of my system staying responsive...
CPUQuota=20%
MemoryLimit=1G

[Install]
WantedBy=multi-user.target

Aiee! It's out of control!

systemctl stop forkd.service
# We catch all the forked processes!

Fork bombs? We don't use that kind of language on this system!

The resource control is described in systemd.resource-control(5), but the ability to stop the fork bomb is given to us automatically by systemd's use of cgroups. Since every service is put into its own cgroup, which it cannot escape, we can be sure that if we (atomically) kill everything in the cgroup, we will catch all the forked processes.

If we have a daemon with privileges that writes to /tmp/foo, PrivateTmp= prevents a malicious user from doing ln -s /tmp/foo /etc/passwd, causing /etc/passwd to be overwritten.

TODO demo this. I don't have a demo for this because I was having trouble creating that malicious symlink :(

There are lots of tools to get an overview of what's going on!

# show a nice process/control group tree
systemd-cgls

# some more recent versions of systemd have a nice status overview with this command
systemctl status

# show every unit systemd knows about
# don't be scared, this complexity all already existed, you just didn't know about it
systemctl

# show active services and their status
systemctl -t service

A cute little integrated documentation-querier.

# show active services and their status
systemctl -t service

# hmm, what's this thing "systemd-logind.service"
# show the man pages, as listed in the .service file!
systemctl help systemd-logind

# Docs are also listed in output of status
systemctl status systemd-logind

Awesome little upgrade to chrooting when you actually want a full system to mess around with! Check systemd-nspawn(1) for examples. Also investigate machinectl(1) allows some overviews and control. Goes great with btrfs!

Don't run your long-running commands or hacked-together daemons through a shell in screen! Instead, run them in a clean environment in systemd! Get logging, limits, store the exit status, all kinds of useful stuff!

# long running commands oh joy!
sudo systemd-run --unit=mycommand sleep 100

# no need for root even!
systemd-run --user sleep 100

Check out systemd-run(1). But wait, what was that –user thing?

user sessions! The best way to run your own daemons, when you first login or after.

# all the same as system-level systemd!
systemctl --user -t service
systemctl --user status mpd.service
systemctl --user stop mpd.service
systemctl --user disable mpd.service
systemctl --user enable mpd.service
systemctl --user start mpd.service

They're like a build-your-own-DE toolkit! You can stick together a bunch of services and manage them in a totally cross-distro-portable way. Read perhaps https://wiki.archlinux.org/index.php/Systemd/User for some tips.

Soon enough your personal user environment will be completely and utterly portable between machines. At least as soon as distros start shipping socket-activatable X by default.

By letting the init system pre-allocate sockets for services, we can just start all services at once. If something depends on NFS or CUPS, they just send their messages to the sockets and get on with bootup; when NFS is actually started, systemd will just give it the pre-allocated socket so it can respond to the messages. In this way, pretty much everything can be started in parallel, by depending on the kernel socket buffers. Ordering, synchronization, parallelism… we get it all for free!

This removes the need to configure service dependencies! This is something CClub sure wishes they could do with machine room boot up after power outages!

It's kinda like inetd, except you don't bother with the whole "activate only when a message comes in". Just allocate all the sockets, and start everything! (Of course, we can also do regular standard socket activation.)

This is better described in this article: http://0pointer.net/blog/projects/systemd.html

Yes, systemd changes things. But it changes them towards standard ways! Red Hat and Fedora people complain about systemd changing towards Debian-style ways of doing things, like putting hostname in /etc/hostname. Debian people complain about, um… I couldn't actually find any Fedora-style things that systemd is forcing Debian to use. But either way, we're moving towards more standardization between distros, which is great.

And finally we can use the same mechanism for managing services across all distros. That's often the only system-level thing I'll need to touch, enabling and disabling and restarting services.

Maybe if you're already greybearded you don't realize how much of a pain this was. My knowledge from Ubuntu didn't carry over to Debian, my knowledge of Debian didn't carry over to Gentoo, my knowledge of Gentoo didn't carry over to Arch, but finally: my knowledge from Arch, which uses systemd, carries over to Fedora, RHEL, Debian Jessie, Ubuntu 15.04, etc. etc..

Can you tell me how to disable or enable a service before systemd on CentOS, OpenSUSE, Gentoo, Arch, etc. etc.? Can you tell me how to do it after systemd? Now the method is universal! systemctl enable and disable!

Also, distributions can share unit files; they could not share sysvinit scripts, because every distro's sysvinit setup was so different. In fact, many upstream projects now ship unit files, moving some of the maintenance workload of the distribution maintainer into the upstream project where it can be shared between distros.

No more forking, no more pid files, no more special debug flags. Just run and log to stdout.

Systemd will run you "detached" automatically, and collect your stdout logs and send them to the appropriate place.

Read daemon(3) for the specificas of the improvements.