howtos/topics/systemd.md

systemd

Init system and service manager — controls what runs, when, and how.

systemctl — Service Management

# Status
systemctl status nginx
systemctl is-active nginx
systemctl is-enabled nginx
systemctl is-failed nginx

# Start / stop / restart
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx                 # reload config without restart
systemctl reload-or-restart nginx      # reload if supported, else restart

# Enable / disable (start on boot)
systemctl enable nginx
systemctl disable nginx
systemctl enable --now nginx           # enable + start immediately

# Mask (prevent starting entirely)
systemctl mask nginx
systemctl unmask nginx

# List
systemctl list-units                   # all loaded units
systemctl list-units --type=service    # services only
systemctl list-units --failed          # failed units
systemctl list-unit-files              # all installed units + state
systemctl list-timers                  # active timers

# Reload systemd after editing unit files
systemctl daemon-reload

# System state
systemctl is-system-running            # running, degraded, maintenance
systemctl --failed                     # shortcut for failed units

journalctl — Logs

# All logs
journalctl

# Specific unit
journalctl -u nginx
journalctl -u nginx -u php-fpm        # multiple units

# Follow (tail -f)
journalctl -f
journalctl -fu nginx                   # follow specific unit

# Recent entries
journalctl -n 50                       # last 50 lines
journalctl -n 50 -u nginx

# Time range
journalctl --since "2025-01-01"
journalctl --since "1 hour ago"
journalctl --since "2025-01-01 08:00" --until "2025-01-01 12:00"

# Current boot only
journalctl -b
journalctl -b -1                       # previous boot

# By priority
journalctl -p err                      # err and above
journalctl -p warning -u nginx

# Kernel messages (dmesg equivalent)
journalctl -k
journalctl -k -b                       # kernel, current boot

# Output formats
journalctl -o json                     # JSON
journalctl -o json-pretty              # formatted JSON
journalctl -o short-iso                # ISO timestamps
journalctl -o cat                      # message only, no metadata

# Disk usage
journalctl --disk-usage
journalctl --vacuum-size=500M          # shrink to 500MB
journalctl --vacuum-time=7d            # keep last 7 days

# By PID / executable
journalctl _PID=1234
journalctl _EXE=/usr/bin/python3
journalctl _UID=1000                   # by user ID

Priority Levels

Level	Keyword	Meaning
0	emerg	System unusable
1	alert	Immediate action needed
2	crit	Critical conditions
3	err	Error conditions
4	warning	Warning conditions
5	notice	Normal but significant
6	info	Informational
7	debug	Debug-level messages

Unit Files

Location and Precedence

Path	Purpose	Precedence
/etc/systemd/system/	Admin overrides	Highest
/run/systemd/system/	Runtime units	Medium
/usr/lib/systemd/system/	Package defaults	Lowest

# Find a unit file
systemctl cat nginx.service

# Show effective config (with overrides)
systemctl show nginx.service

# Edit override (drop-in snippet)
systemctl edit nginx.service           # creates override.conf

# Edit full unit file
systemctl edit --full nginx.service

# Override location
# /etc/systemd/system/nginx.service.d/override.conf

Service Unit

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
Documentation=https://example.com/docs
After=network.target postgresql.service
Requires=postgresql.service
Wants=redis.service

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
Environment=APP_ENV=production
EnvironmentFile=/opt/myapp/.env
ExecStartPre=/opt/myapp/pre-start.sh
ExecStart=/opt/myapp/bin/server --port 8080
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StartLimitBurst=3
StartLimitIntervalSec=60

# Hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/myapp/data /var/log/myapp
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Service Types

Type	Behavior
simple	Main process is the ExecStart process (default)
exec	Like simple, but ready after exec() succeeds
forking	Process forks, parent exits — use PIDFile=
oneshot	Runs once and exits — use with RemainAfterExit
notify	Process sends sd_notify when ready
idle	Like simple, but waits until other jobs finish

Restart Policies

Policy	When it restarts
no	Never (default)
on-failure	Non-zero exit, signal, timeout, watchdog
on-abnormal	Signal, timeout, watchdog (not exit code)
on-abort	Signal only
always	Always, regardless of exit reason

Dependencies

Directive	Effect
Requires	Hard dependency — if it fails, this unit fails too
Wants	Soft dependency — failure doesn't affect this unit
After	Start order (wait for listed units)
Before	Start this unit before listed units
BindsTo	Like Requires, also stops when dependency stops
Conflicts	Cannot coexist — starting one stops the other

After/Before control order. Requires/Wants control activation. Combine them:

# Start after AND depend on postgresql
Requires=postgresql.service
After=postgresql.service

Timers (cron replacement)

Timer Unit

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup timer

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

# /etc/systemd/system/backup.service
[Unit]
Description=Daily backup

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh

systemctl enable --now backup.timer
systemctl list-timers

OnCalendar Syntax

# Format: DayOfWeek Year-Month-Day Hour:Minute:Second

*-*-* 02:00:00              # daily at 2am
Mon *-*-* 09:00:00          # every Monday 9am
*-*-01 00:00:00             # first of every month
*-*-* *:00:00               # every hour
*-*-* *:*:00                # every minute
*-*-* *:00/15:00            # every 15 minutes
Mon..Fri *-*-* 08:00:00     # weekdays at 8am

Monotonic Timers (relative)

[Timer]
OnBootSec=5min              # 5 min after boot
OnUnitActiveSec=1h          # 1 hour after last activation
OnStartupSec=10min          # 10 min after systemd start

# Validate calendar expressions
systemd-analyze calendar "*-*-* 02:00:00"
systemd-analyze calendar "Mon *-*-* 09:00:00" --iterations=5

Targets (runlevels)

Target	Equivalent	Purpose
poweroff.target	runlevel 0	Shut down
rescue.target	runlevel 1	Single user
multi-user.target	runlevel 3	Multi-user, no GUI
graphical.target	runlevel 5	GUI
reboot.target	runlevel 6	Reboot

systemctl get-default
systemctl set-default multi-user.target
systemctl isolate rescue.target        # switch now

Troubleshooting

# Why did a service fail?
systemctl status myapp.service
journalctl -u myapp.service -n 50 --no-pager

# Dependency tree
systemctl list-dependencies nginx.service
systemctl list-dependencies --reverse nginx.service  # what depends on it

# Boot analysis
systemd-analyze                        # total boot time
systemd-analyze blame                  # per-unit boot time
systemd-analyze critical-chain         # critical path

# Verify unit file syntax
systemd-analyze verify myapp.service

# Show effective environment
systemctl show myapp.service -p Environment
systemctl show myapp.service -p MainPID

Gotchas

• Always daemon-reload after editing unit files — systemd caches them
• Requires without After starts both in parallel — the dependency may not be ready
• Restart=always with no StartLimitBurst can thrash — always set rate limits
• EnvironmentFile does not support shell expansion — no $HOME or backticks
• ExecStart must use absolute paths — no PATH lookup
• Type=forking without PIDFile makes systemd guess the main PID — often wrong
• Timer Persistent=true catches up missed runs — can cause a burst after downtime
• systemctl edit without --full creates a drop-in, not a replacement
• User units (systemctl --user) need loginctl enable-linger to run without login

See Also

• systemd.unit(5)
• systemd.service(5)
• systemd.timer(5)
• man systemd.exec — execution environment directives