howtos/topics/systemd.md

# systemd

> Init system and service manager — controls what runs, when, and how.

## systemctl — Service Management

```bash
# Status
systemctl status nginx
systemctl is-active nginx
systemctl is-enabled nginx
systemctl is-failed nginx

# Start / stop / restart
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx                 # reload config without restart
systemctl reload-or-restart nginx      # reload if supported, else restart

# Enable / disable (start on boot)
systemctl enable nginx
systemctl disable nginx
systemctl enable --now nginx           # enable + start immediately

# Mask (prevent starting entirely)
systemctl mask nginx
systemctl unmask nginx

# List
systemctl list-units                   # all loaded units
systemctl list-units --type=service    # services only
systemctl list-units --failed          # failed units
systemctl list-unit-files              # all installed units + state
systemctl list-timers                  # active timers

# Reload systemd after editing unit files
systemctl daemon-reload

# System state
systemctl is-system-running            # running, degraded, maintenance
systemctl --failed                     # shortcut for failed units
```

## journalctl — Logs

```bash
# All logs
journalctl

# Specific unit
journalctl -u nginx
journalctl -u nginx -u php-fpm        # multiple units

# Follow (tail -f)
journalctl -f
journalctl -fu nginx                   # follow specific unit

# Recent entries
journalctl -n 50                       # last 50 lines
journalctl -n 50 -u nginx

# Time range
journalctl --since "2025-01-01"
journalctl --since "1 hour ago"
journalctl --since "2025-01-01 08:00" --until "2025-01-01 12:00"

# Current boot only
journalctl -b
journalctl -b -1                       # previous boot

# By priority
journalctl -p err                      # err and above
journalctl -p warning -u nginx

# Kernel messages (dmesg equivalent)
journalctl -k
journalctl -k -b                       # kernel, current boot

# Output formats
journalctl -o json                     # JSON
journalctl -o json-pretty              # formatted JSON
journalctl -o short-iso                # ISO timestamps
journalctl -o cat                      # message only, no metadata

# Disk usage
journalctl --disk-usage
journalctl --vacuum-size=500M          # shrink to 500MB
journalctl --vacuum-time=7d            # keep last 7 days

# By PID / executable
journalctl _PID=1234
journalctl _EXE=/usr/bin/python3
journalctl _UID=1000                   # by user ID
```

### Priority Levels

| Level   | Keyword  | Meaning                  |
|---------|----------|--------------------------|
| 0       | `emerg`  | System unusable          |
| 1       | `alert`  | Immediate action needed  |
| 2       | `crit`   | Critical conditions      |
| 3       | `err`    | Error conditions         |
| 4       | `warning`| Warning conditions       |
| 5       | `notice` | Normal but significant   |
| 6       | `info`   | Informational            |
| 7       | `debug`  | Debug-level messages     |

## Unit Files

### Location and Precedence

| Path                          | Purpose            | Precedence |
|-------------------------------|--------------------|-----------:|
| `/etc/systemd/system/`        | Admin overrides    | Highest    |
| `/run/systemd/system/`        | Runtime units      | Medium     |
| `/usr/lib/systemd/system/`    | Package defaults   | Lowest     |

```bash
# Find a unit file
systemctl cat nginx.service

# Show effective config (with overrides)
systemctl show nginx.service

# Edit override (drop-in snippet)
systemctl edit nginx.service           # creates override.conf

# Edit full unit file
systemctl edit --full nginx.service

# Override location
# /etc/systemd/system/nginx.service.d/override.conf
```

### Service Unit

```ini
# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
Documentation=https://example.com/docs
After=network.target postgresql.service
Requires=postgresql.service
Wants=redis.service

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
Environment=APP_ENV=production
EnvironmentFile=/opt/myapp/.env
ExecStartPre=/opt/myapp/pre-start.sh
ExecStart=/opt/myapp/bin/server --port 8080
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StartLimitBurst=3
StartLimitIntervalSec=60

# Hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/myapp/data /var/log/myapp
PrivateTmp=true

[Install]
WantedBy=multi-user.target
```

### Service Types

| Type        | Behavior                                          |
|-------------|---------------------------------------------------|
| `simple`    | Main process is the `ExecStart` process (default) |
| `exec`      | Like simple, but ready after exec() succeeds      |
| `forking`   | Process forks, parent exits — use `PIDFile=`      |
| `oneshot`   | Runs once and exits — use with `RemainAfterExit`  |
| `notify`    | Process sends `sd_notify` when ready              |
| `idle`      | Like simple, but waits until other jobs finish    |

### Restart Policies

| Policy          | When it restarts                           |
|-----------------|--------------------------------------------|
| `no`            | Never (default)                            |
| `on-failure`    | Non-zero exit, signal, timeout, watchdog   |
| `on-abnormal`   | Signal, timeout, watchdog (not exit code)  |
| `on-abort`      | Signal only                                |
| `always`        | Always, regardless of exit reason          |

### Dependencies

| Directive   | Effect                                              |
|-------------|-----------------------------------------------------|
| `Requires`  | Hard dependency — if it fails, this unit fails too  |
| `Wants`     | Soft dependency — failure doesn't affect this unit  |
| `After`     | Start order (wait for listed units)                 |
| `Before`    | Start this unit before listed units                 |
| `BindsTo`   | Like Requires, also stops when dependency stops     |
| `Conflicts` | Cannot coexist — starting one stops the other       |

`After`/`Before` control **order**. `Requires`/`Wants` control **activation**. Combine them:

```ini
# Start after AND depend on postgresql
Requires=postgresql.service
After=postgresql.service
```

## Timers (cron replacement)

### Timer Unit

```ini
# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup timer

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target
```

```ini
# /etc/systemd/system/backup.service
[Unit]
Description=Daily backup

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh
```

```bash
systemctl enable --now backup.timer
systemctl list-timers
```

### OnCalendar Syntax

```
# Format: DayOfWeek Year-Month-Day Hour:Minute:Second

*-*-* 02:00:00              # daily at 2am
Mon *-*-* 09:00:00          # every Monday 9am
*-*-01 00:00:00             # first of every month
*-*-* *:00:00               # every hour
*-*-* *:*:00                # every minute
*-*-* *:00/15:00            # every 15 minutes
Mon..Fri *-*-* 08:00:00     # weekdays at 8am
```

### Monotonic Timers (relative)

```ini
[Timer]
OnBootSec=5min              # 5 min after boot
OnUnitActiveSec=1h          # 1 hour after last activation
OnStartupSec=10min          # 10 min after systemd start
```

```bash
# Validate calendar expressions
systemd-analyze calendar "*-*-* 02:00:00"
systemd-analyze calendar "Mon *-*-* 09:00:00" --iterations=5
```

## Targets (runlevels)

| Target              | Equivalent | Purpose            |
|---------------------|------------|--------------------|
| `poweroff.target`   | runlevel 0 | Shut down          |
| `rescue.target`     | runlevel 1 | Single user        |
| `multi-user.target` | runlevel 3 | Multi-user, no GUI |
| `graphical.target`  | runlevel 5 | GUI                |
| `reboot.target`     | runlevel 6 | Reboot             |

```bash
systemctl get-default
systemctl set-default multi-user.target
systemctl isolate rescue.target        # switch now
```

## Troubleshooting

```bash
# Why did a service fail?
systemctl status myapp.service
journalctl -u myapp.service -n 50 --no-pager

# Dependency tree
systemctl list-dependencies nginx.service
systemctl list-dependencies --reverse nginx.service  # what depends on it

# Boot analysis
systemd-analyze                        # total boot time
systemd-analyze blame                  # per-unit boot time
systemd-analyze critical-chain         # critical path

# Verify unit file syntax
systemd-analyze verify myapp.service

# Show effective environment
systemctl show myapp.service -p Environment
systemctl show myapp.service -p MainPID
```

## Gotchas

- Always `daemon-reload` after editing unit files — systemd caches them
- `Requires` without `After` starts both in parallel — the dependency may not be ready
- `Restart=always` with no `StartLimitBurst` can thrash — always set rate limits
- `EnvironmentFile` does not support shell expansion — no `$HOME` or backticks
- `ExecStart` must use absolute paths — no `PATH` lookup
- `Type=forking` without `PIDFile` makes systemd guess the main PID — often wrong
- Timer `Persistent=true` catches up missed runs — can cause a burst after downtime
- `systemctl edit` without `--full` creates a drop-in, not a replacement
- User units (`systemctl --user`) need `loginctl enable-linger` to run without login

## See Also

- [systemd.unit(5)](https://www.freedesktop.org/software/systemd/man/systemd.unit.html)
- [systemd.service(5)](https://www.freedesktop.org/software/systemd/man/systemd.service.html)
- [systemd.timer(5)](https://www.freedesktop.org/software/systemd/man/systemd.timer.html)
- `man systemd.exec` — execution environment directives