# systemd > Init system and service manager — controls what runs, when, and how. ## systemctl — Service Management ```bash # Status systemctl status nginx systemctl is-active nginx systemctl is-enabled nginx systemctl is-failed nginx # Start / stop / restart systemctl start nginx systemctl stop nginx systemctl restart nginx systemctl reload nginx # reload config without restart systemctl reload-or-restart nginx # reload if supported, else restart # Enable / disable (start on boot) systemctl enable nginx systemctl disable nginx systemctl enable --now nginx # enable + start immediately # Mask (prevent starting entirely) systemctl mask nginx systemctl unmask nginx # List systemctl list-units # all loaded units systemctl list-units --type=service # services only systemctl list-units --failed # failed units systemctl list-unit-files # all installed units + state systemctl list-timers # active timers # Reload systemd after editing unit files systemctl daemon-reload # System state systemctl is-system-running # running, degraded, maintenance systemctl --failed # shortcut for failed units ``` ## journalctl — Logs ```bash # All logs journalctl # Specific unit journalctl -u nginx journalctl -u nginx -u php-fpm # multiple units # Follow (tail -f) journalctl -f journalctl -fu nginx # follow specific unit # Recent entries journalctl -n 50 # last 50 lines journalctl -n 50 -u nginx # Time range journalctl --since "2025-01-01" journalctl --since "1 hour ago" journalctl --since "2025-01-01 08:00" --until "2025-01-01 12:00" # Current boot only journalctl -b journalctl -b -1 # previous boot # By priority journalctl -p err # err and above journalctl -p warning -u nginx # Kernel messages (dmesg equivalent) journalctl -k journalctl -k -b # kernel, current boot # Output formats journalctl -o json # JSON journalctl -o json-pretty # formatted JSON journalctl -o short-iso # ISO timestamps journalctl -o cat # message only, no metadata # Disk usage journalctl --disk-usage journalctl --vacuum-size=500M # shrink to 500MB journalctl --vacuum-time=7d # keep last 7 days # By PID / executable journalctl _PID=1234 journalctl _EXE=/usr/bin/python3 journalctl _UID=1000 # by user ID ``` ### Priority Levels | Level | Keyword | Meaning | |---------|----------|--------------------------| | 0 | `emerg` | System unusable | | 1 | `alert` | Immediate action needed | | 2 | `crit` | Critical conditions | | 3 | `err` | Error conditions | | 4 | `warning`| Warning conditions | | 5 | `notice` | Normal but significant | | 6 | `info` | Informational | | 7 | `debug` | Debug-level messages | ## Unit Files ### Location and Precedence | Path | Purpose | Precedence | |-------------------------------|--------------------|-----------:| | `/etc/systemd/system/` | Admin overrides | Highest | | `/run/systemd/system/` | Runtime units | Medium | | `/usr/lib/systemd/system/` | Package defaults | Lowest | ```bash # Find a unit file systemctl cat nginx.service # Show effective config (with overrides) systemctl show nginx.service # Edit override (drop-in snippet) systemctl edit nginx.service # creates override.conf # Edit full unit file systemctl edit --full nginx.service # Override location # /etc/systemd/system/nginx.service.d/override.conf ``` ### Service Unit ```ini # /etc/systemd/system/myapp.service [Unit] Description=My Application Documentation=https://example.com/docs After=network.target postgresql.service Requires=postgresql.service Wants=redis.service [Service] Type=simple User=appuser Group=appuser WorkingDirectory=/opt/myapp Environment=APP_ENV=production EnvironmentFile=/opt/myapp/.env ExecStartPre=/opt/myapp/pre-start.sh ExecStart=/opt/myapp/bin/server --port 8080 ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure RestartSec=5 StartLimitBurst=3 StartLimitIntervalSec=60 # Hardening NoNewPrivileges=true ProtectSystem=strict ProtectHome=true ReadWritePaths=/opt/myapp/data /var/log/myapp PrivateTmp=true [Install] WantedBy=multi-user.target ``` ### Service Types | Type | Behavior | |-------------|---------------------------------------------------| | `simple` | Main process is the `ExecStart` process (default) | | `exec` | Like simple, but ready after exec() succeeds | | `forking` | Process forks, parent exits — use `PIDFile=` | | `oneshot` | Runs once and exits — use with `RemainAfterExit` | | `notify` | Process sends `sd_notify` when ready | | `idle` | Like simple, but waits until other jobs finish | ### Restart Policies | Policy | When it restarts | |-----------------|--------------------------------------------| | `no` | Never (default) | | `on-failure` | Non-zero exit, signal, timeout, watchdog | | `on-abnormal` | Signal, timeout, watchdog (not exit code) | | `on-abort` | Signal only | | `always` | Always, regardless of exit reason | ### Dependencies | Directive | Effect | |-------------|-----------------------------------------------------| | `Requires` | Hard dependency — if it fails, this unit fails too | | `Wants` | Soft dependency — failure doesn't affect this unit | | `After` | Start order (wait for listed units) | | `Before` | Start this unit before listed units | | `BindsTo` | Like Requires, also stops when dependency stops | | `Conflicts` | Cannot coexist — starting one stops the other | `After`/`Before` control **order**. `Requires`/`Wants` control **activation**. Combine them: ```ini # Start after AND depend on postgresql Requires=postgresql.service After=postgresql.service ``` ## Timers (cron replacement) ### Timer Unit ```ini # /etc/systemd/system/backup.timer [Unit] Description=Daily backup timer [Timer] OnCalendar=*-*-* 02:00:00 Persistent=true RandomizedDelaySec=300 [Install] WantedBy=timers.target ``` ```ini # /etc/systemd/system/backup.service [Unit] Description=Daily backup [Service] Type=oneshot ExecStart=/opt/scripts/backup.sh ``` ```bash systemctl enable --now backup.timer systemctl list-timers ``` ### OnCalendar Syntax ``` # Format: DayOfWeek Year-Month-Day Hour:Minute:Second *-*-* 02:00:00 # daily at 2am Mon *-*-* 09:00:00 # every Monday 9am *-*-01 00:00:00 # first of every month *-*-* *:00:00 # every hour *-*-* *:*:00 # every minute *-*-* *:00/15:00 # every 15 minutes Mon..Fri *-*-* 08:00:00 # weekdays at 8am ``` ### Monotonic Timers (relative) ```ini [Timer] OnBootSec=5min # 5 min after boot OnUnitActiveSec=1h # 1 hour after last activation OnStartupSec=10min # 10 min after systemd start ``` ```bash # Validate calendar expressions systemd-analyze calendar "*-*-* 02:00:00" systemd-analyze calendar "Mon *-*-* 09:00:00" --iterations=5 ``` ## Targets (runlevels) | Target | Equivalent | Purpose | |---------------------|------------|--------------------| | `poweroff.target` | runlevel 0 | Shut down | | `rescue.target` | runlevel 1 | Single user | | `multi-user.target` | runlevel 3 | Multi-user, no GUI | | `graphical.target` | runlevel 5 | GUI | | `reboot.target` | runlevel 6 | Reboot | ```bash systemctl get-default systemctl set-default multi-user.target systemctl isolate rescue.target # switch now ``` ## Troubleshooting ```bash # Why did a service fail? systemctl status myapp.service journalctl -u myapp.service -n 50 --no-pager # Dependency tree systemctl list-dependencies nginx.service systemctl list-dependencies --reverse nginx.service # what depends on it # Boot analysis systemd-analyze # total boot time systemd-analyze blame # per-unit boot time systemd-analyze critical-chain # critical path # Verify unit file syntax systemd-analyze verify myapp.service # Show effective environment systemctl show myapp.service -p Environment systemctl show myapp.service -p MainPID ``` ## Gotchas - Always `daemon-reload` after editing unit files — systemd caches them - `Requires` without `After` starts both in parallel — the dependency may not be ready - `Restart=always` with no `StartLimitBurst` can thrash — always set rate limits - `EnvironmentFile` does not support shell expansion — no `$HOME` or backticks - `ExecStart` must use absolute paths — no `PATH` lookup - `Type=forking` without `PIDFile` makes systemd guess the main PID — often wrong - Timer `Persistent=true` catches up missed runs — can cause a burst after downtime - `systemctl edit` without `--full` creates a drop-in, not a replacement - User units (`systemctl --user`) need `loginctl enable-linger` to run without login ## See Also - [systemd.unit(5)](https://www.freedesktop.org/software/systemd/man/systemd.unit.html) - [systemd.service(5)](https://www.freedesktop.org/software/systemd/man/systemd.service.html) - [systemd.timer(5)](https://www.freedesktop.org/software/systemd/man/systemd.timer.html) - `man systemd.exec` — execution environment directives