// linux fundamentals — module 07

Module 7 — systemd: The Modern Init System

Thesis: PID 1 has a name: systemd. It starts every service, supervises every process, mounts filesystems, keeps the logs, and runs the timers — through one universal abstraction (the unit) and two commands (systemctl, journalctl). This is the control panel of a modern Linux system, and the single component NixOS configures most.

Prerequisite: Module 6 — The Boot Sequence — you know exactly how systemd becomes PID 1. And Module 5 — Processes, Signals, and IO is the engine room: everything here is fork/exec, signals, exit codes, and captured stdout, performed professionally.

7.1 What an init system is, and why systemd won

The job: be PID 1. Concretely: (a) start userland — every daemon, mount, and login prompt; (b) supervise it — restart the crashed, reap the orphans (§5.2's re-parenting means PID 1 must reap, or zombies accumulate); (c) shut it all down cleanly.

The old way (SysV init): a numbered pile of shell scripts run sequentially — slow (no parallelism), no real dependency knowledge, and above all no supervision: a script started a daemon, the daemon daemonized itself (§5.6's trick), and init forgot it existed. Whether it was still alive, what it printed, why it died — nobody's job.

systemd's answer (2010, now universal — Debian, Fedora, Arch, Ubuntu, NixOS): declare, don't script. Each service is a small declarative unit file — what to run, not how — so systemd can compute a dependency graph and start everything in parallel that doesn't order-depend (that's systemd-analyze's fast userspace from Lab 6). Because systemd itself forks/execs every service (§5.1) and never lets it detach, it knows every service's PID, state, and exit code, captures its stdout/stderr (§5.5 — into the journal, §7.5), and can restart it on failure. Supervision, delivered by the mechanics you already own.

(systemd is a suite, not one binary: the systemd PID 1 plus components you've brushed against — udevd from §2.6, journald, logind, optional timesyncd/networkd/resolved. Critics call this scope creep; either way, it's the standard, and NixOS embraces it fully.)

7.2 Units: the universal abstraction

systemd manages units — anything it can start, stop, or track. Every unit is named name.type. The types that matter:

Targets deserve one more beat, because boot ends at one. They replaced SysV's "runlevels": multi-user.target = full system, text mode; graphical.target = that plus the display manager; rescue.target = minimal single-user rescue (the thing §6.3 appended to the kernel command line); default.target = a symlink (§3.5!) to whichever one boot aims for. "Booting" in systemd terms = "start default.target and everything it pulls in."

7.3 Anatomy of a unit file

Plain text, INI-style, three sections. Here is a complete real service:

[Unit]
Description=My data sync daemon
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/mysyncd --config /etc/mysyncd.conf
Restart=on-failure
User=syncuser

[Install]
WantedBy=multi-user.target

• [Unit] — identity and dependencies (§7.4): what this is and where it sits in the graph.
• [Service] — the mechanics (only in .service files): ExecStart= is the fork/exec target (§5.1); Type= tells systemd how to know the service is "ready" (simple: the process itself is the service; oneshot: run to completion, then done; forking: legacy self-daemonizers — §5.6 vocabulary); Restart= consults exit codes (§5.3) — on-failure restarts on nonzero; User= drops privileges before exec (§4.1's system users — the fork/exec gap from §5.1 is where it happens; further hardening like AmbientCapabilities (§4.8), ProtectHome=, PrivateTmp= also lives here).
• [Install] — where the unit hooks in when enabled: WantedBy=multi-user.target means "when enabled, become something multi-user.target wants."

Where unit files live — three layers, later shadows earlier: /usr/lib/systemd/system/ (shipped by packages), /etc/systemd/system/ (the admin's: your own units and overrides), /run/systemd/system/ (runtime-generated). systemctl cat <unit> shows the winning file and its path, plus any drop-in overrides — always your first look. (On NixOS all of it is generated into the store and symlinked — §7.7.)

7.4 The dependency graph

Two orthogonal ideas — the classic beginner conflation, so get it precise:

Requirement — must X exist if Y runs?

• Wants=X — please start X too; if X fails, carry on. (Default choice.)
• Requires=X — start X too; if X fails to start, Y fails.

Ordering — who waits for whom?

• After=X / Before=X — sequencing only. Crucially: Wants/Requires imply no order at all. A unit that Wants=network-online.target but lacks After= may start before the network is up — the most common unit-file bug in the wild, and you'll now never write it: dependencies usually come in pairs (Wants= + After=), as in §7.3's example.

At boot, systemd resolves default.target → everything it wants → everything those want, topologically sorts by After/Before, and launches maximal parallelism. Inspect the graph live: systemctl list-dependencies <unit>.

Enable vs. start — the distinction you'll use daily:

• systemctl start x — fork/exec it now. No effect on next boot.
• systemctl enable x — wire it into the graph for future boots. Mechanically delightful: it just creates a symlink (§3.5, again) in /etc/systemd/system/multi-user.target.wants/ pointing at the unit file — that's what [Install]/WantedBy means, and disable deletes the link. Watch the command's output say so.
• enable --now — both.

7.5 Driving it: systemctl and journalctl

The daily verbs:

systemctl status sshd      # the dashboard — read below
systemctl start|stop sshd  # now
systemctl restart sshd     # stop+start; reload = ask it to re-read config (SIGHUP, §5.4!)
systemctl enable|disable sshd    # future boots
systemctl cat sshd         # show the unit file(s)
systemctl list-units --type=service --state=failed   # what's broken
systemctl daemon-reload    # re-read unit files after YOU edit them (easy to forget)

stop is Module 5's etiquette, automated: SIGTERM, a 90-second grace, then SIGKILL.

Learn to read status completely — it's the best diagnostic screen on the system:

● sshd.service - OpenSSH server daemon
     Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; …)
     Active: active (running) since Fri 2026-07-10 09:14:02; 1 day ago
   Main PID: 812 (sshd)
      Tasks: 1
     Memory: 4.1M
     CGroup: /system.slice/sshd.service
             └─812 "sshd: /usr/bin/sshd -D [listener]"
   … last 10 journal lines …

Line by line, with your existing vocabulary: Loaded — which file, and is it enabled (symlink present)? Active — supervision state (active (running) / inactive (dead) / failed / activating (auto-restart) — crash-looping). Main PID — the actual process (§2.5); CGroup — all processes belonging to this service, tracked via cgroups, a kernel grouping mechanism that is how systemd never loses a service's children even when they fork (and how it can kill all of them reliably — the answer to §5.6's runaway-children problem). Then recent log lines, which flow from —

journald. systemd captures every service's stdout/stderr (§5.5 — fd 1 and 2, wired into the journal instead of a terminal at exec time) plus kernel messages (dmesg's stream) into one indexed, structured log. Query it with journalctl:

journalctl -u sshd            # one service's log — the workhorse
journalctl -u sshd -b         # …this boot only
journalctl -b -1              # all of PREVIOUS boot ("why did it crash?")
journalctl -f                 # follow live (tail -f for the whole system)
journalctl -p err -b          # errors and worse, this boot
journalctl --since "10 min ago"

The universal debugging loop, which handles 90% of "why isn't this working" on any modern Linux: systemctl status x → read Active + last lines → journalctl -u x -b for the full story → fix → daemon-reload if you edited units → restart → status again.

7.6 User services and timers

Two more pieces round out daily fluency:

User services. Besides the system instance, each logged-in user gets a personal systemd (systemctl --user …, units in ~/.config/systemd/user/). No root needed, and it solves §5.6's cliffhanger properly: a user service survives terminal closure (and with loginctl enable-linger, even logout). "Background job" for real things = user service, not &.

Timers replace cron. A backup.timer activates backup.service on schedule:

[Timer]
OnCalendar=daily          # or: Mon..Fri 03:00, or OnBootSec=5min
Persistent=true           # if the machine was off at the appointed time, run on next boot

Why bother vs. cron? The service's output lands in the journal, failures show in --state=failed, systemctl list-timers shows every schedule with next/last run — supervision applies to scheduled jobs like everything else. One abstraction, uniformly.

7.7 → NixOS: where the foundations pay off hardest

On a conventional distro you hand-edit files in /etc/systemd/system/. On NixOS you declare services in configuration.nix, and Nix generates the unit files:

systemd.services.mysync = {
  description = "My data sync daemon";
  after = [ "network-online.target" ];
  wants = [ "network-online.target" ];
  wantedBy = [ "multi-user.target" ];
  serviceConfig = {
    ExecStart = "${pkgs.mysyncd}/bin/mysyncd";
    Restart = "on-failure";
    User = "syncuser";
  };
};

Read that against §7.3: it is the same unit file, one field per line, in Nix syntax — every key maps 1:1 to a concept you now own. Higher-level NixOS options (services.openssh.enable = true;) just generate richer versions of the same thing. And crucially, the debugging story is unchanged: on NixOS you still run systemctl status, systemctl cat (revealing generated units symlinked from the store — §3.5 to the last), and journalctl -u, exactly as learned here. Declaration is Nix's; the runtime is pure systemd. Master this module and NixOS service configuration is readable on sight.

Lab 7

1. Read a status screen completely. Pick a running service (systemctl list-units --type=service — try your SSH daemon, display manager, or NetworkManager) and run systemctl status on it. Explain every line out loud, including Loaded/enabled, Active, Main PID (cross-check it in ps -ef), and CGroup.

2. Dissect a unit file. systemctl cat that same service. Identify the three sections; find ExecStart, Type, any Restart, User, and each dependency directive — and say which are requirement vs. ordering.

3. Walk the graph. systemctl get-default (which target does your machine boot to?), then systemctl list-dependencies default.target | less — find your service from step 1 in the tree. Then systemctl list-dependencies --reverse sshd.service — who wants it?

4. Query the journal five ways. For your chosen service: its log this boot (-u X -b); the whole system's errors this boot (-p err -b); the previous boot's last 20 lines (-b -1 | tail -20); live follow (-f, then interact with the service or plug in a USB stick and watch); and one --since "1 hour ago".

5. Write, run, and supervise your own service (the capstone of the module). Create /etc/systemd/system/hello-lab.service:

   [Unit]
   Description=Module 7 lab one-shot
   
   [Service]
   Type=oneshot
   ExecStart=/bin/sh -c 'echo "hello from my first unit at $(date)"'
   
   [Install]
   WantedBy=multi-user.target

   Then: sudo systemctl daemon-reload → sudo systemctl start hello-lab → systemctl status hello-lab (note: inactive (dead) — oneshots finish; is that failure? check the exit path) → find your echo in the journal (journalctl -u hello-lab). Your stdout, captured — §5.5 fulfilled. Then sudo systemctl enable hello-lab and read the output: the symlink it created, exactly as §7.4 promised. ls -l /etc/systemd/system/multi-user.target.wants/ to see it. Disable and remove when done. (On NixOS: declare it via systemd.services.hello-lab as in §7.7 and rebuild — then run the identical start/status/journal steps.)

6. Watch supervision fight back. Make a crash-looper — a service with ExecStart=/bin/false (exits 1 immediately, §5.3) and Restart=on-failure. Start it, then run systemctl status a few times and watch: activating (auto-restart), restart counter climbing, and eventually failed (start-limit hit). Read the journal's account. This screen is what a genuinely broken service looks like — better to meet it first on a toy. Clean up.

7. Break the ordering rule on purpose (thought experiment). Reread §7.3's example unit. Delete After=network-online.target in your head — describe precisely what can now go wrong at boot, and why Wants= alone didn't prevent it.

8. Set a timer. Write hello-lab.timer with OnCalendar=*:0/2 (every 2 minutes) + [Install] WantedBy=timers.target, enable it, run systemctl list-timers to see next-run, wait for two firings, confirm both in the journal, then disable and remove both units.

✅ Mastery Check — do not proceed until true

Answer out loud, without notes:

1. What are PID 1's three jobs? Name two concrete things systemd's model does that SysV scripts couldn't, and tie each to a Module 5 mechanism.
2. What is a unit? Give the six main types with a one-line purpose each. What is a target, and what does "booting" mean in target terms?
3. Sketch a complete .service file from memory — three sections, at least six directives — and explain each directive's mechanism (which syscall, signal, or Module 4/5 concept it drives).
4. Wants= vs. Requires= vs. After=: which are requirement, which is ordering, and what bug does Wants= without After= produce?
5. start vs. enable — including what enable physically does on disk.
6. Recite the universal debugging loop, and decode these Active states: active (running), inactive (dead) after a oneshot, activating (auto-restart), failed.
7. Where do a service's stdout/stderr go, and how do you read: one unit's log this boot; the previous boot; only errors; live?
8. Why prefer a systemd timer over cron, and a user service over nohup … &?

And perform cold:

• Write, install, run, verify (via journal), enable, and cleanly remove a service unit.
• Diagnose a failing service from status + journalctl alone.
• Trace any unit's position in the dependency graph in both directions.

When all of that is effortless: Module 8 — Networking on Linux