# Server Hardening Survey Kit A procedure for hardening a Linux server that one person administers, written so that a person or an AI coding agent can follow it. It is the companion to two jwatte.com articles: - https://jwatte.com/blog/harden-single-admin-linux-server-survey-first/ (the method) - https://jwatte.com/blog/server-hardening-eight-items-three-changes/ (what it found on our own server) Version 2026-11-18. Tested on Ubuntu 24.04 (AIDE 0.18.6) in a clean container and built from what we run on Ubuntu 26.04 (AIDE 0.19.2, auditd 4.1.2). Free to use and adapt. No warranty: read every command before you run it, and try changes on a copy first. The idea in one line: survey the machine, measure before you restrict, write down every decision including the ones you skip, and add detection that stays quiet enough to be read. --- ## 0. Rules for an AI agent using this kit If you are an agent, these rules bind you. If you are a person handing this file to an agent, paste them in. 1. Sections 1 and 2 are read-only. Run them first and report before proposing any change. 2. Never print the contents of a secret, token, key or env file. Report names, owners and modes only. 3. Propose one change at a time. State what it does, what could break, and how to undo it. Wait for a yes. 4. Back up a file before you edit it. Say where the backup is. 5. Never restart, recreate or reboot anything without an explicit yes for that action. 6. Before you add a limit (rate limit, allowlist, dropped capability), measure what the limit would have blocked in the last week. Show the number. 7. "Skip" is a valid decision. Record it with the reason. Do not argue it again unless the facts change. 8. If your first finding turns out wrong or overstated when measured, say so plainly and correct it. 9. Do not send survey output anywhere. It describes how to attack this machine. --- ## 1. Survey first (read-only) Run these as the admin account. Each answers one question. None changes anything. ```bash # 1.1 Subscriptions and entitlements: is any security source switched off? pro security-status # Ubuntu: packages covered, and by what pro status # which services are enabled # 1.2 Updates: do they install by themselves, and from which sources? systemctl is-enabled unattended-upgrades apt-config dump | grep -i 'Unattended-Upgrade::Allowed-Origins' apt list --upgradable 2>/dev/null | tail -n +2 | wc -l # 1.3 Network: what listens, and what does the firewall allow? sudo ss -tlnp sudo ufw status verbose # 1.4 Logins: who can log in, and how? sudo sshd -T | grep -E '^(passwordauthentication|permitrootlogin|pubkeyauthentication|persourcepenalties) ' sudo passwd -S root sudo awk -F: '$7 !~ /(nologin|false)$/ {print $1}' /etc/passwd # 1.5 Mandatory access control sudo aa-status 2>/dev/null | head -4 # 1.6 Kernel: live patching, CPU mitigations pro status | grep -i livepatch grep . /sys/devices/system/cpu/vulnerabilities/* # 1.7 Disks: is anything encrypted at rest? (look for crypto_LUKS) lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINT # 1.8 Containers: privilege flags per container sudo docker ps --format '{{.Names}}' | while read -r n; do sudo docker inspect --format \ '{{.Name}} user={{.Config.User}} secopt={{.HostConfig.SecurityOpt}} capdrop={{.HostConfig.CapDrop}} ro={{.HostConfig.ReadonlyRootfs}} privileged={{.HostConfig.Privileged}}' "$n" done sudo docker ps --format '{{.Names}} {{.Ports}}' # anything published on 0.0.0.0? sudo docker ps -q | xargs -r sudo docker inspect --format '{{.Name}} {{range .Mounts}}{{.Source}} {{end}}' | grep -c docker.sock # 1.9 Detection: is anything watching? systemctl is-active auditd 2>/dev/null command -v aide lynis 2>/dev/null systemctl list-timers --all --no-pager | head -30 # 1.10 Backups: when did a restore last succeed? (your own tooling; write the date down) ``` Write the answers into the record sheet in section 9 before you go further. --- ## 2. Measure before you restrict A limit that would have blocked your own work last week will block it next week. Count first. ```bash # Successful logins in 7 days, and where they came from sudo journalctl -u ssh --since "7 days ago" | grep -c 'Accepted ' sudo journalctl -u ssh --since "7 days ago" | grep 'Accepted ' | grep -oE 'from [0-9a-f.:]+' | sort | uniq -c | sort -rn | head # The busiest single minute (this is the number a rate limit has to clear) sudo journalctl -u ssh --since "7 days ago" -o short-iso | grep 'Accepted ' | cut -c1-16 | sort | uniq -c | sort -rn | head -3 # Failed attempts in 24 hours, and from how many addresses sudo journalctl -u ssh --since "24 hours ago" | grep -cE 'Failed |Invalid user ' sudo journalctl -u ssh --since "24 hours ago" | grep -E 'Failed |Invalid user ' | grep -oE 'from [0-9a-f.:]+' | sort -u | wc -l ``` How to read it: - `ufw limit` denies a source that opens 6 or more connections within 30 seconds (ufw manual page). If your busiest minute from one source is above about 10, that rule will lock out your own automation. - OpenSSH 9.8 and later can penalize sources that repeatedly fail authentication (`PerSourcePenalties`). Check `sshd -T` before adding a second ban tool that does the same job. - If hosted CI runners log in, their addresses change. An address allowlist cannot work. Closing the port means building a tunnel or private network path for CI first. - With key-only login, failed password attempts cannot succeed. They are noise, not progress. The same rule applies to every other limit: for a capability drop, find out what the process reads; for a read-only filesystem, find out what it writes. --- ## 3. The subscription blind spot Hardening checklists and scanners audit settings. They do not audit entitlements. The package manager compares your system only against the repositories that are switched on, so a source you never enabled reports no gap. Check each of these by name. None of them shows up as a failed setting: | Entitlement | How to check | What you miss without it | |---|---|---| | Ubuntu Pro ESM (free for personal use, up to 5 machines) | `pro security-status` | Security fixes for Universe packages | | Livepatch (part of Ubuntu Pro) | `pro status` | High and critical kernel fixes between reboots | | Firmware updates | `fwupdmgr get-updates` | BIOS, drive and device firmware fixes | | Vendor repositories you added by hand | `ls /etc/apt/sources.list.d/` | Whatever that vendor ships; also check each is still wanted | | Support windows | `pro status`, `ubuntu-distro-info --supported` | Any fixes at all once the release ends | Setup steps for the first one are in the Ubuntu Pro kit: https://jwatte.com/downloads/ubuntu-pro-esm-setup-kit.md --- ## 4. Decision table Copy this table. Fill in every row for your machine. "Skip" and "Accept the risk" are real answers when the reason is written down. Review the skipped rows when the facts change (a new service, a new person, a rebuild). | # | Item | Options | Decide by asking | |---|---|---|---| | 1 | Security subscriptions (section 3) | Enable / already on | Is any source off that costs nothing to turn on? | | 2 | Automatic security updates | On / manual with a schedule | Who installs fixes when you are away for two weeks? | | 3 | Firewall default deny | On / fix | Does anything listen that you did not put there? | | 4 | Admin login reachable from the internet | Close behind a private network / keep with key-only / rate limit | What logs in, from where, how often (section 2)? What breaks if it closes? | | 5 | Password login and root login | Disable / keep with reason | Is there any account that still needs a password over the network? | | 6 | Kernel live patching | On / off | How costly is a reboot here, and does a tested reboot routine already exist? | | 7 | CPU side-channel mitigations | Enable / wait for a planned reboot / accept | Does the fix need a boot flag? Who shares this hardware? | | 8 | Container privilege flags | Add no-new-privileges and drop capabilities / leave | Does a probe on a throwaway copy start cleanly (section 5)? | | 9 | Secrets at rest | Fix modes / no change | What role does each readable file have, and is its directory closed (section 6)? | | 10 | Detection layer | Add audit rules and an integrity baseline / none | Would anyone read the alerts? If not, make them quieter first (section 7). | | 11 | Disk encryption at rest | Encrypt at the next rebuild / accept | Who has physical or hypervisor access? Can you enter a passphrase at boot? | | 12 | Update coverage | Close gaps / record them | Is every pending item in your update routine's step list (section 8)? | | 13 | Backups | Test a restore / schedule one | When did a restore last work, not when did a backup last run? | | 14 | Admin key protection | Passphrase or hardware key / accept | Where do copies of the key live, and what protects each? | | 15 | Sudo scope | Narrow to named commands / keep | Does automation hold full sudo, or only what it needs? | | 16 | Outbound traffic | Filter to a short list / leave open | Does anything need the whole internet, or a few hosts? | | 17 | Logs off the machine | Ship a copy / accept | If the server were wiped tonight, would any record survive? | | 18 | Third-party access | Remove / time-limit / keep | Who else has a key or an account, and does it still need to exist? | | 19 | Unused services and accounts | Remove / keep with reason | What is installed or enabled that nothing uses? | Record each decision like this: ```text Item 6, kernel live patching. Decision: ENABLE. Date: 2026-11-18. By: owner. Reason: reboots are rare and take other workloads down. Checked first: the reboot routine reads the reboot-required flag and the newest installed kernel, neither of which live patching changes. Trade-off accepted: it installs through snapd, a second update channel that refreshes by itself. Revisit when: the reboot routine changes. ``` --- ## 5. Probe before you harden containers `no-new-privileges` stops a process gaining privilege through setuid programs and file capabilities. `cap_drop: ALL` removes the capabilities a root process in the container would otherwise hold. Either can stop a service from starting. Find out on a throwaway copy. ### File: probe.sh ```bash #!/bin/bash # probe.sh IMAGE [extra docker run args...] # Starts a throwaway copy of IMAGE with no network and the proposed flags, shows whether it stays up, # then removes it. Nothing running is touched. Extra args (-e, --cap-add, -v) go before the image. set -u [ $# -ge 1 ] || { echo "usage: probe.sh IMAGE [docker run args...]"; exit 2; } IMG="$1"; shift N="nnp-probe-$$" docker run -d --name "$N" --network none --pull never --memory 1g \ --security-opt no-new-privileges:true --cap-drop ALL "$@" "$IMG" >/dev/null || exit 1 sleep "${PROBE_WAIT:-15}" echo "state: $(docker inspect --format '{{.State.Status}} exit={{.State.ExitCode}}' "$N")" docker logs --tail 12 "$N" 2>&1 | cut -c1-200 docker rm -f -v "$N" >/dev/null echo "leftovers: $(docker ps -aq --filter "name=$N" | wc -l)" ``` Examples: ```bash sudo bash probe.sh your-web-image:tag sudo bash probe.sh postgres:16 -e POSTGRES_PASSWORD=probe-only \ --cap-add CHOWN --cap-add DAC_OVERRIDE --cap-add FOWNER --cap-add SETGID --cap-add SETUID ``` What passed for us, as a starting point and not a promise for your images: | Image kind | Flags that passed | |---|---| | Postgres (two versions) | no-new-privileges, drop ALL, add back CHOWN, DAC_OVERRIDE, FOWNER, SETGID, SETUID | | Model server, web UI, remote-desktop gateway and its daemon | no-new-privileges, drop ALL | | LLM proxy that reads owner-only host folders as root | no-new-privileges, drop ALL, add back DAC_READ_SEARCH | Three checks the probe does not do for you: 1. **Existing data.** A database's first start and its second start on existing data are different. Run the probe once without the flags on a throwaway volume, stop it, then run it again with the flags on the same volume. 2. **Live mounts.** A clean copy has no host folders. List what the real container mounts and who owns it: ```bash sudo docker inspect --format '{{range .Mounts}}{{.Source}}{{"\n"}}{{end}}' CONTAINER | \ xargs -r -n1 sudo stat -c '%a %U:%G %n' ``` If root in the container reads a folder that only another user may read, it is using a capability (DAC_READ_SEARCH or DAC_OVERRIDE). Dropping everything breaks it without an obvious error. 3. **Code paths the probe never reached.** A migration, a first-run setup, a plugin load. Say which ones you did not exercise. Then edit the definition only, and prove the edit only added lines: ```yaml services: app: security_opt: - no-new-privileges:true cap_drop: - ALL db: security_opt: - no-new-privileges:true cap_drop: - ALL cap_add: [CHOWN, DAC_OVERRIDE, FOWNER, SETGID, SETUID] ``` ```bash docker compose config > /tmp/before.txt # before the edit cp -p compose.yaml compose.yaml.pre-hardening # ... edit ... docker compose config > /tmp/after.txt diff /tmp/before.txt /tmp/after.txt | grep -c '^<' # must print 0: nothing removed ``` The flags take effect when each container is next recreated. If a service fails right after that, restore the backup file and recreate it. Not covered here, on purpose: user-namespace remapping (Docker's documentation says it is best enabled on a new installation, because it changes ownership of existing data), read-only root filesystems, and process limits. Each is worth its own probe. --- ## 6. Secrets at rest: role before mode A file mode alone does not make a finding. Before you report "readable secret": ```bash # Candidates: names, owners and modes only. Never print contents. sudo find / -xdev -type f \( -name '.env' -o -name '.env.*' -o -name '*.pem' -o -name '*token*' \) \ -printf '%m %u:%g %p\n' 2>/dev/null | sort -k3 # The directory gates above a file: one closed directory protects everything below it namei -l /path/to/the/file ``` For each file that other accounts could read, answer three questions: 1. **Role.** Is it a real secret, an example (`.env.example`), a test fixture, or public by nature (a browser origin-trial token, a content hash)? 2. **Gate.** Does a directory above it deny other accounts? Then the file's own mode does not matter. 3. **Other accounts.** Does any other account or service on this machine exist that should not read it? Optional: `umask 027` in the admin account's profile stops new files being group-writable or world-readable. --- ## 7. Detection that stays quiet A check that fires every day gets ignored. The goal is a small set of signals where each one means something. ### 7.1 Audit rules: narrow on purpose File watches on the places that change who can log in, what runs with privilege, and what starts by itself, plus kernel module loading. No rule that logs every command: sudo already writes an audit record for each command (record type USER_CMD) once the audit service runs, and a host that runs thousands of privileged commands a day through automation would bury everything else. Replace ADMIN with your admin account name. ### File: /etc/audit/rules.d/60-host.rules ```text ## Host audit rules: configuration changes that alter who can log in, what runs with privilege, ## and what starts by itself. File watches only, plus kernel module loading. -w /etc/passwd -p wa -k identity -w /etc/shadow -p wa -k identity -w /etc/group -p wa -k identity -w /etc/gshadow -p wa -k identity -w /etc/sudoers -p wa -k sudoers -w /etc/sudoers.d/ -p wa -k sudoers -w /etc/ssh/sshd_config -p wa -k sshd -w /etc/ssh/sshd_config.d/ -p wa -k sshd -w /home/ADMIN/.ssh/authorized_keys -p wa -k authkeys -w /etc/crontab -p wa -k cron -w /etc/cron.d/ -p wa -k cron -w /var/spool/cron/ -p wa -k cron -w /etc/systemd/system/ -p wa -k systemd -w /home/ADMIN/.config/systemd/user/ -p wa -k systemd-user -w /etc/ufw/ -p wa -k firewall -w /etc/docker/ -p wa -k docker -w /etc/audit/ -p wa -k auditconfig -w /usr/local/sbin/ -p wa -k local-sbin -w /etc/apt/sources.list.d/ -p wa -k apt-sources -a always,exit -F arch=b64 -S init_module,finit_module,delete_module -k modules ``` Notes: - Only list paths that exist. A watch on a missing path fails to load. - The current `auditctl` manual page marks the `-w` form as deprecated in favor of the syscall form, for example `-a always,exit -F arch=b64 -F path=/etc/passwd -F perm=wa -F key=identity`. The `-w` form above is what we loaded on auditd 4.1.2 and it works. Convert when you have time to test. - Raise log retention so a busy day does not rotate the evidence away. In `/etc/audit/auditd.conf` we set `max_log_file = 50` and `num_logs = 8` (the defaults were 8 MB and 5 files). ```bash sudo apt install auditd sudo install -m 640 -o root -g root 60-host.rules /etc/audit/rules.d/60-host.rules sudo augenrules --load sudo auditctl -l | wc -l # how many rules loaded sudo ausearch --input-logs -k sshd -i | tail # who changed the SSH config, and with what ``` Trap: `ausearch` and `aureport` read standard input when it is not a terminal. In a script, a cron job or an agent's shell they will swallow the rest of your script or wait forever. Always pass `--input-logs`. ### 7.2 Integrity baseline: scoped, with package updates sorted out AIDE records a baseline of files and reports differences. Out of the box it reports every package update, which trains people to stop reading it. Two changes fix that: a narrow scope, and a wrapper that asks the package manager about each difference. Scope: the places where a change means someone altered how the machine boots, logs in, schedules work or runs privileged code. Leave out data, caches and anything regenerated by routine updates, and write each exclusion down as a known gap. Replace ADMIN with your admin account name and add your own application paths where marked. ### File: /etc/aide/aide.conf ```text # File-integrity baseline. Driven by /usr/local/sbin/integrity-check (daily timer), not by a stock cron job. database_in=file:/var/lib/aide/aide.db.gz database_out=file:/var/lib/aide/aide.db.new.gz database_new=file:/var/lib/aide/aide.db.new.gz gzip_dbout=yes log_level=warning report_level=changed_attributes # Recorded: permissions, owner, group, file type, link target, SHA-256 of the content. # Timestamps and inode numbers are left out: package updates change them without changing the file. CONTENT = p+u+g+ftype+l+sha256 # --- boot chain: kernels and EFI binaries (initrd and the boot menu are regenerated on kernel updates: left out) /boot/vmlinuz CONTENT /boot/config- CONTENT /boot/System\.map CONTENT /boot/efi CONTENT # --- system programs and service definitions /usr/bin CONTENT /usr/sbin CONTENT /usr/lib/systemd CONTENT /usr/local CONTENT # --- system configuration /etc CONTENT !/etc/ld\.so\.cache$ !/etc/mtab$ !/etc/adjtime$ !/etc/\.pwd\.lock$ !/etc/resolv\.conf$ !/etc/ssl/certs !/etc/alternatives !/etc/(passwd|group|shadow|gshadow|subuid|subgid)-$ !/etc/lvm/(archive|backup|cache) # Units and rules that snapd rewrites on every snap refresh. Known gap: not integrity-checked here. !/etc/systemd/system/snap[-.] !/etc/systemd/system/[^/]+\.wants/snap[-.] !/etc/udev/rules\.d/[0-9]+-snap\. !/etc/dbus-1/[^/]+/snap\. # --- your own application and tooling (add lines like these) # /opt/yourapp CONTENT # !/opt/yourapp/cache # --- login and persistence points of each account that can log in /root/\.ssh CONTENT /home/ADMIN/\.ssh CONTENT !/home/ADMIN/\.ssh/known_hosts /home/ADMIN/\.(bashrc|profile|bash_profile|bash_logout)$ CONTENT /home/ADMIN/\.config/systemd/user CONTENT /home/ADMIN/\.local/bin CONTENT !/home/ADMIN/\.local/bin/__pycache__ ``` The wrapper sorts every difference into one of two kinds: - **package-verified:** the file belongs to an installed package and `dpkg --verify` confirms it matches what the package shipped. That is an ordinary update. When every difference is of this kind, the baseline moves forward by itself and nobody is paged. - **unexplained:** anything else. The baseline stays put, the check exits 1, the systemd unit shows as failed, and the list waits in `integrity-check status` until a person accepts it with a reason. ### File: /usr/local/sbin/integrity-check ```python #!/usr/bin/python3 """integrity-check: file-integrity checks built on AIDE. AIDE keeps a baseline (owner, mode, link target, SHA-256) of the system paths listed in /etc/aide/aide.conf. A plain AIDE check reports every package update as a change, which trains people to ignore it. This wrapper sorts each difference first: package-verified the file belongs to an installed package and `dpkg --verify` confirms it matches that package: an ordinary update. When every difference is of this kind (or a removal that coincides with package activity), the baseline moves forward by itself. unexplained anything else: an edited config file, a new unit or key, a binary no package vouches for. The baseline stays put, the check exits 1 (the systemd unit shows as failed) and the list is in `integrity-check status`. integrity-check check run the comparison (the daily timer does this) integrity-check status last result and the unexplained paths integrity-check accept --reason "why" you reviewed the list: make the current state the baseline integrity-check init build the first baseline Run as root. Nothing here prints file contents. """ import argparse import datetime as dt import json import os import pathlib import re import subprocess import sys CONF = "/etc/aide/aide.conf" DBDIR = pathlib.Path("/var/lib/aide") DB = DBDIR / "aide.db.gz" NEW = DBDIR / "aide.db.new.gz" STATE_DIR = pathlib.Path("/var/lib/integrity-check") STATE = STATE_DIR / "state.json" REPORT = STATE_DIR / "last-report.txt" LOG = pathlib.Path("/var/log/integrity-check.log") UNIT = "integrity-check.service" DPKG_LOGS = ("/var/log/dpkg.log.1", "/var/log/dpkg.log") def now(): return dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") def log(msg): with open(LOG, "a") as f: f.write("%s %s\n" % (now(), msg)) os.chmod(LOG, 0o640) def load_state(): try: return json.loads(STATE.read_text()) except (OSError, ValueError): return {} def save_state(**kw): st = load_state() st.update(kw) STATE_DIR.mkdir(mode=0o755, exist_ok=True) tmp = STATE.with_suffix(".tmp") tmp.write_text(json.dumps(st, indent=2) + "\n") os.chmod(tmp, 0o644) os.replace(tmp, STATE) return st def aide(*args): return subprocess.run(["aide", "--config", CONF, *args], capture_output=True, text=True, timeout=7200) def swap(): if not NEW.exists(): raise RuntimeError("AIDE wrote no new database (%s)" % NEW) os.chmod(NEW, 0o600) os.replace(NEW, DB) def names(report, key): v = report.get(key) or {} if isinstance(v, dict): return dict(v) out = {} for x in v: if isinstance(x, str): out[x] = "" elif isinstance(x, dict): out[x.get("path") or x.get("name") or ""] = x.get("flags", "") out.pop("", None) return out def compare(): """Run AIDE and return (added, removed, changed) as {path: flags}.""" p = aide("--check", "-A", "report_format=json") if p.returncode < 0 or p.returncode >= 14: raise RuntimeError("aide --check failed (exit %d): %s" % (p.returncode, (p.stderr or p.stdout).strip()[-400:])) start = p.stdout.find("{") if start < 0: raise RuntimeError("aide printed no JSON report: %s" % p.stdout.strip()[-300:]) rep = json.loads(p.stdout[start:]) return names(rep, "added"), names(rep, "removed"), names(rep, "changed") def owners(path): for cand in (path, path[4:] if path.startswith("/usr/") else None): if not cand: continue p = subprocess.run(["dpkg-query", "-S", cand], capture_output=True, text=True) pk = set() for line in p.stdout.splitlines(): if line.startswith("diversion "): continue left = line.rpartition(": ")[0] pk.update(n.strip() for n in left.split(",") if n.strip()) if pk: return pk return set() def dpkg_failing(pkgs): """Paths that `dpkg --verify` says differ from what the package shipped.""" bad, pkgs = set(), sorted(pkgs) for i in range(0, len(pkgs), 40): p = subprocess.run(["dpkg", "--verify", *pkgs[i:i + 40]], capture_output=True, text=True) for line in p.stdout.splitlines(): m = re.match(r"^\S+\s+(?:[a-z]\s+)?(/.*)$", line) if m: bad.add(m.group(1)) if not m.group(1).startswith("/usr/"): bad.add("/usr" + m.group(1)) return bad def dpkg_activity_since(stamp): """True when dpkg installed, upgraded or removed something after the baseline was taken.""" try: since = dt.datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ") except (TypeError, ValueError): return False for name in DPKG_LOGS: try: lines = pathlib.Path(name).read_text(errors="replace").splitlines() except OSError: continue for line in lines: m = re.match(r"^(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d) (install|upgrade|remove|purge) ", line) if m and dt.datetime.strptime(m.group(1), "%Y-%m-%d %H:%M:%S") >= since: return True return False def classify(added, removed, changed, baseline): explained, unexplained = [], [] present = {} present.update({p: ("added", f) for p, f in added.items()}) present.update({p: ("changed", f) for p, f in changed.items()}) own = {} for path in present: if os.path.islink(path) or not os.path.isfile(path): continue own[path] = owners(path) failing = dpkg_failing(set().union(*own.values())) if own else set() for path, (kind, flags) in sorted(present.items()): pk = own.get(path) if pk and path not in failing: explained.append((kind, path, "matches package " + ", ".join(sorted(pk))[:60])) elif pk: unexplained.append((kind, path, "differs from package " + ", ".join(sorted(pk))[:60])) elif os.path.islink(path): unexplained.append((kind, path, "symbolic link")) elif os.path.isdir(path): unexplained.append((kind, path, "directory attributes")) else: unexplained.append((kind, path, "no package owns it")) activity = dpkg_activity_since(baseline) if removed else False for path in sorted(removed): (explained if activity else unexplained).append( ("removed", path, "removed alongside package activity" if activity else "removed, no package activity since the baseline")) return explained, unexplained def write_report(explained, unexplained): lines = ["integrity-check at %s" % now(), ""] lines.append("UNEXPLAINED (%d): review, then `sudo integrity-check accept --reason \"...\"`" % len(unexplained)) lines += [" %-8s %s (%s)" % r for r in unexplained] or [" none"] lines += ["", "package-verified or package removals (%d):" % len(explained)] lines += [" %-8s %s (%s)" % r for r in explained[:200]] or [" none"] if len(explained) > 200: lines.append(" ... %d more" % (len(explained) - 200)) REPORT.write_text("\n".join(lines) + "\n") os.chmod(REPORT, 0o644) def cmd_init(_a): DBDIR.mkdir(mode=0o700, exist_ok=True) p = aide("--init") if p.returncode != 0: raise RuntimeError("aide --init failed (exit %d): %s" % (p.returncode, (p.stderr or p.stdout).strip()[-400:])) swap() m = re.search(r"Number of entries:\s*(\d+)", p.stdout) save_state(baseline=now(), baseline_reason="first baseline", status="ok", checked=now(), unexplained=0, explained=0, paths=[], entries=int(m.group(1)) if m else None) log("init: baseline created (%s entries)" % (m.group(1) if m else "?")) print("baseline created: %s entries" % (m.group(1) if m else "?")) return 0 def cmd_check(_a): st = load_state() if not DB.exists(): raise RuntimeError("no baseline yet: run `integrity-check init`") added, removed, changed = compare() total = len(added) + len(removed) + len(changed) if not total: save_state(status="ok", checked=now(), unexplained=0, explained=0, paths=[]) write_report([], []) log("check: no differences") print("ok: no differences from the baseline") return 0 explained, unexplained = classify(added, removed, changed, st.get("baseline")) write_report(explained, unexplained) if not unexplained: aide("--update") swap() save_state(status="ok", checked=now(), unexplained=0, explained=len(explained), paths=[], baseline=now(), baseline_reason="%d package-verified changes" % len(explained)) log("check: %d package-verified changes accepted automatically" % len(explained)) print("ok: %d package-verified changes, baseline moved forward" % len(explained)) return 0 save_state(status="changed", checked=now(), unexplained=len(unexplained), explained=len(explained), paths=["%s %s (%s)" % r for r in unexplained[:60]]) log("check: %d UNEXPLAINED, %d package-verified: %s" % (len(unexplained), len(explained), "; ".join(r[1] for r in unexplained[:12]))) print("CHANGED: %d unexplained, %d package-verified. See: integrity-check status" % (len(unexplained), len(explained))) return 1 def cmd_accept(a): if not (a.reason or "").strip(): print("accept needs --reason \"what changed and why it is fine\"", file=sys.stderr) return 2 added, removed, changed = compare() paths = sorted(set(added) | set(removed) | set(changed)) p = aide("--update") if p.returncode < 0 or p.returncode >= 14: raise RuntimeError("aide --update failed (exit %d)" % p.returncode) swap() who = os.environ.get("SUDO_USER") or os.environ.get("USER") or "root" save_state(status="ok", checked=now(), unexplained=0, explained=0, paths=[], baseline=now(), baseline_reason="accepted by %s: %s" % (who, a.reason.strip()[:200])) log("accept by %s (%s): %d paths: %s" % (who, a.reason.strip()[:200], len(paths), "; ".join(paths[:40]))) try: subprocess.run(["systemctl", "reset-failed", UNIT], capture_output=True) except OSError: pass # no systemd here (a container, for example): nothing to reset write_report([], []) print("accepted %d paths; the current state is the new baseline" % len(paths)) return 0 def cmd_status(_a): st = load_state() if not st: print("no state yet: run `sudo integrity-check init`") return 1 print("status: %s" % st.get("status")) print("checked: %s" % st.get("checked")) print("baseline: %s (%s)" % (st.get("baseline"), st.get("baseline_reason"))) print("last run: %s unexplained, %s package-verified" % (st.get("unexplained"), st.get("explained"))) for p in st.get("paths") or []: print(" - " + p) return 0 if st.get("status") == "ok" else 1 def main(): ap = argparse.ArgumentParser(prog="integrity-check", description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("cmd", choices=["init", "check", "accept", "status"]) ap.add_argument("--reason", default="") a = ap.parse_args() if a.cmd != "status" and os.geteuid() != 0: print("run as root (sudo integrity-check %s)" % a.cmd, file=sys.stderr) return 2 try: return {"init": cmd_init, "check": cmd_check, "accept": cmd_accept, "status": cmd_status}[a.cmd](a) except Exception as e: # an integrity check that cannot run must be loud, not silent msg = "%s: %s" % (type(e).__name__, e) try: save_state(status="error", checked=now(), error=msg[:400]) log("ERROR " + msg[:400]) except OSError: pass print("ERROR " + msg, file=sys.stderr) return 3 if __name__ == "__main__": sys.exit(main()) ``` ### File: /etc/systemd/system/integrity-check.service ```ini [Unit] Description=File-integrity check (AIDE baseline; package-verified changes are accepted automatically) ConditionPathExists=/var/lib/aide/aide.db.gz [Service] Type=oneshot ExecStart=/usr/local/sbin/integrity-check check Nice=10 IOSchedulingClass=idle ``` ### File: /etc/systemd/system/integrity-check.timer ```ini [Unit] Description=Daily file-integrity check [Timer] OnCalendar=*-*-* 09:40:00 UTC RandomizedDelaySec=10min Persistent=true [Install] WantedBy=timers.target ``` Install and self-test: ```bash sudo apt install aide sudo install -d -m 755 /etc/aide sudo install -m 644 -o root -g root aide.conf /etc/aide/aide.conf sudo aide --config /etc/aide/aide.conf --config-check sudo install -m 755 -o root -g root integrity-check /usr/local/sbin/integrity-check sudo install -m 644 integrity-check.service integrity-check.timer /etc/systemd/system/ sudo systemctl daemon-reload sudo integrity-check init # first baseline sudo integrity-check check; echo "exit=$?" # expect 0 sudo sh -c 'echo selftest > /usr/local/sbin/.integrity-selftest' sudo integrity-check check; echo "exit=$?" # expect 1: the planted file is unexplained sudo integrity-check status sudo rm -f /usr/local/sbin/.integrity-selftest sudo integrity-check check; echo "exit=$?" # expect 0 again sudo systemctl enable --now integrity-check.timer ``` Day to day: - After any change you make on purpose in a watched path, run `sudo integrity-check check`, read the list, then `sudo integrity-check accept --reason "what and why"`. The log keeps who, why and which paths. - Wire the failed unit into whatever you already look at: `systemctl --failed`, a health script, a monitor. You do not need a new alert channel. - Do not import the script as a Python module as root. Python writes a cache folder beside it and the next check flags the cache. Known limits, stated so you do not find them later: - `dpkg --verify` checks file contents against the checksum the package recorded. It does not record link targets, so a new or changed symbolic link is always "unexplained", even when a package shipped it. In our container test, installing `less` gave four package-verified files and one unexplained entry: a link the package installs. Expect an occasional accept after installing new software. - Installing a scanner or a snap can add units that change on every refresh. Exclude them the day you see them and note the gap. - A baseline stored on the same machine can be rewritten by anyone who already has root. This catches the tier below that: a changed key file, a new unit, an edited config, a replaced binary. For more, copy the baseline off the machine. ### 7.3 One baseline scan ```bash sudo apt install lynis sudo lynis audit system --quick systemctl list-timers --all --no-pager | grep -i lynis # did installing it schedule it? ``` Treat the hardening index as a baseline for this machine, to compare against later, not as a grade. Most suggestions are generic. On a key-only, single-admin server, password ageing, legal banners and separate partitions matter far less than anything about services you did not know were listening. --- ## 8. Does your update routine cover everything? "We have an update command" and "everything that needs updating is in it" are different claims. List every pending item and write beside it which step of your routine handles it. | Pending item | Covered by the routine? | If not, how is it handled? | |---|---|---| | OS packages (including ESM sources) | | | | Kernel (and reboot, or live patch) | | | | Container engine | | | | Each service image | | | | Command-line tools installed outside the package manager | | | | Snaps | | | | Firmware | | | | Bundled certificate stores (inside images, language package caches, runtimes) | | | | Anything flagged "manual" | | | On our server this took ten minutes and found three gaps. --- ## 9. Record sheet ```text Machine role: Date: Surveyed by: Security subscriptions: ESM infra [ ] ESM apps [ ] Livepatch [ ] firmware source [ ] Automatic updates: on [ ] off [ ] origins include ESM [ ] Listening services: ____ expected: ____ Firewall default: deny [ ] other: ____ Password login: off [ ] on, because: ____ Accounts with a shell: ____ Successful logins / 7 days: ____ busiest minute from one source: ____ Failed logins / 24 hours: ____ from ____ addresses Containers: total ____ with no-new-privileges ____ with dropped capabilities ____ mounting the Docker socket ____ Disk encryption: yes [ ] no [ ] decision: ____ Audit service: running [ ] rules loaded: ____ Integrity baseline: entries ____ last check ____ unexplained ____ Last successful restore test: ____ Decisions (one line each: item, decision, reason, revisit when): 1. 2. 3. ``` --- ## 10. What this does not protect against - Someone who already has root, or who has your admin key and its passphrase. - A vulnerable application you run. Hardening the host does not fix the code. - A compromised dependency or image. That needs pinning, a release hold and a supply-chain check. - Physical or hypervisor access when disks are not encrypted. - Anything you skipped in section 4. That is what the reasons are for. --- ## 11. Prompt for an AI coding agent ```text Read the attached Server Hardening Survey Kit in full. Its section 0 rules bind you. 1. Run sections 1 and 2 on this machine. They are read-only. Do not print the contents of any secret, token, key or env file: names, owners and modes only. 2. Fill in the record sheet from section 9 with what you found. 3. Fill in the decision table from section 4. For each row give your recommendation, the measurement it rests on, what could break, and how to undo it. Say "skip" where skipping is reasonable, and why. 4. Stop. Wait for me to decide each row. Change nothing until I answer for that row. 5. For each row I approve: back up first, make the smallest change, show me the proof it worked, and tell me what will only take effect later (after a recreate or a reboot). 6. If a number you gave me earlier turns out wrong when you look closer, tell me and correct it. 7. Finish with: what changed, what was skipped and why, what is still open, and what none of this covers. Do not restart, recreate or reboot anything without my explicit yes for that action. ```