# Hardening checklist for a self-hosted AI server

Kit file for guide doc 2 ("Build in layer order" and "From-scratch checklist"). Work top to bottom.
Each line has the check, the command that reads the current state, and why it matters. Versions and
defaults as of October 2026 on Ubuntu 26.04 LTS. Read before you change; keep a second SSH session open
for every access change; take a provider snapshot before risky steps.

## Before anything else

- [ ] **Out-of-band console works.** Log in through the provider's KVM or rescue console once and set a
  local console password. Why: SSH keys do not work at a KVM console, and a firewall or SSH mistake
  otherwise means a support ticket.
- [ ] **Provider snapshot or image backup exists.** Why: the fastest rollback for any early mistake.

## Accounts and SSH

- [ ] **A normal admin user with sudo; root login off.** Read: `sudo sshd -T | grep -E 'permitrootlogin|passwordauthentication|kbdinteractiveauthentication'`.
- [ ] **Key-only SSH.** Copy your key (`ssh-copy-id <user>@<server-ip>`), open a second session with the
  key, then create `/etc/ssh/sshd_config.d/00-hardening.conf` with `PasswordAuthentication no`,
  `KbdInteractiveAuthentication no`, `PermitRootLogin no`. The first value read wins and the drop-ins are
  read in name order, so the file must sort before any cloud-init drop-in. Test with `sudo sshd -t`,
  apply with `sudo systemctl restart ssh`, and confirm with `sudo sshd -T` from the second session. Why:
  password logins are guessed all day; an edited file is not proof the running daemon changed.
- [ ] **Agent accounts are separate and unprivileged.** No sudo, not in the `docker` group. Why: Docker
  group membership is root in practice.
- [ ] **Every automation key is a forced-command key.** Read: `grep -c 'command=' ~/.ssh/authorized_keys`.
  Shape: `restrict,command="/path/to/one-script" ssh-ed25519 AAAA... purpose`. Why: a leaked automation
  key can then run one script, not a shell.

## Firewall and exposure

- [ ] **ufw denies inbound except SSH.** `sudo ufw status verbose`; set with `sudo ufw allow OpenSSH`
  first, then `sudo ufw default deny incoming`, `sudo ufw enable`.
- [ ] **No container port on a public address.** Read: `sudo ss -ltnp` and `docker ps --format '{{.Names}} {{.Ports}}'`.
  Every published port must start with `127.0.0.1:`. Why: Docker-published ports bypass ufw.
- [ ] **Test exposure from outside.** From another network, try each service port on `<server-ip>` over
  IPv4 and IPv6; all must fail except SSH. Why: local checks cannot prove public exposure.
- [ ] **Consider closing public SSH.** With Tailscale SSH or Cloudflare's SSH access, port 22 can be shut.
  Keep the provider console as the way back in.

## Cloudflare Tunnel and Access

- [ ] **Access application before the route.** For each hostname: create the Access application (deny by
  default, allow named emails), then add the tunnel route with "Protect with Access". Why: a route created
  first is public.
- [ ] **Refusal test.** In a private browser window, every hostname must send you to sign-in; a second
  email address not on the policy must be refused; the web socket path of the desktop must be refused too.
- [ ] **Apps verify the Access JWT themselves** (audience, issuer, signature) when they make decisions on
  identity. Header-trusting apps (Guacamole header auth) must listen on loopback only.

## Services

- [ ] **Images pinned** by tag and then by digest in the deployed Compose files; updates are deliberate.
- [ ] **Secrets in owner-only files** (`chmod 600`), referenced by name from Compose or systemd; never in
  briefs, chats, command lines, logs or repos. Scan before every commit with gitleaks.
- [ ] **One model gateway.** Callers get LiteLLM virtual keys limited to their models, with budgets; the
  master key stays with the admin. Open WebUI's direct Ollama link is off.
- [ ] **Open WebUI**: sign-up off; `ENABLE_PERSISTENT_CONFIG=False` if you manage settings from Compose.
- [ ] **Qdrant**: admin key for writers, read-only key for searchers, port on loopback.
- [ ] **n8n**: owner account set, Execute Command and file nodes excluded, encryption key backed up.
- [ ] **Desktop Commander**: `allowedDirectories` set to project folders after setup; telemetry decided;
  unused devices revoked. Its blocked-commands list is a convenience, not a sandbox.

## Updates, backups, alerts

- [ ] **Security updates automatic**: `cat /etc/apt/apt.conf.d/20auto-upgrades`; decide a reboot window
  (`Unattended-Upgrade::Automatic-Reboot` and `Automatic-Reboot-Time`) or reboot by hand weekly.
- [ ] **Backups off the server**, encrypted, with the password stored off the server; `restic check` and a
  test restore to a scratch folder on a schedule. RAID is not a backup.
- [ ] **Alerts reach your phone**, and the ntfy topic is long, random and private (or protected by tokens).
- [ ] **Record the baseline**: `docker ps`, `systemctl --user list-timers`, `ss -ltnp` and versions in a
  dated file, so the next change has something to compare against.