# The Safe Update Command Kit A template for one `update` command on a self-hosted Linux server that runs containers, a desktop in a container, a virtual machine and a few command-line tools. Type `update`, read the checks, pick everything or a few parts, and it patches them in a safe order, proves each part works, keeps the result across reboots and rolls back what fails. There's also a GitHub Actions workflow that runs the same actions on request without giving GitHub a shell. It comes from the command we built for our own server, described in [Updated on Disk, Old in Memory](https://jwatte.com/blog/safe-update-command-self-hosted-ai-server/). Names, paths and services below are placeholders. Replace them with yours. ## 1. What "updated" has to mean A part counts as updated only when all five are true: 1. **Installed:** the new version is on disk. 2. **Running:** the process in memory is the new version (a binary replaced on disk keeps running old until restart). 3. **Checked:** a test that exercises the service passes, after the change and again after a restart. 4. **Pinned:** the version is recorded by digest or exact version, so a reboot or a stray `pull` can't change it. 5. **Reversible:** the previous version and its data are saved, and the rollback has been written before the change. Each step in the command follows the same contract: detect, back up, apply, verify, roll back on failure, record. ## 2. Inventory worksheet Fill this in before writing any code. Every row becomes a step or part of one. | Part | How it's installed | Tracks | Health check | Restart cost | Auto-restart allowed? | |---|---|---|---|---|---| | Ubuntu packages | apt | security + updates | `dpkg --audit` empty, SSH banner | none, sshd briefly | yes | | Docker engine | apt (docker.com repo) | patch releases | every container still running, same start time | all containers, unless live-restore | yes, with live-restore | | Tunnel (cloudflared) | binary + launcher symlink | GitHub releases, SHA256 | service active after restart | a few seconds of remote access | yes | | AI CLIs | npm / native installer | registry latest | `--version`, `--help` | none | yes | | Database container | image by digest | same major (e.g. `postgres:16`) | `pg_isready` | dependent services reconnect | yes | | App containers | image by digest | `latest` or a channel tag | an HTTP check that touches real data | seconds each | yes | | Desktop container | locally built image | weekly build | login page answers, browser profile intact | closes the browser | ask first | | Windows VM container | image by digest | version tags | VM SSH port answers after boot | Windows restarts | ask first, never automatic | ## 3. Order, and why | Step | Why it sits here | |---|---| | Preflight | Refuse to start with low disk, a half-configured dpkg or Docker down. Save a baseline: every container's image pin and start time. | | Host packages | Security fixes first. One apt transaction in its own root unit, so a dropped session can't kill dpkg halfway. | | Docker engine | Before images, so containers are recreated by the new engine. Live-restore keeps them running during the upgrade. | | CLIs and binaries | Cheap and independent. Restart any service still running an old binary. | | Service images | One at a time, database first, each verified after a restart. | | Locally built images | Only when the base image or locked dependencies changed, and only after their tests pass. | | VM image | Opt-in, because it reboots the guest. | | Desktop image | Build and test on a separate TEST container first, then promote the exact tested image. | | Persistence | Write pins the boot-time restore reads; set restart policies. | | Verification | The full health check. Anything failed is reported and alerted. | A step that fails and rolls back cleanly doesn't stop later independent steps. A rollback that can't be verified stops the run, but persistence and verification still run. ## 4. Run every change in a detached unit Updates that run inside your SSH session die with it. Start the work as a transient systemd unit instead, then watch its log. Ctrl-C stops watching, not the update. ```sh sudo systemd-run --unit "update-$(date -u +%Y%m%dT%H%M%SZ)" --uid "$USER" --gid "$USER" \ --property WorkingDirectory="$HOME" \ --property StandardOutput=append:"$RUN_DIR/log.txt" --property StandardError=append:"$RUN_DIR/log.txt" \ --setenv HOME="$HOME" --collect /usr/bin/python3 "$HOME/ops/update/update.py" execute "$RUN_ID" tail -f "$RUN_DIR/log.txt" ``` Anything that runs dpkg goes one level further: its own root unit with `--wait`, writing to the same log, so dpkg finishes even if the orchestrator is killed. `NEEDRESTART_SUSPEND=1` matters on Ubuntu 24.04 and later, where needrestart restarts affected services by itself after every apt run. The update restarts services deliberately, one at a time, with checks. ```sh sudo systemd-run --quiet --wait --collect --unit "update-apt-$(date +%s)" \ --property StandardOutput=append:"$RUN_DIR/log.txt" --property StandardError=append:"$RUN_DIR/log.txt" \ --setenv DEBIAN_FRONTEND=noninteractive --setenv NEEDRESTART_SUSPEND=1 \ apt-get install -y --no-remove -o Dpkg::Options::=--force-confold --only-upgrade pkg1=VER pkg2=VER ``` Keep state in a JSON file per run (plan, each step's status, every change with its before and after). Resuming and rolling back both read it. ## 5. Host packages with a real rollback Save the installed version of every package before upgrading. The archive often no longer serves the old version once an update supersedes it, so fall back to the apt cache, then to `dpkg-repack`, which rebuilds a .deb from the installed files. ```sh mkdir -p "$RUN_DIR/apt-rollback" && cd "$RUN_DIR/apt-rollback" for p in $(apt list --upgradable 2>/dev/null | awk -F/ 'NR>1{print $1}'); do v=$(dpkg-query -W -f='${Version}' "$p") apt-get download "$p=$v" 2>/dev/null || sudo dpkg-repack "$p" done ``` Verify after the install: every package at its new version, `dpkg --audit` empty, and if OpenSSH changed, `sshd -t` passes and **port 22 answers with an SSH banner**. On Ubuntu 22.10 and later, sshd is socket-activated: right after an upgrade `ssh.service` can be inactive while `ssh.socket` is listening, which is healthy. Check what a client sees: ```python import socket, time def ssh_ok(wait=120): end = time.monotonic() + wait while time.monotonic() < end: try: with socket.create_connection(("127.0.0.1", 22), timeout=10) as s: if s.recv(64).startswith(b"SSH-"): return True except OSError: pass time.sleep(3) return False ``` On failure: `dpkg --configure -a` if needed, then `apt-get install --allow-downgrades` with the saved files. ## 6. Docker engine without stopping containers Docker's live-restore keeps containers running while the daemon restarts. It can be switched on with a reload, and Docker supports it across patch releases only (29.9.0 to 29.9.1, not 29.9 to 30.0). 1. Save `/etc/docker/daemon.json` (or note that it doesn't exist) and every running container's `StartedAt`. 2. Write the file with `"live-restore": true` and run `sudo systemctl reload docker`. Confirm `docker info --format '{{.LiveRestoreEnabled}}'` says `true` and no start time changed. 3. Upgrade `docker-ce`, `docker-ce-cli`, `docker-ce-rootless-extras` and `containerd.io` in one transaction. 4. Check every container is still running with the same start time and passes its health check. If one lost its published port, restart that one container before blaming the engine. 5. Put live-restore back to what it was. **Write `{"live-restore": false}` explicitly** if there was no file: a reload with no config file keeps the old value. Confirm `docker info` agrees. Why switch it back off: with live-restore on, stopping the daemon leaves containers running, so a host shutdown no longer passes through Docker's own graceful stop, and a VM in a container loses its configured shutdown grace period. Rollback: reinstall the saved engine packages, with live-restore still on, then restore the setting. ## 7. Binaries that are updated on disk but not in memory Linux keeps running the program a process started with. Replacing the file, or repointing a symlink, changes nothing until the service restarts. `/proc//exe` shows what is really running, with ` (deleted)` appended when the file it started from is gone. ```sh pid=$(systemctl --user show -p MainPID --value my-tunnel.service) running=$(readlink "/proc/$pid/exe" | sed 's/ (deleted)$//') installed=$(readlink -f "$HOME/.local/bin/cloudflared") [ "$running" = "$installed" ] || echo "stale: restart my-tunnel.service" ``` For each binary: install the new version beside the old, smoke test it (`--version`, `--help`), swap the launcher symlink atomically (`ln -s new tmp && mv -T tmp launcher`), restart the service that runs it, and confirm the service is active. If it isn't, point the launcher back and restart again. `needrestart` is a useful second opinion: its report lists services and user sessions running outdated binaries. User sessions are only reported, not restarted, and that is exactly where a tunnel started as a user service hides. ## 8. Container images by digest For each service: compare the registry digest of the tag you track with the digest the container runs. Skip if equal. ```sh docker buildx imagetools inspect postgres:16 --format '{{json .Manifest}}' | jq -r .digest docker image inspect --format '{{join .RepoDigests " "}}' "$(docker inspect -f '{{.Image}}' my-db)" ``` Then: record the old `repo@sha256` pin, pull, stop the container, archive its named volumes (and dump its database if it has one), recreate it from a Compose override that pins `image: repo@sha256:...`, run its health check, restart it, run the check again. On failure: stop it, restore the archived data, recreate from the old pin, check again. Write the override to a file that both the update command and your boot-time restore read, so the pin survives. ## 9. A desktop or VM: test first, promote the exact image For a desktop in a container: 1. Build the candidate and deploy it to a separate TEST container. Restart TEST and check it again. 2. Run the production checks against the candidate with a **copy** of the live profile while the real desktop keeps running. 3. Close the browser gracefully, stop the desktop, archive its profile volume, deploy the same image digest. 4. Check: health, login page, browser running, profile intact. Close the browser gracefully again, restart, check again. 5. On failure: restore the old image, the old Compose files and the archived profile, then check the old one. Compare browser extensions by ID only (32 letters a to p) and fail only when one is **missing**. Browsers leave temporary folders while updating, and an exact match fails on those. For a VM in a container, never restart it automatically. Ask first, then wait for its SSH port before calling it healthy. ## 10. The menu A plain numbered menu works in every terminal, including a phone and a browser-based console. ``` NEEDED Host packages (Ubuntu) 15 upgrades (11 security) NEEDED Docker engine docker-ce 29.9.0 -> 29.9.1, containerd.io 2.4.1 -> 2.4.2 NEEDED Host CLIs tunnel restart (runs an older cloudflared than installed) ok Windows 11 VM image current NEEDED Desktop image (TEST -> live) 8 package updates 1 Update everything that is needed (asks before restarting your desktop) 2 Choose what to update 3 Full health check 4 Restart something 5 Roll back something 6 Recover an interrupted run / resume the last run 7 Watchdog: status, run now, enable, disable 8 Last run report and log ``` Offer the same actions as subcommands (`update check`, `update all --yes`, `update run apt,docker`, `update rollback `, `update restart `, `update health`) so scripts and the workflow can call them. ## 11. A watchdog that heals only what is safe Run it every five minutes from a systemd timer. It skips while an update holds the lock. | Situation | Action | |---|---| | A run died with a container half-updated | Restore its old image and data | | dpkg interrupted and no apt running | `dpkg --configure -a` in a root unit | | Docker or sshd down | Start them | | Tunnel or remote-access service down | Restart it (at most 3 times in 6 hours) | | live-restore left on by a cut-off upgrade | Put the saved setting back | | Pinned container missing | Recreate it from its pin | | Container unhealthy for 15 minutes | Restart it (at most 3 times in 6 hours) | | Updated in the last 24 hours and still failing after two restarts | Roll its image back, keep its data | | VM container unhealthy | Alert only | | Desktop on probation after a promotion | Leave it to the desktop's own watchdog | | Disk below 30 GB | Alert | Read back every record you write with the same code that consumes it. A missing field in a record can switch off another watchdog without a single error you'd notice. ## 12. Running it from GitHub Actions Create a dedicated SSH key whose `authorized_keys` line forces one script and drops everything else: ``` restrict,command="/home/you/ops/update/gh-dispatch.sh" ssh-ed25519 AAAA... github-actions-update ``` `restrict` disables port, agent and X11 forwarding, PTY allocation and `~/.ssh/rc`. `command=` runs the script for every login and hands it the requested command in `SSH_ORIGINAL_COMMAND`. The script allows a fixed list and refuses the rest: ```bash #!/bin/bash set -u REQ=${SSH_ORIGINAL_COMMAND:-} STEP='(apt|docker|cli|services|desktop-test|desktop)' FLAGS='( --(yes|no-desktop|no-follow))*' log() { printf '%s %s %s\n' "$(date -u +%FT%TZ)" "${SSH_CLIENT%% *}" "$1" >> "$HOME/ops/update/gh-dispatch.log"; } if [[ $REQ =~ ^(check|health|status|report)$ ]] || [[ $REQ =~ ^logs\ --from\ [0-9]{1,12}$ ]] \ || [[ $REQ =~ ^all$FLAGS$ ]] || [[ $REQ =~ ^run\ $STEP(,$STEP)*$FLAGS$ ]]; then log "RUN update $REQ" exec "$HOME/.local/bin/update" $REQ &2; exit 64 ``` Test it from the server itself before trusting it: `status` must run; `id; hostname`, `status; id` and a reboot request must be refused, and the log must show all four. The workflow (`.github/workflows/server-update.yml`): ```yaml name: Server update on: workflow_dispatch: inputs: action: { type: choice, options: [check, health, status, update-all, report], default: check } promote_desktop: { type: boolean, default: true } concurrency: { group: server-update, cancel-in-progress: false } permissions: { contents: read } jobs: update: runs-on: ubuntu-latest timeout-minutes: 330 env: { ACTION: "${{ inputs.action }}", PROMOTE: "${{ inputs.promote_desktop }}" } steps: - name: SSH with a pinned host key env: { KEY: "${{ secrets.UPDATE_SSH_KEY }}", KNOWN: "${{ secrets.UPDATE_KNOWN_HOSTS }}", HOST: "${{ secrets.UPDATE_HOST }}" } run: | install -m 700 -d ~/.ssh printf '%s\n' "$KEY" > ~/.ssh/k && chmod 600 ~/.ssh/k printf '%s\n' "$KNOWN" > ~/.ssh/known_hosts printf 'Host srv\n HostName %s\n User you\n IdentityFile ~/.ssh/k\n StrictHostKeyChecking yes\n ServerAliveInterval 30\n' "$HOST" > ~/.ssh/config - name: Run run: | case "$ACTION" in update-all) f=""; [ "$PROMOTE" = true ] || f=" --no-desktop"; ssh srv "all --yes --no-follow$f" ;; *) ssh srv "$ACTION"; exit $? ;; esac off=0 while :; do out=$(ssh srv "logs --from $off") || { sleep 30; continue; } printf '%s\n' "$out" | grep -v '^#UPDATE-LOG' meta=$(printf '%s\n' "$out" | grep '^#UPDATE-LOG' | tail -1) off=$(sed -n 's/.*offset=\([0-9]*\).*/\1/p' <<<"$meta") st=$(sed -n 's/.*status=\([a-z_]*\).*/\1/p' <<<"$meta") case "$st" in running|queued) sleep 30 ;; *) break ;; esac done [ "$st" = finished ] ``` Inputs reach the script only as environment variables, never pasted into the script text. The `logs --from N` subcommand prints the run log from byte N plus one `#UPDATE-LOG run=... offset=... status=...` line, so the job can poll in short SSH calls and survive a dropped connection. The run itself keeps going on the server regardless. Store three repository secrets: the private key, the server's host keys in `known_hosts` format, and the address. Generate the key on the server, store it with `gh secret set NAME < keyfile`, then shred the local copy. Re-run the setup to rotate. ## 13. Test before you trust it - `update check` on a quiet day. Read every NEEDED line and decide whether you agree. - Run one cheap step alone: `update run cli`. - Kill the run mid-step (`systemctl stop update-`) and confirm `update recover` or the watchdog puts the container back. - Upgrade one container to a tag you know is broken and watch it roll back. - Promote the desktop once while you can watch, and keep the old image until a week has passed. - Run the workflow with `status` before `update-all`. ## 14. Traps we hit, so you don't - sshd socket activation makes `systemctl is-active ssh` lie after an OpenSSH upgrade. Check the banner. - `systemctl reload docker` with no `daemon.json` keeps the old live-restore value. Write the default explicitly. - A tunnel ran a week-old binary for three days after its launcher was updated. Compare `/proc//exe` with the launcher target and restart. - A record written without one required field silently disabled a watchdog. Read every record back with its consumer. - An exact browser-extension comparison failed on a temporary folder and rolled back a good desktop. Compare IDs, fail on missing. - A user-level systemd unit that runs apt can be killed mid-dpkg. Give dpkg its own root unit. - A `pkill -f` pattern that also appears in your own SSH command line kills your SSH. Find PIDs first, kill numbers.