One Safe Update Command for a Self-Hosted AI Server

The tunnel that carries our remote desktop to the internet is a single program called cloudflared. On October 12, 2026 an update job installed version 2026.10.0, pointed the launcher at it, ran the new binary's version check and called the job done. The tunnel kept running the old version for another 70 hours.

Nothing was broken, which is why nobody noticed. Linux keeps running the program a process started with, and replacing the file on disk changes nothing until that process restarts. Cloudflare's own update guide says it in one line: after updating, "you must restart the service for it to take effect." We found out by accident, while installing an unrelated package, when Ubuntu's needrestart printed this:

User sessions running outdated binaries:
 ubuntu @ user service: jwatte-cloudflared.service[302641]

That line is why this post exists. On a server that runs a desktop, a Windows virtual machine, AI services and a pile of command-line tools, "updated" has to mean more than "the new version is on disk". So we built one command, update, that checks what is out of date, updates it in a fixed order, restarts what needs restarting, proves each part works, keeps the result across reboots and rolls back whatever fails. Then we ran it for real, and it caught four mistakes, three of them ours.

Download the Safe Update Command Kit in Markdown. It's a generalized version you can adapt: the order and why, the check and rollback for each step, the menu, a self-healing watchdog, a GitHub Actions workflow that can't open a shell, and the traps below.

What "updated" has to mean

We settled on five conditions. A part only counts as updated when all of them hold:

Installed. The new version is on disk.
Running. The process in memory is the new version.
Checked. A test that touches the real service passes, right after the change and again after a restart.
Pinned. The version is recorded by exact version or image digest, so a reboot or a stray pull can't change it.
Reversible. The old version and its data are saved before anything changes, and the rollback is part of the same step.

The cloudflared update met the first, the third if you squint, and none of the rest. Most update scripts stop at the first.

Why the binary on disk isn't the binary in memory

When a service starts, the kernel maps the executable into memory. If you then overwrite the file, or repoint a symlink at a newer one, the running process doesn't care. It keeps the old code until it exits. The Linux proc manual describes the tell: /proc/<pid>/exe is a link to the program the process actually runs, with " (deleted)" appended when the original file is gone.

So the check takes two lines. Compare what the service runs with what the launcher points at:

pid=$(systemctl --user show -p MainPID --value jwatte-cloudflared.service)
readlink "/proc/$pid/exe"                 # what is running
readlink -f ~/.local/bin/cloudflared      # what is installed

If they differ, restart the service and confirm it came back. Cloudflare warns that the restart "will impact traffic currently being served", and for us that meant our browser sessions blinked for a few seconds. That's the honest price of an update, and paying it on purpose beats discovering three days later that you never did.

Two details made our case easy to miss. We run the tunnel with --no-autoupdate, on purpose, because we want versions pinned rather than changing under us. That puts the restart on us. And the tunnel runs as a user service. needrestart reports user sessions running outdated binaries but leaves them running, so its warning only helps if someone reads it. (Since Ubuntu 24.04, needrestart does restart affected system services by itself after an apt run, which is a separate reason to control when it runs. Our update suspends it and restarts services one at a time, with checks.)

Our update now does this for every long-running binary: install the new version beside the old, smoke test it, swap the launcher atomically, restart the service that runs it, check the service is active, and put the old launcher back if it isn't. When we ran it, the CLI step printed "tunnel restarted on cloudflared version 2026.10.0", and the Cloudflare dashboard showed the connector on 2026.10.0 with seven minutes of uptime.

The order, and why each step sits where it does

Typing update opens a menu. It runs the checks first, so you see what needs doing before anything changes:

  NEEDED  Host packages (Ubuntu)         15 upgrades (11 security)
  NEEDED  Docker engine                  containerd.io 2.4.1 -> 2.4.2, docker-ce 29.9.0 -> 29.9.1
  NEEDED  Host CLIs                      claude 2.1.287 -> 2.1.300, netlify 27.12.0 -> 27.12.1
  NEEDED  Service images                 Enclave Postgres, LiteLLM, Ollama, Open WebUI, n8n newer
  ok      Guacamole                      current
  ok      Windows 11 VM image            current (dockur 6.06)
  NEEDED  Webtop image (TEST -> desktop) 8 package updates

  1  Update everything that is needed (asks before restarting your desktop)
  2  Choose what to update
  3  Full health check
  4  Restart something
  5  Roll back something
  6  Recover an interrupted run / resume the last run
  7  Watchdog: status, run now, enable, disable
  8  Last run report and log

Option 1 runs the steps in this order:

Scroll to compare columns and rows when the table extends beyond the box.

Step	What it changes	How it proves itself	How it undoes itself
Preflight	Nothing	50 GB free, dpkg not interrupted, Docker up; saves every container's image pin and start time	Not needed
Host packages	Ubuntu updates	New versions installed, `dpkg --audit` empty, SSH answers	Reinstalls the saved packages
Docker engine	docker-ce and containerd	Every container still running with the same start time	Reinstalls the saved engine
Host CLIs	Claude Code, Codex, Netlify, gh, cloudflared	Version and help output, then services restarted	Points the launcher back
Service images	Postgres, Qdrant, LiteLLM, Ollama, Open WebUI, n8n	Service checks, a restart, the checks again	Old image and archived data
Integrations	A locally built image	Its own test suite, before anything moves	Old image
Guacamole, Windows VM	Compose files pinned by digest	Health, and for Windows the VM's SSH port	Old Compose file and data
Webtop build	A new desktop image on a TEST container	Managed stage checks, a TEST restart	Managed TEST rollback
Desktop promotion	The live desktop	16 checks after promotion and after a restart	Old image, Compose files and profile
Persistence	Pin files, restart policies	The boot-time restore reads them	Not needed
Verification	Nothing	The full health check	Not needed

The order matters in three places. Security packages go first because they're the reason to update at all. The Docker engine goes before any container is recreated, so new containers start on the new engine. And the desktop goes last, because it's the step that interrupts a person.

Every run executes in its own systemd unit, not in the terminal that started it. A dropped SSH session or a closed browser tab doesn't stop an update halfway; Ctrl-C only stops watching. Anything that runs dpkg goes one level deeper, into a root unit of its own, so a killed orchestrator can't leave packages half-configured. We learned that one on October 12, when systemd killed a user-level unit that was running apt while the boot image was being rebuilt, and the boot tooling stayed half-configured until it was repaired by hand.

Four things the first real run caught

We ran the command against our own server at 00:55 UTC on October 15. Then we fixed what it found and ran it twice more.

1. Ubuntu's sshd isn't running most of the time, and that's healthy

The first pass upgraded 15 packages, including OpenSSH, and apt finished cleanly. Then our own check failed the step and rolled every package back. The check asked systemd whether ssh.service was active, and the answer was no.

Ubuntu has socket-activated sshd by default since 22.10. Its announcement puts it plainly: sshd "will not be started until an incoming connection request is received." The OpenSSH upgrade stopped the service and left ssh.socket listening, which is exactly right. Our check was wrong. It now does what a client does: connect to port 22 and wait for the SSH- banner. The second pass upgraded all 15 packages in 29 seconds.

The rollback did its job, and that matters as much as the bug. Before upgrading, the step saves each installed package as a .deb file. When the archive no longer serves the old version, it falls back to dpkg-repack, which "creates a .deb file out of a Debian package that has already been installed on your system."

2. Docker can keep containers running through its own upgrade, if you ask

A Docker engine upgrade restarts the daemon, and by default that stops every container. On this server that would mean the Windows VM, the desktop, Guacamole and the AI services all going down mid-session.

Docker's live-restore setting keeps containers running while the daemon restarts. Docker's documentation adds two conditions that shaped the step: it's "only supported when installing patch releases (YY.MM.x), not for major (YY.MM) daemon upgrades", and you can switch it on "by reloading the Docker daemon" rather than restarting it. 29.9.0 to 29.9.1 is a patch release. So the step reloads with live-restore on, confirms no container restarted, upgrades the engine, and checks every container again. All 17 kept running with their original start times.

Then the step tried to switch live-restore back off, and that failed. Our server had no /etc/docker/daemon.json at all, so the step deleted the file it had created and reloaded. The daemon kept live-restore on. A reload with no config file doesn't fall back to the defaults; it keeps what it has. The fix is to write {"live-restore": false} explicitly.

Why switch it off at all? With live-restore on, stopping the daemon leaves containers running. A host shutdown then no longer passes through Docker's own graceful stop, and our Windows VM loses the two-minute shutdown grace period it's configured with. We'd rather Windows shut down properly.

3. One missing field had switched off a watchdog for three days

The desktop's managed updater has its own watchdog. For 24 hours after a promotion it probes the desktop and rolls it back if the new image goes bad. While reading its logs, we found it had failed on every run since October 12, along with the weekly check and backup retention. Every failure carried the same line: "current default record: schema_version must be 1".

The October 12 promotion had written the desktop's release record without one field. Nothing crashed. Three timers simply failed in a log nobody was reading: the new desktop's 24-hour probation passed with no watchdog behind it, and the weekly check and backup retention stayed broken for nearly three days. The fix took one field. The lesson took longer: whatever writes a record must read it back with the same code that will consume it. Our promotion now does that, and the new watchdog repairs that exact defect if it ever comes back.

4. A good desktop image rolled itself back

The webtop candidate built and passed its TEST container checks in 18 minutes. Promotion then ran the production checks against a copy of the live browser profile while the real desktop kept running, closed Chrome gracefully, archived the 7.8 GB profile, deployed the new image and passed all 16 checks. Then, by design, it restarted the desktop and checked again.

The second check failed: "Chrome extensions changed across promotion". The rollback put back the old image, the old Compose files and the archived profile in a little over three minutes, and the old desktop passed its own checks.

The image was fine. Our check compared the exact list of folders in Chrome's extension directory. Chrome had run for about half a minute with network access, and the restart cut it off mid-task. We think it left a temporary folder behind, the kind Chrome uses while updating an extension; we can't prove which folder it was, because the rollback restored the profile from its archive. Either way, the check was wrong to want an exact match. It now compares extension IDs only, fails only if one goes missing, and closes Chrome gracefully before the second restart too.

The retry hit one more snag. The first attempt had already filed its backups under that release's ID, and the managed backup code refuses to overwrite a file. So the retry promotes the same tested image under a fresh release ID. The third run promoted the desktop at 01:46 UTC, verified after promotion and after a restart, and the final health check reported 27 checks with none failed.

The watchdog: what it heals and what it refuses

A five-minute timer runs a separate watchdog between updates. It sticks to things that are safe to do without a person:

recovers a container left half-updated by an interrupted run, using that run's saved image and data;
repairs interrupted dpkg, starts Docker, restores sshd, and restarts the tunnel and our remote-desktop service;
recreates a pinned container that went missing;
restarts a container that stays unhealthy for 15 minutes, at most three times in six hours;
rolls a container back to its previous image, keeping its data, if it was updated in the last 24 hours and still fails after two restarts.

It refuses two things. It never restarts the Windows VM; it only raises an alert, because restarting Windows uninvited is how you lose someone's unsaved work. And it leaves the desktop alone during the 24 hours after a promotion, because the desktop's own watchdog owns that window. Two watchdogs disagreeing about the same container is worse than one.

Running it from GitHub without giving GitHub a shell

We also wanted to start the same actions from GitHub Actions: check, health, update everything, roll back one thing. Our first instinct was to put an SSH key in a repository secret. A plain key would give anyone who obtained that secret a full shell on the server.

OpenSSH has had the answer for years. A line in authorized_keys can force a command and strip everything else. The restrict option will "enable all restrictions, i.e. disable port, agent and X11 forwarding", and it also disables PTY allocation. The command= option runs a fixed script for every login, and per the sshd manual "the command originally supplied by the client is available in the SSH_ORIGINAL_COMMAND environment variable." Our script compares that request with a short list of allowed patterns and refuses everything else:

restrict,command="/home/you/ops/update/gh-dispatch.sh" ssh-ed25519 AAAA... github-actions-update

The setup script tests the key before storing it: status must run; id; cat /etc/hostname, status; id and restart host must all be refused. Our log shows exactly that, then the first real request from a GitHub runner, which ran status, report and health and nothing else. The workflow pins the server's host key, takes its inputs as environment variables rather than pasting them into a script, and uses a concurrency group so, as GitHub's documentation puts it, "there can be at most one running job or workflow in a concurrency group at any time." Long runs don't hold one SSH connection open for an hour. The job polls the run log in short calls, and the update keeps going on the server regardless.

What we left alone, on purpose

Desktop Commander had a new minor version. Its remote pairing may change between versions, so the check lists it as a manual update instead of applying it.
Claude Code updates through its own installer channel, which gave us 2.1.292 while npm's latest was 2.1.300. We take the installer's version rather than mixing install methods.
The Windows VM image is opt-in every time, because it restarts Windows.
Kernel updates get installed, but the command never reboots by itself. The menu offers a reboot that asks you to type REBOOT, stops Windows gracefully first and closes the desktop browser.

Adapting this to your server

Our command is about 2,400 lines of Python because it carries our specific services. Yours won't need ours. It needs the same contract for each part: detect, back up, apply, verify, roll back, record. The kit starts with an inventory worksheet. List every part, how it's installed, what it tracks, the check that proves it works, what a restart costs and whether anything may restart it without asking. Fill that in honestly and most of the code writes itself.

Then test it on a quiet day before you trust it. Kill a run halfway and watch it recover. Point one container at a tag you know is broken and watch it roll back. Promote your desktop once while you're sitting in front of it.

Fact-check notes and sources

Run records: our own logs from October 15, 2026 (UTC): runs at 00:55, 01:36 and 01:42; Docker engine 29.9.0 to 29.9.1 with 17 containers kept running; 15 host packages rolled back on the first pass and upgraded on the second; webtop build and TEST from 01:08 to 01:26; the desktop rollback from 01:30:16 to 01:33:30; promotion at 01:46. The tunnel's launcher moved to 2026.10.0 at 03:29 UTC on October 12 and the tunnel restarted on it at 01:37 UTC on October 15.
cloudflared updates: Cloudflare, Update cloudflared (restart after update; restarts affect traffic; package-manager installs) and run parameters (--no-autoupdate).
Running binaries: Linux man-pages, proc_pid_exe(5).
needrestart on Ubuntu 24.04 and later: Ubuntu Discourse, needrestart changes in Ubuntu 24.04: service restarts.
Socket-activated sshd: Ubuntu Discourse, SSHd now uses socket-based activation (Ubuntu 22.10 and later).
Package rollback: Debian, dpkg-repack(1).
Docker live-restore: Docker Docs, Live restore (patch releases only; enable by reload; skipped releases).
Forced commands: OpenBSD, sshd(8), AUTHORIZED_KEYS FILE FORMAT (restrict, command=, SSH_ORIGINAL_COMMAND).
GitHub Actions: GitHub Docs, Workflow syntax (workflow_dispatch input types and concurrency).

Updated on Disk, Old in Memory: The One Update Command We Trust on Our AI Server