Yesterday we wrote about turning on Ubuntu Pro ESM, a free subscription that put security fixes behind a third of the packages on our server. Then came the uncomfortable question: we had asked for best practices on this machine many times, so why had ESM never come up?
The answer is a blind spot, and it applies to more than one subscription. Hardening checklists check settings. An entitlement is not a setting. The package manager compares a system only against the repositories that are switched on, so a source that was never enabled reports nothing missing. "Up to date" meant up to date against what we had asked for.
So the next question was the right one: what else is missing? We answered it by surveying the machine instead of reciting a generic list. The survey produced eight candidate items. We went through them one at a time. Three became changes. Three of the first recommendations turned out wrong or overstated once we measured, and this article is more useful if it says which.
The general method, with a kit you can run yourself, is in the companion guide.
What was already in place
A survey that only lists gaps makes a machine sound open. This one was not.
- The firewall denies by default, and no container publishes a port to the internet.
- Logins are key-only. The root password is locked. Most authorized keys are tied to one forced command each, so the key can run that command and nothing else.
- AppArmor enforces 107 profiles. No container mounts the Docker socket.
- Unattended upgrades are on and include the ESM sources.
- Backups go offsite and a timer verifies a restore. A health check with 29 rows was passing.
- Every change runs through one update tool with a 24-hour hold on new releases, a supply-chain check and a rollback. We described it in Updated on Disk, Old in Memory.
The eight items
Scroll to compare columns and rows when the table extends beyond the box.
| Item | Decision | Why |
|---|---|---|
| Kernel live patching was off | Enabled | Reboots here are rare and take other workloads down |
| No detection layer | Added | One baseline scan, narrow audit rules, a scoped integrity check |
| Containers without privilege limits | Added to the definitions | Tested on throwaway copies first |
| Secret files that looked readable | Surveyed, no change needed | The readable ones were examples and test fixtures |
| Pending patches | Checked against the update tool | Most are covered by one command. Three things are not |
| Three more items | No change now | See below |
Three items stayed as they were. Each was a judgment about cost against benefit on this particular machine, and each describes something an attacker would like to know, so they are not in this article. The companion guide's decision table covers a wider set of such decisions in general terms. Leaving an item alone with the reason written down is a decision. Leaving it alone because nobody looked is the thing we are trying to stop doing.
Wrong answer 1: a rate limit that would have locked us out
One early suggestion was a connection rate limit on the login service, using the firewall's built-in limit rule. It sounds harmless. The ufw manual says what it does: it will "deny connections if an IP address attempts to initiate 6 or more connections within 30 seconds."
Then we counted our own logins. In seven days there were 4,572 successful ones, and 4,498 of them came from a single internal virtual machine that runs our automation. Its busiest minute held 19 logins. The rule would have started refusing our own jobs within the first burst.
There was a second reason not to add it. OpenSSH has had its own penalty system since version 9.8: the server records a penalty against addresses that repeatedly fail authentication and refuses them for a while. It was already doing that job here.
The lesson is not about SSH. It is that a limit should be tested against last week's legitimate traffic before it is switched on. The numbers were sitting in the journal the whole time.
Change 1: Livepatch, and a reversal
Our ESM article said we had left Livepatch off on purpose, because the server has a guarded reboot routine and we wanted one mechanism responsible for the kernel. One day later we turned it on. That deserves an explanation.
Livepatch applies fixes for high and critical kernel vulnerabilities to the running kernel. Canonical's description is that it removes "the immediate need to reboot to upgrade the kernel, instead allowing the downtime to be scheduled." On this machine a reboot is rare and expensive, because it takes a desktop and a virtual machine down with it. So the window between a kernel fix and our next reboot can be long. That window is what Livepatch covers.
The worry was that it would confuse the update automation. We checked before enabling:
- The reboot logic reads four things: the reboot-required flag, the newest installed kernel against the running one, pending firmware and pending microcode. Livepatch changes none of them. The tool still asks for a reboot when a new kernel package lands, which is what we want. Livepatch covers the gap. It does not replace the reboot.
- The tool's subscription report reads only the two ESM services.
- The automation key cannot run the command that enables or disables services, and a test asserts that.
After enabling, the kernel reported as supported with nothing to apply yet, and the update tool's reports read exactly as before.
The trade-off we accepted. Livepatch arrives as a snap. That brings in snapd, a second update channel that by default refreshes four times a day outside our release hold and supply-chain check. On a server where "every change goes through one tool" is a principle, that is a real exception. It can be paused with snap refresh --hold. We also have no row for live patches in the update tool's status yet, so they are visible only by asking Livepatch directly.
Change 2: a detection layer that stays quiet
The server had prevention and no detection. If someone had added a key or a scheduled job, nothing would have said so. We added three things, and kept each one narrow.
One baseline scan. Lynis 3.1.6 ran 270 tests and reported a hardening index of 66, one warning and 45 suggestions. The warning was the missing integrity tool, fixed the same hour. Most of the suggestions are generic: password ageing on a key-only server, legal banners, separate partitions. We treat the index as a number to compare against later, not a grade.
One thing to watch: installing the package enabled a daily timer by itself. We had agreed to one scan, so we turned the timer off. Installing a scanner can quietly schedule it.
Audit rules, 24 of them. File watches on the identity files, sudo rules, the SSH server configuration, authorized keys, scheduled jobs, service definitions, the firewall, the container engine configuration, the audit configuration itself, the package sources, and kernel module loading. There is no rule that records every command. Sudo already writes an audit record for each command once the audit service runs, and a server that runs thousands of privileged commands a day through automation would bury the signal. We raised log retention from 5 files of 8 MB to 8 files of 50 MB.
An integrity baseline that does not cry wolf. AIDE records a baseline of files and reports what changed. Out of the box it reports every package update, which trains people to stop reading it. Two decisions fixed that.
The first is scope. We baseline the places where a change means someone altered how the machine boots, logs in, schedules work or runs privileged code: kernels and boot binaries, system programs, service definitions, system configuration, our own stack's configuration, and the login and persistence points of the accounts that can log in. That came to 5,034 entries, and the baseline built in 8 seconds. Each entry records permissions, owner, group, type, link target and a SHA-256 hash. Timestamps are left out, because updates change them without changing the file.
The second is a small wrapper that asks the package manager about every difference. If the file belongs to an installed package and dpkg --verify confirms it matches what the package shipped, it is an ordinary update and the baseline moves forward by itself. Anything else is unexplained: the baseline stays put, the daily check fails, and our existing health check shows the failed unit as a warning. No new alert channel was needed. A person clears it with an accept command that requires a reason and logs who, why and which paths.
We tested it by planting a file (flagged), editing a packaged configuration file (classed as differing from its package), adding a script no package owns (classed that way), and removing a file with no package activity (flagged).
What went wrong, which is the useful part:
- Our own edits later that day, and a cache folder one of our tests created, showed up as unexplained within minutes. That is the tool working, and it made a good first run of the accept flow.
- Installing Livepatch added a dozen unit files that would change on every snap refresh. We caught it the same day and excluded them, with the gap written into the configuration.
- A package database stores no link targets, so a changed symbolic link is always unexplained, even when a package made the change.
ausearch, the audit log search tool, reads standard input when it is not attached to a terminal. Run from a piped script it swallowed the rest of the script. The manual page has the answer:--input-logs, which it says "is needed if you are using ausearch from a cron job."- A baseline stored on the same machine can be rewritten by someone who already has root. This is detection for the tier below that.
Change 3: container privilege flags, and wrong answer 2
We surveyed 14 running containers. Six were already hardened; two of those run as a non-root user with every capability dropped, a read-only filesystem and a process limit. Eight lacked the no-new-privileges flag.
The first count was wrong. The initial report said six containers lacked the flag and called all of them root. The careful count was eight, of which four stay root and four drop to a service user after starting. A count that feeds a decision gets checked twice now.
One of the eight, a virtual machine runner, was left out, because recreating it restarts the machine it runs.
For the other seven the method mattered more than the result. Before changing anything, we started a throwaway copy of each image with the proposed flags, with no network, no image pull and capped memory, and removed it afterwards.
Scroll to compare columns and rows when the table extends beyond the box.
| Image kind | Flags that passed |
|---|---|
| Postgres (two versions) | no-new-privileges, drop all capabilities, add back CHOWN, DAC_OVERRIDE, FOWNER, SETGID, SETUID. Also tested as a second start on existing data, which is the real case |
| Model server, web UI, remote-desktop gateway and its daemon | no-new-privileges, drop all capabilities |
| LLM proxy | no-new-privileges, drop all capabilities, add back DAC_READ_SEARCH |
The LLM proxy is the instructive one. On a clean copy it started fine with everything dropped. The live container, though, reads two host folders that only their owner may read. Root inside the container was reading them through a capability. Per the capabilities manual, DAC_READ_SEARCH is what lets a process "bypass file read permission checks and directory read and execute permission checks." Dropping everything would have broken that service with no obvious error. Only checking who owns the real mounts showed it. A probe on a clean copy does not prove the live case.
What the flag does, so nobody oversells it: the kernel's no_new_privs documentation describes it as stopping a program from gaining privileges through execve, where "the most obvious examples are setuid/setgid programs and file capabilities." A container that already runs as root gains little from that alone. For those four, dropping capabilities is the larger change.
The edits went into the service definitions only. As proof that they only added, we rendered the full configuration before and after and compared: added lines, zero removed. Nothing was restarted. The flags take effect when each container is next recreated, which the next update run does for most of them.
Not done: user-namespace remapping, read-only root filesystems, and process limits on those eight. The probe of the LLM proxy image used a minimal configuration with no database, so its migration path was not exercised.
No change needed: secrets at rest, and wrong answer 3
A shallow search reported that four of seven env files were readable by other accounts. That sounded like a finding. The full survey found 43 env-style files. Twenty-eight hold real values, and every one of those was already readable by its owner only. The 15 readable files were .env.example templates and test fixtures.
So the first claim was overstated. A permission finding needs the file's role, not only its mode. Two more things the survey turned up:
- The home directory itself is closed to other accounts, so files beneath it are unreachable by them whatever their own mode says. Check the directories above a file before reading its mode.
- Some files with "token" in the name were public by nature, such as browser origin-trial tokens, or were content hashes.
Checked: does the update tool cover what is pending?
We listed every pending item and wrote beside it which step of the update tool handles it. The container engine, the service images, the desktop image and most command-line tools are covered by one command. Three things are not:
- The firmware scan. It is a separate command. The routine check only warns when the last scan is stale.
- Bundled certificate stores. A certificate authority that the operating system no longer trusts can live on in copies: inside a language package manager's vendored bundle within an image, in package caches, in a runtime. Each copy clears a different way and none clears through the update run.
- One tool flagged manual, because updating it may need a pairing step repeated.
"We have an update tool" and "everything that needs updating is in it" are different claims. Checking one against the other took ten minutes.
What the three corrections have in common
Each wrong first answer came from the same place: a reasonable-sounding recommendation made before measuring.
Scroll to compare columns and rows when the table extends beyond the box.
| First answer | What measuring showed |
|---|---|
| Add a firewall rate limit to logins | Our own automation peaks at 19 logins a minute. The rule trips at 6 in 30 seconds |
| Four of seven env files are readable | 43 files. All 28 real ones were owner-only. The readable 15 were examples and fixtures |
| Six containers run as root without the flag | Eight lacked the flag. Four stay root, four drop to a service user |
None of these would have been a disaster. All three would have cost time or trust: a locked-out job, an afternoon fixing permissions that were fine, a change plan built on the wrong list. When an assistant, a scanner or a colleague hands you a finding, the fair question is what it was measured against.
What we did not do
We did not reboot. We did not recreate any container. We did not run the pending updates as part of this work. The first scheduled integrity check runs the morning after this article. The details of three items are withheld on purpose, and the audit rule list and integrity scope described here are summaries, not the exact set.
Fact-check notes and sources
- Our records: survey and changes on our own server on November 18, 2026. 4,572 successful logins in 7 days, 4,498 from one internal VM, busiest minute 19. Lynis 3.1.6: index 66, 270 tests, 1 warning, 45 suggestions. auditd 4.1.2 with 24 rules. AIDE 0.19.2: 5,034 entries, baseline built in 8 seconds. 14 running containers: 6 already hardened, 8 without
no-new-privileges. 43 env-style files: 28 real, 15 examples and fixtures. - The earlier article and its counts: A Third of Our Server's Packages Had No Security Promise, including the choice to leave Livepatch off, which this article reverses.
- The firewall limit rule: ufw manual page, "6 or more connections within 30 seconds".
- OpenSSH penalties: OpenSSH 9.8 release notes, the
PerSourcePenaltiesoption. - Livepatch: Canonical, Canonical Livepatch Client for the quoted description and delivery as a snap, and Which services are for me?.
- Snap refresh schedule and holds: Canonical, Manage updates, "scheduled to refresh four times per day".
- Capabilities: capabilities(7). The flag: kernel documentation, No New Privileges Flag. Docker options: docker container run and user namespace remapping.
- Package verification: dpkg(1),
--verify: "the only functional check performed is an md5sum verification of the file contents". - AIDE: project page and manual. Audit search: ausearch(8). Audit record types: the Linux audit project's message dictionary, USER_CMD. Lynis: CISOfy.
Related reading
- Hardening a One-Admin Linux Server: the method behind this article, with a downloadable kit.
- A Third of Our Server's Packages Had No Security Promise: the subscription that started the question.
- Updated on Disk, Old in Memory: the update tool these checks sit beside.
- The check was green because it measured a state no visitor is ever in: more cases of a passing check that told us nothing.
- Build a Self-Hosted AI Server From Scratch: the server this runs on, layer by layer.
This post is informational, not security-consulting or legal advice. Test any change on a copy of your own system first. Mentions of Ubuntu, Canonical, Livepatch, OpenSSH, Docker, PostgreSQL, AIDE, Lynis, CISOfy and AppArmor are nominative fair use. No affiliation is implied.