← Back to Blog

Hardening a One-Admin Linux Server: Survey First, Measure Before You Restrict

· 15 min read A small lantern casting a circle of warm amber light on a deep navy floor, with a closed padlock lying in the light.

Most hardening advice is a list of settings. Turn this off, set that to no, install this tool. We followed a fair amount of it on our own server and still missed a free subscription that covered a third of our packages. The list was not wrong. It was answering a different question from the one we needed answered.

This guide is the method we used the day after, when we asked what else was missing. It is written for the common small-business case: one Linux server, one person who administers it, a handful of services in containers, and no security team. The worked example, with our own numbers and three recommendations that turned out wrong, is in the companion article.

Everything here is also in a single file you can follow by hand or give to an AI coding agent: the Server Hardening Survey Kit. It includes the commands, a decision table, a container probe script, audit rules, an integrity-check script with its timer, and a paste-ready agent prompt. We ran the kit's install and self-test sequence in a clean Ubuntu 24.04 container before publishing it.

The method in seven steps

  1. Survey the machine with read-only commands.
  2. Measure your own traffic before adding any limit.
  3. Check entitlements as well as settings.
  4. Decide each item on purpose, and let "skip" be a real answer.
  5. Probe a throwaway copy before hardening a container.
  6. Add detection that stays quiet enough to be read.
  7. Check that your update routine covers everything pending.

1. Survey first

A generic checklist cannot know what your machine runs. A survey can. Every command in this step only reads.

pro security-status                 # which packages have security coverage, and from where
sudo ss -tlnp                       # what is listening
sudo ufw status verbose             # what the firewall allows
sudo sshd -T | grep -E '^(passwordauthentication|permitrootlogin|persourcepenalties) '
grep . /sys/devices/system/cpu/vulnerabilities/*
lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINT
systemctl is-active auditd
systemctl list-timers --all --no-pager

The kit has the full set, including one line per container that shows its user, privilege flags and dropped capabilities.

Two habits make the survey worth more than its output. Write down what was already right, so the result does not read as though the machine were open. And treat the survey output as sensitive. It is a description of how to attack the machine, so it does not belong in a chat with an outside service or a public ticket.

2. Measure before you restrict

A limit that would have blocked your own work last week will block it next week. Before adding a rate limit, an allowlist or a dropped permission, count what it would have stopped.

For logins, three numbers are enough:

# successful logins in a week
sudo journalctl -u ssh --since "7 days ago" | grep -c 'Accepted '

# the busiest single minute
sudo journalctl -u ssh --since "7 days ago" -o short-iso | grep 'Accepted ' \
  | cut -c1-16 | sort | uniq -c | sort -rn | head -3

# failed attempts in a day
sudo journalctl -u ssh --since "24 hours ago" | grep -cE 'Failed |Invalid user '

Then compare with what the limit does. The firewall's limit rule, per the ufw manual, will "deny connections if an IP address attempts to initiate 6 or more connections within 30 seconds." If your own automation logs in more often than that from one address, the rule locks it out. Ours peaked at 19 logins in a minute.

Three more things to know before restricting logins:

  • OpenSSH may already be doing it. Since version 9.8 the server can penalize addresses that repeatedly fail authentication and refuse them for a time. Check sshd -T for persourcepenalties before installing a second tool for the same job.
  • Hosted CI runners change address. If a build service logs in to deploy, an address allowlist cannot work. Closing the port means building a tunnel or private network path for those jobs first.
  • With key-only login, failed password attempts cannot succeed. They are noise in the log. They are not progress for the attacker.

The same rule holds for every other restriction. Before dropping a capability from a container, find out what the process reads. Before making a filesystem read-only, find out what it writes.

3. The subscription blind spot

This is the step no settings checklist has. A scanner audits configuration. An entitlement, a thing you are allowed to switch on, is outside its view. And the package manager compares your system only against the repositories that are enabled, so a source you never turned on reports no gap at all.

Check these by name:

Scroll to compare columns and rows when the table extends beyond the box.

Entitlement Check What it covers
Ubuntu Pro ESM pro security-status Security fixes for community-maintained packages. Free for personal use on up to five machines
Livepatch pro status High and critical kernel fixes applied without an immediate reboot
Firmware updates fwupdmgr get-updates BIOS, drive and device firmware
Repositories you added by hand ls /etc/apt/sources.list.d/ Whatever that vendor ships. Confirm each is still wanted
The release's support window pro status Everything, once it ends

We wrote up the first one in detail in the Ubuntu Pro ESM article. On our server it surfaced 21 waiting fixes in ten minutes.

4. Decide each item, and let "skip" be an answer

The survey gives you a list of candidates. The mistake is to treat the list as a to-do list. Some items are cheap and obviously right. Some cost more than they return on your machine. Some cannot be done without rebuilding. The goal is that every row has a decision and a reason, not that every row is fixed.

Scroll to compare columns and rows when the table extends beyond the box.

Item The question that decides it
Security subscriptions Is any source off that costs nothing to turn on?
Automatic security updates Who installs fixes when you are away for two weeks?
Firewall default deny Does anything listen that you did not put there?
Admin login reachable from the internet What logs in, from where, how often? What breaks if it closes?
Password and root login Does any account still need a password over the network?
Kernel live patching How costly is a reboot, and is there already a tested reboot routine?
CPU side-channel mitigations Does the fix need a boot flag and a reboot? Who shares the hardware?
Container privilege flags Does a probe on a throwaway copy start cleanly?
Secrets at rest What role does each readable file have, and is its directory closed?
Detection Would anyone read the alerts? If not, make them quieter first
Disk encryption Who has physical or hypervisor access? Can you enter a passphrase at boot?
Update coverage Is every pending item in your update routine's step list?
Backups When did a restore last work, not when did a backup last run?
Admin key protection Is the key passphrase-protected or hardware-backed, and where do copies live?
Sudo scope Does automation hold full sudo, or only the commands it needs?
Outbound traffic Does anything need to reach the whole internet, or a short list of hosts?
Logs off the machine If the server were wiped tonight, would any record of what happened survive?
Third-party access Who else has a key or an account, and does it still need to exist?
Unused services and accounts What is installed or enabled that nothing uses?

Many of these have honest "not now" answers. Moving the admin login behind a private network breaks any outside job that logs in until that job has a path. A CPU mitigation that needs a boot flag can wait for a planned reboot. Disk encryption on a running server usually means a reinstall, so it is a decision for the next rebuild. Outbound filtering is a real project on a server that pulls packages and images from many places. Live patching is right for a server nobody reboots on a schedule and optional for one with a tested reboot routine. What matters is that the reason is written down with a date and a condition for looking again.

Two ways a first answer goes wrong, both of which happened to us:

  • A count that was not checked. A quick look said six containers needed a fix. The careful count was eight, and they were not all the same kind.
  • A mode without a role. A quick search said four of seven env files were readable by other accounts. The full survey found 43 files. All 28 real ones were already private. The readable ones were templates and test fixtures.

For secrets, three questions settle it. What is the file's role: a real secret, an example, a fixture, or public by nature? Is a directory above it closed to other accounts, which makes the file's own mode irrelevant? And does any other account on the machine exist that should not read it? namei -l /path/to/file shows every directory gate on the way down.

5. Probe before you harden containers

Two container settings are worth having on almost every service.

no-new-privileges stops a process from gaining privilege when it runs another program. The kernel's documentation names what it blocks: "the most obvious examples are setuid/setgid programs and file capabilities." Dropping capabilities removes specific root powers. Docker's security page says the engine already drops many by default, "an allowlist instead of a denylist approach", and cap_drop: ALL takes away the rest.

Either one can stop a service from starting. So try it on a copy that nothing depends on: same image, no network, capped memory, removed afterwards. The kit has a 15-line script for this. We ran it on the public Postgres 16 image two ways.

With every capability dropped, the container exited at once:

state: exited exit=1
chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
chmod: changing permissions of '/var/run/postgresql': Operation not permitted
error: failed switching to 'postgres': operation not permitted

With five capabilities added back (CHOWN, DAC_OVERRIDE, FOWNER, SETGID, SETUID), it started and reported "database system is ready to accept connections". Those five are what its startup script uses to fix ownership of its data folder and switch to its service user. The capabilities manual describes each one.

In a Compose file the result looks like this:

services:
  db:
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    cap_add: [CHOWN, DAC_OVERRIDE, FOWNER, SETGID, SETUID]

A probe is necessary and not sufficient. Three things it does not show:

  • Existing data. A database's first start and its second start on existing data take different paths. Probe the second one too.
  • Live mounts. A clean copy mounts none of your host folders. If root in the real container reads a folder that only another user may read, it is relying on a capability, and dropping everything breaks it with no clear error. List the real mounts and who owns them. This is how we found that one of our services needed DAC_READ_SEARCH added back.
  • Paths the probe never ran. Migrations, first-run setup, plugins. Say which ones you did not exercise.

Then change the definition only, and prove the change only added lines: render the full configuration before and after with docker compose config and compare. The flags take effect the next time the container is recreated. Keep the old file so you can go back.

Be honest about how much this buys. A container that already runs as root gains little from no-new-privileges alone. The capability drop is the larger change for those. And two stronger measures are left out here on purpose. User-namespace remapping changes who owns existing data, and Docker's own guide says it is best enabled on a new installation. Read-only root filesystems need their own probe.

6. Detection that stays quiet

Prevention fails silently. If someone adds a login key or a scheduled job, nothing in a firewall or a patch level will tell you. Detection is the answer, and the usual way it fails is noise. A check that fires every day gets ignored by the second week.

Audit rules: watch configuration, not commands. Put file watches on the places that change who can log in, what runs with privilege and what starts by itself: the account files, sudo rules, the SSH server configuration, authorized keys, scheduled jobs, service definitions, the firewall, the container engine's configuration, the package sources, and the audit configuration itself. Add kernel module loading. Do not add a rule that records every command. Sudo already writes an audit record for each command once the audit service runs, and on a server with automation the flood hides everything else. The kit has a 20-rule file to start from.

One trap for scripts and agents: the audit search tools read standard input when they are not attached to a terminal. In a script they wait forever or swallow the rest of the script. The manual gives the fix, --input-logs, which it says "is needed if you are using ausearch from a cron job."

An integrity baseline that sorts out package updates. AIDE records what a set of files looks like and reports differences. Out of the box, every package update is a difference. Two changes make it readable.

First, narrow the scope to places where a change means someone altered how the machine boots, logs in, schedules work or runs privileged code. Leave out data, caches and anything that routine updates regenerate, and write each exclusion down as a known gap. Record ownership, permissions, link targets and a content hash. Leave out timestamps, which change on every update without the file changing.

Second, ask the package manager about each difference. If the file belongs to an installed package and dpkg --verify says it matches what the package shipped, it is an ordinary update, and the baseline can move forward by itself. Anything else waits for a person. The kit's wrapper script does this and requires a written reason to accept a change.

We tested the published script in a clean container. The results show the behavior better than a description:

Scroll to compare columns and rows when the table extends beyond the box.

What we did Result
Built the baseline 644 entries
Planted a file in a system folder Flagged: "no package owns it"
Installed a package of plain files Accepted automatically, baseline moved forward
Edited a packaged configuration file Flagged: "differs from package"
Added a line to authorized_keys Flagged
Tried to accept with no reason Refused
Removed a package Accepted automatically
Deleted a watched file with no package activity Flagged

The test also found a limit and a bug. The limit: installing a package that ships a symbolic link left one entry for a person, because the package database stores no link targets to verify against. The dpkg manual says "the only functional check performed is an md5sum verification of the file contents." Expect an occasional accept after installing new software. The bug: the accept command failed on a system without systemd. That is fixed in the published version, and it is the reason we test a kit before handing it to anyone.

Wire the failed check into something you already look at, such as the list of failed units or an existing health script. A new alert channel is one more thing to ignore.

One baseline scan. Lynis is a good one-time second opinion. Treat its hardening index as a number to compare against on this machine later, not a grade. Most of its suggestions are generic, and on a key-only server with one administrator, password ageing and legal banners matter far less than a service you did not know was listening. Check your timers afterwards: on our server, installing the package scheduled a daily scan by itself.

Know the limit of all of this. A baseline stored on the same machine can be rewritten by someone who already has root. This catches the tier below that: a changed key file, a new unit, an edited configuration, a replaced binary.

7. Does your update routine cover everything?

"We have an update command" and "everything that needs updating is in it" are different claims. List every pending item and write beside it which step of your routine handles it: operating system packages, the kernel, the container engine, each service image, tools installed outside the package manager, snaps, firmware, and bundled certificate stores inside images and language runtimes.

On our server this took ten minutes and found three gaps. One of them is easy to miss anywhere: when an operating system stops trusting a certificate authority, copies of the old list can live on inside a language package manager's vendored bundle, in caches and in runtimes. None of those is updated by updating the operating system.

If you add live patching, count its delivery channel too. On Ubuntu it arrives as a snap, and snaps by default refresh four times a day by themselves. If your rule is that every change goes through one reviewed routine, that is an exception to record, and snap refresh --hold is the control.

What this does not protect against

  • Someone who already has root, or who has your admin key and its passphrase.
  • A vulnerable application. Hardening the host does not fix the code running on it.
  • A compromised dependency or image. That needs version pinning, a hold on brand-new releases and a supply-chain check.
  • Physical or hypervisor access when disks are not encrypted.
  • Anything you chose to skip. That is what the written reasons are for.

Using the kit

Download the Server Hardening Survey Kit. It is one Markdown file. Sections 1 and 2 are read-only and safe to run first. Section 0 holds rules for an AI agent: read-only before anything else, never print the contents of a secret, one change at a time with a way back, no restart without a yes, and correct a wrong first answer plainly. The last section is a prompt you can paste with the file attached.

If you only do three things from this guide: run pro security-status, count your own logins before adding any limit, and write one line of reason beside everything you decide to leave alone.

Fact-check notes and sources

Related reading

This post is informational, not security-consulting or legal advice. Read every command before you run it and test changes on a copy of your own system first. Mentions of Ubuntu, Canonical, Livepatch, OpenSSH, Docker, PostgreSQL, AIDE, Lynis and CISOfy are nominative fair use. No affiliation is implied.

← Back to Blog

Accessibility Options

Text Size
High Contrast
Reduce Motion
Reading Guide
Link Highlighting
Accessibility Statement

J.A. Watte is committed to ensuring digital accessibility for people with disabilities. This site conforms to WCAG 2.1 and 2.2 Level AA guidelines.

Measures Taken

  • Semantic HTML with proper heading hierarchy
  • ARIA labels and roles for interactive components
  • Color contrast ratios meeting WCAG AA (4.5:1)
  • Full keyboard navigation support
  • Skip navigation link
  • Visible focus indicators (3:1 contrast)
  • 44px minimum touch/click targets
  • Dark/light theme with system preference detection
  • Responsive design for all devices
  • Reduced motion support (CSS + toggle)
  • Text size customization (14px–20px)
  • Print stylesheet

Feedback

Contact: jwatte.com/contact

Full Accessibility Statement • Privacy Policy

Last updated: April 2026