# How a Self-Hosted AI Server Works, Layer by Layer
A description of a working self-hosted AI server, generalized so you can build your own. It covers what
each layer does, how it connects to the next, the config shape, and the settings that matter for cost and
speed. Every hostname, address and account in this guide is a placeholder.
Read as of October 2026. Versions and defaults change often; anything marked "as of October 2026" was
checked against the vendor's documentation on 2026-10-15.
The companion guide, [02-adding-jev-typesafe-and-full-build.md](02-adding-jev-typesafe-and-full-build.md),
is the step-by-step build. The setup kit in [kit/](kit/) holds the templates both guides refer to. The
guides and the kit come together as one zip, linked from .
## How to read this guide
Each layer has four parts: what it does, how it connects, the config shape, and what to tune. Two labels
keep the facts honest:
- **In a working build**: how a server built this way does it, generalized.
- **Recommended**: an addition or a stricter setting worth adding. These are suggestions.
Placeholders used everywhere:
| Placeholder | Stands for |
| --- | --- |
| `example.com` | your domain on Cloudflare |
| `llmhub.example.com` | the prompt hub |
| `ops.example.com` | the fixed-command operations portal and link directory |
| `chat.example.com` | Open WebUI |
| `llm.example.com` | the LiteLLM admin page and API |
| `desk.example.com` | the browser desktop |
| `win.example.com` | Guacamole, which opens the Windows VM |
| `code.example.com` | the web code editor |
| `n8n.example.com`, `status.example.com` | the n8n editor and a host metrics page, if you route them |
| ``, ``, ``, `` | your server address, Linux user, tunnel ID and Access team name |
| `YOUR_API_KEY`, `` | secrets you create and never paste into a chat |
## The whole stack on one page
```text
+-----------------------------------------------------------------------------------------+
| L8 CONTROL PLANE prompt hub: prompt -> plan -> Run -> job log -> phone alert |
| ops portal: fixed commands with approval, plus the link directory |
+-----------------------------------------------------------------------------------------+
| L7 JOBS systemd user timers | detached agent jobs | parallel runners | n8n |
+-----------------------------------------------------------------------------------------+
| L6 PREPROCESSING raw text -> small local model -> JSON schema -> validate -> hand-off |
+-----------------------------------------------------------------------------------------+
| L5 AGENTS Claude Code | Codex CLI | Cowork | Desktop Commander | Hermes 3 8B |
+-----------------------------------------------------------------------------------------+
| L4 MEMORY Qdrant vectors | SQLite FTS5 keywords | notes as files in git |
+-----------------------------------------------------------------------------------------+
| L3 GATEWAY LiteLLM: one OpenAI-compatible endpoint, keys, budgets, spend logs |
+-----------------------------------------------------------------------------------------+
| L2 INFERENCE Ollama on CPU: qwen2.5 1.5B, hermes3 8B, nomic-embed-text |
+-----------------------------------------------------------------------------------------+
| L1 ACCESS Cloudflare Tunnel + Access | SSH keys | browser desktop | Guacamole |
+-----------------------------------------------------------------------------------------+
| L0 HARDWARE/OS CPU server, 64 to 96 GB RAM, NVMe RAID 1, Ubuntu 26.04 LTS, Docker |
+-----------------------------------------------------------------------------------------+
Cross-cutting: ntfy alerts | restic backups | update tool + GitHub Actions | monitors
```
The design rule behind the stack: the expensive, smart models do the writing and the judgment; cheap local
models do triage and extraction; code does everything deterministic; and a person approves anything that
leaves the server. Every web page sits behind one identity check, and no service listens on a public port.
## Component inventory
Everything a complete build carries, generalized. **Core** means the rest of the stack depends on it;
**optional** means add it when you need what it does. Versions tested in October 2026 sit next to the
commands in doc 2 and in the kit.
| Component | What it does | Layer | Started by | Talks to | Core or optional |
| --- | --- | --- | --- | --- | --- |
| Ubuntu 26.04 LTS | operating system | L0 | firmware | everything | core |
| Linux software RAID 1 over 2 NVMe | survives one disk failure (not a backup) | L0 | kernel | disks | core |
| Docker Engine with Compose | runs the service containers | L0 | systemd | containers | core |
| ufw | host firewall; allows only SSH in | L1 | systemd | kernel | core |
| OpenSSH, keys only | admin logins and forced-command keys | L1 | systemd | shell, tools | core |
| cloudflared | outbound tunnel, one hostname per service | L1 | user unit | Cloudflare edge, loopback ports | core |
| Cloudflare Access | sign-in in front of every hostname | L1 | Cloudflare | cloudflared | core |
| Tailscale | private mesh network; an alternative to public SSH | L1 | systemd | your devices | optional |
| Browser desktop (Selkies webtop) and a test copy | Linux desktop in a browser tab | L5 | Compose | hub, Chrome, Cowork | optional |
| Windows 11 VM (dockur/windows on KVM) | Windows-only apps | L5 | Compose | Guacamole (RDP) | optional |
| Apache Guacamole, guacd, PostgreSQL | Windows desktop in a browser tab | L1 | Compose | VM over RDP | optional |
| code-server | VS Code in a browser tab | L5 | user unit | files | optional |
| Ollama | runs local models on the CPU | L2 | Compose | gateway, hub, indexers | core |
| Local models | qwen2.5:1.5b, hermes3:8b, nomic-embed-text | L2 | Ollama | callers | core |
| LiteLLM and PostgreSQL | single model endpoint, keys, budgets, spend logs | L3 | Compose | Ollama, named cloud routes | core |
| Open WebUI | chat page for people | L3 | Compose | LiteLLM only | optional |
| Qdrant | vector search for notes and reports | L4 | Compose | indexers, hub, search API | optional |
| Memory indexer | embeds notes into collections nightly | L4 | user timer | Ollama, Qdrant | optional |
| Keyword index (SQLite FTS5) | keyword search over project notes | L4 | user timer | SQLite | core |
| Search API | typed, read-only search for automations | L4 | Compose | Qdrant (read-only key) | optional |
| Grounded Q&A helper | Hermes 3 answers only with exact quotes | L5 | CLI, gateway plugin | FTS5, Ollama | optional |
| Claude Code | coding agent (subscription) | L5 | hub, terminal, Remote Control | files, MCP servers | core |
| Codex CLI | coding agent (ChatGPT plan) | L5 | hub, briefs | files, MCP servers | core |
| Claude Cowork | desktop agent in the browser desktop | L5 | the person | its own output folder | optional |
| Desktop Commander (remote device) | lets claude.ai run commands here | L5 | user unit | host shell | optional |
| Remote Control | Claude Code sessions reachable from claude.ai | L5 | user unit + watchdog | Claude Code | optional |
| MCP servers | agent tools: page fetching, image generation, design files, desktop control | L5 | agents | outside APIs | optional |
| Hermes Agent (Nous Research) | agent runtime with memory and skills | L5 | the person | the gateway | optional |
| n8n | automations and approvals | L7 | Compose | LiteLLM, Qdrant, search API | optional |
| systemd user timers | backups, indexing, monitors, sync | L7 | linger | scripts | core |
| Prompt hub (FastAPI) | plan, Run, logs, alerts | L8 | user unit | agents, Ollama, Qdrant, ntfy | core |
| Operations portal (FastAPI) | fixed commands with approval, link directory | L8 | user unit | allowlisted commands | optional |
| ntfy | phone alerts | cross | alert script | phone app | core |
| restic and rclone | encrypted local and off-site backups | cross | user timers | cloud drive | core |
| Update tool and pinned-container restore | ordered, reversible updates | cross | user units, GitHub Actions | Docker, apt, CLIs | core |
| gitleaks, trivy | secret and image scans in gates | cross | scripts | repos, images | core |
| unattended-upgrades | security patches, with a reboot window | cross | apt timers | apt | core |
| Host metrics dashboard | CPU, RAM, disk history | cross | Compose or systemd | the host | recommended |
| Jev / System One | typed decisions with confidence | L6/L8 | a wrapper function | TypeSafe or Ollama | optional (doc 2) |
### Containers in detail
Pin every image by digest. A tidy way is a second Compose file that overrides the first, so the readable
file shows the tag and the pin file records the exact build; small stacks can pin inline
(`image: name@sha256:`).
| Container (generalized) | Image | Published port | Data | Restart | Networks |
| --- | --- | --- | --- | --- | --- |
| ollama | ollama/ollama | 127.0.0.1:11434 | named volume (models) | unless-stopped | app |
| litellm | LiteLLM main-stable | 127.0.0.1:4000 | config file, read-only bind | unless-stopped | app, database |
| litellm-db | postgres 16 | none | named volume, dumped before backup | unless-stopped | database (internal, no route out) |
| open-webui | Open WebUI | 127.0.0.1:8080 | named volume | unless-stopped | app |
| qdrant | qdrant | 127.0.0.1:6333 | named volume, snapshot before backup | unless-stopped | automation, search |
| n8n | n8n | 127.0.0.1:5678 | PostgreSQL plus its encryption key | unless-stopped | automation, app, search |
| search-api, search-indexer | local build | 127.0.0.1:8095 (API only) | binds: embedding model, state, notes (read-only), key files | unless-stopped | search |
| desktop | Selkies-based webtop | 127.0.0.1:3000-3001 | named volume (profile) | unless-stopped | its own |
| desktop-test | same image, test copy | none | named volume (test profile) | no | its own |
| guacamole, guacd | guacamole | 127.0.0.1:8007 (web app only) | none | unless-stopped | guac; guacd also on the VM network |
| guacamole-db | postgres 16 alpine | none | data folder, dumped before backup | unless-stopped | guac |
| windows | dockur/windows (KVM) | 127.0.0.1:8006 viewer, 127.0.0.1:3389 RDP | bind folder (VM disk) | unless-stopped | the VM network |
Compose projects: the core AI stack, an optional stack (Qdrant, n8n, the desktop) with the update tool's
pin overrides, Guacamole, the Windows VM, and the search integration. Logs rotate at 10 MB times 3 files per
container. n8n blocks environment access in nodes and excludes the Execute Command, Read/Write File and
Local File Trigger nodes.
### Services, timers and packages
- **System services that matter:** Docker and containerd, OpenSSH, unattended-upgrades, smartmontools and
the RAID monitor, and the update tool's maintenance service.
- **User services (with linger):** the tunnel, the prompt hub, the operations portal, the web code editor,
Desktop Commander in remote mode, and a Remote Control session in tmux.
- **User timers:** backups, indexing, monitors and sync, listed in layer L7. Prefer timers to cron: they log
to the journal, catch up after downtime and can run inside a resource slice.
- **Runtimes:** the system Python, Node through a version manager for the agents, and uv for Python
environments. Use each vendor's own packages rather than snaps.
- **Global Node tools:** Playwright, Puppeteer, Lighthouse and axe-core for browser checks, plus the deploy
CLIs your own sites use.
- **Other tools:** Claude Code, Codex CLI, GitHub CLI, gitleaks, trivy, restic, rclone and code-server.
- **Add-ons:** the three Ollama models; a LiteLLM callback and custom provider that answer from project
notes with exact quotes; Qdrant collections; Guacamole's PostgreSQL and header authentication
extensions; Chrome with the Claude extension and Claude Desktop (Cowork) inside the browser desktop;
OpenSSH, a non-admin job account and Desktop Commander inside the Windows VM; MCP servers for the agents.
## L0. Hardware and operating system
**What it is.** The example build is a rented dedicated server: two 12-core Intel Xeon processors (Cascade
Lake), 48 threads in total; about 96 GB of RAM; two data-center NVMe drives in Linux software RAID 1; no
GPU. Ubuntu 26.04 LTS with Docker Engine and Compose. The memory is split across two NUMA nodes, one per
processor.
**CPU features and what uses them.** Processors of this generation report AVX2, FMA, F16C and AVX-512 (F,
BW, CD, DQ, VL) with VNNI. They have no AMX and no AVX-512 BF16. Ollama's log shows it loading its Cascade
Lake CPU backend with AVX-512 and VNNI turned on:
```text
load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-cascadelake.so
system_info: n_threads = 16 / 48 | CPU : AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 |
AVX512 = 1 | AVX512_VNNI = 1
```
So the model runner already uses the widest vector instructions this CPU has, with no extra flags. Why it
matters: token generation on a CPU is limited mostly by memory bandwidth, and prompt processing by vector
math; the right backend gets the vector half for free.
**Ubuntu defaults you will likely find (none tuned for inference):**
| Setting | Typical value | What it means |
| --- | --- | --- |
| CPU frequency governor | `schedutil` on `intel_cpufreq` | demand-based default; doc 2 shows how to test `performance` |
| Transparent hugepages | `madvise` | only programs that ask get huge pages |
| Swap | often none on provider images; `vm.swappiness` 60 | with no swap device, the swappiness value has no effect |
| NVMe I/O scheduler | `none` | the right default for NVMe |
| cgroups | v2; the user manager gets `cpu`, `memory`, `pids` | user services can be given CPU and memory limits, not I/O weights |
| Container CPU pinning | none unless you set it | threads float across both processors |
**How it connects.** Every layer above is either a Docker container or a systemd user service. User
services keep running after logout because "linger" is enabled for the account.
**What to tune.** Leave room. A Windows VM holds its whole memory (32 GB in the example) while it runs,
Ollama may use up to its cap (48 GiB here), and browsers and builds need the rest. The operating notes set
an admission budget for a box this size:
about eight hosted-model agent jobs, four browsers and two builds at once, and pause new work when the
one-minute load passes 24 or free memory drops under 20 GiB
([the operating guide](https://jwatte.com/blog/sovereign-ai-workstation-operating-system/)). Doc 2 has the
full CPU and Ubuntu tuning section.
## L1. Network and access
**What it does.** Lets you reach every web page on the server from a phone or laptop without opening a
single inbound port for those pages. Only SSH is open to the internet.
```text
you: laptop or phone browser the server
--------------------------- ----------
| https://llmhub.example.com cloudflared (user service)
v | opened FROM the server:
+------------------------------+ | 4 outbound connections
| Cloudflare edge |<-----------------------+
| Access: sign in, check the |
| email policy, add a signed | allowed +---------------------------------------------+
| JWT to the request |------------>| ingress rules, first match wins |
+------------------------------+ | llmhub.example.com -> 127.0.0.1:8790 hub |
| not allowed | chat.example.com -> 127.0.0.1:8080 chat |
v | desk.example.com -> 127.0.0.1:3000 desk |
refused at the edge, | win.example.com -> 127.0.0.1:8007 guac |
never reaches the server | ...one rule per service you add |
| anything else -> 404 |
+---------------------------------------------+
every rule: access.required = true, so the
tunnel also rejects requests without a valid
Access token; the hub and portal check it again
```
**In a working build.**
- **Cloudflare Tunnel.** cloudflared runs as a user service with `--no-autoupdate` and its metrics on
loopback. A remotely managed tunnel keeps the ingress rules in the Cloudflare dashboard, so the server
holds only a token file. Each hostname points at one loopback port, a final catch-all returns 404, and
every rule requires a valid Access token at the tunnel. See Cloudflare's
[Tunnel docs](https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/).
- **Cloudflare Access.** Each hostname is an Access application with an allow policy for named email
addresses. The edge adds a signed JWT in the `Cf-Access-Jwt-Assertion` header. The prompt hub and the
operations portal verify that JWT themselves against the team's public keys, check the issuer and the
audience (the application's AUD tag, set in their config up front), and then check the email against
their own allowlist
([Cloudflare: validate JWTs](https://developers.cloudflare.com/cloudflare-one/access-controls/applications/http-apps/authorization-cookie/validating-json/)).
- **Loopback everywhere.** `ss -ltnp` shows only port 22 on public interfaces. Every container port is
published as `127.0.0.1:`. Docker-published ports bypass ufw, so loopback binding is what actually
keeps them private ([Docker: packet filtering and firewalls](https://docs.docker.com/engine/network/packet-filtering-firewalls/)).
- **SSH, keys only.** ufw allows only port 22, and SSH accepts keys only: password, keyboard-interactive and
root password logins are off (doc 2, section 4.1, shows the safe way to switch). Each automation gets its
own forced-command key that can run exactly one program: for example the prompt hub's terminal (from the
browser desktop), a chat and email dispatcher (from the automation network only), and the update tool
(from GitHub Actions).
- **Browser desktops.** The Selkies webtop desktop runs a full Linux desktop with Chrome in a browser tab
at `desk.example.com`, with sudo, screen sharing and collaboration disabled. Guacamole at
`win.example.com` opens the Windows 11 VM over RDP. Guacamole trusts the Access email header
(`Cf-Access-Authenticated-User-Email`) for sign-in, which is safe only because it listens on loopback
and the tunnel enforces Access. On Guacamole 1.6, turn that on with `HTTP_AUTH_ENABLED=true` set
explicitly (doc 2, section 4.4). It asks for the Windows password at connect time and never stores it
([Guacamole header authentication](https://guacamole.apache.org/doc/gug/header-auth.html)).
**Why this shape.** One identity check covers every page; nothing new is exposed when you add a service;
and a stolen URL is useless without a sign-in. The tradeoff: Cloudflare sees the traffic, and a Cloudflare
outage takes the web pages down (SSH still works).
## L2. Local inference: Ollama
**What it does.** Runs small open-weights models on the CPU for triage, extraction, short drafts and
embeddings. It is not the main worker: the hosted coding agents do the heavy writing.
**In a working build (tested with Ollama 0.40, October 2026).** Ollama in Docker, pinned by image
digest, port `127.0.0.1:11434`, capped at 24 CPUs and 48 GiB. Settings: `OLLAMA_NUM_PARALLEL=4`,
`OLLAMA_MAX_LOADED_MODELS=3`, `OLLAMA_KEEP_ALIVE=5m`. Flash attention is on (`auto`). Models:
| Model | Size on disk | Job | Measured on a 48-thread CPU server |
| --- | --- | --- | --- |
| `qwen2.5:1.5b` (Q4_K_M) | 0.99 GB | routing and triage, short JSON | routing median 0.416 s at 12 threads, 0.365 s at 24 |
| `hermes3:8b` (Q4_0) | 4.7 GB | JSON extraction, short drafts, grounded answers | 9 of 9 valid JSON records; median 3.0 s, 13.7 tokens/s; first call 12.5 s including 7.5 s load |
| `nomic-embed-text` | 274 MB | embeddings for Qdrant | one embed 0.81 s cold; four at once 0.10 s |
The speed figures come from published benchmark runs on that server
([small quantized models on a 96 GB server](https://jwatte.com/blog/small-quantized-models-96gb-ai-server/))
and its operating notes. With four parallel slots and 24 threads the small model produced about 104
tokens per second in total against 39 serially, a 2.7x gain, at the cost of each request taking longer.
**How it connects.** LiteLLM calls Ollama by container name. The prompt hub calls Ollama directly for
its quick triage. The memory indexer calls it for embeddings. People never call it directly: Open WebUI has
its Ollama connection turned off and goes through LiteLLM.
**Config shape** (Compose, generalized):
```yaml
services:
ollama:
image: ollama/ollama@sha256: # pinned; update deliberately
ports: ["127.0.0.1:11434:11434"] # loopback only
environment:
OLLAMA_NUM_PARALLEL: "4" # parallel requests per loaded model
OLLAMA_MAX_LOADED_MODELS: "3" # small + 8B + embedder resident together
OLLAMA_KEEP_ALIVE: "5m" # unload idle models after 5 minutes
volumes: [ollama_data:/root/.ollama]
cpus: 24
mem_limit: 48g
restart: unless-stopped
```
**What to tune and why.**
- Parallel slots multiply context memory: each slot holds its own context, so 4 slots of 8,192 tokens
need four contexts' worth of RAM ([Ollama FAQ](https://docs.ollama.com/faq)).
- Keep-alive trades memory for latency: the 8B model's first call spent 7.5 s loading. Batch jobs can
pass `keep_alive` per request instead of keeping a model resident all day.
- Quantization: Q4 builds of 1.5B to 8B models fit easily in RAM on this box. Bigger files may fit but
are not fast: a 70B Q4 file is about 40 GB and its interactive speed on this CPU was never demonstrated.
- Threads: Ollama chose 16 threads on that server. Short tests ran faster at 24 than at 12, so measure before
pinning to one processor.
## L3. The model gateway: LiteLLM
**What it does.** Gives every caller one OpenAI-compatible endpoint, `http://127.0.0.1:4000/v1`, with
named model aliases, per-caller keys, budgets, retries and fallbacks, caching, and a spend log. Callers
ask for `local-hermes3`, not for a host and port.
**In a working build (tested with LiteLLM 1.105, October 2026).** LiteLLM from the `main-stable` image,
pinned by digest, with PostgreSQL 16 for keys and spend logs on an internal-only Docker network. Prompt
text is not stored in the spend logs (`store_prompts_in_spend_logs: false`) and telemetry is off. A first
config can be two local models, one of them a custom provider:
```yaml
model_list:
- model_name: local-hermes3
litellm_params:
model: ollama_chat/hermes3:8b
api_base: http://ollama:11434
timeout: 180
- model_name: local-hermes3-grounded # custom provider: answers only with exact quotes
litellm_params:
model: grounded/hermes3
timeout: 300
general_settings:
store_prompts_in_spend_logs: false
litellm_settings:
callbacks: [notes_hook.add_context] # your own callback module
custom_provider_map:
- provider: grounded
custom_handler: grounded.handler # your own provider module
```
Starting local-only is sound: add cloud models, fallbacks and a cache once you know which callers need
them. The hosted coding agents (Claude Code and Codex) run on subscriptions, and subscription logins must
not be put into an API gateway: they are a different billing lane with their own terms
([the playbook](https://jwatte.com/blog/sovereign-ai-playbook-remote-desktop-commander/)).
**Recommended: local first, cloud only by explicit rule.**
```text
caller (hub, n8n, Open WebUI, a job script)
| POST /v1/chat/completions model="triage" key=
v
+----------------------------- LiteLLM 127.0.0.1:4000 ------------------------------+
| 1 key check: which models this key may use, its budget, its rate limit |
| 2 cache lookup (exact match, or semantic in Qdrant) hit -> return, no model |
| 3 route "triage" -> ollama_chat/qwen2.5:1.5b (local) |
| 4 error or timeout -> fallback "extract" -> ollama_chat/hermes3:8b (local) |
| 5 input too long for the local context -> context-window fallback "cloud-long" |
| (only on keys allowed to use it, only within the budget) |
| 6 write a spend row: key, model, tokens, cost; prompt text off |
+-----------------------------------------------------------------------------------+
| | |
v v v
Ollama 1.5B (about 0.4 s) Ollama 8B (about 3 s) cloud API (metered; never a silent
fallback for private material)
```
The kit's [config.yaml](kit/litellm/config.yaml) has this shape with
[router fallbacks](https://docs.litellm.ai/docs/proxy/reliability),
[budgets on virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys) and
[caching](https://docs.litellm.ai/docs/proxy/caching).
**Why one gateway.** Every model call gets the same keys, limits and logs, so you can answer "what did
this cost and who called it" in one place, and swap a model by editing one alias. Give each caller,
including the prompt hub and Open WebUI, its own virtual key limited to the models it needs, with a
budget. The master key is for creating keys, never for an app to call models with.
**Two cost traps the gateway cannot fix.** An OpenAI-style chat endpoint cannot express a provider's
batch API, so batchable work sent through it loses half-price batch processing; send batch jobs to the
provider's batch API directly, or through LiteLLM's provider pass-through route. And a gateway fallback to
a paid API on a private document sends that document out; make cloud routes opt-in per key.
## L4. Memory and retrieval: Qdrant plus keyword search
**What it does.** Lets agents and people find the right note, report or rule before they act, without
pasting whole folders into a prompt.
**In a working build (tested with Qdrant 1.19, October 2026).**
- **Qdrant** in Docker, loopback port 6333, with an admin API key and a separate read-only API key,
telemetry off, capped at 4 CPUs and 8 GB. Three collections: `memory` (operating notes), `handoff`
(agent briefs and results) and `audits` (report documents). Embeddings come from `nomic-embed-text`
(768 dimensions), pinned by digest. Each point's payload carries the file path, title, date and chunk
number.
- **Indexer.** A user service indexes at start, and a nightly timer, scheduled after the backup, adds a
few minutes of random jitter and catches up after missed runs. It hashes file content, so unchanged files cost
nothing; changed files replace their vectors and deleted files remove theirs.
- **Keyword search.** A separate project-knowledge snapshot refreshes on a short timer into a SQLite
FTS5 index and is ranked with BM25. Exact rules, IDs and phrases are found by keywords; in practice,
pure vector ranking missed exact phrases
([mobile workspace lessons](https://jwatte.com/blog/mobile-ai-workspace-lessons/)).
- **Search API.** A small typed service for automations runs searches through a read-only Qdrant key with
a CPU embedding model (`bge-small-en-v1.5`, ONNX, quantized). It accepts no shell input and no arbitrary
URLs, limits request bodies to 8 KB and runs at most two searches at once.
**How it connects.** The prompt hub runs a memory search for every plan and passes the top five hits
(score at least 0.66) to the agent as titles and file paths, not as pasted text, so the agent reads the
file itself. n8n reaches Qdrant and the search API over private Docker networks.
**Config shape** (a collection and a filtered query):
```json
PUT /collections/memory
{ "vectors": { "size": 768, "distance": "Cosine" } }
PUT /collections/memory/index
{ "field_name": "path", "field_schema": "keyword" }
POST /collections/memory/points/query
{ "query": [0.012, -0.044, "...768 numbers..."],
"filter": { "must": [ { "key": "date", "range": { "gte": "2026-08-20" } } ] },
"limit": 5, "with_payload": true }
```
**What to tune and why.** Index payload fields you filter on, or filters scan every point
([Qdrant indexing](https://qdrant.tech/documentation/concepts/indexing/)). Pin the embedding model and
record its dimension: a different model needs a new collection and a full re-index. Keep both keyword and
vector search; each misses things the other finds. Keep notes as files in git; the vector index must be
rebuildable, never the only copy.
## L5. Agents: Hermes, the coding agents and the desktops
**In a working build.**
- **Hermes 3 8B (the model).** Nous Research's open-weights chat model runs in Ollama as the server's local
writer. It is the hub's last-resort "local" engine and the engine of the grounded Q&A helper. The helper
retrieves the top three passages by keyword, asks Hermes for JSON in a fixed schema (each finding has a
statement, a source label from an enum, and a quote), and then rejects any finding whose quote is not a
literal substring of that source. If nothing survives, it says it could not produce a source-verified
answer instead of guessing. Settings: temperature 0, fixed seed, 8,192-token context, 15-minute
keep-alive.
- **Hermes Agent (the runtime) is optional.** It is Nous Research's open-source agent with persistent
memory, skills and chat gateways. Doc 2 covers where it fits if you want it.
- **Claude Code and Codex CLI.** The main workers, on subscriptions (Claude plan and ChatGPT plan). They
run headless from briefs (`claude -p ... --output-format stream-json`, `codex exec --json -`), each in
its own job folder. Codex reads a global rules file (`~/.codex/AGENTS.md`) with a few standing sections:
how to fetch a page before calling it blocked, what may never appear in a shareable report, how to run
jobs in parallel, and where the memory and local tools live.
- **MCP servers for the agents.** Tools for page fetching, image generation and design files, and
Desktop Commander for the browser desktop.
- **Claude Cowork.** Run by the person inside the browser desktop with a handoff prompt file; it writes
only to its own output folder.
- **Desktop Commander in remote mode.** A user service links the server to claude.ai as a remote device,
so Claude on the web can run commands and edit files on the host. The same tool runs as a scheduled
task inside the Windows VM.
- **Remote Control.** A spawner service keeps a Claude Code session in tmux that can be picked up from
claude.ai/code; the hub can also open any finished job as a Remote Control session.
- **The desktops.** The browser desktop is where the person's own Chrome, extensions and Cowork live; the
Windows VM (for example 16 cores, 32 GB and a 150 GB thin disk) runs Windows-only software.
**How it connects.** Agents receive work from the hub, from briefs, or from a person, write into their
own folders, and report through a result file and an ntfy alert. They reach models through their own
subscriptions, not through LiteLLM; local models are used through LiteLLM or for triage.
**What to tune and why.** Give every unattended job a narrow permission profile: read-only for questions,
a writable job folder for work, never a blanket bypass. An allowlist of tools is not a sandbox
([best practices](https://jwatte.com/blog/sovereign-ai-workstation-best-practices/)). Keep the person's
browser profile separate from automated browsers, and serialize anything that drives the person's
desktop Chrome.
## L6. Preprocessing into JSON
**What it does.** Turns raw pages, feed items, emails and documents into small, strict JSON records before
any expensive model sees them. The expensive model then reads a record of a few hundred tokens instead of
a page of many thousands.
```text
raw input: HTML page, feed item, PDF text, email
|
v
[1] deterministic clean strip markup and scripts, dedupe, regex dates and prices,
| hash the source so the record can prove what it came from
v
[2] budget check fits the context budget? no -> split at headings, keep overlap
|
v
[3] local model Ollama with format=, temperature 0, small model
|
v
[4] validate JSON Schema, then meaning: enums, dates parse, sums add up,
| every quoted excerpt is a literal substring of the source
|-- fails --> [5] repair one retry with the validator's errors in the prompt
| |-- fails again --> escalate: larger local model, then a person or an
| approved cloud model; log the reason
v
[6] store record + source hash + excerpt -> database; embed -> Qdrant
|
v
[7] hand-off only the compact record goes to the expensive model or report
```
**In a working build.** Three places use this pattern:
1. The hub's triage asks `qwen2.5:1.5b` for `{"kind": ..., "changes_files": ...}` with Ollama's JSON
format, temperature 0 and a 120-token cap, checks the kind against a fixed list, and falls back to a
keyword heuristic if the model is slow or wrong.
2. The grounded Q&A helper uses a full JSON schema (`enum` for source labels, `maxItems: 3`,
`additionalProperties: false`) and a quote check, which is the strongest validation on the box.
3. Audit and research jobs keep only measured, cited excerpts from the pages they read.
**Config shape** (Ollama structured output with a schema):
```json
POST http://127.0.0.1:11434/api/chat
{ "model": "hermes3:8b", "stream": false, "keep_alive": "15m",
"format": { "type": "object", "additionalProperties": false,
"required": ["title", "published", "price_usd", "evidence"],
"properties": {
"title": { "type": "string" },
"published": { "type": ["string", "null"], "format": "date" },
"price_usd": { "type": ["number", "null"] },
"evidence": { "type": "string", "maxLength": 300 } } },
"options": { "temperature": 0, "seed": 42, "num_ctx": 8192 },
"messages": [ { "role": "user", "content": "Extract the fields. Use null when absent. Source: ..." } ] }
```
See [Ollama structured outputs](https://docs.ollama.com/capabilities/structured-outputs). The kit has a
full [schema](kit/schemas/page-extract.schema.json) and an [extractor](kit/preprocess/extract.py).
**The cost and latency reasoning.** A measured example from one site: the same page was 71,005 bytes as
HTML and 5,743 bytes as clean Markdown
([agent readiness beyond llms.txt](https://jwatte.com/blog/agent-readiness-beyond-llms-txt/)). At roughly
four characters per token for English text, that is about 17,700 tokens against about 1,400, and a JSON
record of the facts you need is often a few hundred. As arithmetic only: at an example rate of $3 per
million input tokens, 1,000 raw pages a day is about $53 a day of input; 1,000 records of 400 tokens is
about $1.20. The local model is not free: it costs CPU time, and on a CPU a long input spends most of its
time in prompt processing, so measure on your own pages before promising a turnaround. Put bulk
extraction in batch jobs, not in the interactive path. The bill also depends on output, retries and
escalations, so a 90 percent token cut is not a 90 percent bill cut.
**What to tune and why.** Prefer `null` to an invented value. Never gate an automatic action on the
model's own stated confidence; gate on evidence present and rules you can inspect. Keep a fixed set of
hard test cases (missing, contradictory, hostile, injected instructions) and re-run them after any model,
quantization or prompt change ([small quantized models](https://jwatte.com/blog/small-quantized-models-96gb-ai-server/)).
Doc 2 shows how typed decisions with calibrated confidence change this step.
## L7. Jobs and scheduling
**What it does.** Runs work unattended and safely: nightly backups and indexing, monitors, agent jobs that
take hours, and parallel batches.
```text
systemd user timer (OnCalendar, Persistent=true, RandomizedDelaySec)
|
v
service unit (Nice=10; recommended: Slice=agent-jobs.slice with CPUWeight and MemoryMax)
|
v
runner: flock -n lock -> write a "started" receipt -> list the work units
|
+--> worker 1 own folder, own result file --+
+--> worker 2 own folder, own result file --+--> -RESULT.md (first line = verdict)
+--> worker N capped by load and free RAM --+ |
v
terminal state always written: ok | ok_nothing_to_do | partial |
quota_stopped | approval_required | failed
|
v
ntfy: job id + status + link (never the prompt or any secret)
```
**In a working build.**
- User timers rather than cron: knowledge sync, memory indexing, restic backup and prune, the off-site
copy, desktop and data-volume backups and their weekly verification, site monitors, maintenance polling,
and watchdogs for the update tool and Remote Control. A job that spends an external quota runs only
while an "armed" flag file exists.
- Long agent jobs start detached with a launcher (`setsid nohup ./run.sh > run.log 2>&1 < /dev/null &`),
write `-RESULT.md`, and append "exit" lines to a progress file that dependent jobs wait on. A
parallel launcher runs independent jobs at once, one per site or target.
- n8n carries light integration flows such as service health and knowledge sync. Flows that take
commands by chat or email stay inactive until their rejection tests pass.
**Why it is built this way.** A job that dies with your SSH session, or that two schedulers both run, is
the most common failure. Detached launches, a lock, a receipt for every run, and one ledger per external
quota fix most of it. Background jobs run at `Nice=10` so they never starve the person's desktop.
**Quotas are account-wide.** External limits such as a search console's indexing requests are shared by
the whole account, not per site, and two systems spending the same quota must share one ledger. On one
working server, running independent checks in parallel cut a full gate run from 169 s to 106 s.
## L8. The control plane: the prompt hub
**What it does.** One place to send any prompt, from a phone or a desktop, that decides which engine and
which machine should run it, shows the plan, waits for your Run, streams the log, and pings your phone when
the job ends. It ties the stack together: local models for triage, Qdrant for context, subscription agents
for the work, ntfy for the alert.
```text
phone or laptop
| 1 open https://llmhub.example.com/?q= (a GET only shows the page)
v
Cloudflare Access sign-in -> tunnel -> hub on 127.0.0.1:8790
| hub verifies the Access JWT (signature, issuer, audience) and the email allowlist
| 2 the page POSTs /api/plan (JSON only)
v
PLAN, nothing runs yet
- triage: qwen2.5:1.5b returns {"kind", "changes_files"}; keyword heuristic if slow
- memory: Qdrant search, top 5 hits scoring 0.66 or more (titles and paths)
- engine: first available of claude -> codex -> local; skip any engine cooling down
- target: host | browser desktop | Windows VM mode: ask (read only) | work
- returns the plan and a one-time run token
| 3 you press Run (POST with the run token; the token is then deleted)
v
QUEUE in SQLite -> worker, at most 2 jobs at once -> engine runs in the job's own folder
| events stream to log.txt (the page polls it); result.md at the end
| usage limit hit -> mark that engine cooling down (30 or 20 min) -> next engine
v
4 ntfy push "job done via codex "; the link opens the job page behind Access
```
**How a working hub does it:**
- Listens on `127.0.0.1:8790` only, one uvicorn worker, as a user service with `UMask=0077`.
- Two ways in: a verified Cloudflare Access JWT (keys fetched from the team's certs endpoint, RS256,
issuer checked, the audience set in the config from the Access application's AUD tag, email allowlist)
or a local token file for the terminal client, compared in constant time (doc 2, section 4.3).
- A GET never executes. A URL prompt (`/?q=...`) fills the box and creates a plan. Running needs a POST with
the one-time run token, and every POST must be `application/json`, which blocks simple cross-site forms.
- Engine order `claude, codex, local`. A rate-limit event or a limit message in the output marks that
engine as cooling down and the job falls back to the next one.
- Modes: `ask` runs Claude with a read-only tool list (`Read, Grep, Glob, WebSearch, WebFetch`) or Codex
with `--sandbox read-only`; `work` lets the engine edit files: Codex is sandboxed to the job folder
(`--sandbox workspace-write`), while Claude runs in auto mode with the job folder as its working
directory and an instruction to stay there and ask before anything outward-facing.
- Targets: the host (a job folder), the browser desktop (`docker exec` as the desktop's normal user), or
the Windows VM (SSH with a key, as a non-admin account that exists only for these jobs).
- Every job has `prompt.md`, `log.txt`, `events.jsonl` and `result.md`. A restart marks running jobs as
interrupted instead of pretending they are alive.
- Status page: service health pills, Claude's usage window, cooling-down engines, today's jobs and cost,
and the queue.
- Alerts carry the job id, status and link only, because they cross a public relay. Prompts never do.
- Strict headers: no framing, no referrer, no caching, a content security policy, and no auto-generated
API docs.
- A terminal client mirrors the web page (`/target`, `/engine`, `/mode`, `/jobs`, `/open`, `/remote`,
`/cancel`, `/status`, `/dc`), and the browser desktop has a launcher that opens it through a
forced-command SSH key that can run the hub client and nothing else.
**Why subscription agents first.** The hosted coding agents are paid by subscription and are far stronger
than an 8B model on a CPU, so they get the real work. The local model costs only CPU time and answers in
under a second, so it gets triage. Cloud API tokens are spent only where a script explicitly asks for them.
See [build your own AI prompt hub](https://jwatte.com/blog/build-your-own-ai-prompt-hub/) for the four bugs
found while building it.
**The operations portal** at `ops.example.com` is the hub's stricter sibling: a fixed list of actions
(health, disk, memory, uptime, projects), each shown as a proposal you approve before it runs, with no
generated shell at all. It also serves the link directory: one page that lists every service, the
provider's rescue console and renewal dates, so you can find them when something is broken.
## Portals and dashboards
| Portal | URL (generalized) | What it manages | Efficiency it shows or controls | Protection | Priority |
| --- | --- | --- | --- | --- | --- |
| Prompt hub | `llmhub.example.com` | prompts, plans, jobs, engines, targets | which engine ran, cost per job today, queue depth, cooling-down engines | Access + JWT check in the app + loopback | core |
| Ops portal and link directory | `ops.example.com` | fixed health actions, links to everything | quick health without a shell | Access + JWT + host check | optional |
| LiteLLM admin | `llm.example.com/ui` | models, virtual keys, budgets, spend | spend per key and model, tokens saved by local routing, cache hits once caching is on | Access + LiteLLM login | core |
| Open WebUI | `chat.example.com` | chat with the gateway's models | none directly; all traffic shows in the gateway's spend log | Access + its own login, sign-up off | optional |
| Browser desktop | `desk.example.com` | Linux desktop, Chrome, Cowork | interactive work without a local machine | Access + desktop password | optional |
| Guacamole | `win.example.com` | the Windows VM | Windows apps without a Windows server licence on your laptop | Access + header sign-in | optional |
| Web code editor | `code.example.com` | files, terminal | edits without SSH | Access + editor password | optional |
| n8n editor | loopback (SSH tunnel) or `n8n.example.com` | automations | runs, failures, execution history | Access, or not routed | optional |
| Qdrant dashboard | loopback `/dashboard` (SSH tunnel) | collections, points | index size, collection health | API key, not routed | optional |
| GitHub Actions page | github.com | dispatch-only update workflow | update reports kept 30 days | GitHub account | core |
| ntfy app | phone | alerts | job failures, backup and update results | topic secrecy | core |
| Host metrics (Netdata or Uptime Kuma) | `status.example.com` | CPU, RAM, disk, uptime history | model load, memory pressure, disk growth | Access | recommended |
```text
PORTAL MAP (which portal watches which layer)
L8 control [ llmhub.example.com ] [ ops.example.com + link directory ]
| |
L7 jobs [ n8n editor, loopback ] [ GitHub Actions run page ] [ ntfy on the phone ]
|
L5 agents [ desk.example.com ] [ win.example.com ] [ code.example.com ]
|
L3 gateway [ llm.example.com/ui ] <- every model call shows up here
|
L2/L4 [ chat.example.com ] [ Qdrant dashboard, loopback ]
|
L0 host ( host metrics: recommended )
every bracketed URL sits behind Cloudflare Access; loopback ones need an SSH tunnel
```
## Working together through web URLs
A person and several agents share the same server through URLs, without stepping on each other.
```text
person on a phone or a desktop browser
| every URL behind Cloudflare Access
+--> llmhub.example.com plan -> Run --------------------------+
+--> desk.example.com Cowork in the browser desktop |
+--> win.example.com Windows-only tools |
+--> claude.ai / ChatGPT Desktop Commander, Remote Control |
v
+----------------------------- server ------------------------------+
| briefs: ~/handoff/.md, one per job, written before start |
| Codex -> ~/agents/codex// writes only here |
| Cowork -> ~/agents/cowork// writes only here |
| hub jobs -> ~/llmhub/jobs// writes only here |
| \ | / |
| v v v |
| ONE merger job reads all three and writes the result |
+-------------------------------------------------------------------+
|
v
ntfy -> phone -> the person reviews, then sends or deploys
```
**How it was used.** For one audit, the person ran Cowork in the browser desktop with a handoff prompt
file, Cowork wrote to its own folder, and a later Codex job merged those notes into the reports when they
existed. Codex ran on the ChatGPT plan and Cowork on the Claude plan, so each agent's tokens came from a
subscription that already covered them.
**The rules that make it work:**
- A GET never acts. URL prompts and phone-alert links open a plan or a job page; only a POST with a
one-time token runs anything. Link previewers and crawlers follow GETs.
- Every URL sits behind Access, and each portal has its own narrow key: the browser desktop's key can open
the hub's terminal and nothing else; the dispatcher's key runs one script.
- One writer per folder, per branch and per deploy target. Agents never write into each other's folders.
- Agents fill, people send. An agent can prepare a message, a form or a deploy; a person presses send.
**Which desktop for which job.** The browser desktop is for the person's own Chrome, extensions and Cowork.
Guacamole and the Windows VM are for Windows-only tools. Desktop Commander and Remote Control are for
working with an agent on the host from claude.ai without opening a desktop at all.
## How the pieces cooperate without fighting
```text
Cloudflare Access + Tunnel (the only way in from the web)
|
+-------------+--------------+-------------+--------------+---------------+
v v v v v v
prompt hub ops portal Open WebUI LiteLLM UI browser desk Guacamole
| | | |
| runs | key | v
v v | Windows VM
Claude Code / Codex +--> LiteLLM <-------+--- n8n, job scripts
(own folders, | |
own subscriptions) | v
| | Ollama (CPU): Hermes 3, qwen, nomic <-- hub triage, indexers
| |
+-- memory search --> Qdrant <--- memory indexer (nightly), search API (read-only key)
|
+--> ntfy <---- timers, backups, update tool, GitHub Actions
```
| Rule | What it prevents |
| --- | --- |
| Every service binds to `127.0.0.1` on its own port; the tunnel maps one hostname to one port | port clashes, and an accidental public port that Docker would open past ufw |
| One gateway for model calls (LiteLLM); Open WebUI's direct Ollama link is off | invisible spend, models swapped behind a caller's back, keys scattered in many apps |
| One memory store per kind of memory (Qdrant for vectors, FTS5 for keywords, git for the files) | two indexes disagreeing about which note is current |
| Container limits (Ollama 24 CPUs and 48 GiB, Qdrant 4 and 8 GB, n8n 4 and 4 GB, gateway 4 GB) | a model load starving the desktop or the database |
| `Nice=10` on background jobs; at most 2 hub jobs at once | backups and indexing making the desktop lag |
| One folder per agent and per job; one merger | two agents overwriting each other's edits |
| One production deploy per site and one push per repo at a time | half-applied deploys and rejected pushes |
| Scheduling windows and shared locks: the backup first, an off-site pass that waits on the same lock, the memory index after the backup, volume backups in their own window; updates only when no agent job runs | two jobs writing one repository; a backup copying a half-written index; an update restarting a service mid-job |
| Owner-only steps are listed in a checklist, never improvised by an agent | an agent clicking through a sign-in, DNS change or payment |
## Observability, alerts, maintenance and backups
**Alerts.** One script, `alert.sh`, sends every alert. Its backend is ntfy: the public `ntfy.sh` relay with
a long random topic name stored in an owner-only file. The topic is effectively the password: anyone who
knows it can read the alerts, so messages carry only job ids, statuses and links. Callers are the hub,
the monitors, the update tool and its watchdog. The public relay has limits worth knowing (the operating
guide notes 250 messages a day and a 12-hour cache) and no end-to-end encryption; a self-hosted ntfy
with access tokens removes both concerns ([ntfy publishing](https://docs.ntfy.sh/publish/)).
**Monitors.** Nightly and weekly deterministic monitors check the sites the server looks after, with an
incremental nightly pass and a full weekly sweep, and report planned, measured and deferred counts so a
green subset never looks like a full pass. A host metrics dashboard (Netdata or Uptime Kuma, doc 2,
section 4.16) adds the history these checks lack.
**Updates.** A local update tool applies changes in a fixed order, one component at a time, with a pinned
"before" record, health checks after each step, and automatic rollback: host packages, then Docker, then the
agent CLIs, then service images, and the person's desktop last, only when nobody is using it. A GitHub
Actions workflow can trigger it on demand: dispatch only, a choice list of actions, inputs checked against
strict patterns, one run at a time, read-only repository permission, and an SSH key whose forced command is
the update tool, so the workflow cannot open a shell. It polls the log, then saves the report as an
artifact for 30 days. See [a safe update command](https://jwatte.com/blog/safe-update-command-self-hosted-ai-server/),
which documents four mistakes the checks caught, including a tunnel binary that ran the old version for 70
hours after its file was replaced.
```text
UPDATE AND BACKUP FLOW
nightly restic local snapshot -> restic copy to the off-site drive -> check a 5% sample
-> local retention; Sunday: 7 daily / 5 weekly / 12 monthly off-site
then memory index (after the backup, so the backup never copies a half-written index)
|
update request (portal or GitHub) ----> | waits until no agent job is running
v
record pins -> backup -> apt -> Docker -> agent CLIs -> service images -> desktop (last)
each step: pull candidate -> start -> health + real smoke test -> pass? keep : roll back
|
v
report + ntfy; cleanup only after rollback points are verified
```
**Backups** have their own section below.
**Security summary.** Loopback-only services, one Access policy, forced-command keys for every automation,
secrets in owner-only files and never in briefs or logs, gitleaks before commits, trivy on images, and
unattended security updates with a planned reboot window, and SSH that accepts keys only.
## Backups and restore
**A design that works** (names generalized; the kit's version adds the database dumps and the failure alarms):
```text
nightly dumps first: pg_dump, sqlite3 .backup, Qdrant snapshots --> a staging folder
restic backup of home, configs and staging --> local restic repository
excludes: caches, node_modules, build output, model blobs, the VM disk, credentials
lock first: a second run refuses instead of reporting a skipped success
|
v (same job, right after)
off-site step:
Drive free space >= local repository size? no -> stop, keep everything local
restic check (local)
restic copy --from-repo local --> restic repository on Google Drive (rclone backend)
every local snapshot present off-site? no -> stop, keep everything local
restic check --read-data-subset=5% (off-site)
local retention, then restic check again
|
Sunday off-site retention: 7 daily, 5 weekly, 12 monthly, prune, full check
daily Qdrant and n8n volumes: pause each container for seconds, tar its volume with the
service's own image, unpause, checksum; copy to Drive and verify checksums
before local retention
weekly restore the newest Qdrant and n8n archives into throwaway containers and check them
daily single exports (a project's data, the update tool's rollback history) copied to Drive
```
**Why it is built this way:**
- **restic** gives encrypted, deduplicated snapshots; a nightly run only stores what changed.
- **Two repositories joined by `restic copy`**, not a folder mirrored with `rclone sync`. A mirror would
copy local pruning to the cloud and delete the off-site history; `restic copy` only adds snapshots.
- **Google Drive through rclone's Drive backend, with its own OAuth client.** The operating notes measured
under 1 MiB/s on rclone's shared client against about 27 MiB/s on an own client
([sovereign AI made simple](https://jwatte.com/blog/sovereign-ai-made-simple/)); a shared client's quota
is shared with every other user of it.
- **Verify before deleting.** Local copies are pruned only after the off-site copy is checked, so a failed
upload never leaves you with nothing.
- **Pause and tar for small live volumes.** A paused container is a crash-consistent point that Qdrant's
write-ahead log and SQLite both recover from, and the weekly restore into throwaway containers proves it.
- **The restic password and the Drive token live outside the backups** they protect.
**Build these in from the start:**
- **Failures reach the phone.** Give every backup unit an `OnFailure=` hook, and add a dead-man's switch
outside the server, so a missed night is noticed instead of sitting in a log.
- **Every data store is covered.** Docker named volumes (the gateway's PostgreSQL, Open WebUI's data) live
outside the home folder, and configs live under `/opt` and `/etc`. A home-folder backup misses all of
them unless you add them: list your volumes with `docker volume ls` and match each one to a backup step.
- **Databases are dumped, not copied live.** `pg_dump` for PostgreSQL (the gateway, n8n, Guacamole),
`sqlite3 .backup` for SQLite (the hub), and Qdrant's snapshot API. A copy of a live database file can
capture a torn state.
- **The local repository sits on a second disk if you can.** On the same NVMe mirror as the live data it
protects against deletion and corruption, not against losing the server; the off-site copy covers that.
- **`IOSchedulingClass=idle` does nothing on NVMe** with the `none` scheduler, so do not count on it to keep
backups out of the way.
- **No low rclone rate cap with your own OAuth client.** A cap such as 0.5 requests per second mostly slows
the copy; rclone's own backoff handles quota errors.
Doc 2 has the full build, the kit templates for each of these, and the restore runbook.
## Three requests, end to end
**1. A scraping or audit job.**
1. A person types "audit https://www.example.org" into the hub. Triage marks it `research`, memory search
attaches the audit method notes, the plan picks Codex on the host in `work` mode. The person presses Run.
2. Codex fetches each page through the full proxy chain: direct, then the approved proxy tiers, then a
passive browser render. A challenge page is recorded as an access finding, not as missing content.
3. Each page is reduced to a JSON record (title, status, the facts measured, short excerpts) by code and
the local model; only these records and the measurements go into the report.
4. A shareable-report gate checks for local paths and private names. Result file written, ntfy alert sent.
The person reads and decides whether to send it.
**2. A news or analysis job.**
1. A timer starts the nightly job at its window, takes the lock, and writes a "started" receipt.
2. Feed items are cleaned and deduplicated in code; a small local model tags each item into a fixed set of
categories with JSON output; items failing validation are retried once, then set aside.
3. The compact records, not the raw articles, go to the hosted model that writes the analysis.
4. The job writes its terminal state and counts (items in, items processed, items set aside), and alerts
only on a change or a failure.
**3. A retrieval question.** "What did we decide about container updates?"
1. The question goes to the grounded helper (or the hub in `ask` mode). Keyword search finds the update
rules by exact phrase; vector search finds the related notes.
2. Hermes 3 answers in the fixed JSON schema with quotes; the validator drops any quote not found in the
source.
3. The person sees each statement next to its quote and source, or a plain "could not establish this"
when the notes do not support an answer.
## Further reading
- [The work around the model: operating a sovereign AI workstation](https://jwatte.com/blog/sovereign-ai-workstation-operating-system/)
- [Sovereign AI workstation best practices](https://jwatte.com/blog/sovereign-ai-workstation-best-practices/)
- [Small quantized models on a 96 GB server](https://jwatte.com/blog/small-quantized-models-96gb-ai-server/)
- [Build your own AI prompt hub](https://jwatte.com/blog/build-your-own-ai-prompt-hub/)
- [A safe update command for a self-hosted AI server](https://jwatte.com/blog/safe-update-command-self-hosted-ai-server/)
- [A Windows desktop and Drive backups on the same server](https://jwatte.com/blog/sovereign-windows-desktop-drive-backups/)
- [Mobile AI workspace lessons](https://jwatte.com/blog/mobile-ai-workspace-lessons/)
- [The sovereign AI playbook with Remote Desktop Commander](https://jwatte.com/blog/sovereign-ai-playbook-remote-desktop-commander/)
- [Agent readiness beyond llms.txt](https://jwatte.com/blog/agent-readiness-beyond-llms-txt/)
- [Sovereign AI made simple](https://jwatte.com/blog/sovereign-ai-made-simple/)
- [Jev and System One models in a sovereign AI stack](https://jwatte.com/blog/typesafe-jev-system-one-sovereign-ai-stack/)
- [n8n for small-business automation](https://jwatte.com/blog/n8n-small-business-automation-guide/)
## Sources
Read on 2026-10-15. Server examples come from a working build, generalized.
- Ollama FAQ (parallel requests, loaded models, keep-alive): https://docs.ollama.com/faq
- Ollama structured outputs: https://docs.ollama.com/capabilities/structured-outputs
- LiteLLM proxy configuration: https://docs.litellm.ai/docs/proxy/configs
- LiteLLM fallbacks and retries: https://docs.litellm.ai/docs/proxy/reliability
- LiteLLM virtual keys and budgets: https://docs.litellm.ai/docs/proxy/virtual_keys
- LiteLLM caching: https://docs.litellm.ai/docs/proxy/caching
- Qdrant indexing: https://qdrant.tech/documentation/concepts/indexing/
- Cloudflare Tunnel: https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/
- Cloudflare Access JWT validation: https://developers.cloudflare.com/cloudflare-one/access-controls/applications/http-apps/authorization-cookie/validating-json/
- Docker packet filtering and firewalls: https://docs.docker.com/engine/network/packet-filtering-firewalls/
- Apache Guacamole header authentication: https://guacamole.apache.org/doc/gug/header-auth.html
- ntfy publishing: https://docs.ntfy.sh/publish/