# Building a Self-Hosted AI Server From Scratch, Then Adding Typed Decisions With Jev

A complete build guide: how to choose the server, how to set up every layer in order, how to keep it
updated, and what typed decisions (TypeSafe's Jev and the System One API) would add on top. It stands on
its own. For a description of how the finished layers work together, read
[01-self-hosted-ai-server-layers.md](01-self-hosted-ai-server-layers.md). Templates for every config
file are in [kit/](kit/), mapped in [kit/README.md](kit/README.md). Both guides and the kit come
together as one zip, linked from <https://jwatte.com/blog/self-hosted-ai-server-from-scratch/>.

Read as of October 2026. Commands and versions were checked against vendor documentation on 2026-10-15;
anything marked "as of October 2026" changes often, so check the linked page before you run it.

Two labels used throughout:

- **A working build**: how a server built this way does it, generalized.
- **Recommended**: what this guide suggests, including stricter settings worth adding from day one.

**Jev is an optional addition**, built last. If you run Ollama 0.35 or later, it already serves the System
One API at `/v1/systemone` (a GET there returns 405 Method Not Allowed, which shows the route exists); the
route does nothing until you pull a decision model.

Placeholders: `example.com` and its subdomains, `<server-ip>`, `<user>`, `<tunnel-id>`, `<team>`,
`YOUR_API_KEY`, `<your-random-topic>`. Replace them with your own values on your own server; never paste
real secrets into a chat with an assistant.

## Contents

1. What to decide before you build, and why
2. Choosing the dedicated server
3. Setting this up with limited experience: Remote Desktop Commander plus ChatGPT
4. The build, in layer order
5. CPU and Ubuntu tuning
6. What Jev and typed decisions add
7. Updates and maintenance
8. Lessons learned
9. From-scratch checklist
10. Troubleshooting
11. The setup kit

## 1. What to decide before you build, and why

Each row is a decision you will make whether you plan it or not. The recommended choice is what a working
build and its operating notes support; the source column says where the reasoning comes from.

| Decision | Recommended choice | Why | Source |
| --- | --- | --- | --- |
| Hardware tier | CPU-only, 16 to 24 cores, 64 to 96 GB RAM, two NVMe in RAID 1, when hosted agents do the main work | five unattended hosted-agent jobs used about 2 percent of a 48-thread server; the local models only triage and extract | [operating guide](https://jwatte.com/blog/sovereign-ai-workstation-operating-system/) |
| GPU or not | No GPU until a measured workload needs local interactive generation; then size by VRAM | RAM lets a model fit, not run fast; a 70B file fits in 96 GB but was never shown to be interactive on CPU | [small quantized models](https://jwatte.com/blog/small-quantized-models-96gb-ai-server/) |
| Local first or cloud | Local for triage, extraction, embeddings and private material; subscriptions for coding agents; metered APIs only on routes you name | three cost lanes with different terms; a silent paid fallback sends private data out and surprises the bill | [sovereign advantage](https://jwatte.com/blog/sovereign-advantage-dedicated-ai-workstation/) |
| What to preprocess into JSON | Bulk inputs (pages, feeds, emails, documents) before any expensive model; not interactive chat | a clean record is a few hundred tokens where the raw page is many thousands, and it can be validated and reused | doc 1, layer L6 |
| Retrieval or none | Keyword search (SQLite FTS5) over notes kept in git first; add Qdrant when you need meaning-based search over hundreds of files | keyword search finds exact rules; an unfinished vector index proves nothing | [mobile workspace lessons](https://jwatte.com/blog/mobile-ai-workspace-lessons/) |
| How agents act | A fixed permission profile per job (read-only for questions, a writable job folder for work); the agent fills, a person sends | an allowlist is not a sandbox, and an agent that can send or sign can commit you to something | [best practices](https://jwatte.com/blog/sovereign-ai-workstation-best-practices/) |
| How access is protected | Cloudflare Tunnel plus Access in front of every page; every service on 127.0.0.1; key-only SSH | one identity check for everything, no inbound ports for web pages, nothing new exposed when a service is added | [best practices](https://jwatte.com/blog/sovereign-ai-workstation-best-practices/) |
| Control plane | A plan-then-run prompt hub once you have two engines or two machines | one place for prompts, plans, logs and alerts; a GET never runs anything | [build your own prompt hub](https://jwatte.com/blog/build-your-own-ai-prompt-hub/) |
| Automation (n8n) | Only for webhook glue, approvals and integrations a non-coder must read and edit | it is the wrong place for long agent jobs or heavy data pipelines | [n8n guide](https://jwatte.com/blog/n8n-small-business-automation-guide/) |
| Typed decisions | Hosted Jev for material that may leave the server; a local decision model through Ollama for material that may not; shadow mode first | typed answers with probabilities remove "parse the JSON and hope", but thresholds must be measured on your own data | [Jev in a sovereign stack](https://jwatte.com/blog/typesafe-jev-system-one-sovereign-ai-stack/) |
| Update cadence | Security patches automatic and daily; everything else weekly, one component at a time, through a tool with rollback; the desktop last | an update is only done when it is installed, running, checked, pinned and reversible | [safe update command](https://jwatte.com/blog/safe-update-command-self-hosted-ai-server/) |
| Backups | restic locally plus an encrypted off-site restic copy on Google Drive through rclone, with your own OAuth client and the `drive.file` scope; the password off the server; a dead-man's switch; a test restore every month | RAID is not a backup, a backup that was never restored is a hope, and rclone's shared Drive client stops working during 2026 | [desktop and Drive backups](https://jwatte.com/blog/sovereign-windows-desktop-drive-backups/); [rclone Drive docs](https://rclone.org/drive/) |
| Quotas and costs | One ledger per external quota; a budget on every gateway key; count server, subscriptions, API, backup storage and your time | external quotas are account-wide, and "local" is not free | operating notes; [sovereign advantage](https://jwatte.com/blog/sovereign-advantage-dedicated-ai-workstation/) |
| Alerts | ntfy, with a long random private topic or a self-hosted server with tokens; ids and links only | the topic is the password on the public relay | [operating guide](https://jwatte.com/blog/sovereign-ai-workstation-operating-system/) |
| Remote help during setup | Desktop Commander with an approval for every step; narrow its folders afterwards | it can reach the whole server by default | [Desktop Commander guide](https://jwatte.com/blog/desktop-commander-codex-claude-small-business-guide/) |

## 2. Choosing the dedicated server

**Start from the work, not the model.** If the main writing and coding is done by hosted agents on
subscriptions, the server's job is to run services, desktops, schedules, small models and backups. That
is a CPU and RAM job. If a local model must write long answers at interactive speed, that is a GPU job,
and VRAM decides which models you can run.

| Workload | CPU | RAM | GPU | Disk | Fits |
| --- | --- | --- | --- | --- | --- |
| Agents and schedules, no local inference | 4 cores | 8 GB | none | 100 GB NVMe | hub, timers, Tunnel, backups |
| Plus a persistent browser desktop and containers | 8 cores | 32 GB | none | 250 GB NVMe | the above plus a desktop, Open WebUI, n8n |
| Plus small models on the CPU | 16 cores with AVX2 (AVX-512 helps) | 64 GB | none | 500 GB NVMe | 1.5B to 8B models for triage, JSON extraction, embeddings |
| Plus a Windows VM and parallel jobs (the example build) | 2 x 12 cores, 48 threads, AVX-512 VNNI | about 96 GB | none | 2 x 1 TB class NVMe, RAID 1 | everything in doc 1 |
| Interactive local generation, 7B to 14B (32B at Q4 is tight) | 8 cores or more | 64 GB | 1 GPU with 24 GB VRAM | 1 TB NVMe | a local writer that answers in seconds |
| Interactive 30B to 70B at Q4 | 16 cores or more | 128 GB | 48 GB VRAM or more | 2 TB NVMe | larger local models; still benchmark before you buy |

The first four rows are planning examples from the operating notes, not measured minimums. The GPU rows
follow the published rule of thumb for Q4_K_M weights: about 8 GB of VRAM for a 7B model, 12 to 16 GB for
13B to 14B, about 24 GB for 30B to 32B, and 35 to 48 GB for 70B, before the context cache
([on-prem versus cloud](https://jwatte.com/blog/local-ai-on-prem-vs-cloud/),
[Qwen at home](https://jwatte.com/blog/qwen-open-weights-self-hosted-at-home/)).

**What to look for in a provider's dedicated range:**

- **CPU generation and vector instructions.** AVX2 is the floor for fast CPU inference; AVX-512 with VNNI
  speeds prompt processing; AMX (Intel Xeon 4th generation and later) speeds it further. Check with
  `lscpu` on the provider's rescue system before you commit to a long contract.
- **Memory bandwidth and ECC.** Token generation on a CPU is limited mostly by memory bandwidth: more
  memory channels per socket matter more than clock speed. Two sockets double the channels but add NUMA
  effects (section 5).
- **Two NVMe drives** you can mirror, preferably data-center models. Headroom matters more than speed:
  cleanups on one working server freed 82 GB and 139 GB at different times, mostly old desktop images.
- **An out-of-band console** (KVM over IP or a rescue system). It is your way back in after a firewall or
  SSH mistake.
- **Virtualization** (`/dev/kvm` present) if you want a Windows VM.
- **Bandwidth terms, IPv6, a provider firewall, snapshots or backup storage, setup fees and minimum
  terms**, and a data-center location whose jurisdiction you accept.

**Examples, as of October 2026 (prices change; check the provider's page):** a CPU server of the
example class rents for about $125 a month plus a setup fee in a budget dedicated range such as
OVHcloud's Eco line
([sovereign advantage](https://jwatte.com/blog/sovereign-advantage-dedicated-ai-workstation/)). For a
24 GB GPU, Hetzner's current GEX45 (RTX PRO 4000 Blackwell, 24 GB) and OVHcloud's Scale-GPU line (NVIDIA
L4, 24 GB, listed from $1,145 a month and marked "soon available") are examples; Hetzner's older GEX44
(20 GB) is no longer sold. For 48 GB, OVHcloud's HGR-AI-2 lists an NVIDIA L40S at $4,355 a month; Hetzner
jumps from 24 GB to 96 GB (GEX63 and GEX131, RTX PRO 6000 Blackwell).

**Renting cloud GPUs instead.** Hourly GPU rental is the right tool for bursts: evaluating a 70B model
for a day, a large batch, a fine-tuning run. A dedicated server is the right tool for always-on services
whose cost should not move with use. The published comparison: a $20 a month subscription is $720 over
three years, so owned hardware wins only with heavy use or a privacy requirement, and a mid-2026 memory
shortage pushed the math further toward cloud ([on-prem versus cloud](https://jwatte.com/blog/local-ai-on-prem-vs-cloud/)).
Many builders end up with both: a CPU server for services and a rented GPU when a job needs one.

## 3. Setting this up with limited experience: Remote Desktop Commander plus ChatGPT

**Who this is for.** You can log in to the provider's console or SSH, and you have a claude.ai plan
and possibly a ChatGPT plan, but you have not run a Linux server before. An assistant can do the typing
and explain each step, as long as you keep the approvals and the snapshots.

```text
 you (laptop or phone)
   |
   +--> claude.ai chat --(Desktop Commander connector)--> device link --> the server
   |       proposes ONE step, explains it, shows the command          | runs it as your
   |       waits for "approve"                                         | normal user
   |                                                                   v
   |                                          output read back, verified, appended to
   |                                          ~/setup-log.md (time, command, result, undo)
   |
   +--> ChatGPT chat: plans, explains errors, reviews configs (no server access)
   |
   +--> Codex CLI on the server (ChatGPT plan): read-only review in ~/agents/codex/<job>/
   |
   +--> provider console: snapshot before risky steps; rescue console if SSH breaks
```

**Install Desktop Commander and link the server (as of October 2026).** Desktop Commander is an MCP
server that lets Claude run commands and edit files on a machine; its remote mode links that machine to
your claude.ai account. It needs Node.js 22.12 or later
([npm package metadata](https://registry.npmjs.org/@wonderwhy-er/desktop-commander/latest)).

```bash
# 1. Node through a version manager, as your normal user (not root); take the current
#    pinned version from nvm's README (v0.40.3 shown)
curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
. ~/.nvm/nvm.sh && nvm install 22 && node --version

# 2. Start the remote device. It prints a code; confirm the same code in the browser page it opens
#    (or open the printed URL on your laptop). Pin the version you tested.
npx @wonderwhy-er/desktop-commander@0.3.0 remote
```

Then add the Desktop Commander connector to your claude.ai account from the connectors directory and
follow the instructions shown there; accounts and devices are managed at the vendor's Remote MCP
dashboard ([Desktop Commander for Claude on the web](https://desktopcommander.app/mcp/claude-web/),
[remote device README](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/remote-device/README.md)).
The device answers only while the process runs; keep it in a systemd user service once it works, and
tmux is fine for a first session. Its credential lives in `~/.desktop-commander-device/device.json`.

**A safe working loop:**

1. The assistant proposes one step and explains what it does and what could go wrong.
2. It shows a command that reads the current state, then the command that changes it.
3. You type "approve". Nothing runs without it.
4. It runs the step, reads the whole output, and verifies with a second read-only check.
5. It appends a line to `~/setup-log.md`: time, step, command, result, how to undo it.
6. Before firewall, SSH, disk, kernel or Docker network changes, you take a provider snapshot first.

The starter prompt in [kit/prompts/remote-desktop-commander-setup-prompt.md](kit/prompts/remote-desktop-commander-setup-prompt.md)
builds this loop and the build order into the conversation.

**Least privilege: change the defaults most people never change.** Desktop Commander can do most of a
build, mostly by starting processes and reading and writing files, which is exactly why its defaults
matter. As of version 0.3.0 (which adds reconnect and token-rotation fixes and a diagnostic report), reads
and writes are limited to 1,000 and 50 lines per call by default, and:

- `blockedCommands` holds a default list (disk formatting, account tools, shutdown, firewall and similar).
  It is a convenience, not a sandbox: the vendor's own security notes say it can be bypassed through
  substitution, absolute paths or another interpreter
  ([SECURITY.md](https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/SECURITY.md)).
- `allowedDirectories` starts empty, which means file operations can reach the whole file system, and
  terminal commands can reach files outside it anyway. After the base install, set it to your project
  folders.
- `telemetryEnabled` is on by default. Turn it off if you prefer.
- The tool keeps a local log of tool calls with their arguments, so anything typed into a command (a
  password, a token) lands in that log. Keep secrets out of commands.
- Run it as a normal user with narrow sudo, never as root.
- Remove devices you no longer use: revoke them in the Remote MCP dashboard, then run
  `npx @wonderwhy-er/desktop-commander@0.3.0 remote --logout` on the machine. Logging out alone does not
  revoke the device.
- Never paste API keys or passwords into the chat. Create the `.env` files yourself, with `chmod 600`.

A sensible pattern is to make configuration changes (folders, telemetry) in a separate chat from the one
doing the build, so the building assistant never relaxes its own limits. The settings live in
`~/.claude-server-commander/config.json`, and the tool's `get_config` and `set_config_value` calls read and
change them ([Desktop Commander README](https://github.com/wonderwhy-er/DesktopCommanderMCP)).

**ChatGPT's role (as of October 2026).** Check your own plan's settings, because availability differs by
plan:

- **Codex CLI on the server** signs in with a ChatGPT plan (`codex login`, or `codex login --device-auth`
  on a headless server once device-code login is enabled in ChatGPT's security settings) and runs
  non-interactive reviews with `codex exec --sandbox read-only` in its own folder
  ([Codex non-interactive mode](https://learn.chatgpt.com/docs/non-interactive-mode)). The old
  `--full-auto` flag is gone; use `--sandbox workspace-write` when it must edit.
- **ChatGPT itself** can add custom MCP servers as plugins (developer mode, plan-dependent), and Desktop
  Commander is listed in the ChatGPT plugin directory
  ([custom MCP servers in ChatGPT](https://developers.openai.com/api/docs/guides/custom-mcp-server)).
- A split that suits each tool: ChatGPT for planning, explaining errors and a second opinion on a config;
  Claude through Desktop Commander for the hands-on setup; Codex for read-only reviews in its own folder;
  and, once built, the prompt hub (section 4.12) as the one place you send prompts.

**The rule that keeps two assistants from colliding:** they never edit the same files at the same time.
Each works in its own folder, and one of them (or you) merges. The second starter prompt,
[kit/prompts/chatgpt-codex-setup-prompt.md](kit/prompts/chatgpt-codex-setup-prompt.md), sets this up.

## 4. The build, in layer order

Every step reads the current state before it changes anything. Work as your normal user with `sudo`.
Take a provider snapshot before steps 4.1 and 4.3.

### 4.1 Operating system, users, SSH and the firewall

Install Ubuntu 26.04 LTS ("Resolute Raccoon", released 23 April 2026; 26.04.1 is the current point
release as of October 2026) from the provider's image, with the two NVMe drives in RAID 1 if the
installer offers it.

```bash
# Read first
cat /etc/os-release; uname -r; lscpu | grep -E 'Model name|^CPU\(s\)|NUMA'; free -h; lsblk
cat /proc/mdstat                        # RAID state if you mirrored the drives

# A normal admin user (skip if the provider created one)
sudo adduser <user> && sudo usermod -aG sudo <user>

# From your laptop: copy your key, then open a SECOND session with it before changing SSH
ssh-copy-id <user>@<server-ip>
```

Key-only SSH. Ubuntu reads `/etc/ssh/sshd_config.d/*.conf` first and the first value it reads for a
setting wins, so the file name must sort before any cloud-init drop-in
([sshd_config](https://manpages.ubuntu.com/manpages/resolute/en/man5/sshd_config.5.html)):

```bash
sudo sshd -T | grep -E 'passwordauthentication|kbdinteractiveauthentication|permitrootlogin'
printf 'PasswordAuthentication no\nKbdInteractiveAuthentication no\nPermitRootLogin no\n' | sudo tee /etc/ssh/sshd_config.d/00-hardening.conf
sudo sshd -t && sudo systemctl restart ssh
sudo sshd -T | grep -E 'passwordauthentication|kbdinteractiveauthentication|permitrootlogin'
```

Keep the first session open until a new key login works. Do this on day one, before any service goes on
the server.

Firewall. Allow SSH first, then deny the rest ([Ubuntu firewall docs](https://ubuntu.com/server/docs/how-to/security/firewalls/)):

```bash
sudo ufw status verbose
sudo ufw allow OpenSSH
sudo ufw default deny incoming && sudo ufw default allow outgoing
sudo ufw enable
```

Automatic security updates are on by default in Ubuntu
([automatic updates](https://ubuntu.com/server/docs/how-to/software/automatic-updates/)). Check them and
decide on a reboot window:

```bash
cat /etc/apt/apt.conf.d/20auto-upgrades
grep -n 'Automatic-Reboot' /etc/apt/apt.conf.d/50unattended-upgrades
sudo unattended-upgrade -v --dry-run
```

Agents get their own unprivileged accounts later: no sudo, and not in the `docker` group, because Docker
group membership is root in practice.

### 4.2 Docker Engine

Use Docker's own apt repository, not the snap: the snap package confines Docker differently and its
paths differ from the documentation ([install on Ubuntu](https://docs.docker.com/engine/install/ubuntu/)).

```bash
sudo apt update && sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
printf 'Types: deb\nURIs: https://download.docker.com/linux/ubuntu\nSuites: %s\nComponents: stable\nArchitectures: %s\nSigned-By: /etc/apt/keyrings/docker.asc\n' \
  "$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")" "$(dpkg --print-architecture)" \
  | sudo tee /etc/apt/sources.list.d/docker.sources
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo docker run --rm hello-world
```

Two rules for every Compose file from here on:

- **Publish ports on `127.0.0.1` only.** Docker writes its own firewall rules ahead of ufw, so a port
  published as `8080:8080` is open to the internet even with ufw denying it
  ([packet filtering and firewalls](https://docs.docker.com/engine/network/packet-filtering-firewalls/)).
- **Pin images.** Start from a version tag; after the first good start, record the digest
  (`docker inspect --format '{{index .RepoDigests 0}}' <image>`) and pin `image: name@sha256:<digest>`.
  A tidy pattern keeps the digests in a second Compose file that overrides the first, so the main file
  stays readable and the pins are one reviewed change.

### 4.3 Cloudflare Tunnel plus Access

You need a domain on Cloudflare. The order matters: **create the Access application first, then the
tunnel route**, or the route is public until you notice
([self-hosted Access applications](https://developers.cloudflare.com/cloudflare-one/access-controls/applications/http-apps/self-hosted-public-app/)).

1. **Access application.** In Zero Trust, Access controls, Applications: create a self-hosted
   application for `llmhub.example.com` (repeat per hostname, or one application covering several), with
   an Allow policy for your own email address. Applications deny by default. Copy its Application
   Audience (AUD) tag.
2. **Tunnel.** In Networking, Tunnels: create a tunnel and copy the install command. A remotely managed
   tunnel keeps its rules in the dashboard and needs only a token on the server.
3. **cloudflared** from Cloudflare's apt repository ([packages](https://pkg.cloudflare.com/index.html)):

```bash
sudo mkdir -p --mode=0755 /usr/share/keyrings
curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg | sudo tee /usr/share/keyrings/cloudflare-main.gpg >/dev/null
echo 'deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared any main' | sudo tee /etc/apt/sources.list.d/cloudflared.list
sudo apt-get update && sudo apt-get install cloudflared
cloudflared --version
```

4. **Run it with the token kept in a file**, not on a command line where `ps` shows it (`--token-file`
   needs cloudflared 2025.4.0 or later). Create `/etc/cloudflared/tunnel-token` (mode 600, owned by the
   user that runs the tunnel) by pasting the token yourself, then run
   `cloudflared tunnel --no-autoupdate --metrics 127.0.0.1:20241 run --token-file /etc/cloudflared/tunnel-token`
   from a systemd unit. The simpler `sudo cloudflared service install <TOKEN>` also works but puts the
   token in the unit file ([run parameters](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/configure-tunnels/cloudflared-parameters/run-parameters/)).
5. **Routes.** For each hostname, add a published application route to its loopback port
   (`http://127.0.0.1:8790` and so on) and turn on **Protect with Access**, which makes cloudflared itself
   reject requests without a valid Access token. A locally managed tunnel uses the same rules in a file:
   see [kit/cloudflared/config.yml](kit/cloudflared/config.yml), and run
   `cloudflared tunnel ingress validate` before every restart.
6. **Refusal test.** In a private window, each hostname must go to the Cloudflare sign-in; an address not
   on the policy must be refused; the desktop's WebSocket path must be refused too.
7. **Apps that decide on identity verify the token themselves.** The token arrives in the
   `Cf-Access-Jwt-Assertion` header; check its signature against
   `https://<team>.cloudflareaccess.com/cdn-cgi/access/certs`, its audience (your AUD tag) and its issuer
   ([validate JWTs](https://developers.cloudflare.com/cloudflare-one/access-controls/applications/http-apps/authorization-cookie/validating-json/)):

```python
import jwt
from jwt import PyJWKClient

TEAM = "https://<team>.cloudflareaccess.com"
AUD = "<aud-tag>"
jwks = PyJWKClient(TEAM + "/cdn-cgi/access/certs")

def verify_access(token: str) -> dict:
    key = jwks.get_signing_key_from_jwt(token).key
    return jwt.decode(token, key, algorithms=["RS256"], audience=AUD, issuer=TEAM,
                      options={"require": ["exp", "iat", "aud", "iss"]})
```

Pin the audience in configuration from the start. An app that learns its audience from the first
verified request it sees trusts whoever arrives first; the AUD tag belongs in the config file.

**Machines that need in without a person** (a GitHub Actions health check, for example) use an Access
service token: a policy with the Service Auth action, and the client sends `CF-Access-Client-Id` and
`CF-Access-Client-Secret` headers ([service tokens](https://developers.cloudflare.com/access/service-auth/service-token/)).

**Closing public SSH.** Port 22 can stay open with key-only and forced-command logins, or Tailscale SSH
or Cloudflare's SSH access can replace it, with the provider console as the way back in
([Tailscale on Linux](https://tailscale.com/kb/1031/install-linux)).

### 4.4 Browser desktops and Guacamole

Three kinds of remote desktop, for three jobs:

| Desktop | Reached at | Use it for |
| --- | --- | --- |
| Browser desktop (Selkies webtop container) | `desk.example.com` | your own Chrome, extensions and Cowork on the server |
| Guacamole to a Windows VM | `win.example.com` | Windows-only software |
| none: Desktop Commander or Remote Control | claude.ai | working with an agent on the server without a desktop |

**Guacamole 1.6.0** ([Docker install](https://guacamole.apache.org/doc/gug/guacamole-docker.html)):

```bash
mkdir -p ~/guacamole/init && cd ~/guacamole
docker run --rm guacamole/guacamole:1.6.0 /opt/guacamole/bin/initdb.sh --postgresql > init/initdb.sql
# copy kit/guacamole/docker-compose.yml here, create .env with POSTGRES_PASSWORD (chmod 600)
docker compose config --quiet && docker compose up -d
```

Sign-in through Access uses Guacamole's header authentication, which trusts the email header Access adds.
That is safe only because the web app listens on 127.0.0.1 and the tunnel route requires Access; anyone who
can reach the port directly could type the header
([header authentication](https://guacamole.apache.org/doc/gug/header-auth.html)). **Version trap:**
Guacamole 1.6.0 calls the switch `HTTP_AUTH_ENABLED`; the 1.5 name `HEADER_ENABLED` is ignored. Because
1.6.0 also turns a feature on when any variable with its prefix (`HTTP_AUTH_HEADER`) is present, a config
that still carries the old name can appear to work by accident. Set `HTTP_AUTH_ENABLED=true` explicitly. Change
or disable the default `guacadmin` account after the first sign-in, and raise `API_SESSION_TIMEOUT` if
sessions drop.

**A Windows VM** (optional) with dockur/windows needs KVM (`ls -l /dev/kvm`); publish its ports on
127.0.0.1 only (the project README publishes them on all interfaces), and you need your own Windows
licence ([dockur/windows](https://github.com/dockur/windows)). A workable size is 16 cores, 32 GB and a
150 GB thin disk; budget its memory as fully used while it runs, and exclude its disk image from the host
backup by checking the real exclude file, not the plan.

**The webtop browser desktop** gives a full Linux desktop in a tab, but its documentation warns that it
has no authentication by default and its terminal has passwordless sudo
([webtop docs](https://docs.linuxserver.io/images/docker-webtop/)). Keep it on loopback behind Access, set
a password, disable sudo, give the container a fixed `hostname:` so Chrome's profile lock survives
recreation, and update it through a test copy (section 7).

### 4.5 Ollama

Native install ([Linux](https://docs.ollama.com/linux)) or Docker ([Docker](https://docs.ollama.com/docker)).
The kit uses Docker ([kit/core/docker-compose.yml](kit/core/docker-compose.yml)); a
native install uses the drop-in in [kit/systemd/ollama-override.conf](kit/systemd/ollama-override.conf).

```bash
curl -fsSL https://ollama.com/install.sh | sh          # native; or: docker compose up -d ollama
ollama --version
ollama pull qwen2.5:1.5b        # triage, about 1 GB
ollama pull hermes3:8b          # extraction and short drafts, 4.7 GB
ollama pull nomic-embed-text    # embeddings, 768 dimensions
ollama ps                       # loaded models, their context and CPU/GPU split
```

Settings that matter (defaults from the [FAQ](https://docs.ollama.com/faq), as of October 2026):

| Variable | Default | Kit value | Why |
| --- | --- | --- | --- |
| `OLLAMA_NUM_PARALLEL` | 1 | 4 | parallel slots; RAM grows with parallel x context |
| `OLLAMA_MAX_LOADED_MODELS` | 3 on CPU | 3 | triage, extraction and embedding models resident together |
| `OLLAMA_KEEP_ALIVE` | 5m | 5m | how long an idle model stays loaded |
| `OLLAMA_CONTEXT_LENGTH` | small (the docs disagree: 2048, 4096, or by VRAM) | 8192 | set it explicitly on a CPU server |
| `OLLAMA_FLASH_ATTENTION` | auto | 1 | needed for a quantized KV cache |
| `OLLAMA_KV_CACHE_TYPE` | f16 | q8_0 | `q8_0` halves context memory |
| `OLLAMA_HOST` | 127.0.0.1:11434 | container port published on loopback | never expose the model API |

Measure before you change: on a 48-thread CPU server the small model gave 2.7 times the total throughput
with 4 parallel slots, but each request took longer
([small quantized models](https://jwatte.com/blog/small-quantized-models-96gb-ai-server/)). Section 5
covers threads, NUMA and the CPU backend.

### 4.6 LiteLLM (and Open WebUI)

One endpoint for every model call: [kit/core/docker-compose.yml](kit/core/docker-compose.yml) runs
LiteLLM v1.105.1 with PostgreSQL, and [kit/litellm/config.yaml](kit/litellm/config.yaml) defines the
aliases `triage`, `extract`, `embed` and an optional `cloud-long`
([config](https://docs.litellm.ai/docs/proxy/configs), [routing](https://docs.litellm.ai/docs/routing)).

```bash
cd ~/ai-core            # holds docker-compose.yml from kit/core and ../litellm/config.yaml
umask 077
printf 'POSTGRES_PASSWORD=%s\nLITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\nWEBUI_SECRET_KEY=%s\n' \
  "$(openssl rand -hex 24)" "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" > .env
docker compose config --quiet && docker compose up -d litellm-db ollama litellm
curl -s http://127.0.0.1:4000/health/liveliness
```

Never change `LITELLM_SALT_KEY` after models are stored: it encrypts them. Then give every caller its own
virtual key limited to the models it needs, with a budget
([virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys)):

```bash
mkdir -p ~/.config/litellm && grep '^LITELLM_MASTER_KEY=' .env | cut -d= -f2- > ~/.config/litellm/master_key
curl -s http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $(cat ~/.config/litellm/master_key)" -H "Content-Type: application/json" \
  -d '{"models": ["triage", "extract", "embed"], "key_alias": "prompt-hub"}' | python3 -m json.tool
```

Local models cost nothing per call, so for a local-only key the model list is the limit. Give every key
that lists a metered model a budget, for example `"max_budget": 20, "budget_duration": "30d"`.

Store each returned key in the file its caller reads (mode 600). Add the chat page's key to `.env` as
`WEBUI_LITELLM_KEY`, then start Open WebUI. Two Open WebUI traps: settings it saves on first start override
the environment afterwards unless `ENABLE_PERSISTENT_CONFIG=False`, and its direct Ollama connection
should be off so every chat shows up in the gateway's spend log
([Open WebUI environment reference](https://docs.openwebui.com/reference/env-configuration)).

**Do not put subscription logins (Claude or ChatGPT plans) into the gateway.** They are a different
billing lane with their own terms. Route only metered API keys through LiteLLM, and only on keys that
list those models.

### 4.7 Qdrant (and keyword search)

[kit/qdrant/docker-compose.yml](kit/qdrant/docker-compose.yml) runs Qdrant v1.19.2 on loopback with an
admin key and a read-only key ([security](https://qdrant.tech/documentation/security/)). Create a
collection sized for the embedding model, index the fields you filter on, and snapshot before upgrades
([collections](https://qdrant.tech/documentation/concepts/collections/),
[indexing](https://qdrant.tech/documentation/concepts/indexing/),
[snapshots](https://qdrant.tech/documentation/snapshots/)):

```bash
set -a; . ~/qdrant/.env; set +a
curl -s -X PUT http://127.0.0.1:6333/collections/memory -H "api-key: $QDRANT_API_KEY" \
  -H 'Content-Type: application/json' -d '{"vectors": {"size": 768, "distance": "Cosine"}}'
curl -s -X PUT http://127.0.0.1:6333/collections/memory/index -H "api-key: $QDRANT_API_KEY" \
  -H 'Content-Type: application/json' -d '{"field_name": "path", "field_schema": "keyword"}'
curl -s -X POST http://127.0.0.1:6333/collections/memory/snapshots -H "api-key: $QDRANT_API_KEY"
```

Embed through the gateway's `embed` alias so the model is pinned in one place. `nomic-embed-text` expects
the prefixes `search_document:` for stored text and `search_query:` for questions
([model card](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5)). Snapshots restore only to the same
minor version or the next one, so take one before every Qdrant upgrade. In Qdrant 1.19, the quantization
option `always_ram` became `"memory": "pinned"` (the old key still works).

Keep keyword search beside it: a SQLite FTS5 table over the same notes takes a few lines of Python and
finds exact rules and identifiers that vectors miss.

### 4.8 Hermes: the model and, optionally, the agent

**Hermes 3 (the model)** is already in place after 4.5 (`hermes3:8b`, 128K context in the Ollama library).
Use it through the gateway's `extract` alias for JSON extraction and short drafts, with a schema and
temperature 0.

**Hermes Agent (the runtime)** is Nous Research's MIT-licensed agent with persistent memory, skills and
chat gateways; v0.22.2 as of October 2026
([repository](https://github.com/NousResearch/hermes-agent)). It is optional. If you
add it:

```bash
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
source ~/.bashrc && hermes doctor
```

Point it at the gateway, not at providers directly, so its calls appear in the same spend log
([providers](https://hermes-agent.nousresearch.com/docs/integrations/providers)):

```yaml
# ~/.hermes/config.yaml
model:
  default: cloud-long          # a LiteLLM alias; see the context note below
  provider: custom
  base_url: http://127.0.0.1:4000/v1
  key_env: HERMES_LITELLM_KEY  # a LiteLLM virtual key with its own budget, kept in ~/.hermes/.env
```

Two things to know. Hermes Agent refuses to start with less than 64,000 tokens of context, and an 8B model
at 64K context on a CPU needs a lot of memory and is slow; point it at a hosted model through the gateway,
or at a GPU-served local model. And its chat gateway (Telegram, Slack and others) is default-deny: only
allowlisted or paired users reach it, and the docs warn against allowing everyone on a bot with terminal
access. Where it fits: a persistent personal assistant with memory and skills. Where it does not: running
unattended jobs that change files, which belong to the job runner.

### 4.9 Coding agents and the job runner

**Install the agents as your normal user** (as of October 2026):

```bash
curl -fsSL https://claude.ai/install.sh | bash          # Claude Code (Pro, Max, Team or Enterprise plan)
npm install -g @openai/codex                             # Codex CLI (ChatGPT plan); or its install script
claude --version && codex --version
```

([Claude Code setup](https://code.claude.com/docs/en/setup), [Codex CLI](https://github.com/openai/codex))

**Headless runs, one folder per job.** A hub runs the same shapes
([Claude Code headless](https://code.claude.com/docs/en/headless),
[permission modes](https://code.claude.com/docs/en/permission-modes)):

```bash
# Read-only question
claude -p --output-format stream-json --verbose --permission-mode default \
  --allowedTools "Read,Grep,Glob" < prompt.md > events.jsonl
codex exec --sandbox read-only --skip-git-repo-check --json -o result.md - < prompt.md > events.jsonl

# Work that may edit files inside the job folder
codex exec --sandbox workspace-write --skip-git-repo-check --json -o result.md - < prompt.md > events.jsonl
```

Never use the bypass mode (`--dangerously-skip-permissions`, `bypassPermissions`) outside a disposable
container or VM. For locked-down scripts, Claude Code's `dontAsk` mode denies anything not pre-approved.
Deny rules beat allow rules, and argument patterns are fragile (`Bash(curl *)` does not match
`sh -c 'curl ...'`), so isolation comes from the account and the folder, not from the allowlist.

**Standing rules file.** Codex reads `~/.codex/AGENTS.md` before every run. A useful set is
four sections: the full fetch chain before calling a page blocked, no local paths in shareable
reports, one job per target when running in parallel, and where memory and local tools live
([AGENTS.md](https://learn.chatgpt.com/docs/agent-configuration/agents-md)).

**Detached launches that survive SSH.** Put the job in a script file and start it so it owns nothing of
your session:

```bash
mkdir -p ~/handoff && cd ~/handoff
setsid nohup ./run-audit.sh > run-audit.log 2>&1 < /dev/null &
echo $! > run-audit.pid
```

Do not chain it after `&&` in one remote command line (the whole list goes to the background and the SSH
session hangs), and stop it by the recorded PID, never with `pkill -f` on a pattern that also appears in
your own SSH command.

**The job runner.** Scheduled work runs as systemd user units in one slice so it cannot starve the
desktop ([kit/systemd/](kit/systemd/)):

```bash
mkdir -p ~/.config/systemd/user
cp kit/systemd/agent-jobs.slice kit/systemd/nightly-job.service kit/systemd/nightly-job.timer ~/.config/systemd/user/
systemd-analyze --user verify ~/.config/systemd/user/nightly-job.service
systemctl --user daemon-reload
systemctl --user enable --now nightly-job.timer
sudo loginctl enable-linger <user>
systemctl --user list-timers
```

Every run takes a lock, writes a "started" receipt, works in its own folder, and always writes a terminal
state (`ok`, `ok_nothing_to_do`, `partial`, `quota_stopped`, `failed`) even when the agent reports nothing.
Parallel runs get one job per target and their own result files; deploys and pushes stay one at a time.
A starting admission budget for a 48-thread server with about 96 GB: about eight hosted-agent jobs, four browsers
and two builds at once; pause new work at a one-minute load of 24 or under 20 GiB free memory
([operating guide](https://jwatte.com/blog/sovereign-ai-workstation-operating-system/)).

### 4.10 ntfy alerts

```text
 timers, backups, update tool ----+
 prompt hub (job done/failed) ----+--> notify.sh --> ntfy topic --> phone app
 n8n workflows -------------------+    title, priority 1-5,          tap opens the Click URL
 GitHub Actions (on failure) -----+    tags, Click URL               (a job page behind Access)
```

The kit's [notify.sh](kit/alerts/notify.sh) and [notify.ps1](kit/alerts/notify.ps1) read the topic URL
from an owner-only file and send a title, a priority (5 urgent, 4 high, 3 default, 2 low, 1 min), tags and
a Click URL ([publishing](https://docs.ntfy.sh/publish/)):

```bash
mkdir -p ~/.config/notify && umask 077
printf 'NTFY_URL=https://ntfy.sh/%s\n' "$(openssl rand -hex 16)" > ~/.config/notify/ntfy.env
bash kit/alerts/notify.sh "test" "hello from the server" 3 "white_check_mark" "https://llmhub.example.com/"
```

Subscribe to the same topic in the ntfy phone app (read the topic from the file on the server; do not
send it through a chat). **On the public relay the topic name is the password**: anyone who learns it can
read your alerts, so make it long and random, keep it private, and put only ids, statuses and links in
messages. For more control, self-host ntfy with `auth-default-access: "deny-all"` and access tokens
([configuration](https://docs.ntfy.sh/config/)).

### 4.11 n8n: where it fits

n8n is the automation and integration layer beside the job runner and the hub. It does not replace either.

```text
 SaaS and forms --webhook--> n8n --HTTP--> LiteLLM (local model, strict JSON) --> validate
                              |                                                     |
                              |<-------------------- record --------------------------+
                              |
                              +--> Qdrant (search, read-only key)
                              +--> ntfy alert with a link to an approval form
                              +--> Wait (form): a person approves  --> person sends
                              +--> hub plan URL (still needs the person's Run)

 not in n8n: long coding-agent jobs (hub + job runner), heavy data pipelines (timers or GitHub
 Actions), anything that must live in version control first (GitHub Actions)
```

**Where it works best, and why:**

- Webhook glue between SaaS tools (forms, email, a CRM, sheets, GitHub): it handles the auth and retries
  for each service so you do not write them.
- Human-in-the-loop approvals: the Wait node can pause until a person submits a form, and several nodes
  have a "send and wait for response" operation.
- Scheduled multi-step integrations a non-coder must read and edit: the canvas is the documentation.
- Fan-out notifications through ntfy with one HTTP Request node.
- Simple AI steps through the local gateway (`http://litellm:4000/v1`, or the OpenAI credential's Base URL
  pointed at it), so every model call still has the gateway's budgets and logs.

**Where it is the wrong tool:** long-running coding-agent jobs (use the hub and the job runner); heavy
data pipelines (use scripts under systemd timers or GitHub Actions); anything that must be reviewed in
version control first (GitHub Actions); and anything that sends, books, pays or signs for a person without
an approval step, which breaks "agent fills, person sends".

**Self-hosting it safely (n8n 2.43.6 as of October 2026):** use
[kit/n8n/docker-compose.yml](kit/n8n/docker-compose.yml): a pinned version, PostgreSQL instead of the
default SQLite for real use, an external task runner for Code nodes, loopback port, and behind Access at
`n8n.example.com`. Public webhooks need their own protection (header authentication on the Webhook node,
or an Access service token). Back up the database and the encryption key together; without the key a
restored database cannot decrypt its credentials. Use queue mode with Redis and workers when executions
outgrow one process. The Execute Command and Local File Trigger nodes are disabled by default in 2.x; the
kit also excludes Read/Write Files. The licence is n8n's Sustainable Use License: internal business and
personal use are allowed; hosting n8n for clients to build their own workflows is not
([licence](https://docs.n8n.io/n8n-community-license),
[Docker Compose install](https://docs.n8n.io/deploy/host-n8n/),
[security](https://docs.n8n.io/deploy/host-n8n/configure-n8n/security)).

Start with low-risk flows such as service health and knowledge sync. Keep any flow that takes commands
by chat or email inactive until its rejection tests pass: a forced-command SSH key, an exact sender
allowlist, email that must pass DMARC and DKIM through a receiver that strips forged authentication
headers, fixed verbs only, rate limits, a kill switch and an audit log ([n8n guide](https://jwatte.com/blog/n8n-small-business-automation-guide/)).
The kit's example workflow, [kit/n8n/workflow-intake-approval.json](kit/n8n/workflow-intake-approval.json),
is the safe shape in miniature: webhook, local model to strict JSON through LiteLLM, validation in code,
an ntfy alert whose link opens an approval form, and a wait for the person.

### 4.12 A prompt hub (an LLMhub-style control plane)

The hub turns "which assistant, on which machine, with what permissions" into one page: prompt, plan, Run,
live log, phone alert. Doc 1 (layer L8) describes a working hub in detail; the kit's
[kit/hub/server.py](kit/hub/server.py) is a reduced version to build from.

**The shape:**

- FastAPI, one process, `uvicorn server:app --host 127.0.0.1 --port 8790 --workers 1`, as a user service
  ([kit/systemd/hub.service](kit/systemd/hub.service)). FastAPI's `fastapi run` binds 0.0.0.0 by default;
  call uvicorn with an explicit host ([FastAPI deployment](https://fastapi.tiangolo.com/deployment/manually/)).
- Authentication: the Access JWT (section 4.3) with a pinned audience and an email allowlist, or a local
  token file for the terminal client compared with `hmac.compare_digest`.
- Plan then run: `POST /api/plan` stores a job in SQLite with status `planned` and returns a one-time run
  token; `POST /api/jobs/{id}/run` with that token moves it to `queued`; a worker loop runs at most two
  jobs. A GET never changes state, which matters because link previewers and crawlers follow GETs.
- Every POST requires `application/json`, which stops plain cross-site form posts.
- Each job folder holds `prompt.md`, `log.txt` and `result.md`; on restart, running jobs are marked
  `interrupted`.
- Alerts carry the job id, status and a link to `https://llmhub.example.com/#job=<id>`.
- A terminal client and a browser-desktop launcher that uses a forced-command key
  (`restrict,pty,command="/usr/local/bin/llmhub"`) so the desktop can open the hub and nothing else. The
  kit does not include the terminal client; it is a short script that calls the same two POST endpoints
  with the local token and streams `/api/jobs/{id}/log`.

```bash
mkdir -p ~/hub/app && cp kit/hub/server.py ~/hub/app/ && cp kit/hub/requirements.in ~/hub/
cd ~/hub && uv venv .venv && uv pip compile --generate-hashes requirements.in -o requirements.lock
uv pip sync --python .venv/bin/python --require-hashes requirements.lock
mkdir -p ~/.config/hub && umask 077 && printf 'ACCESS_TEAM=https://<team>.cloudflareaccess.com\nACCESS_AUD=<aud-tag>\nALLOWED_EMAILS=<your-email>\nGATEWAY_KEY_FILE=%s/.config/hub/litellm.key\n' "$HOME" > ~/.config/hub/hub.env
cp kit/systemd/hub.service ~/.config/systemd/user/ && systemctl --user daemon-reload
systemctl --user enable --now hub.service && curl -s http://127.0.0.1:8790/healthz
```

Engine order and why: subscription coding agents first (strongest, already paid for), the local model for
triage and short drafts (fast and free per call), metered APIs only where a script names them. When an
engine hits a usage limit, mark it cooling down and fall back to the next. Section 6 shows how typed
decisions sharpen the plan.

### 4.13 GitHub Actions for maintenance

You do not need a self-hosted runner: GitHub-hosted runners reach the server through Access (health
checks) or through a forced-command SSH key (updates). Patterns that work, generalized:

| Pattern | Trigger | What it does | Why |
| --- | --- | --- | --- |
| Health and freshness watchdogs | `schedule` plus `workflow_dispatch` | read health endpoints and data freshness; open one issue or alert | a green deploy says nothing about whether last night's data arrived |
| Keepalives | `schedule` | ping async pipelines so they drain on time | one scheduler is a single point of failure |
| Weekly or quarterly data jobs | `schedule` plus `workflow_dispatch`, no `push` | refresh data, build, publish | a push-triggered rebuild of a data site can publish stale or partial data |
| Smoke tests | `push`, `pull_request` | render checks before merge | catch breakage before it ships |
| Workflow lint | `push`, `pull_request` | assert every data workflow ends with its failure gate | a gate that silently stopped running looks like success |
| Server update | `workflow_dispatch` only, choice inputs | ask the server's update tool for one verb over a forced-command key; save the report | changes happen only when a person asks, and the workflow cannot open a shell |

The kit's [kit/github/maintenance.yml](kit/github/maintenance.yml) combines the first and last patterns,
and [kit/github/dependabot.yml](kit/github/dependabot.yml) opens weekly update pull requests for pinned
actions and images. Facts to design around, as of October 2026: scheduled workflows run in UTC unless an
entry sets `timezone:`, only on the default branch, at most every 5 minutes, can be delayed at busy times,
and are disabled after 60 days without activity in public repositories; set `permissions: contents: read`;
pin actions to a full commit SHA; use self-hosted runners only with private repositories
([events](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows),
[secure use](https://docs.github.com/en/actions/reference/security/secure-use)).

Two operating rules: never dispatch a production data workflow by hand just to test it (one repeated
dispatch cost 20 to 100 minutes of runtime each and can overwrite fresh data), and when many runs fail at
once with identical short durations, check GitHub's status page before debugging YAML.

### 4.14 Typed decisions (Jev or a local decision model)

Build this last, in shadow mode, once everything above works. Section 6 explains what it adds.

```bash
python3 -m venv ~/jev && ~/jev/bin/pip install typesafe-sdk      # official SDK, MIT
# Hosted Jev: put TYPESAFE_API_KEY in ~/.config/jev/hosted.env yourself (chmod 600)
# Local decision model through Ollama 0.35 or later (no key needed for local calls):
ollama pull tev1:0.8b        # small; or nimble (9B, about 9.4 GB) for higher accuracy
```

Point the same SDK at the local server for private material
([Ollama decision models](https://docs.ollama.com/capabilities/decision),
[Nimble](https://ollama.com/library/nimble)):

```bash
TYPESAFE_BASE_URL=http://127.0.0.1:11434 TYPESAFE_API_KEY=ollama TYPESAFE_DEFAULT_MODEL=nimble \
  ~/jev/bin/python route.py
```

Run both backends over a labelled set from your own history and log what they would have decided for a
week before anything acts on them.

### 4.15 Backups and restore

Build backups before you trust the server with anything, and test a restore before you call them done.

```text
 BACKUP AND RESTORE FLOW

 nightly    backup.sh (restic-backup.timer, OnFailure -> phone alert)
   |
   +--> consistent dumps into a staging folder
   |      Qdrant snapshot API per collection      (never the live storage folder)
   |      pg_dump -Fc per database, read back      (gateway, n8n, Guacamole)
   |      sqlite3 .backup, then integrity_check    (hub, anything on SQLite)
   |      inventory: model list, image digests, timers, packages
   |
   +--> restic backup: home, configs, staging --> LOCAL repository (a second disk if you can)
   |      excludes: caches, venvs, node_modules, model blobs, VM disks,
   |                the secrets that open the backups
   +--> restic check (local)
   +--> restic copy --> OFF-SITE repository on Google Drive
   |      (rclone backend, own OAuth client, drive.file scope)
   +--> restic check --read-data-subset=5% (off-site)
   +--> status file ok; re-arm the dead-man's switch (delayed ntfy alert 26 h out)
   |
 Sunday     retention: local 7 daily + 4 weekly; off-site 7 daily + 5 weekly + 12 monthly;
            then full checks
 monthly    restore-test.sh: restore into a scratch folder, list dumps, integrity-check copies
 daily      outside the server: GitHub Actions reads the status through Access
            (a second dead-man's switch)

 RESTORE
   one file:   restic restore latest --target /tmp/r --include /home/<user>/path/file
   one service: stop it -> restore its dump -> pg_restore, SQLite copy or Qdrant
                snapshot recover -> start
   the server: new box -> section 4 in order with the kit -> restic restore -> dumps
               -> start services
```

**3-2-1, and how this design meets it.** Three copies on two kinds of storage with one off-site: the live
data on the NVMe mirror, a local restic repository, and an off-site restic repository on Google Drive. Put
the local repository on a second disk or the provider's backup storage if you can: on the same mirror it
protects against deletion and corruption, not against losing the server. RAID is not a backup.

**Why restic.** Snapshots are encrypted before they leave the server and deduplicated, so a nightly run
stores only what changed, and any snapshot restores on its own. The password lives in a password manager
and in an owner-only file, never inside the repository it opens
([restic docs](https://restic.readthedocs.io/en/stable/)).

**Google Drive through rclone, two ways:**

| Way | Fits | Why |
| --- | --- | --- |
| restic with the rclone backend (`rclone:drive-offsite:restic/server`) | whole-server backups | restic encrypts, deduplicates and verifies; Drive only stores opaque packs |
| plain `rclone copy` of archives that are already encrypted | single exports another tool must read (a database dump for a colleague, a volume archive) | simple, but you manage encryption and history yourself; use `copy`, not `sync`, which deletes on the destination |

A sensible split: the first way for the home folder, the second (with checksums verified after upload)
for paused-volume archives.

**Google API setup with the least privilege** (full steps in
[kit/backup/rclone-drive-setup.md](kit/backup/rclone-drive-setup.md)):

- **Your own OAuth client.** rclone's shared client is being retired and stops working during 2026; its
  quota is also shared with everyone using it ([rclone Drive docs](https://rclone.org/drive/)).
- **The `drive.file` scope**, so rclone sees only the files it created. Use full `drive` only if it must
  read files uploaded another way.
- **Publish the OAuth app "In production".** An External app in "Testing" gets refresh tokens that expire
  after 7 days, and the nightly copy then fails quietly
  ([Google OAuth](https://developers.google.com/identity/protocols/oauth2)).
- **Service accounts need a Workspace shared drive.** They have no storage quota of their own, so a personal
  Drive needs a user token.
- **Quotas.** As of October 2026 the Drive API counts quota units per minute (changed on 1 May 2026), and
  uploads are limited to about 750 GB per user per day; rclone's pacer handles rate limits, so do not add a
  low `--tpslimit` of your own ([Drive API limits](https://developers.google.com/workspace/drive/api/guides/limits)).
- **Token storage.** Encrypt the rclone config (`rclone config encryption set`), keep it at mode 600 outside
  the backup set, and pass it explicitly with `--config` or `RCLONE_CONFIG`.
- **Trash.** Set `--drive-use-trash=false` on the restic remote, or pruned packs keep using your quota.

**What to include, and how:**

| Item | How | Why this way |
| --- | --- | --- |
| Qdrant collections | `POST /collections/<name>/snapshots`, download, delete the server copy ([snapshots](https://qdrant.tech/documentation/snapshots/)) | copying the storage folder of a running server can capture a half-written state; restore works to the same or next minor version |
| PostgreSQL (gateway, n8n, Guacamole) | `docker exec <db> pg_dump -U <user> -d <db> -Fc`, then `pg_restore --list` to prove it reads | consistent without stopping the database |
| SQLite (the hub, anything small) | `sqlite3 file ".backup 'copy'"`, then `PRAGMA integrity_check` | the online backup gives a consistent copy of a live file |
| Small volumes (alternative) | pause the container, tar the volume with the service's own image, unpause | a crash-consistent point that write-ahead logs recover from; it suits Qdrant and n8n when a weekly restore into throwaway containers proves it |
| Configs | Compose files, `.env` files, systemd units, the cloudflared config (not its credentials file), the LiteLLM config, the kit-equivalent files | the server can be rebuilt from them; secrets inside the repository are encrypted by restic |
| Model list | `ollama list` (names and digests), not the models | models are large and re-downloadable; the list pins what to pull |
| Hermes Agent, if installed | `~/.hermes/config.yaml`, `memories/`, `skills/` | memory and skills are the part you cannot re-download |
| n8n | its database and its encryption key, together | a restored database without the key cannot decrypt its credentials |
| Inventory | image digests, timers, installed packages | the rebuild checklist for a new box |

**What to exclude, and why:** caches and build output (rebuildable), Python virtual environments and
`node_modules` (rebuild from lock files), model blobs (re-pull), VM disk images (back them up inside the VM
or snapshot them separately; check the real exclude file after adding one), Docker image layers (pull by
digest), and the secrets that open the backups: the restic password files, the rclone config and the backup
`.env`. The kit's [excludes.txt](kit/backup/excludes.txt) groups them.

**Retention and verification.** Locally, 7 daily and 4 weekly; off-site, 7 daily, 5 weekly and 12 monthly,
pruned weekly. After every copy run `restic check --read-data-subset=5%` on the off-site repository; once a
week a full `restic check`; once a month [restore-test.sh](kit/backup/restore-test.sh), which restores into
a scratch folder, lists every PostgreSQL dump, integrity-checks every SQLite copy and reads every Qdrant
snapshot, then records which snapshot passed.

**Alerts and the dead-man's switch.** A backup that never runs sends no failure message, so silence needs
its own alarm:

- `OnFailure=notify-failure@%n.service` on the backup units sends a phone alert when the script dies
  ([kit/backup/notify-failure@.service](kit/backup/notify-failure@.service)).
- At the end of every good run, `backup.sh` publishes a delayed ntfy message ("no backup in 26 hours") with
  a fixed sequence ID; each run replaces the pending one, so it reaches your phone only if runs stop
  succeeding. This needs ntfy server 2.16 or later and a server other than the one being watched
  ([scheduled and updated messages](https://docs.ntfy.sh/publish/)).
- A daily GitHub Actions workflow reads the status file through Access and alerts when it is missing,
  failed or older than 26 hours ([kit/github/backup-heartbeat.yml](kit/github/backup-heartbeat.yml)). Keep
  the repository private and active, because scheduled workflows in public repositories stop after 60 days
  without activity.

**Install the kit's backup (as your normal user):**

```bash
mkdir -p ~/bin ~/.config/backup ~/.local/state/backup ~/.config/systemd/user && chmod 700 ~/.config/backup
cp kit/backup/backup.sh kit/backup/restore-test.sh kit/alerts/notify.sh ~/bin/ && chmod 700 ~/bin/*.sh
cp kit/backup/excludes.txt ~/.config/backup/
# write ~/.config/backup/backup.env and the two password files yourself (chmod 600); see backup.sh's header
restic init --repo /srv/backup/restic --password-file ~/.config/backup/restic.password
# create the off-site repository with --copy-chunker-params (kit/backup/rclone-drive-setup.md, step 4)
cp kit/backup/restic-*.service kit/backup/restic-*.timer kit/backup/notify-failure@.service ~/.config/systemd/user/
systemd-analyze --user verify ~/.config/systemd/user/restic-backup.service
systemctl --user daemon-reload && systemctl --user enable --now restic-backup.timer restic-prune.timer
systemctl --user start restic-backup.service && cat ~/.local/state/backup/status.json
```

**Restore runbook.**

*One file:*

```bash
restic snapshots --path /home/<user>
restic restore latest --target /tmp/restore --include /home/<user>/notes/plan.md
diff /tmp/restore/home/<user>/notes/plan.md /home/<user>/notes/plan.md
```

*One service* (the gateway's database as the example): stop the service, restore the staged dump from the
newest snapshot, `docker exec -i litellm-db pg_restore -U litellm -d litellm --clean --if-exists < litellm.dump`,
start the service, check `/health/readiness`. For Qdrant, recover the collection from its snapshot through
the snapshot API; for SQLite, stop the service and put the `.backup` copy back.

*The whole server, from bare metal:*

1. Order the replacement; confirm its console; take note of the new address.
2. Section 4.1 and 4.2: OS hardening, users, firewall, Docker.
3. Install restic and rclone; put the rclone config (from your password manager) and the restic password
   files back by hand; run `restic snapshots` against the off-site repository.
4. `restic restore latest --target /` for the home folder and configs (or into a scratch folder and copy
   back what you need).
5. Section 4.3: cloudflared with the tunnel token; the hostnames move with the tunnel, so no DNS change is
   needed.
6. Bring the stacks up in order: core (Ollama, LiteLLM and its database), Qdrant, Guacamole, n8n, the hub.
   Restore each database from its dump and Qdrant from its snapshots before the first real use.
7. `ollama pull` every model in the saved model list; re-run your fixed test prompts.
8. Re-enable timers, then run one backup and one restore test on the new server.
9. Keep the old server until restores and jobs pass on the new one.

### 4.16 Portals and dashboards

Doc 1 has the portal map. When you build, give every portal the same three properties: it listens on
127.0.0.1, it sits behind an Access application, and anything that acts is a POST with an explicit approval.

| Portal | Generalized URL | Build it with | Shows or controls | Priority |
| --- | --- | --- | --- | --- |
| Prompt hub | `llmhub.example.com` | section 4.12 | engine used, cost per job, queue, cooling-down engines | core |
| Link directory and fixed-command portal | `ops.example.com` | a small FastAPI app with an allowlist of actions | every service link, the provider console, renewal dates, quick health | optional |
| LiteLLM admin | `llm.example.com/ui` | section 4.6 (sign in with `UI_PASSWORD`) | spend per key and model, budgets, how much traffic stayed local | core |
| Open WebUI | `chat.example.com` | section 4.6 | chat for people; its traffic shows in the gateway | optional |
| Browser desktop and Guacamole | `desk.example.com`, `win.example.com` | section 4.4 | interactive desktops | optional |
| n8n editor | `n8n.example.com`, or an SSH tunnel | section 4.11 | executions, failures, approvals | optional |
| Qdrant dashboard | SSH tunnel to `127.0.0.1:6333/dashboard` | section 4.7 | collections and their sizes | optional, not routed |
| Host metrics | `status.example.com` | Netdata (bind its dashboard to `localhost:19999`; its installer defaults to the nightly channel, so pass `--release-channel stable`) or Uptime Kuma (it defaults to port 3001, which clashes with the webtop's 3001, so remap it) | CPU, memory, disk growth, model load, uptime history | recommended |

([Netdata install](https://learn.netdata.cloud/docs/netdata-agent/installation/linux/),
[Netdata security](https://learn.netdata.cloud/docs/netdata-agent/configuration/securing-agents),
[Uptime Kuma](https://github.com/louislam/uptime-kuma))

## 5. CPU and Ubuntu tuning

Everything in this section is a recommendation: read the current value first, change one thing, measure,
and keep the old value to roll back.

### 5.1 Which instruction sets matter, and how to check them

| Instruction set | What uses it | How to check |
| --- | --- | --- |
| AVX2 and FMA | the baseline for fast CPU inference in llama.cpp and Ollama; Qdrant's float and int8 distance code | `grep -o -w -E 'avx2\|fma' /proc/cpuinfo \| sort -u` |
| AVX-512 (F, BW, DQ, VL) | llama.cpp CPU variants for Skylake-SP and later Xeons, and Zen 4 | the Ollama log line `load_backend: loaded CPU backend from .../libggml-cpu-<variant>.so` |
| AVX-512 VNNI | one-instruction INT8 dot products for quantized matrix math (Cascade Lake and later, Zen 4) | `system_info: ... AVX512_VNNI = 1` in the Ollama log |
| AVX-512 BF16 and FP16 | newer llama.cpp variants (BF16 from Cooper Lake and Zen 4; FP16 from Sapphire Rapids) | `/proc/cpuinfo` flags `avx512_bf16`, `avx512_fp16` |
| AMX (INT8, BF16) | Sapphire Rapids (4th generation Xeon) and later; speeds prompt processing | flags `amx_tile`, `amx_int8`, `amx_bf16` |
| AVX-VNNI (256-bit) | client chips from Alder Lake, and Zen 5 | flag `avx_vnni` |
| AVX-512 VPOPCNTDQ | Qdrant's only AVX-512 path, in binary quantization (Ice Lake and later) | flag `avx512_vpopcntdq` |
| BLAS (OpenBLAS or MKL) | custom llama.cpp builds; helps prompt processing at batch sizes over 32, not generation | your build flags |
| oneDNN | PyTorch on the CPU, not Ollama | `torch.backends.mkldnn` |

```bash
grep -o -w -E 'avx2|fma|avx512f|avx512_vnni|avx512_bf16|avx512_fp16|amx_tile|avx_vnni|avx512_vpopcntdq' /proc/cpuinfo | sort -u
docker logs ollama 2>&1 | grep -E 'load_backend|system_info' | tail -2      # after a model has loaded
```

Two things to know as of October 2026. Ollama 0.40 runs llama.cpp's `llama-server` with every CPU variant
built in and loads the best one at model load, so there is nothing to compile or select; on a Cascade Lake
Xeon the log shows the cascadelake variant with AVX-512 and VNNI. And the `system_info` flags describe
how the loaded variant was built, not what the CPU supports; read `/proc/cpuinfo` for the hardware
([llama.cpp build docs](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md),
[Ollama releases](https://github.com/ollama/ollama/releases)). Qdrant has no AVX-512 path on Cascade Lake:
it uses AVX2 with FMA, which this CPU has, so nothing is lost
([Qdrant distance code](https://github.com/qdrant/qdrant/blob/master/lib/segment/src/spaces/simple.rs)).

### 5.2 Threads and NUMA

Token generation on a CPU is limited mostly by memory bandwidth. On a two-socket server each processor has
its own memory; a thread on one socket reading model weights from the other socket's memory crosses the
interconnect, and llama.cpp's own issue tracker reports two-socket runs slower than one socket
([llama.cpp issue 1437](https://github.com/ggml-org/llama.cpp/issues/1437)). On the other hand, one
48-thread server measured its small model faster at 24 threads than at 12 in short tests (0.365 s against
0.416 s median routing). So measure on your own prompts before pinning anything.

What to know before you test:

- Ollama has no NUMA setting (there is no `OLLAMA_NUMA`), and it passes no `--numa` flag.
- With `num_thread` unset, llama-server counts physical cores across all sockets from the host, and it does
  not see a Docker cpuset. If you pin the container to one socket, also set `num_thread` to that socket's
  physical cores. `num_thread` still works through the API `options` or a Modelfile `PARAMETER`, although it
  no longer appears in the parameter tables.
- A container capped at 24 CPUs but not pinned showed 16 threads in its log, so read the log line.

```bash
# Read
lscpu | grep -E 'NUMA|Socket|Core|Thread'; numactl --hardware        # sudo apt install numactl
sysctl kernel.numa_balancing
docker inspect -f 'cpus={{.HostConfig.NanoCpus}} cpuset={{.HostConfig.CpusetCpus}} mems={{.HostConfig.CpusetMems}}' ollama

# Recommended experiment: pin to node 0 and match the threads, then compare tokens/s and latency
docker update --cpuset-cpus="<node0-cpu-list>" --cpuset-mems="0" ollama
curl -s http://127.0.0.1:11434/api/generate -d '{"model":"hermes3:8b","prompt":"<your test prompt>","stream":false,"options":{"num_thread":12}}' | python3 -c 'import json,sys; d=json.load(sys.stdin); print(round(d["eval_count"]/(d["eval_duration"]/1e9),1), "tokens/s")'
# Roll back: docker update --cpuset-cpus="<all-cpu-list>" --cpuset-mems="0,1" ollama
```

Compose has no key for memory-node pinning, so a pin made with `docker update` is lost when the container
is recreated; write it into your start procedure if it wins
([Docker resource constraints](https://docs.docker.com/engine/containers/resource_constraints/)).

### 5.3 Quantization by tier

Smaller weights mean fewer bytes read per token, so on bandwidth-limited hardware a smaller quantization
is faster. llama.cpp's own table for an 8B model: Q4_K_M at 4.89 bits per weight is 4.58 GiB and generates
71.9 tokens/s on its test machine, Q8_0 is 7.95 GiB at 50.9, F16 is 14.96 GiB at 29.2
([quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md)).

| Tier | Suggested quantization | Why |
| --- | --- | --- |
| CPU only | Q4_K_M (or the Q4_0 the library ships) for 1.5B to 8B models | speed is bandwidth-bound; Q4 keeps quality adequate for triage and extraction |
| 24 GB VRAM | Q8_0 for 7B to 8B; Q4_K_M for 14B to 32B | weights plus context must fit in VRAM; anything that spills to the CPU slows sharply |
| 48 GB VRAM and up | Q4_K_M for 70B; Q6_K or Q8_0 for 30B | trade quality against the room the context cache needs |

Set `OLLAMA_FLASH_ATTENTION=1` explicitly if you quantize the context cache with
`OLLAMA_KV_CACHE_TYPE=q8_0`; the cache can only be quantized with flash attention on
([Ollama FAQ](https://docs.ollama.com/faq)).

### 5.4 Ubuntu settings

| Setting | Typical Ubuntu value | Recommendation and why | Read, then change |
| --- | --- | --- | --- |
| Kernel | 7.0 GA | The 26.04 HWE package tracks the same 7.0 kernel as of October 2026, so there is nothing to gain yet; revisit at the next point release | `apt policy linux-generic linux-generic-hwe-26.04`; `sudo apt-get install --install-recommends linux-generic-hwe-26.04` |
| CPU governor | `schedutil` on `intel_cpufreq` | Test `performance` for steadier latency; it costs power, and a BIOS power profile can override it ([intel_pstate](https://docs.kernel.org/admin-guide/pm/intel_pstate.html)) | `cpupower frequency-info`; `sudo apt install linux-tools-common linux-tools-$(uname -r)`; `sudo cpupower frequency-set -g performance` (not persistent; use a oneshot unit if it wins) |
| Transparent hugepages | `madvise` | Leave it. Model weights are memory-mapped files, which this setting barely affects ([transhuge](https://docs.kernel.org/admin-guide/mm/transhuge.html)) | `cat /sys/kernel/mm/transparent_hugepage/enabled` |
| Swap | often none on provider images (then swappiness has no effect) | Add a small zram swap as a buffer so a memory spike slows a job instead of waking the out-of-memory killer; keep `MemoryMax` limits so a runaway job is the one killed ([zram-generator](https://github.com/systemd/zram-generator)) | `swapon --show; zramctl`; `sudo apt install systemd-zram-generator`; `printf '[zram0]\nzram-size = 8192\ncompression-algorithm = zstd\n' \| sudo tee /etc/systemd/zram-generator.conf`; `sudo systemctl daemon-reload && sudo systemctl start /dev/zram0` |
| Swappiness | 60 | 10 if you add disk swap, so the page cache gives way before process memory ([vm sysctl](https://docs.kernel.org/admin-guide/sysctl/vm.html)) | `sysctl vm.swappiness`; `echo 'vm.swappiness=10' \| sudo tee /etc/sysctl.d/90-swappiness.conf && sudo sysctl --system` |
| NVMe I/O scheduler | `none` | Leave it: the kernel picks `none` for multi-queue devices ([switching schedulers](https://docs.kernel.org/block/switching-sched.html)) | `cat /sys/block/nvme0n1/queue/scheduler` |
| I/O priority for background jobs | often `IOSchedulingClass=idle` in backup units | That setting does nothing with `none`: only BFQ and mq-deadline honor I/O priorities ([ioprio](https://docs.kernel.org/block/ioprio.html)). Limit background I/O with cgroup I/O controls in a system slice instead, or accept it | `systemctl --user show <unit> -p IOSchedulingClass` |
| Background jobs vs the desktop | no limits until you add them | Put agent and batch jobs in one slice with `CPUWeight`, `MemoryHigh` and `MemoryMax` ([kit/systemd/agent-jobs.slice](kit/systemd/agent-jobs.slice)). The user manager gets only the cpu, memory and pids controllers, so `IOWeight` and `AllowedCPUs` in user units are ignored ([resource control](https://manpages.ubuntu.com/manpages/resolute/man5/systemd.resource-control.5.html)) | `cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers` |
| Security updates and reboots | daily updates, no automatic reboot | Pick a reboot window so kernel fixes take effect: `Unattended-Upgrade::Automatic-Reboot "true";` and `Unattended-Upgrade::Automatic-Reboot-Time "04:30";`, or reboot by hand weekly ([automatic updates](https://ubuntu.com/server/docs/how-to/software/automatic-updates/)) | `grep -n Automatic-Reboot /etc/apt/apt.conf.d/50unattended-upgrades`; `ls /var/run/reboot-required` |
| Snap or apt | the installer offers snaps | Use the vendors' own packages. The Docker snap is built by Canonical, keeps files under your home folder, and cannot start NVIDIA GPU containers since v29.2.0; the Ollama snap is a community build several versions behind ([Docker snap README](https://github.com/canonical/docker-snap/blob/main/README.md), [Ollama on Linux](https://docs.ollama.com/linux)) | `snap list` |

### 5.5 If you have a GPU

- **Driver:** list and install the server branch with `sudo ubuntu-drivers list --gpgpu` and
  `sudo ubuntu-drivers install --gpgpu nvidia:<branch>-server`, then hold it (`sudo apt-mark hold` on the
  driver packages) so an unattended upgrade cannot change it under a running model
  ([NVIDIA drivers on Ubuntu](https://ubuntu.com/server/docs/how-to/graphics/install-nvidia-drivers/)).
- **Persistence:** use the `nvidia-persistenced` daemon; `nvidia-smi -pm 1` is the legacy mode NVIDIA calls
  near end of life ([persistence daemon](https://docs.nvidia.com/deploy/driver-persistence/persistence-daemon.html)).
- **Containers:** install the NVIDIA Container Toolkit, run `sudo nvidia-ctk runtime configure --runtime=docker`,
  restart Docker, and test with `docker run --rm --gpus all ubuntu nvidia-smi`
  ([toolkit install](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)).
- **Ollama** needs compute capability 5.0 or newer and driver 550 or newer ([Ollama GPU](https://docs.ollama.com/gpu)).
- **Why VRAM decides the model:** the weights, plus the context cache (which grows with context length times
  parallel slots), plus about a gigabyte of overhead must fit in free VRAM, or layers spill to the CPU and
  generation slows sharply. `ollama ps` shows the split (for example `100% GPU` or `48%/52% CPU/GPU`).

## 6. What Jev and typed decisions add

**What it is (as of October 2026).** TypeSafe's Jev is the first "System One" model: you send a state
(text or JSON) and typed questions, and it returns typed answers with probabilities. A Choice picks one of
your options, a Score places the state on your rubric, a Noul gives the probability that a statement is
true; a fourth type for images (Box) is in preview. It does not write text. The hosted API is
`POST https://api.typesafe.ai/v1/systemone` with a bearer key; `jev-latest` is an alias for `jev-1.13.0`;
input costs $0.042 per million tokens and output is free; a request holds up to 64K tokens of context
([quick start](https://docs.typesafe.ai/introduction/quickstart), [models](https://docs.typesafe.ai/models)).
The speed and accuracy figures in the announcement are the company's own. Ollama 0.35 and later serves a
local endpoint that follows the same API, with open decision models (Bespoke Labs' Nimble 9B, Together
AI's Tev1 4B and 0.8B, and Cloudflare's Clef models), and the official SDK works against it
([Ollama decision models](https://docs.ollama.com/capabilities/decision)). Nimble's page reports 75.7
percent mean accuracy against 76.0 percent for Jev 1.13 on its maker's evaluation: a vendor claim to test,
not a result to quote.

**Where it sits.** Beside the gateway, not behind it: LiteLLM speaks the OpenAI chat API, and System One
is a different API. Wrap it in one function that picks the backend by what the state contains, and logs
the model version, the probabilities and the threshold beside every decision.

```text
 consumers: prompt hub, n8n, job scripts, the preprocessing step, agent wrappers
        |                                          |
        | typed questions ("which, how much,       | text ("write, summarize, explain")
        |  is it true")                            |
        v                                          v
 decide(state, questions, private?)          LiteLLM gateway
   private  -> Ollama /v1/systemone            -> Ollama (local models)
               (nimble or tev1, local)         -> cloud APIs (named routes, budgets)
   shareable -> TypeSafe /v1/systemone
               (jev-1.13.0 pinned)
   logs: model, probabilities, threshold, outcome
```

**What it adds, concretely:**

1. **Typed answers for every decision a model makes.** Without it, a decision such as "which job type is this"
   is a small model asked to return JSON, a parser, a retry and an enum check. A Choice can only return one
   of your labels, with a probability for each; nothing to parse, nothing outside the schema.
2. **Validation and repair become questions.** Instead of only checking that a quote appears in the source,
   you ask a Noul per field ("does every amount in `extracted.prices` appear in `source` with the same
   meaning?") and repair only the fields that fail.
3. **Confidence and intent routing.** TypeSafe's confidence for a Choice with n options is
   `(p_top - 1/n) / (1 - 1/n)`: 0 for an even split, 1 for certainty. Its patterns set a floor (0.5 or
   0.6) below which a person decides, and a higher bar (0.85 or 0.9) for actions with consequences
   ([confidence](https://docs.typesafe.ai/confidence),
   [confidence routing](https://docs.typesafe.ai/patterns/confidence-routing),
   [intent routing](https://docs.typesafe.ai/patterns/intent-routing)). Cheap local first; escalate only
   when confidence is low.
4. **Guardrails as a battery of Nouls.** Screen every inbound message and every outbound reply with one
   request: asks for credentials, asks for money, a jailbreak attempt, a policy breach, plus a severity
   Score. Code maps each to pass, review or block ([LLM guardrails cookbook](https://docs.typesafe.ai/cookbooks)).
5. **Self-reported confidence is replaced by probabilities you can calibrate.** The confidence number is a
   summary of the probabilities, not a promise of accuracy. It becomes useful once you bucket a labelled
   set by confidence and see accuracy rise with it.

**Before and after: the preprocessing step.**

Before (the usual pattern): code cleans the page; a local model fills the JSON schema; code
checks the shape, that dates parse and that each quote is a literal substring of the source; one repair
retry; then escalation. The validator can say a quote exists, but not whether the price is the current one
or the date is the publication date.

After: two typed requests around the same extraction.

```text
 page --> clean --> [A] Choice page_type, Noul "states a price", Noul "asks for credentials"
                        |  no prices and not a product page -> skip extraction (store type only)
                        |  asks for credentials (p >= 0.5)  -> quarantine, alert, never auto-process
                        v
                    local model fills the schema (as before)
                        |
                        v
                    [B] Nouls: prices_supported, date_supported, title_supported;
                        Score: record_quality (3 levels)
                        |  all Nouls >= 0.8 and quality confidence >= 0.6 -> store
                        |  any Noul < 0.3 -> repair that field only, then re-ask [B] once
                        |  otherwise -> larger model or a person, with the probabilities attached
                        v
                    record + probabilities + model version -> database and Qdrant
```

The request for [B] is in [kit/schemas/page-extract.jev-questions.json](kit/schemas/page-extract.jev-questions.json).
What it saves: extraction runs only on pages that have something to extract; repairs target one field
instead of the whole record; and the escalation decision has a number behind it that you can tune. The
thresholds above are starting points; set yours from a labelled set.

**Before and after: one agent decision (the hub's plan).**

Before: a typical hub asks a 1.5B model for `{"kind", "changes_files"}`, picks the mode from
`changes_files` (true means work mode), and picks the engine from a fixed order. A wrong `changes_files`
gives a read-only job that cannot do the work, or write access for a question. There is no confidence, so
the plan cannot say "unsure".

After: one System One request per prompt, and code that owns the outcome.

```text
 prompt --> questions (one request, answered in parallel):
              Choice  mode:        ask | work
              Choice  engine_fit:  script | local | codex | claude | person
              Noul    outward:     "asks to send, deploy, pay, sign or publish"
              Noul    credentials: "asks for or contains a password, key or token"
              Score   complexity:  simple | multi-step | unusual
        |
        v
  credentials >= 0.5 ------------------------------> refuse; tell the person; nothing stored
  outward >= 0.5 ----------------------------------> plan says "a person sends"; mode forced to ask
                                                      for the outward part
  mode confidence < 0.6 ---------------------------> default to ask; the plan shows the alternative
  engine_fit confidence < 0.5 or "person" ---------> show the top two engines; the person picks
  complexity "unusual" (confidence >= 0.6) --------> strongest engine; note it in the plan
  otherwise -----------------------------------------> plan as decided
        |
        v
  the plan still waits for the person's Run; every probability is stored with the job
```

**How it sharpens the hub:**

- **A typed plan schema.** The kit's hub already validates its plan with Pydantic (`Literal` fields,
  `extra="forbid"`), so a plan outside the allowed values cannot exist. System One answers map straight
  onto those fields.
- **Confidence-based engine routing** replaces the fixed order for prompts that do not name an engine.
- **Validated job results.** After a job, a Noul battery over `result.md` ("claims a deploy happened",
  "contains a secret-like string", "answers the prompt") decides whether the alert says done, needs review
  or blocked.
- **Typed triage from the local model** for private prompts, through Ollama's `/v1/systemone`, so private
  prompts never leave the server.

**What it optimizes:** fewer retries (no parse failures), fewer cloud tokens (skipped extractions and
targeted repairs), deterministic downstream code (every branch is an `if` on a typed value), and safer
agents (an outward or credential flag stops the plan before it reaches Run).

**What leaves the server, and the rule to write down.** Calling hosted Jev sends the state and questions to
TypeSafe. Its privacy policy says it does not train on your input; retention is "as long as reasonably
necessary"; hosting is in the United States; zero retention is offered to enterprise customers; whether it
can run in your own VPC is asked on its enterprise page but not answered
([privacy](https://typesafe.ai/privacy), [enterprise](https://typesafe.ai/enterprise)).

| What is in the state | Where the decision runs |
| --- | --- |
| Public pages, your own published content, synthetic tests | hosted Jev (pinned version) |
| Internal notes with no personal or client data | hosted Jev after a redaction pass, or local |
| Client, health, legal or financial records, or anything under a sharing limit | local decision model only |
| Credentials, keys, session tokens | never in any state |

**How to adopt it without risk:** pin `jev-1.13.0` rather than `jev-latest` once thresholds are tuned,
because the alias moves; build a labelled set of a few hundred past decisions; run hosted Jev, the local
decision model and your current method over it; bucket by confidence and read the accuracy per bucket;
run a week in shadow mode; then let it decide one low-stakes question with the local fallback wired in
([Jev in a sovereign stack](https://jwatte.com/blog/typesafe-jev-system-one-sovereign-ai-stack/)). Costs:
at the listed price a 2,000-token decision is about $0.000084, so 10,000 a day is about $0.84. A local
decision model costs CPU time instead; its latency on a CPU-only server has not been measured, so measure
it before you put it in an interactive path.

```text
 JEV CONFIDENCE AND INTENT ROUTING (decision tree for one request)

 incoming message or prompt
   |
   v
 private material? --yes--> local decision model (Ollama /v1/systemone)
   | no                                    |
   v                                       |
 hosted Jev (pinned version) <-------------+  same questions, same answer shape
   |
   v
 credentials or money requested (Noul >= 0.5)? --yes--> hold for the person; alert; stop
   | no
   v
 intent Choice confidence < 0.5? --yes--> person decides (plan shows the options)
   | no
   v
 intent = read-only action and confidence >= 0.6 --> run (after Run in the hub)
 intent = outward action:  confidence >= 0.85 --> prepare it; the person sends
                           0.5 to 0.85        --> ask the person to confirm the intent first
   |
   v
 log: model version, probabilities, threshold, what happened
```

## 7. Updates and maintenance

**The definition that keeps updates honest:** an update is done when the new version is installed,
running, checked, pinned and reversible. On one working server, the tunnel ran its old binary for 70 hours
after the file was replaced, because nothing compared the running process with the file
([safe update command](https://jwatte.com/blog/safe-update-command-self-hosted-ai-server/)). Check the
running binary with `readlink /proc/<pid>/exe`, or the container's image digest with `docker inspect`.

```text
 UPDATE FLOW (one component at a time)

 trigger: weekly window, the ops portal, or a manual GitHub dispatch
   |  wait until no agent job is running (a plain shell wait, outside any agent)
   v
 record: current digests, versions, configs, health  --> pins file
 backup: restic snapshot; Qdrant snapshot; database dumps; desktop profile snapshot
   |
   v
 host packages (apt) --> Docker Engine --> agent CLIs --> service images --> desktop (last)
   each step: read release notes -> pull candidate -> start -> health + a real read/write smoke test
              -> pass: pin the new digest
              -> fail: restore the old digest and config, re-check, stop
   |
   v
 report + ntfy alert; cleanup ONLY after the rollback points are verified
```

**Per component:**

| Component | How to update safely | Watch for |
| --- | --- | --- |
| Ubuntu packages | unattended-upgrades for security; `sudo apt full-upgrade` in the window, then reboot if `/var/run/reboot-required` exists | never run apt from a service that can be killed mid-install; after an interruption run `sudo dpkg --configure -a` and check the boot image before rebooting |
| Docker Engine | apt, in the window; containers with restart policies come back | read the release notes for firewall and networking changes |
| Agent CLIs (Claude Code, Codex) | each through its own channel; confirm `--version` from a new shell | a wrapper or launcher can keep pointing at the old version |
| Ollama | new image or install script; re-run your fixed test prompts | a new version can change defaults (context length, scheduling) |
| Models | pin the tag and record the digest (`ollama list`); re-run the held-out test cases; never change the embedding model in place | a changed embedding model needs a new Qdrant collection and a full re-index |
| LiteLLM | new pinned tag; keep the same salt key; check `/health/readiness` | database migrations; never regenerate the salt key |
| Qdrant | snapshot first; one minor version at a time | snapshots restore only to the same or next minor version |
| Open WebUI | new pinned tag | saved settings override the environment |
| Guacamole | read the release notes | renamed variables are ignored silently (1.6.0 renamed the header switch) |
| n8n | pinned version, runner image at the same version, database and key backed up first | 2.x changed several defaults |
| cloudflared | apt; then compare the running binary with the installed one | auto-update is off for package installs |
| Browser desktop | build or pull the candidate, test it on a copy of the live profile with the same file ownership, promote only when nobody is using it, keep the old image and the profile snapshot | a root-owned folder inside the live profile broke one tool only on the live desktop; retries that rebuild a 34 GB image fill the disk |

**Pinning and cleanup.** Keep a pins file with every digest and version you run, change it in reviewed
commits, and prune old images only through the updater's own records, after the rollback points are
verified. Digest pinning does not deliver security fixes; the weekly window does.

**Backups before every update.** Take a restic snapshot, a Qdrant snapshot and the database dumps first
(section 4.15), and keep the previous image digests until the new versions have run cleanly for a while.
The LiteLLM salt key, the n8n encryption key and the browser profile must be restorable together with
their data ([desktop and Drive backups](https://jwatte.com/blog/sovereign-windows-desktop-drive-backups/)).

## 8. Lessons learned

Each rule comes from something that went wrong, or nearly did, on a working server. Names are removed.

**Running jobs**

1. **Detach long jobs properly.** Start them from a script with `setsid nohup ./run.sh > run.log 2>&1 < /dev/null &`.
   Why: a job tied to your SSH session dies or hangs when the session does.
2. **Stop jobs by recorded PID, never by `pkill -f` on a pattern in your own command line.** Why: the
   pattern matches the SSH command that runs it, and you kill your own session.
3. **Killing a wrapper does not kill its child.** Judge liveness by the age of the job's log or session
   file, not by whether a process name exists. Why: a stalled job looked alive, and a killed wrapper left
   the real worker running.
4. **Unattended runs do not read messages sent after they start.** Stop, amend the brief, and relaunch.
   Why: a correction queued to a running job was never seen.
5. **Never make maintenance wait for "no agents" while an agent waits for maintenance.** Put the wait in a
   plain shell loop outside both. Why: a mutual wait deadlocked an update.
6. **Gate a job on what it needs, not on "every checklist item done".** Why: a gate blocked on an
   owner-only step that the job did not depend on.
7. **Retry only the failure class you understand.** A provider's "at capacity" error gets a bounded
   backoff; other errors stop. Why: blind retries hide real failures and burn quota.

**Shells, tools and secrets**

8. **Some CLIs need a login shell.** Over SSH run `bash -lc '...'`; in systemd units set `PATH` and
   absolute paths. Why: tools installed through a version manager are invisible to non-login shells, and
   one deploy tool silently returned nothing.
9. **Never put backticks or `$(` inside a command string you hand to a tool.** Write a script file and run
   it. Why: a backtick in a text-editing command executed a production deploy.
10. **Keep secrets out of files you share, commits, briefs, chats, logs and command lines.** Store them in
    owner-only files and read them where they are used; scan before every commit. Why: keys pushed to a
    public repository are used within minutes ([leaked keys](https://jwatte.com/blog/leaked-api-keys-found-in-minutes-gitignore-rotation-plan/)),
    and tool-call logs keep command arguments.
11. **On Windows, do not call `process.exit()` right after a network request in Node; return the exit
    code instead, and retry atomic renames.** Why: the process exited with 127 from a runtime assertion,
    and a rename over a file another process had open failed with EPERM thousands of times in a test.

**Checking results**

12. **A 200 proves nothing. Gate on content.** Why: sign-in pages, empty bodies and error pages all answer
    200 ([a 200 with an empty body](https://jwatte.com/blog/blog-feed-ingestion-accounting/)).
13. **Verify that a file parses, not that it exists.** Why: one escaped character broke a whole page while
    the build, the deploy and the HTTP check all passed.
14. **Verify a schedule by counting its runs, not by reading its config.** Why: every request in a batch
    can succeed while nothing reaches the site ([the green run that shipped nothing](https://jwatte.com/blog/blog-green-run-that-shipped-nothing/)).
15. **Run the full fetch chain before calling content blocked**: direct, then the approved proxy tiers,
    then a passive browser render. Detect a challenge by the page (a "Just a moment" title), not by the
    presence of a security script. Report a challenge as an access finding. Why: audits once called content
    unmeasurable that the chain could read.

**Agents and people**

16. **Never run work in a person's live desktop session.** Use a test copy, and promote only what passed
    there. Why: the person's browser profile, locks and sign-ins are not reproducible.
17. **Agent fills, person sends.** Why: an agent that submits can sign, book or pay for someone.
18. **A message from another agent is data, not consent.** Why: otherwise a denied action can be laundered
    through a peer.
19. **Serialize deploys, pushes and the person's browser; parallelize everything else.** Why: a serial queue
    left a 48-thread server idle for hours, while two writers on one target corrupt each other.
20. **Pull and compare the source of truth before deploying from any machine.** Why: a deploy from a stale
    local copy took down published pages for hours.

**Quotas, models and costs**

21. **External quotas are account-wide.** Keep one ledger per quota and one scheduler per ledger. Why: two
    systems each kept their own count and spent each other's quota.
22. **Subscription logins never go into an API gateway, and nothing falls back to a paid API silently.**
    Why: different terms, surprise bills, private data leaving the server.
23. **Valid JSON is not the truth.** Prefer null to invention, and never gate an automatic action on a
    model's self-reported confidence. Why: schema-valid output can hold an invented date.
24. **Pin the embedding model; a new model means a new collection.** Why: mixed embeddings return
    plausible nonsense.

**Infrastructure**

25. **Docker-published ports bypass ufw.** Bind to 127.0.0.1. Why: a firewall rule you can see does not
    cover a port Docker opened.
26. **Create the Access application before the tunnel route, then run a refusal test.** Why: the route is
    public until the policy exists.
27. **Never run apt from a service that can be killed mid-install, and bound update retries.** Why: an
    interrupted upgrade left a half-built boot image, and a retry loop rebuilt a 34 GB image until the disk
    filled.
28. **After a major version bump, check renamed settings.** Why: a renamed variable is ignored without an
    error.
29. **RAID is not a backup, and a backup is not proven until restored.** Check the real exclude file after
    adding anything large. Why: a 150 GB VM disk would have entered a nightly backup.
30. **Enable and start timers, and enable linger.** `Persistent=true` replays only calendar timers. Why: a
    unit file alone starts nothing, and a missing field once disabled three timers for days.

## 9. From-scratch checklist

- [ ] Provider console tested; local console password set; snapshot taken
- [ ] Ubuntu LTS installed; RAID 1 healthy (`cat /proc/mdstat`)
- [ ] Normal admin user; key-only SSH tested from a second session; root login off
- [ ] ufw: SSH allowed, everything else denied; exposure tested from outside over IPv4 and IPv6
- [ ] Automatic security updates on; reboot window decided
- [ ] Docker from Docker's apt repository; every published port on 127.0.0.1
- [ ] Domain on Cloudflare; Access application per hostname (deny by default); tunnel routes with Protect with Access; refusal test passed
- [ ] Guacamole (and a Windows VM if needed) behind Access; default admin changed; `HTTP_AUTH_ENABLED=true`
- [ ] Browser desktop on loopback, password set, sudo off, fixed hostname
- [ ] Ollama with one triage, one extraction and one embedding model; backend and threads checked in the log
- [ ] LiteLLM with PostgreSQL; aliases defined; a virtual key with a budget for every caller
- [ ] Open WebUI through LiteLLM only; sign-up off
- [ ] Notes in git; FTS5 keyword search; Qdrant only if needed, with admin and read-only keys
- [ ] Claude Code and Codex installed; standing rules file; one folder per job
- [ ] Job runner slice, timers enabled and started, linger on
- [ ] ntfy topic long, random, private; test alert received
- [ ] n8n only if needed: PostgreSQL, task runner, key backed up, rejection tests before activation
- [ ] Prompt hub on loopback with JWT verification, plan then Run, alerts with links
- [ ] GitHub Actions: health checks through an Access service token; updates by dispatch only
- [ ] Backups: dumps and snapshots staged; local and off-site restic repositories (off-site created with `--copy-chunker-params`); own Google OAuth client, `drive.file`, published "In production"; rclone config encrypted; password off the server
- [ ] Backup alarms: `OnFailure` alert, ntfy dead-man's switch, GitHub heartbeat; first restore test passed
- [ ] Desktop Commander folders narrowed; unused devices revoked; telemetry decided
- [ ] Baseline recorded: versions, digests, `ss -ltnp`, timers
- [ ] Typed decisions in shadow mode for a week before any unattended use

## 10. Troubleshooting

| Symptom | Likely cause | Check | Fix |
| --- | --- | --- | --- |
| A hostname shows a Cloudflare error page | service not listening, or the tunnel is down | `ss -ltnp \| grep <port>`; `systemctl status cloudflared` | start the service; fix the route's port |
| A page opens without signing in | route created before the Access application, or not protected | private-window test | create the application; turn on Protect with Access |
| A service is reachable from the internet despite ufw | Docker published it on all addresses | `docker ps --format '{{.Names}} {{.Ports}}'` | publish on `127.0.0.1:` |
| SSH change did not take effect | an earlier drop-in sets the value first | `sudo sshd -T` | rename your drop-in to sort first (`00-`) |
| First Ollama call takes many seconds | model loading | `ollama ps` | longer keep-alive, or per-request `keep_alive` for batches |
| Ollama runs out of memory | parallel slots multiply context | `ollama ps` (CONTEXT column) | fewer slots, smaller context, `OLLAMA_KV_CACHE_TYPE=q8_0` |
| An Ollama setting is ignored | set in a shell, not in the unit or Compose file | `systemctl show ollama -p Environment` or `docker inspect` | set it where the service reads it; restart |
| `/v1/systemone` returns 404 | Ollama older than 0.35 | `ollama --version` | update; a 405 on GET is normal |
| LiteLLM returns 401 | the key is not allowed that model | `/key/info?key=...` with the master key | issue a key with the right model list |
| Open WebUI ignores a changed variable | it saved the old value on first start | its admin settings | change it there, or set `ENABLE_PERSISTENT_CONFIG=False` |
| Qdrant answers "Must provide an API key" | the client sends no key | the request headers | send `api-key: <key>` |
| Search results got worse after a model change | mixed embedding models | the collection's vector size and model note | new collection, full re-index |
| Guacamole header sign-in does nothing on 1.6 | old variable name | `docker inspect` the environment | `HTTP_AUTH_ENABLED=true` |
| Guacamole drops idle sessions | session timeout or a hidden browser tab | `API_SESSION_TIMEOUT` | raise it; keep the desktop in its own window |
| A timer never ran | timer not started, or linger off | `systemctl --user list-timers`; `loginctl show-user <user> -p Linger` | `enable --now` the timer; enable linger |
| A job vanished when SSH dropped | not detached | the job's log time | relaunch with `setsid nohup ... < /dev/null &` |
| A CLI is "not found" in a unit or over SSH | the PATH comes from a login shell | `bash -lc 'command -v <tool>'` | absolute paths or `Environment=PATH=` |
| Hermes Agent refuses to start | context window under 64K | `hermes doctor` | larger context, or a hosted model through the gateway |
| n8n credentials unreadable after a restore | encryption key missing | the `.n8n/config` file | restore the key with the database |
| The off-site copy fails about a week after setup | the Google OAuth app is still in "Testing" | the consent screen's publishing status | publish it "In production" and authorize again |
| rclone Drive access stops working in 2026 | the remote uses rclone's shared client | `client_id` in `rclone config show <remote>` (on the server, not in a chat) | create your own OAuth client and authorize again |
| Drive storage keeps growing after restic prunes | deleted packs sit in the Drive trash | Drive storage page | `--drive-use-trash=false` on the restic remote; empty the trash once |
| A restore fails to decrypt | wrong or lost restic password | `restic snapshots` with the password file | use the copy in your password manager; without it the backup is unreadable |
| A scheduled GitHub workflow stopped | 60 days without activity in a public repository | the Actions tab | re-enable it; keep the repository private |
| Many CI runs fail at once with identical durations | a GitHub outage | GitHub's status page | wait; do not re-run during the outage |

## 11. The setup kit

Every file has a header comment saying what it is and which section uses it. They are templates for your
own server, not copies of any live server's files. See [kit/README.md](kit/README.md) for the full
map and the validation results.

## Further reading

- [Jev and System One models in a sovereign AI stack](https://jwatte.com/blog/typesafe-jev-system-one-sovereign-ai-stack/)
- [The work around the model: operating a sovereign AI workstation](https://jwatte.com/blog/sovereign-ai-workstation-operating-system/)
- [Sovereign AI workstation best practices](https://jwatte.com/blog/sovereign-ai-workstation-best-practices/)
- [The sovereign advantage of a dedicated AI workstation](https://jwatte.com/blog/sovereign-advantage-dedicated-ai-workstation/)
- [Small quantized models on a 96 GB server](https://jwatte.com/blog/small-quantized-models-96gb-ai-server/)
- [A safe update command for a self-hosted AI server](https://jwatte.com/blog/safe-update-command-self-hosted-ai-server/)
- [Build your own AI prompt hub](https://jwatte.com/blog/build-your-own-ai-prompt-hub/)
- [Desktop Commander, Codex and Claude for a small business](https://jwatte.com/blog/desktop-commander-codex-claude-small-business-guide/)
- [The sovereign AI playbook with Remote Desktop Commander](https://jwatte.com/blog/sovereign-ai-playbook-remote-desktop-commander/)
- [n8n for small-business automation](https://jwatte.com/blog/n8n-small-business-automation-guide/)
- [A Windows desktop and Drive backups on the same server](https://jwatte.com/blog/sovereign-windows-desktop-drive-backups/)
- [Podman versus Docker for a Windows 11 dedicated server](https://jwatte.com/blog/podman-vs-docker-windows-11-dedicated-server/)
- [Running your own AI on-prem versus cloud](https://jwatte.com/blog/local-ai-on-prem-vs-cloud/)
- [Qwen open weights, self-hosted at home](https://jwatte.com/blog/qwen-open-weights-self-hosted-at-home/)

## Sources

Read on 2026-10-15. Vendor claims are marked as such where they appear.

- TypeSafe: announcement https://typesafe.ai/blog/introducing-system-one-models-and-jev ; quick start https://docs.typesafe.ai/introduction/quickstart ; models and prices https://docs.typesafe.ai/models ; confidence https://docs.typesafe.ai/confidence ; confidence routing https://docs.typesafe.ai/patterns/confidence-routing ; intent routing https://docs.typesafe.ai/patterns/intent-routing ; cookbooks https://docs.typesafe.ai/cookbooks ; adapter https://github.com/typesafe-ai/system-one-adapter-python ; privacy https://typesafe.ai/privacy ; enterprise https://typesafe.ai/enterprise
- Ollama: FAQ https://docs.ollama.com/faq ; Linux https://docs.ollama.com/linux ; Docker https://docs.ollama.com/docker ; structured outputs https://docs.ollama.com/capabilities/structured-outputs ; decision models https://docs.ollama.com/capabilities/decision ; System One API https://docs.ollama.com/api/systemone ; Nimble https://ollama.com/library/nimble ; Hermes 3 https://ollama.com/library/hermes3
- LiteLLM: configs https://docs.litellm.ai/docs/proxy/configs ; routing https://docs.litellm.ai/docs/routing ; virtual keys https://docs.litellm.ai/docs/proxy/virtual_keys ; caching https://docs.litellm.ai/docs/proxy/caching ; Ollama provider https://docs.litellm.ai/docs/providers/ollama
- Qdrant: security https://qdrant.tech/documentation/security/ ; collections https://qdrant.tech/documentation/concepts/collections/ ; indexing https://qdrant.tech/documentation/concepts/indexing/ ; snapshots https://qdrant.tech/documentation/snapshots/
- Open WebUI environment reference: https://docs.openwebui.com/reference/env-configuration
- Hermes Agent: https://github.com/NousResearch/hermes-agent ; providers https://hermes-agent.nousresearch.com/docs/integrations/providers
- Cloudflare: packages https://pkg.cloudflare.com/index.html ; run parameters https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/configure-tunnels/cloudflared-parameters/run-parameters/ ; self-hosted applications https://developers.cloudflare.com/cloudflare-one/access-controls/applications/http-apps/self-hosted-public-app/ ; JWT validation https://developers.cloudflare.com/cloudflare-one/access-controls/applications/http-apps/authorization-cookie/validating-json/ ; service tokens https://developers.cloudflare.com/access/service-auth/service-token/
- Apache Guacamole 1.6.0: Docker https://guacamole.apache.org/doc/gug/guacamole-docker.html ; header authentication https://guacamole.apache.org/doc/gug/header-auth.html ; feature switches https://raw.githubusercontent.com/apache/guacamole-client/1.6.0/guacamole-docker/entrypoint.d/700-configure-features.sh
- Ubuntu: firewalls https://ubuntu.com/server/docs/how-to/security/firewalls/ ; automatic updates https://ubuntu.com/server/docs/how-to/software/automatic-updates/ ; sshd_config https://manpages.ubuntu.com/manpages/resolute/en/man5/sshd_config.5.html ; 26.04 release notes https://documentation.ubuntu.com/release-notes/26.04/
- Docker: install on Ubuntu https://docs.docker.com/engine/install/ubuntu/ ; packet filtering and firewalls https://docs.docker.com/engine/network/packet-filtering-firewalls/
- dockur/windows https://github.com/dockur/windows ; webtop https://docs.linuxserver.io/images/docker-webtop/ ; Tailscale https://tailscale.com/kb/1031/install-linux
- ntfy: publishing https://docs.ntfy.sh/publish/ ; configuration https://docs.ntfy.sh/config/
- restic: https://restic.readthedocs.io/en/stable/
- n8n: licence https://docs.n8n.io/n8n-community-license ; hosting https://docs.n8n.io/deploy/host-n8n/ ; security https://docs.n8n.io/deploy/host-n8n/configure-n8n/security
- Desktop Commander: https://github.com/wonderwhy-er/DesktopCommanderMCP ; security https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/SECURITY.md ; remote device https://github.com/wonderwhy-er/DesktopCommanderMCP/blob/main/src/remote-device/README.md ; Claude on the web https://desktopcommander.app/mcp/claude-web/
- OpenAI: Codex https://github.com/openai/codex ; non-interactive mode https://learn.chatgpt.com/docs/non-interactive-mode ; AGENTS.md https://learn.chatgpt.com/docs/agent-configuration/agents-md ; custom MCP servers https://developers.openai.com/api/docs/guides/custom-mcp-server
- Claude Code: setup https://code.claude.com/docs/en/setup ; headless https://code.claude.com/docs/en/headless ; permission modes https://code.claude.com/docs/en/permission-modes
- GitHub Actions: events https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows ; secure use https://docs.github.com/en/actions/reference/security/secure-use
- FastAPI deployment: https://fastapi.tiangolo.com/deployment/manually/
- Server examples: Hetzner GPU matrix https://www.hetzner.com/dedicated-rootserver/matrix-gpu/ ; OVHcloud GPU servers https://www.ovhcloud.com/en/bare-metal/gpu-dedicated-server/