New to this? Start with Sovereign AI Made Simple.
I care less about whether an AI workstation can answer a question than whether it can finish useful work, leave a result I can trust, and pick up the next task without another briefing.
The Sovereign Advantage covers the hardware, model choices and financial limits. Sovereign AI Workstation Best Practices covers desktop persistence, credentials and what earlier maintenance checks actually proved.
I want to extend those posts with the operating habits that turn the workstation into a place where work accumulates.
The examples below are proposed workflows for a small business, a publisher and an agency. They are not customer case studies or measured returns. I do not attach invented performance or savings figures to them. The payoff I would test is simpler: does the machine complete more of the work I already need, with less supervision and a recoverable record?
Download the Markdown operator kit. The kit includes adaptable job rules, service templates, monitoring, memory, backups and update checklists.
Give independent work its own lane
A queue can waste a roomy machine by putting unrelated work behind one long task. I would split it by a unit that can finish independently: one website, one repository, one report. Each job gets a brief, its own working copy, a result location and a clear completion condition.
Before launching the next batch, I check available memory, CPU load and what is already running. I leave room for the desktop and local model service. I reduce concurrency when memory pressure, storage contention or slow responses show that the machine is busy.
Hardware capacity does not tell me how many inference requests will fit; Ollama's documentation describes the additional memory needed for concurrent requests and longer contexts.
An agency could collect evidence for separate websites while another job prepares a finished report. A publisher could build one site while checking another site's public pages. Within a job, independent checks can overlap too. Fetching and interpreting the same page are dependent steps, so interpretation waits for the evidence.
Ownership prevents concurrency from becoming a repair job. Two workers should not edit the same branch, overwrite the same report or deploy the same site together. I serialize production deployments per site and pushes per repository. I also share the request budget across jobs visiting the same origin.
Parallel work across different sites is useful; a burst of workers against one site is poor manners and unreliable evidence.
I want a durable record of queued, started and finished states, with a checkpoint after each completed unit. If the machine restarts halfway through a batch, the runner should know which reports exist and which pages remain. Restarting everything can duplicate work and external actions.
The agent jobs themselves need the same discipline. In one measured window, five unattended coding agent jobs running together used about 2 percent of a 48 core server; model response time and per site politeness set the pace.
The best practices guide covers permission profiles by job role, safe brief changes, log based liveness checks and the shell traps that cost time when launching them.
Keep access failures in the audit
Website research needs an explicit retrieval policy. I start with a direct request. If the page blocks that request, I retain that result and try the configured alternate route: direct access, then a Cloudflare worker, then a Deno service. If those routes remain blocked, I try a passive browser render, loading the page and reading what appears without interacting with a challenge.
For each planned page, I record the time, status, route attempted, timeout state and final address. I inspect the returned page for a challenge or an unexpectedly small interstitial. A script associated with challenge handling can appear on an ordinary page too; its presence alone does not establish a block.
A successful alternate route lets me measure the content. It does not erase the original access finding. My report should say that the tested direct route was blocked and explain the consequence for tools using that route. I would ask the site operator to review the relevant security rule and provide an intentional policy for legitimate tools.
One failed request does not prove that all AI agents, datacenter addresses or search engines are blocked, and it does not establish the operator's reason.
I keep the access finding separate from the content findings. If the browser can read a missing description, I can report that missing description with the public page address and observation time. If every permitted route fails, I mark that page unmeasured and list the failed attempts. I do not label unseen content compliant.
The coverage statement belongs beside the conclusion: pages planned, pages measured, the route used for each, and pages unmeasured with reasons. Every page in the plan gets the retrieval sequence; a successful sample does not establish access to the rest of a site.
This policy stops at a CAPTCHA, login or paywall. I do not solve challenges, impersonate a search engine or treat an alternate network route as permission to enter a restricted area. For an agency, that boundary makes the audit easier to explain: the report describes what the permitted checks established.
Make the result useful outside the workstation
A report is unfinished if the recipient needs my filesystem to understand it. I want the public page address, observation time, relevant excerpt, finding and proposed fix together. If a screenshot is necessary, I embed it in the document.
The private evidence can contain full page captures and diagnostic records. The shared document should contain the evidence a reader needs, with private account details and credentials removed. I scan the finished Markdown, HTML and PDF for machine paths, private links and preview addresses. For a PDF, I check both its extracted text and its link annotations.
For HTML, I inspect image sources and links as well as visible prose.
A small business owner could send a report to a designer and get a usable response without forwarding a chat history. An agency could deliver a fix list with public sources and explicit coverage. A publisher could keep a release note that still makes sense after changing hosting providers.
I also want the acceptance condition in the brief. “Write an audit” is vague. “Produce a shareable report, account for every planned page and verify each proposed fix against the evidence” tells the worker what finished means.
Watch the work, then watch the result
A watchdog and a scheduled monitor answer different questions. The watchdog asks whether a running job has died, stalled or exceeded its allowed duration. The scheduled monitor asks whether the system still produces the expected result after yesterday's success.
I would keep a heartbeat and last-progress time for long jobs, with bounded retries and an alert after repeated failure. A heartbeat proves that a process can update a timestamp. It does not prove that the workload is advancing. A useful watchdog compares both.
The alert needs enough context to act: what failed, when it last worked, whether any output was produced, what recovery was attempted and which step needs a person. Repeated identical alerts should be grouped. I would deliberately trigger a harmless failure during setup and confirm that the alert arrives through the chosen channel.
Scheduled checks should exercise behavior. A publisher's monitor could confirm that a newly published article appears in the feed, that its canonical address is correct and that a representative tool returns an expected result. A small business could check its contact form using an approved test route and confirm receipt.
An agency could revisit agreed public pages and flag changed findings without sending a new report automatically.
I preserve a baseline and report changes against it. A check that suddenly measures fewer pages should flag lost coverage, even if the remaining pages pass. A stale result should fail the freshness condition. A successful response from a homepage is weak evidence for a broken application feature.
Send the alerts to your phone with ntfy
The chosen channel needs a name. Ours is ntfy, an open-source service, dual licensed Apache 2.0 and GPLv2, that turns an HTTP PUT or POST into a push notification on a phone or desktop. Anything that can run curl can send one. The Android app comes from Google Play, F-Droid or GitHub, the iOS app from the App Store, and the web app works too once added to a home screen. On our server one script, alert.sh, sends everything to one topic: update runs, watchdog repairs, prompt hub job results and monitors. That comes to about 25 messages a day.
The simplest publish needs no account:
curl -d "Backup successful" ntfy.sh/your-long-random-topic
Topics are created on first use, which is why the publishing guide says "the topic is essentially a password". Headers add the useful parts: Title; Priority from 1 (min) to 5 (urgent); Tags, where emoji short codes such as warning turn into emoji; Click, a link the notification opens; up to three Actions buttons; and At or Delay for scheduled delivery, from 10 seconds to 3 days ahead on ntfy.sh.
Six small-business examples, one line each. NTFY_URL holds the server and topic, for example https://ntfy.sh/ followed by a long random name. Load it from a protected environment file rather than writing it into the script.
-
Website or booking page down, from a timer every few minutes:
curl -fsS --max-time 10 -o /dev/null https://example.com/book/ || curl -H "Title: Booking page down" -H "Priority: urgent" -H "Tags: rotating_light" -H "Click: https://example.com/book/" -d "No answer within 10 seconds." "$NTFY_URL" -
Nightly backup failed:
restic backup "$BACKUP_SOURCE" || curl -H "Title: Nightly backup failed" -H "Priority: high" -H "Tags: warning" -d "restic exited with code $?. Read the backup log before the next run." "$NTFY_URL" -
New booking or form submission, sent by the form handler after it saves the record:
curl -H "Title: New booking request" -H "Tags: calendar" -H "Click: https://example.com/admin/bookings/2417" -d "Booking 2417 is waiting for confirmation." "$NTFY_URL" -
Failed payment or low stock:
curl -H "Title: Payment failed" -H "Priority: high" -H "Tags: credit_card" -H "Click: https://example.com/admin/orders/1088" -d "Order 1088: the card payment failed." "$NTFY_URL" -
Long AI job finished:
./weekly-price-check.sh; curl -H "Title: Price check finished" -H "Tags: white_check_mark" -d "Exit code $?. The draft is ready for review." "$NTFY_URL" -
Scheduled reminder, computed in your own time zone and delivered by the server:
curl -H "At: $(date -d 'tomorrow 09:00' +%s)" -H "Title: Reminder" -H "Tags: alarm_clock" -d "Swap the backup drive and take one copy off site." "$NTFY_URL"
Notice what the messages leave out. They carry a booking or order number, a status and a link to a page behind your own sign-in. They never carry prompt text, a customer's name, a card detail or anything you would not print on a postcard. The public server caches messages for 12 hours by default and does not log their content by default; connections use TLS, but the privacy policy describes no end-to-end encryption and suggests self-hosting for sensitive messages.
Keep the topic private. Generate the name, for example with openssl rand -hex 16, keep it out of Git and screenshots, and replace it if it leaks. For a private server, run the binwiederhier/ntfy image with auth-file and auth-default-access: deny-all, create a write-only user for the server and a read-only user for the phone with ntfy user add and ntfy access, and give scripts a token from ntfy token add, sent as Authorization: Bearer tk_.... An iPhone needs upstream-base-url: https://ntfy.sh on a self-hosted server; without it, the configuration guide warns, "delivery can take hours".
Know the free limits. The public ntfy.sh server allows 250 messages a day per visitor, bursts of 60 requests that refill at one every 5 seconds, and attachments of 2 MB each and 20 MB per visitor, deleted after 3 hours. Our 25 a day is a tenth of the message allowance. Forwarding to email needs a verified address and is limited to 5 a day on the free plan; phone calls need a verified number and a paid plan. Paid tiers raise the limits.
Then test the route the way I described above: break something harmless on purpose, confirm that the phone buzzes once rather than every minute, fix it and confirm the recovery message. The ntfy alerts cookbook has more recipes, and the operator kit has an alert.sh you can adapt.
Make services survive the session
The runner, local model endpoint, memory refresh and monitoring schedule need a defined lifecycle. On a systemd machine, I would use enabled user services for user-owned work and timers for scheduled checks. Where they must continue after logout and start without an interactive login, I configure lingering for the service account.
That is a service design, not a promise that a graphical browser or credential store will be ready. I test those dependencies separately. I keep operational scripts and private logs outside publishing repositories, use a dedicated configuration source and give the runner only the access its jobs require.
I distinguish process supervision from an application watchdog. systemd can restart a failed service; its watchdog support depends on the service sending the required notifications. An arbitrary shell script does not gain a meaningful heartbeat simply because I set a watchdog option. Restart limits matter too: a missing credential will not improve through endless relaunches.
For containers, I preserve required state in declared storage and put repeatable changes into the image or supported startup configuration. I test recreation, not just a restart of the current container. An installation that disappears during replacement is still temporary.
Put bulk work through a tested local route
I would use ordinary code for extraction, counting, link checks and exact comparisons. A local model can then help classify documents, draft short summaries or group similar findings. I keep source references attached and test a representative batch before trusting the output.
A publisher could use this route to propose categories for an archive. An agency could group repeated findings into a draft fix list. A small business could summarize approved internal notes for review. These are candidate uses, not claims that the installed model has passed them.
I choose a model and context size that fit alongside the other services, bound the batch size and preserve failed items for retry. I review samples and exceptions. If a summary drops a deadline or turns uncertainty into a fact, shorter output has made the work worse.
Local inference avoids sending that request to an outside model provider, provided the application uses the intended local endpoint. It still uses the machine's resources and needs maintenance. I do not silently switch a failed local batch to a paid provider. That changes the data destination and spending decision.
Make memory searchable and current
I want to ask, “What did we decide, and what source supports it?” A searchable memory should return the relevant note with its date and provenance. SQLite FTS5 offers full-text search; semantic retrieval is another option, but requires its own relevance checks.
I would start with concise records: the decision, its reason, the last verified state, failed approaches and the next step. I keep secrets out. Source documents remain authoritative, and the index records which revision it contains and when it refreshed.
Historical notes are useful until an old status answers a current-state question. I distinguish superseded records from the latest verified state, remove deleted material from retrieval and test questions whose answer should be “not established.” A confident answer without a supporting record is a failure worth keeping in the test set.
For a publisher, this could preserve the reason for a redirect or an editorial convention. For an agency, it could preserve an approved audit scope. Access controls must keep one client's material out of another client's answer.
Back up the ability to recover
Mirrored storage does not protect me from deleting the wrong directory, losing the host or damaging a database. I want an encrypted off-site backup containing the project state, configuration, job records and memory sources needed to rebuild. I handle credentials through an appropriate protected recovery process rather than mixing them into ordinary notes.
I set retention and backup frequency according to the work I can afford to lose. I also schedule a restore into a separate location and verify representative files and application state. A backup command reporting success is not a restore test.
For a live database, I use its supported backup or snapshot method rather than assuming a copied file is consistent. I preserve the information needed to recover the backup itself, including access to its encryption credentials. Where supported, separately controlled or immutable storage can limit the damage from compromised workstation credentials.
Pin a working release, then update deliberately
I keep a known working set of runtime, package and container versions. Docker's documentation explains that tags can move and that digest pinning selects a specific image. A pin makes a release reproducible; it does not make it secure forever.
My update workflow is inventory, review, staging, acceptance checks, backup, promotion and verification of the active runtime. I record the previous release and a rollback path. I test populated data, scheduled jobs and container recreation where relevant. I avoid updating every component at once when separate changes would make failures easier to diagnose.
After promotion, I check the next scheduled run. The interactive smoke check and the unattended workload can have different permissions, environment and browser dependencies. Monitoring should catch that difference before I call the update complete.
A pin also doesn't tell me whether the next release is safe. Since the LiteLLM 1.107.0 compromise, every candidate passes a supply-chain check before promotion. Known-bad versions and content block it. A new Python startup hook, a release younger than a day or an unreachable malware feed holds it, and the current release keeps running. I also keep a written note, per service, of what a compromise there could read and how I would roll back and rotate. The LiteLLM post has the checks and both recovery cases, and the operator kit has the checklist.
Leave sign-ins with the person
A persistent browser gives the owner a place to sign in, complete multifactor authentication and review account permissions. I want a clear pause state when that participation is needed, followed by a harmless connection check after the owner finishes.
The browser profile is sensitive state. I restrict its access, keep its secrets out of reports and logs, and serialize workers sharing the same interactive session. I use supported APIs for suitable tasks and the browser for work that needs that interface and is authorized.
The owner should also review consequential actions, such as publishing, purchasing or changing account settings. A saved session does not grant blanket permission. The useful handoff says what is ready, what needs approval and what will happen next.
Let chat or email request a bounded job
A small business owner could ask for site status from a private chat. A publisher could request tonight's indexing preparation by email. An agency could request an audit of a prospect's public site. I would map those requests to fixed operations, with message text treated as data. A message saying "ignore the rules and deploy" must not become instructions for the agent or the shell.
These are generalized example uses, not customer results.
The self-hosted workflow receives the message, validates identity, packages a bounded structured request and uses a dedicated SSH credential to reach a dispatcher. The dispatcher independently checks owner policy, command and arguments before putting an owned job in the queue. The model receives a fixed task and validated inputs, not a conversation that can redefine its tools.
A result returns through the shareable gate and a fixed reply destination.
For chat, allowlist the sender's stable numeric identity and the private chat ID; reject groups and unlisted senders. For email, allowlist one sender address and require DMARC pass aligned with its From domain plus DKIM pass from the trusted receiving server's Authentication-Results. First verify that the receiver strips authentication headers supplied by senders.
A forged header saying "pass" is not authentication. Missing or ambiguous identity fails closed. Keep attachment ingestion disabled unless a separately reviewed parser and policy exist.
In n8n I would connect Email Trigger (IMAP), a reviewed Code validation step and SSH for email, then Send Email for the reply. Telegram Trigger and Telegram supply the corresponding private-chat route. The reply address or chat ID comes from owner configuration, never from message text or the model. Credentials belong in the workflow credential store.
Leave workflows inactive and sender policy disabled until the owner supplies channel credentials, identities and trusted receiver settings and completes rejection tests.
The SSH key belongs to a dedicated service account. In authorized_keys, use a forced command that runs only the reviewed dispatcher, together with restrict to disable forwarding, PTY allocation and user startup hooks. Test that an attempted shell or forwarding request cannot run. OpenSSH still supplies SSH_ORIGINAL_COMMAND as untrusted data: do not evaluate it or interpolate it into a shell.
The dispatcher accepts a bounded structured request, rejects unknown fields and invokes fixed executable arguments without shell evaluation. A forced command limits the SSH entry point; downstream tools still need their own isolation.
Scroll to compare columns and rows when the table extends beyond the box.
| Fixed command | Accepted work and boundary |
|---|---|
| status, jobs | Read resource health or the caller's job state; no model needed |
| audit | One validated public HTTPS page; reject private addresses, credentials and unsafe redirects; apply the full retrieval policy |
| gsc-prep | Prepare a dated candidate list, timing and attempt ledger without an indexing click |
| gsc-check | Check the owner's signed-in desktop browser and intended property without spending indexing quota |
| gsc | Execute only the prepared, scoped indexing plan after readiness and confirmation |
| stop | Stop only an owned job after confirmation; retain its checkpoint |
| confirm | Consume a single-use code for the exact pending action |
This is a command contract to implement, not permission to infer a new command from prose. Validate URL length and scheme, public DNS at each connection, redirects, job ownership and every argument. Reject shell syntax and unexpected arguments. Do not expose deploy, push, arbitrary sending or general shell execution as channel commands.
Request IDs and persisted locking prevent redelivery from duplicating work.
Run agents in a sandbox or with a fixed tool set and a restricted account. Never use skip-permissions or another global bypass for a channel job. Give public-page collection its own constrained network access; analysis can work on supplied captures without network access. Restrict accessible files and outputs. Refusal produces a recorded stop state instead of a less restricted retry.
Set persisted request rate limits, cooldowns, bounded concurrency and timeouts. A kill switch must block new execution independently of the workflow interface. Keep an append-only audit log of requester identity, request ID, selected command, validation outcome, confirmation state, job transitions and gated delivery result.
Restrict write access, retain failed attempts and omit secrets, cookies and sensitive message bodies. Rotation preserves audit history rather than truncating it.
Before anything outward-facing, require a one-time confirmation code tied to the same sender and channel, the exact action, payload and destination, with an expiry. Publishing, sending and indexing clicks cannot rely on earlier broad approval. Return a preview through the owner's private review interface, consume the code atomically and reject replay.
The fixed status or report reply is outward-facing too: bind confirmation to the exact gated reply, or keep it unsent for owner retrieval. No automatic reply escapes this rule. Scan the final reply and every attachment for private paths, private links and secrets before release. For PDFs include extracted text and link annotations; for HTML include links and image sources.
The dispatcher integration reviewed on October 12 passed manual channel tests through its restricted SSH key and passed again after the workflow service restarted. Its workflows remained inactive without channel credentials, and host sender policy remained disabled. That establishes a tested dormant integration, not a live messaging service.
Its current implementation asks for codes for stop and indexing clicks, while ordinary replies rely on sender policy and output gates. The stricter outward-reply confirmation above remains a requirement before activating that variant; I do not claim that every reply already has it.
Give every assistant one front door
The chat and email doorway runs fixed commands. For open-ended requests we built a second doorway: a small prompt hub that sends each request to the right assistant and shows its plan before anything runs. It is a FastAPI service listening only on the server's loopback address. We reach it four ways: a web page on a private hostname behind Cloudflare Access, with the same sign-in as the remote desktop; an llmhub terminal command; an icon on the browser desktop; and two shortcuts on the Windows 11 VM's desktop, one for the terminal and one for the web page.
Routing follows what we already pay for. Claude Code goes first, run as claude -p --output-format stream-json. Its stream includes a rate_limit_event with a status, the limit window (for example five_hour) and a reset time. When Claude reports a usage limit, the hub cools Claude down until that reset, shows the window on its status panel and sends the request to ChatGPT through the Codex CLI: codex exec, with the prompt on standard input. Local models, served by Ollama through LiteLLM, get only triage, short drafts, or the work when both subscriptions are unavailable: a 1.5B Qwen model sorts requests and Hermes 3 8B writes drafts. Claude Pro or Max and the ChatGPT plans work only through their own CLIs, and LiteLLM handles API keys, not subscription sign-ins, so the hub drives the CLIs.
Memory goes in before the prompt goes out. A search over the Qdrant memory attaches the most relevant notes, by title and file path, before Claude or Codex sees the request.
A link builds a plan; only Run executes. A URL such as https://hub.example.com/?q=...&target=host&mode=ask fills in the plan: engine, target machine, mode and the memory it will attach. Nothing executes until you press Run on the page or Enter in the terminal. A GET request never runs anything, so a link preview, a browser prefetch or a stale bookmark cannot start a job. Ask mode gives read-only tools. Work mode uses Claude Code's auto permission mode inside a folder made for that job. We never pass --dangerously-skip-permissions, and deploys, pushes and email still need their own confirmation.
The hub checks identity itself. cloudflared connects to the hub from loopback, so "it came from localhost" proves nothing. The hub verifies the Cloudflare Access token in the Cf-Access-Jwt-Assertion header: its signature, issuer and audience, plus an email allowlist. Run requires a one-time token that comes back only with the plan. POST bodies must be JSON, and the hub sends no CORS headers, so another website cannot drive it from your browser.
Each target has its own way in. A job can run on the host, inside the browser desktop as the desktop user, or on the Windows VM as a dedicated non-admin local account, reached over SSH with its own key; signing that account in to Claude and Codex is a one-time step for the owner. The browser desktop's key is pinned in authorized_keys as restrict,pty,command="/usr/local/bin/llmhub", so it can open the hub and nothing else.
A job can move to your phone. One button opens a job as a Claude Code Remote Control session in its own folder, inside tmux, so you can continue it from claude.ai/code or the Claude app. Another checks Desktop Commander on the server and on the VM and starts it if it is down. The hub itself is covered by the update tool's health check and self-healing watchdog.
The first real job was a read-only question answered by Claude Code on the server. The CLI reported an equivalent API cost of $0.11, an estimate rather than a bill. Local triage is slow on this CPU-only machine: a short Hermes 3 answer took about 12 seconds, which is why local models never do the main work.
Three lessons from the first days:
- Keep a reference to every background task. An asyncio task started without one can be garbage-collected; the hub's worker silently vanished until we kept references. Python's documentation warns that "The event loop only keeps weak references to tasks."
- Do not let a tiny model name things. The 1.5B model wrote silly job titles. Titles now come from the prompt's first sentence.
- Set the memory threshold high enough. A threshold that was too low attached irrelevant notes. Raise it until the attached notes are ones you would have picked yourself.
For a small business, the hub turns repeat requests into bookmarks. "Check the booking page and the contact form and tell me if anything looks broken" runs in ask mode. "Change the Saturday hours on every page of the website copy and show me the changes" runs in work mode, and publishing still waits for a confirmation. Each one stops at a plan you approve. Build your own AI prompt hub has the full build, and the prompt hub kit has a generalized version with routing, safety rules and code.
Keep indexing preparation separate from signed-in action
Some useful work still needs the owner's browser. A search engine's manual indexing request is an example: preparation can verify public URLs and assemble a plan, but it cannot replace property access in the signed-in interface. The owner signs the desktop browser in and completes account or multifactor challenges. The agent never handles the credential.
Use gsc-prep to prepare the candidates and ledger, then gsc-check to verify the intended desktop browser, property access and readiness without making a request. A connected extension or installed browser does not prove readiness. If only another computer is connected, stop rather than silently selecting it. Readiness expires and must be rechecked before action.
Only then may gsc arm the exact plan and request same-sender one-time confirmation. Recheck readiness, quota timing and the accepted/refused ledger before the first click and between slots. Record each attempt's outcome; an inspection result is not an accepted indexing request. Stop at a refusal, missing evidence or unexpected page. Do not replay an armed job automatically after reboot.
Preparation and dry checks spend no quota. The reviewed integration's Linux property access and first-click-stop proof still awaited the owner's connection and sign-in, so I make no claim that this browser execution was verified.
Promote a desktop image only after matching-profile proof
A clean test profile can hide a failure in the live desktop. Test the candidate image using an isolated copy of the current live profile, preserving ownership, numeric UIDs and GIDs, modes and required volume layout. Confirm the effective runtime user after startup as well as the image's default identity.
Never share the live profile volume with the test container or read its cookies and credentials for diagnostics.
The reviewed desktop update first failed because a generated health check expanded a shell variable at the wrong layer. A corrected attempt then exposed a cache directory owned by a different user: a CLI could run with the test profile but could not create its extraction directory in the live profile.
The later successful proof used a copied live profile and a narrow ownership repair with modes preserved. Those observations explain the acceptance rule; they do not mean every profile needs that repair. Avoid recursive ownership changes and permissive modes as a generic fix.
Validate generated health-check quoting before deployment. Check both Compose rendering and the command that will actually run inside the container; parsing YAML alone does not prove that dollar signs and quotes survive correctly. Each probe attempt needs its own receipt so a retry cannot reuse or overwrite an earlier result.
Promote only the identical immutable image digest that passed copied-profile checks after start and passed again after a second start caused by restart. Bound proof runs and promotion attempts in advance; a failed gate stops the sequence. Restore the test copy's original profile when proof ends.
For the live transition, announce the interruption and require owner authorization if the desktop is active. Close the browser gracefully, stop the container and take a fresh consistent profile-volume snapshot before replacement. Validate the archive and checksum, and retain the previous image, configuration and profile together.
Promote the proved digest, run health plus real application checks after start, then repeat them after a second restart. Include effective user, mounted profile, CLI operations and retained state. Separately check browser connection readiness; directories being present are insufficient.
Automatically restore the previous image, configuration and profile if either live verification fails. Verify the restored desktop and stop if rollback fails. A profile snapshot contains sensitive state and stays private. Update the persistent container definition and release record through managed promotion, keep rollback pins out of cleanup, and preserve the enabled user maintenance service.
A forced update changes who authorizes interruption; it does not remove proof, snapshot or rollback gates.
Build out the workflow around the queue
The chat and email dispatcher described above gives the queue a constrained workflow entry point. Manual integration checks establish the dormant route; owner credentials, channel activation and browser execution retain their separate gates. A nightly digest is another candidate workflow, with the same exact-payload confirmation and gated delivery requirements.
For searchable memory, I would pair Qdrant with a local embedding model. The ingestion job would split approved notes into passages, attach source revision, date and access scope, and store vectors alongside those references. A question would use the same embedding model and retrieve candidate passages. I would open the source before relying on the answer.
Changing the embedding model would require rebuilding the collection; deleting a source would also remove its indexed passages.
Ollama would handle local bulk triage after ordinary code extracts the facts. I would give each item an identifier, constrain the requested output, validate its structure and route uncertain or failed items to review. The original evidence stays attached. I would test relevance and missed facts before increasing the batch, with no silent fallback to an outside provider.
Add checks that can run beside the work
I would add a browser pool running without a visible window through Playwright for isolated public-page checks, with Lighthouse runs and screenshot diffs against a dated baseline. Each browser worker would own its output. A changed screenshot would be a review cue, since fonts, motion and changing content can move pixels without a defect.
Lighthouse would describe the measured browser run; it would not certify content hidden behind an access block.
Repository gates could run in parallel after their shared preparation and build finish. In one workstation comparison, the same repository revisions took 169.255 seconds serial and 106.310 seconds parallel. Those end-to-end runs included fresh clones, dependency setup and builds, so warm caches can affect the comparison.
Some application checks depended on external probes disabled for that local benchmark and remained unmeasured. I would use this as a reason to test concurrency, not a promise that every gate passes or every workload gets the same gain.
Git worktrees would give parallel edits separate branches and directories while sharing repository history. I would fetch the intended base first, assign one owner per worktree, and serialize merges, pushes and production deploys. The shared fetch cache would key each public URL and its freshness conditions, coordinate the origin request budget and retain retrieval provenance.
A challenge page would never enter the content cache. Losing coverage would count as a monitor change even if the remaining findings stayed green.
I would keep scheduled monitoring quiet when nothing changes, but alert on a changed finding, failed check, stale result or lost coverage. The digest would group identical failures and show the last good observation. A separate watchdog would watch the runner's heartbeat and actual progress. These are different checks and deserve separate failure tests.
Recover storage and remote access deliberately
I would version non-secret notes and instructions in git, then use encrypted restic snapshots for files and supported database exports. The backup layers would include local recovery and a separately controlled off-site destination. One completed workstation snapshot covered 210,018 files totaling 51.519 GiB before compression, with caches, dependencies, generated sites and model blobs excluded.
That measurement describes the selected local snapshot. It does not establish an off-site copy. I would restore representative data before calling either layer ready.
Retention-based cleanup would expire disposable browser captures, fetch caches and finished job scratch space after their review window. I would preview the deletion list and protect active jobs, source notes, credentials and backup repositories. Restic retention would use its own snapshot policy rather than deleting repository files.
Reaching a disk threshold would alert before cleanup touches anything outside those approved boundaries.
For persistent containers, I would record the working image digest and configuration, verify a backup, then update one service at a time. A failed health check would restore the previous digest and rerun the check. Database migrations need a separate data rollback plan; restarting an old image cannot undo a schema change.
Automatic rollback belongs only on changes whose state compatibility has been established. I would also check the next unattended run before accepting the update.
For remote use, Cloudflare Tunnel could connect the workstation to a protected entry point without an inbound port on the host. I would put an Access policy in front of the application, limit identities and sessions, and test an unauthorized request. The tunnel connection alone does not supply an access policy.
I would keep the local service private and leave interactive sign-ins and consequential browser actions with the owner.
Choose one repeatable job first
For a small business, I would start with a weekly website check and a reviewed fix list. For a publisher, I would start with publication checks across independently owned site jobs. For an agency, I would start with a repeatable audit scope, complete coverage records and a report someone can act on.
I would measure completion, failures, review time, recovery and actual operating costs before expanding. More running agents are useful only if their results are worth keeping.
The service packaging question connects to The $20 Dollar Agency: define work another person can understand and use. Here, that starts with a brief, a bounded runner, a shareable result and proof that the next run still works.
Capacity and scheduling lessons you can apply
Find the bottleneck before adding threads. Agent jobs calling a hosted model often spend most of their time waiting for that model. In that workload, the provider's usage and rate limits, plus real dependencies between jobs, can govern throughput even while local CPUs are idle.
Check the machine before and after a representative batch, and compare elapsed time, failures and queue length alongside these readings:
uptime
free -h
ps -eo pcpu,rss,comm --sort=-pcpu
Load, available memory and per-process CPU help distinguish local contention from waiting elsewhere. The process listing is a snapshot, not a complete performance trace. Increase independent work only while memory, response quality and provider limits permit it; adding threads cannot remove a dependency or raise a remote quota.
Local model defaults can serialize a busy queue too. The official Ollama FAQ, checked October 12, 2026 at 03:15 UTC, says OLLAMA_NUM_PARALLEL limits simultaneous requests per model and defaults to 1. OLLAMA_MAX_LOADED_MODELS limits concurrently loaded models that fit in memory; its documented default is 3 times the number of GPUs, or 3 for CPU inference.
The FAQ also notes a Windows Radeon exception of 1 loaded model. These are documented defaults, not a measurement of an installed server. Parallel requests increase context memory requirements, and model weights must fit too.
Change those variables in the persistent container definition, alongside a pinned image digest, rather than in a shell that disappears on restart. Choose values for the RAM and GPU memory actually available after other services have their share.
Save the previous definition and digest, recreate the container, then check readiness, an embedding call with an embedding model, a short generation with a generation model, and concurrent requests. Confirm valid responses, memory use and whether requests actually overlap. Repeat after recreation to verify persistence.
If health or memory checks fail, restore the prior definition and digest and rerun the checks. No local inference timing is claimed here.
Apply request politeness per website, across all workers. For sites you do not own, aim for about one request a second, respect published restrictions and slow down on errors. This is an operating recommendation, not a measured safe rate for every site.
Your own CDN-hosted sites can use a higher measured rate when their capacity allows it, with bounded concurrency, backoff on 429 or 503, and respect for Retry-After. Ownership does not remove limits on an upstream service or a shared proxy.
Make nightly checks incremental without losing visibility. Always read the full sitemap, then check new pages, changed pages, previously failing pages and a rotating slice of the rest. Use sitemap modification dates as a signal, not proof that everything else is unchanged. Run a full sweep weekly.
Every report should state pages planned, pages measured, the access tier used, unmeasured pages with reasons, and what was deferred by the incremental policy. A green subset must not imply that the whole site passed.
Two scheduling traps deserve an explicit gate. A maintenance task waiting for no agent jobs, while a queued agent waits for that maintenance task, creates a circular wait. Put the waiting in a plain shell loop outside the agent job accounting, and start the agent only after the gate opens. Check the condition again when acquiring the gate so another launcher cannot race it.
Use a timeout and a visible failure state instead of an endless silent wait.
A job's exit also does not prove its work finished when it hands work to a background service. Gate the next step on that service's own terminal state and result, tied to the submitted work identifier. A timer being enabled, a process being alive or a submission being accepted is not completion. Failed or unknown states should stop dependent work and preserve evidence for review.
Order destructive cleanup after the update has finished and created verified rollback points. Wait for the service's own successful completion and acceptance checks, then prune only objects outside the retention policy. Keep every image, configuration and backup referenced by current rollback pins. Resolve those references before deletion; if they cannot be resolved, stop cleanup.
This keeps an update from losing its recovery path while it is still running.
Close the remaining maintenance and evidence gaps
A shareability gate must report zero hits before release. Apply it to every shared file and every member of an archive, not just the top-level report. For PDFs inspect both extracted text and link annotations; for HTML inspect URL attributes as well as visible text. Keep raw evidence outside the bundle.
Distinguish a monitor failure from a site failure before reporting an outage. Check the monitor's own time budget, expected paths and execution environment first. Save progress by stable item identity and source revision, so adding targets to the inventory preserves completed work instead of failing a strict inventory-equality check. Report lost coverage as a change.
Use hybrid retrieval for memory: vector similarity helps find related decisions, while keyword search finds exact phrases, rules and identifiers. Validate both against the authoritative source.
Test updates in a TEST copy before live promotion, skip the live desktop while someone uses it, and bound retries before large candidate images fill storage. Never run OS package upgrades from a service that can be killed during installation. After any interrupted upgrade, verify package-database consistency, repair incomplete configuration and verify the intended boot image before any reboot.
Where no suitable API exists, prepare the human browser handoff completely. For a manual indexing request, check candidate URLs, order, timing and the current published quota rules; use the signed-in browser for the authorized click and record every attempt. A successful URL inspection is not proof of a successful indexing request.
Keep secret values out of files, briefs, logs and repositories. Supply them through the process environment from protected credential storage, limit inherited access, and scan before every commit.
Prune old images through the updater's own records
Container images and profile backups pile up quietly. Delete them only through the records your updater keeps, never by guessing from names or dates:
- Run the prune as a dry run and read the list.
- Verify each file's checksum against the record the updater registered for it. A mismatch means the record and the disk disagree; stop and look.
- Hold the updater's lock while deleting, so no update starts and selects something you are removing.
- Update the registry so it matches what you removed.
- Keep four sets every time: the current release, the previous one, anything marked protected, and anything a rollback would restore from.
If a protected marker looks stale, keep what it protects and flag it for a person. Overriding it in a cleanup pass removes the one copy someone deliberately kept.
Measured on October 13, 2026: removing seven unused test images and superseded profile backups freed 82 GB, taking disk use from 472 GB to 390 GB on an 877 GB disk. Removing images made their build cache reclaimable too, about 69 GB in this case.
We left that cache in place; docker system df reports reclaimable space by type, and build cache is cleared by a separate command, so decide on it separately.
Verify a promotion by the updater's records, not an exit code
A promotion job that exited cleanly has only told you that it stopped. Check the updater's own current and previous records, confirm the running container uses the promoted image and reports healthy, check that its restart count is not climbing, and confirm the rollback set still exists on disk.
Do this before cleanup, so the prune in the previous section never removes the image a failed promotion would need. The operator kit has the commands for both checks.
Review host security as an operating task
Check security-update timers and logs, actual SSH listeners, active Fail2ban jails, host and provider firewall rules, container publications and the last successful restore. Review again after a service or access change.
The ops kit adds routine commands; the host-hardening guide explains safe changes and their limits.
Turn measured headroom into a bounded queue
Independent validation can use the spare CPU too. Eighteen repository checks ran after their serial preparation and build steps. Two batches per condition averaged 5.62 seconds with one worker and 2.04 seconds with four, about 2.76 times faster. All 72 check executions passed. The order was one, four, four, one workers to reduce a simple warm-cache explanation.
These were local checks with no external requests; they do not certify live website behavior or make a dependent build step parallel. Browser throughput was not measured because the installed browser could not start with its sandbox enabled. I retained the security policy and reported that limit rather than claiming a browser speedup.
The October 13 capacity review found an exhausted file-notification instance budget even with abundant available RAM. Raising that specific limit let a repeat probe open 256 instances where it previously opened only 14. Existing model parallelism also worked: four short Qwen requests reached 103.98 aggregate output tokens/s versus 39.15 serially, with longer batch latency.
The capacity and rollback procedure and dated model test give the conditions and limits.
Start with eight hosted-model jobs, four independent browsers and two builds on a 48-logical-CPU, 94-GiB-class machine, sharing admission limits of load below 24 and more than 20 GiB available. These are initial budgets, not a claim that every mix is safe. The local model queue has its own latency and memory test. Serialize shared writes and admissions, and keep a checkpoint per completed unit.
Backup API quotas can be the remote bottleneck too. A shared Drive OAuth client produced a requests-per-minute quota error; a one-transaction cap slowed the copy without fixing the client quota.
The Drive client correction explains why a private client comes before experimenting with eight transfers and 128 MiB chunks. The operator kit supplies the acceptance steps. No network throughput improvement is claimed.
Fact-check notes and sources
Prompt hub and phone alert update, October 15, 2026: the hub's routing, run rules, targets, the rate_limit_event fields, the $0.11 estimate, the 12-second local answer and the 25 alerts a day are our own observations of our build, not vendor documentation. The small-business requests and alert examples are illustrations, not customer results.
Sources read October 15, 2026 (UTC): ntfy publishing for the topic warning, headers, scheduled delivery range and the ntfy.sh limits; ntfy configuration for auth-file, auth-default-access, the user, access and token commands and upstream-base-url; ntfy privacy for the 12-hour cache, content logging and self-hosting advice; ntfy phone apps; and the ntfy source for the licence. Claude Code programmatic use documents stream-json output and calls its cost figures "client-side estimates". Permission modes lists bypass mode for "Isolated containers and VMs only". Remote Control says code execution and filesystem access stay on your machine. Codex non-interactive mode documents codex exec - reading the prompt from standard input. Cloudflare Access token validation recommends validating the Cf-Access-Jwt-Assertion header against the team's public keys, audience tag and issuer.
Capacity and backup guidance updated October 13, 2026. The linked dated capacity procedure cites Docker, Linux, rclone and Ubuntu sources read at 21:42 to 21:44 UTC, with both retrieval routes checked. The model article reports the two-repeat short-context experiment; job budgets are recommendations. No new backup throughput or application accuracy measurement is implied.
Dedicated-server security update, October 13, 2026: the new access and maintenance guidance links to the dated Ubuntu hardening sources, checked October 13, 2026, and is a recommendation, not a claim about an existing server.
The added operating checks concern security updates, access controls, service exposure and restore evidence.
Cleanup update, October 13, 2026: the storage figures were read before and after the cleanup on October 13, on a single 877 GB disk, and the build cache figure is what the container tool reported as reclaimable afterwards. The five job CPU observation is from the same window.
Sources checked October 13, 2026 from 05:57:28 to 05:57:52 UTC, each read directly and through the alternate route with readable content, none unmeasured.
docker system df shows a RECLAIMABLE column per type, and docker buildx prune "Clears the build cache of the selected builder." The cleanup and promotion checks are operating practices, not vendor requirements.
Workflow sources checked October 12, 2026 from 06:40:54 to 06:41:03 UTC: Email Trigger (IMAP), Telegram Trigger, Code, SSH, Send Email, Telegram and OpenSSH authorized_keys options.
Seven pages planned, seven measured with readable direct content and alternate-route checks, zero unmeasured. OpenSSH says the supplied command "is ignored" when the key has a forced command. The n8n nodes supply transport and workflow steps; sender authentication, argument validation, approval, sandboxing and output gates are application requirements, not guarantees supplied by those nodes.
Official documentation rechecked October 12, 2026 from 05:08:04 to 05:08:09 UTC: Ollama concurrency, Docker on Ubuntu, Qdrant hybrid queries and systemd service completion.
The Ollama FAQ states “default 1” for request parallelism. These sources describe interfaces and limits; the operational acceptance rules remain recommendations.
Operational guidance updated October 12, 2026. The new practices are recommendations, not new measurements or claims that every integration is deployed. Existing dated tests retain their original conditions. The workstation operator guide supplies the detailed retrieval, scheduling, update and evidence policy.
These workflows are proposed additions, not a claim that every component is installed or has passed acceptance on my workstation. The dormant dispatcher has manual acceptance evidence as stated above; live channel activation and signed-in indexing execution are not represented as finished.
The two local measurements above describe one gate comparison and one selected backup snapshot, with their conditions and limits stated. They establish no customer result, model accuracy or cost saving.
I reviewed official documentation on October 12, 2026, from 03:03 to 03:07 UTC. Coverage was 22 URLs planned, 21 valid source pages measured from readable direct content, and one nonexistent candidate returning 404 through direct and alternate routes, excluded from the sources below. Every planned URL also received the full alternate-route check.
Three systemd pages produced a bot-check interstitial through that service; passive browser checks also read their documentation. No valid source page was left unmeasured. That access limitation concerns the tested alternate route, not every reader. The capabilities documented below support the proposed designs; each installed workflow still needs its own acceptance checks.
- Services and recovery limits: systemd service documentation describes restart behavior and notification-based watchdogs: “The service must call” its notification function regularly. Timer documentation describes scheduling and persistent calendar timers. loginctl documentation describes lingering: “kept around after logouts.”
- Capacity update source coverage: The Ollama FAQ was rechecked October 12, 2026 at 03:15 UTC: one URL planned, one measured directly and through the alternate-route service, zero unmeasured. Both returned readable content. The scheduling, cleanup and incremental-check advice above is an operating recommendation, not a vendor benchmark.
- Local inference: Ollama FAQ describes concurrent requests, context memory and queuing, including “parallel request processing.” It supports capacity planning, not a benchmark for this machine.
- Searchable memory: SQLite FTS5 documentation describes full-text indexing and queries. Retrieval correctness still needs application tests.
- Recovery: restic restore documentation documents restoring snapshots and checking existing file content before overwriting it. It does not prove that my own backup or a live database is recoverable.
- Pinned containers: Docker image pull documentation describes selecting a specific image by immutable digest. Security updates still need a deliberate new pin.
- Workflow automation: n8n self-hosting covers operating the workflow service. Credential encryption documents protecting stored credentials with an encryption key. Human approval describes pausing tool execution for approval. The queue bridge and nightly digest above are proposed application workflows.
- Semantic retrieval: Qdrant FastEmbed documentation describes local embeddings and Qdrant integration. Source revisions, deletion handling, access scopes and reindexing are acceptance requirements I would add.
- Browser checks: Playwright screenshots documents capturing browser pages. Lighthouse overview describes audits and regression checks. Neither establishes the quality of an untested browser pool.
- Backup snapshots: restic backup documentation explains creating snapshots. Off-site copies and recovery testing remain separate operator responsibilities.
- Remote entry: Cloudflare Tunnel documentation describes an outbound connection. Access policies describes identity rules that control access. I would configure and test both.
- Retrieval policy: the direct, alternate-route and passive-render sequence is my operating policy. It is not a vendor guarantee of access, permission or universal coverage. Each audit needs its own dated page evidence.
Related reading
- LiteLLM 1.107.0 Ran When Python Started: IOC checks before updates and recovery if a bad release gets through.
- Sovereign AI Workstation Best Practices
- The Sovereign Advantage
- The Sovereign AI Playbook
- Mobile AI Workspace Lessons
- Build Your Own AI Prompt Hub, with the prompt hub kit and the ntfy alerts cookbook
- Local models, Codex drafts, Claude review: measured job overlap, the log calculator and the brief and review templates.
Download the Markdown operator kit. Start with one job, adapt the placeholders, and test its failure and recovery paths before expanding.