Architecture brittleness audit — the work–queue system

Private research document · 2026-10-18 · not for publication — names machine-specific ports, paths, and processes. Companion records: ~/manifest.md §3 (cockpit / punchclock / pipeline lines), flags 13–16 (~/flags.txt), checklist 48 (~/todo.checklist), mem #141/#142 (topic infra).

Question asked: where is this system brittle — where can it fail in a way that is silent, self-inflicted, or unrecoverable without a human?

Method. Every claim below is from one of three places: (a) the actual source, read this session; (b) the machine’s own files (ledgers, ~/.run/*, process table); (c) the 12 hours of incident the searxng chain just lived — a queue clobber, a 36-second gate flap, one deck kill/rebuild, a 59-minute agent job dispatched mid-session. Nothing here is recalled.

The system, as it runs

One deno serve process on 127.0.0.1:8443 is, at once: the HUD (the owner’s dashboard), the tailnet front’s target (443 → / and /studio), the queue’s dispatch pump, the deep-work gate’s consumer, and the pipelines card. It drains one FIFO, one job at a time, from ~/.run/queue.json, which three writers touch: the card (both books: ~/pipelines/<slug>/meta.json + the schedule), the pipeline CLI (both books), and the pump itself (persist() on state change). The focus gate (a 30-second watch loop under launchd) flips a boolean that throttles the pump and ComfyUI.

Findings, ranked

1 demonstrated The gate detects silence, not work

Mechanism. pi-activity (~/projects/punchclock/adapters/pi/pi-activity.py) builds lanes from pi-ledger events and transcript mtimes; idle = 300 s of quiet, active = an event within 120 s, poll every 30 s. Thinking writes no transcript bytes. A sustained ≈5-minute thinking stretch is indistinguishable from the agent being gone.

Evidence (this session). The focus ledger shows flip-off 16:50:33Z → the pump dispatched the searxng plan at 16:50:19Z-window, and start-comfyui ran — then flip-on 16:50:49Z, 36 s later, as my session resumed. A 59-minute agent job ran, and ComfyUI loaded its weights (the unified-memory hog the 10-14 ruling exists to keep away), while the working session was mid-thought. The 10-14 purpose — the agent gets the machine while the agent is present — inverts exactly in its most load-bearing window: a long think.

Severity. High. The throttle’s failure direction is the dangerous one: a false off runs heavy work under the operator; a false on merely defers work, which the queue absorbs.

Fix. The ledger already defines a beat work-kind and the harness plumbs it (work_kinds=("start","beat")) — the missing piece is emission: the pi extension writes a beat when a session begins thinking, so an in-flight think reads as activity. Fallback if emission proves coarse: never flip off while a session started < 30 min ago has not recorded a stop.

2 demonstrated The pump has no liveness, and one wedged row wedges all

Mechanism. makePump().tick() (~/projects/cockpit/lib/studio/queue.ts ≈L330):

// the dispatch slot — one at a time
if (store.runningCount() >= 1) return;
...
await deps.runPipeline(job);   // `pipeline run   --wait`, NO timeout

Adjacent. A corrupt queue.json makes load() return an empty store (graceful boot — good) but silently loses every queued image job (the card’s self-heal recovers only pipeline rows; image jobs leave no ledger trail).

Severity. High. A work system that cannot revive itself and in which one wedged row freezes the whole queue fails invisibly. Both modes require a human staring at the HUD to notice.

Fix. (a) A non-reviving watchdog: a periodic launchd agent that curls /api/who and restarts the deck only when dark and no manual-off marker exists (cockpit stop writes the marker; the watcher respects it) — supervision that preserves the clean off/on standard, which raw KeepAlive could not. (b) A dispatch deadline: the pump tracks started; a pipeline row silent for N minutes with a dead/absent --wait pid is marked failed, the slot frees, the queue lives. (c) On corrupt file: rename to queue.json.corrupt-<ts>, boot empty, and surface the loss in the HUD — never silent.

3 demonstrated A write-back path clobbered a book nobody can see

Mechanism. QueueStore.persist() saves its in-memory view over the whole file; the only merge, syncFromFile(), runs at the top of tick() — which early-returns while busy > 0. A card- or CLI-written row is invisible to the pump for as long as any job runs. (This is the generalization of manifest rule 1(e), added today: a write-back path is a writer.)

Evidence. Your 16:29:56Z searxng click wrote the row correctly (the two-books fix was deployed). The co-running system-heat plan finished at 14:15Z–window → remove() → persist() → the file was rewritten from the pump’s blind view → the row was gone. LEDGER said queued; SCHEDULE said []; the HUD said “queued” (it renders the ledger). The books diverged because the pump’s writer doesn’t read before it writes.

Severity. High — silent work loss with a cheerful UI. Status: queued as checklist 48 + flag 14; the fix is one function (syncFromFile() before the write) + a regression test. Awaiting your green light for a rebuild window.

4 demonstrated The record and the machine disagree, in both directions

Three real divergences found while verifying the above — all of them of the class rule 1(c) warns about (a record written from intent is a lie waiting to happen), today biting in both directions:

Severity. Medium–high, compounding. Every other finding is easier to hide when the deploy surface itself is unprovable. Fix. (a) Correct manifest lines to the machine (do it the moment a finding lands). (b) A cockpit start writes _fresh/BUILD-INFO (HEAD, branch, build time) and a /api/deploy endpoint serves it; a mismatch with HEAD prints at start. Turns “is the fix live?” from an investigation into a one-liner.

5 latent Concurrency is enforced by sentences

“Never run both decks (two pumps on one queue.json double-dispatch)” is a manifest sentence. Nothing mechanical stops a second deno serve: bin/cockpit manages 8090 (its pidfile is .run/cockpit.pid), while the live 8443 deck was started by ad-hoc bash with a non-canonical, stale pidfile (8443-serve 25776 — flag 13; 25776 is dead, the real pid is 62577). cockpit stop silently no-ops on the deck that’s actually running. Image double-dispatch (two pumps → two engines → one ComfyUI) is unguarded; pipeline double-dispatch rests on the racy check of finding 4. Fix. Make 8443 the bin/cockpit instance (the script already honors COCKPIT_PORT — one wrapper: start --prod) + a single-instance lock (the repo’s lib/locks.ts) taken at boot, checked on every dispatch.

6 latent Three clock formats cross every boundary

File mtimes are local CDT; the ledgers write epoch-ms; the pipeline ledger writes ISO-Z; the schedule writes epoch-seconds. I misread the incident timeline twice this session (13:59Z appeared “before” 08:26 — it was 08:59 CDT, after) and one misread rode into a compaction boundary. Any reader that pairs an mtime with a Z-timestamp makes a 5-hour coin flip. Fix. A rule, not code: epochs are canonical; mtimes are display-only — written into the manifest so every future reader inherits the discipline.

7 latent One FIFO slot mixes 8-second renders with 59-minute plans

The 10-15 design deliberately chose one slot, FIFO, one at a time. It was the right call for memory (safety: the focus gate’s whole premise). The cost is throughput: a long plan blocks a quick render, and vice versa; the searxng plan’s 59 minutes was a full day’s worth of render capacity. Fix (design call, not urgent). Kind-aware lanes — heavy agent works stay gated; light renders may take the slot in between — or, cheaper: at least make the slot’s policy visible in the HUD (why is my render waiting 40 minutes?) so the ceiling reads as a choice.

8 accepted The whole tailnet face is one toggle

cockpit on/off takes / and /studio together; the serve map is tailscaled’s in-memory state (10-15 flag — a tailscaled restart darkens the maps until the front is re-toggled). A known, ruled tradeoff of one-front simplicity. Listed for completeness; the watchdog of finding 2 should health-check the face, not just the app.

What is sound

Recommended sequence

#2–#5 ride one rebuild so the owner pays one dark window; #1 and #6 need none. Findings 6–7 are rule-level and cost a manifest line, not a rebuild.

Decisions needed

(a) The rebuild window for #2–#5 — your call (the earlier ask was cancelled). (b) Whether the watchdog’s manual-off marker pattern satisfies your clean off/on standard (it is designed to: the marker is the standard, made machine-readable). (c) The searxng build itself stays terminal-only (“pipeline run searxng-self-host build”) — the plan is in ~/implementation-plans/searxng-self-host-impl-plan.html, chain is planned, ready when you are.