An idea pipeline: research → plan → build across agent sessions

Research, 2026-10-17. Question from the owner: a system where a loose inspiration gets researched into a public domain-expertise document, then turned (with a human-supplied goal) into a private implementation plan, then built — each step an independent job, with unbounded waits between steps. This report answers the owner's ten questions and proposes a concrete design. It is a research document, not an implementation plan: it contains no folder hierarchies, hooks, or build steps that are system-unique beyond pointing at the existing tools it would ride. Companion documents: ~/manifest.md, ~/research/fresh-deno-cockpit-research.html, ~/implementation-plans/checklist-impl-plan.html (the house patterns this borrows), ~/projects/orbit/README.md, the relay skill.

1. The shape of the system

Each idea moves through five named states, and each state is crossed by exactly one kind of job — an independent, bounded agent session:

Three properties make this work across sessions that may be days apart:

1. Every artifact outlives the session. The session is disposable; the folder it wrote is the permanent record. A session that died mid-research leaves the partial doc in place — the next re-run of that job reads it and continues, like the orchestration skill for multi-context work (relay, the relay-runner profile) already does with its baton. The pipeline ledger plays the relay roles: ledger = stable agreement, per-job output = baton, log.md = history.
2. Handoffs are human-gated. Research never auto-becomes a plan. The owner sits at the seams where judgment lives: the idea (what to research) and the goal (what to build). Those two are the only inputs a job can't produce for itself.
3. Nothing in the chain needs the prior session to be alive. The handoff medium is a file path. "Wait an undefined amount of time" is just "the next job runs when its input exists and the owner releases it" — there is no timer, no daemon waiting, no state to expire.

2. Answer: how is the data organized as it flows?

Three distinct homes, one rule: the filesystem is the source of truth (same ruling orbit's checklist item 2.11 carries — per-task dirs, filesystem is SoT).

~/research/                      the PUBLIC tree — domain expertise
  ├── index.html                 generated catalog (never hand-edited)
  └── searxng-self-host/
        ├── doc.html                   the readable document (self-contained)
        ├── sources.md                 source list with access dates
        └── assets/                      images, diagrams — only if doc.html needs them
        (sibling) searxng-self-host-2026-10-17.tar.gz   portable bundle

~/implementation-plans/          the PRIVATE tree — exactly what's there today
  └── searxng-self-host-impl-plan.html   (+ its reference tarball, existing pattern)

~/pipelines/                     the LEDGER — one small file per idea (the new part)
  └── searxng-self-host/meta.json
~/pipeline-jobs/                 job WORKSPACES — ephemeral, disposable once done
  └── p002-searxng-self-host-research/
        prompt.md  output.md  progress.md  log.md  lock  meta.json

The key separations:

• Artifacts vs. workspaces. The research/ and plans/ trees are stable homes for finished artifacts. The workspace is where a job lives while working and is garbage after — the ledger's pointer is the bundle, the workspace is the desk.
• Public vs. private is a directory, not a permission. The publish step copies only ~/research/. Nothing in ~/pipelines/, ~/pipeline-jobs/, or ~/implementation-plans/ ever leaves the box. Keeping private and public in separate trees makes the leak a non-event: you cannot publish the plan by accident, because the plan is not in the tree you publish.
• Names match along a chain. One slug per idea: searxng-self-host names the research folder, the plan file, the ledger, and the job workspaces. A human can follow an idea from idea to artifact to code by name alone. Date suffixes sit on *versions* (-2026-10-17) so a re-research is a new artifact, not a mutation of the old one — the old one stays readable, same way mem keeps its raw rows under the rollup.

3. Answer: how are expertise documents organized, and in what format?

Format: self-contained single-file HTML. One decision, all ten questions get easier. It is already the house standard (AGENTS rule 8): renders identically by an agent reading it, by the owner on a phone, by a browser on the tailnet, or on the public web — and it requires zero tooling to view, which is what matters for a public audience who will not clone a repo.

The one thing Markdown does better is diffing. The trade is acceptable: expertise docs change rarely (re-research = new file, per §2), and where diff matters — code — git already does it natively. For documents that genuinely have many multi-kilobyte edits across a long life, the doc becomes a folder: doc.html is the assembled reader view; large tables (source lists, benchmarks) live in sibling files it links. That covers "what a doc needs more than one file" (§7) without breaking the one-file rule for the 95% case — most house documents will always be one file, because the boilerplate keeps them small.

What goes in an expertise doc (per the owner's spec, made concrete):

• The domain itself — what the thing is, how it works, the design space.
• Required software — the exact tools/versions a correct implementation needs, with what each is for.
• Comparable systems — "tools that sort of work the same way", each with what it does well and where it fails. Reference material, not a product list.
• Patterns and pitfalls — the failure modes an implementer will meet, each with the standard way to defuse it.
• Sources — sources.md in the folder: URL, publisher, access date, one-line claim-each. Dates are the currency here: sources rot, and the reader (agent or human) is trained to check dates, not trust the text (same no-supersedence rule as mem).

The no system-specific content rule gets a concrete enforcement: the research job's prompt ends with a checklist the agent verifies before finishing — no paths, no daemon names, no hostnames, no machine specs that aren't generic. (orbit checklist item 2.16, "every lane prompt ends with verification instructions", is the existing pattern.) A doc that fails its own audit gets flagged, not shipped.

4. Answer: public web

Because the document is a self-contained HTML file, publishing is a copy:

pipeline publish            # regenerates index.html, copies ~/research/* → ~/research-public/
                            # (tar.gz alongside; a single rsync/ftp of one folder to the host)

Three viable hosts, in order of friction:

1. A git remote + GitHub Pages (or Cloudflare Pages): publish rsyncs into a git worktree, commits, pushes. Pages serves the folder; the generated index.html is the catalog. The box gets a git remote for the first time in the system — a new hook that needs a manifest row the moment it lands (AGENTS rule 6). Note this deliberately differs from the no-remotes-standing policy in manifest §5, which covers code repos as backup; this one publishes prose, not code, and the owner rules on it.
2. Tailscale funnel on the cockpit: /research/… served from the box. Instant, no remote, but the tailnet-identity gate means it is visible to the owner's devices, not the open web — fine if "public" really means "me, on any device".
3. A home server: rsync the folder. Most moving parts, least magic.

The generated index.html lists each doc with its title, date, and a one-paragraph abstract (read from the doc's <title> + first <p>), newest first — same newest-first convention as ~/tapes/INDEX.md. Regenerating it is derivable work like mem sum: you can rm and rebuild it any time, because the docs are the source.

5. Answer: database, or a folder of files?

Files. Specifically: one JSON file per idea. SQLite would be the wrong shape.

• Scale. This is a handful of ideas a year, each with three to six jobs. A database buys you query power at thousands of rows; here it buys you nothing and costs you the ability to read the whole system's state in cat while debugging.
• Ownership style. Every tool in this system has exactly one plain-text state file that is its source of truth: ~/todo.checklist, ~/flags.txt, ~/tags.txt, and mem is the exception (SQL for high-churn ledger rows). The pipeline ledger is a small, slowly-changing set of records — the checklist shape, not the mem shape. cat ~/pipelines/*/meta.json is the honest ls command.
• The two-record rule. Each idea = exactly two durable records: ~/pipelines/<slug>/meta.json (the ledger) and its artifact folders. No third place where state can hide — the standing anti-pattern that burned the manifest before.

The ledger, concretely:

{
  "slug": "searxng-self-host",
  "inspiration": "replace the Exa free tier with a local search backend",
  "goal": "orbit research jobs get search that costs nothing and can't be rate-limited",
  "state": "planned",
  "created": 1763323333,   // unix epoch seconds (owner 2026-10-17: decisive sort);
                              // legacy day-strings normalize at read, never rejected
  "jobs": [
    {"id": 1, "stage": "research", "status": "shipped",
     "workspace": "~/pipeline-jobs/p001-searxng-self-host-research/",
     "artifact": "~/research/searxng-self-host/doc.html",
     "started": "2026-10-17T21:00:00Z", "finished": "2026-10-17T21:14:00Z",
     "attempts": 1},
    {"id": 2, "stage": "plan",     "status": "shipped",
     "artifact": "~/implementation-plans/searxng-self-host-impl-plan.html",
     "started": "2026-10-17T21:30:00Z", "finished": "2026-10-17T21:44:00Z",
     "attempts": 1},
    {"id": 3, "stage": "build",    "status": "queued", "artifact": null, "attempts": 0}
  ]
}

That file is the answer to "how do I track an idea end to end": a state field for the chain and a status field per job, with timestamps that a fresh session can read without asking anyone. It is also the cockpit card's data source (§8) and the publish script's input. One file, three consumers — the inverse of the lesson in manifest §5, where state spread over tailscaled, the app, and a note, and one of them was wrong.

6. Answer: how are jobs organized?

They don't get a new queue. orbit is the queue. The research and plan steps are the research lane with a lane-level prompt orbit's checklist already specifies (2.3: 27B-class model + web search tools); the build step is the code lane with the plan as its prompt; the verify step is the test model (checklist 2.8: "build ↔ test model handoff"). What orbit v2 gives the pipeline for free, item by item:

So: pipeline new/research/plan/build <slug> resolves to dispatch_task with a composed prompt (template + inputs), writes the job row into the ledger, and orbit runs it. When the job finishes, the workspace's output files land in the artifact home, the ledger row flips to shipped, the chain's state advances, and — per the owner ruling in checklist 1.2's kin — that is the end of the automated part. The next step waits for the owner.

7. Answer: multi-file jobs stay one job, because a job is a directory

The orbit pattern already answers this: a job is a directory, and a directory is atomic to the filesystem — there is no "bundle" step, no zip step, no archive to re-attach. The workspace's contents are the bundle:

~/pipeline-jobs/p002-searxng-self-host-research/
  prompt.md        ← exactly what the agent was told (re-runnable: this is the job, verbatim)
  output.md        ← working draft (markdown is fine here; rendered at the end)
  sources.md
  progress.md      ← the baton: what's done, what's next, for a successor after a death
  log.md           ← append-only: what the agent tried, what failed, dates
  lock             ← the single-writer token (orbit 2.5)
  meta.json        ← attempt count, state machine for workspace, model, lane

Two moves keep it clean. First, the finished bundle is the artifact folder (§2), which carries the stable name in the stable home; the workspace gets tarred to a sibling .tar.gz (the existing pattern — every house implementation plan ships with its reference tarball) and the workspace itself is deleted. The tarball is the process record; the folder is the product record. Second, the ledger row points at the folder, never at workspace fragments, so a re-run of a job can never be ambiguous about which work belonged to it. "One job, many files" then has exactly one meaning: a path, plus a machine-readable manifest inside it.

8. Answer: visualizing progress

Two surfaces, same source. Both read the ledgers; neither can write them (the dashboard observes, the terminal mutates — the standing ruling from the cockpit's task card, checklist 1.2).

• pipeline ls — one line per idea, and the stepper is five cells: new ▸ researched ▸ planned ▸ built ▸ shipped, the current state highlighted, per-job attempts in the margin, newest chain on top. Identical output philosophy to checklist ls (open by default, --all for the dead):

  $ pipeline ls
  ▸ searxng-self-host    researched(1) planned(1)  built[queued]      2026-10-17
    voice-baragein       researched(1)  planned[queued]              2026-10-16
    focus-adaptive       new                                               2026-10-15

• The cockpit card. The HUD renders the same stepper as a small SVG — five connected nodes per row, colored by state, attempts as badges; a read-only GET /api/pipelines that does one readdir of ~/pipelines/ (same one-file-read shape as the tasks card's GET /api/tasks). It shows the pipeline's shape, live: what is moving what is stuck, at a glance from any device. It can also serve the generated research catalog (§4) behind the tailnet gate — public and private views, one server.

Both surfaces are derivations of the same five ledgers, so they cannot disagree in any way a human would notice — and if a ledger is corrupt, both fail loudly together, which is the failure mode you want.

9. Answer: resource management — not taking the system down

The box is one 64 GB M4 Mac where the owner lives, so the doctrine is the one already in the house: bounded concurrency, gated admissions, hard caps, clean stops. Each clause maps to something that exists:

The unsaid clause, stated now: the research and plan jobs are the cheap ones — a few minutes of 27B inference and some web fetches. The build job is the expensive one, and its budget is exactly the project's test suite + the off/on drill, both bounded and existing. Nothing in this pipeline can take the box down, because nothing in it runs unbounded in the background: the only daemons involved (Ollama, orbit worker, focus watch) all have manifest rows with named stops.

10. How this fits what already exists — and what that means for building it

Piece by piece, the new system is a ledger, a lane prompt, a script, and a card:

Sequencing, as ruled (2026-10-17): (1) the pipeline CLI with one-shot execution — writes the ledger, launches the session, lands the artifact; (2) publish + cockpit card, each an afternoon and independent; (3) orbit v2 as written in its checklist — the same verbs start reading from the queue, and running chains through it becomes the daemon's proof of concept. Nothing here needs a new service, a new port, or a new launchd job in the first two phases — the pipeline CLI is on-call, like mem/tags/checklist/flags. The manifest gets one new row when orbit v2 lands, not a fleet.

11. Decisions (ruled, 2026-10-17)

Consequence for scope of the first build: the pipeline CLI (one-shot verbs), the three prompt templates, the ledger writer, and the SearXNG research job. Everything else waits its phase.

One file, zero JS, dark mode, phone-readable per house rule 8. Companion: ~/implementation-plans/ — the plan job for whichever chain the owner releases first lands there as <slug>-impl-plan.html + tarball. Written 2026-10-17.