ADR-0008: MVP-2 deployment topology — per-client Fly apps, run-capable sizing, `/data` layout, pack roster, backups¶

Status: Accepted — records the founder decisions of 2026-08-23 (Q3 per-client apps, Q4 Postgres OUT, V5 env-var + restart) and fixes the deployment contract that T6.2 (per-client apps), T6.3 (resize + live ceiling test), T6.4 (backup extension) and T4.4 (pack emission) implement. Nothing described here is built by this ADR. Date: 2026-08-24 Driver: Khalil Related: ADR-0007 (the DiscoveryRun rows whose file artifacts land under /data/runs/<id>/; merged via PR #286), docs/product/mvp-scope.md §8 (the MVP-1 deployment shape this ADR extends), docs/deployment.md (the operating runbook this ADR amends), MVP-2 tasks T6.1 (this ADR) → T6.2 / T6.3 / T6.4 / T4.4. The plan of record (docs/product/mvp2-plan-final.md) is on branch docs/mvp2-plan-final, not yet on main; its §11 decision table is the decision source cited throughout.

Context¶

What exists, deployed, today (MVP-1 / OGUR-67 / OGUR-68):

One Fly app, ogur (fly.toml), region cdg, one warm machine (auto_stop_machines = "off", min_machines_running = 1, deploys --ha=false), shared-cpu-1x / 512MB, one 1GB volume ogur_data mounted at /data with automatic daily snapshots, 14-day retention (fly.toml [mounts] snapshot_retention = 14).
The live DB is /data/ogur.db (Dockerfile:75 — DATABASE_URL="sqlite:////data/ogur.db"), WAL mode on every connection (ogur/store/database.py::_configure_sqlite). SQLite on a volume permits exactly one writer machine; the single machine is that writer.
Frozen packs are baked into the image, read-only, at /app/packs/* (Dockerfile:56-58, chmod 444 at :71). The served roster is EXPLORE_REPORT_PACK_PATH, a pathsep-joined list of pack paths (Dockerfile:73), consumed by ogur/config.py::explore_report_pack_path.
Backups are two layers (docs/deployment.md §Backups and restore): automatic daily volume snapshots, plus a manual offsite WAL-consistent DB copy (make fly-backup → scripts/ops/backup_db.sh, which uses sqlite3.backup() semantics and clamps the copy to 0600). The offsite layer copies the DB only.
Spend is ceilinged per identity per UTC day (OGUR-65, ogur/store/spend.py); ANTHROPIC_API_KEY is deliberately absent from the deployed app until a live ceiling test has run (fly.toml [env] comment; the ceiling is proven by unit tests only today).

What MVP-2 adds that this topology must absorb: metered discovery runs with per-run budgets (ADR-0007, T1.5), in-engine pack emission that writes to disk (T4.4), run artifacts and the Gate C manifest, and two clients whose data must be disjoint.

Founder decisions already taken (docs/product/mvp2-plan-final.md §11, 2026-08-23, recorded there with their reversal conditions):

Decision	Verdict
Q3 — per-client Fly apps vs shared	Per-client Fly apps
Q4 — Postgres	OUT; ADR trigger = a dedicated worker machine
Q5 — per-run cap	$10 default, revisited once T5.4 publishes measured per-stage costs
V2 — run execution for Sept 7	Persisted status machine + operator CLI; in-app thread is a conditional stretch
V5 — pack roster for Sept 7	Env-var + restart; DB-backed restart-free roster is post-gate

This ADR records those decisions and derives the remaining mechanics from them. It re-opens none of them.

Decision¶

Decision 1 — One Fly app per client (Q3 — decided 2026-08-23; recorded here, not re-opened)¶

Chosen: one Fly app per client — ogur-genfit, ogur-polygon (T6.2) — each with its own volume, its own users, its own secrets (including ANTHROPIC_API_KEY), its own spend ceiling, and its own pack roster. Zero application code changes — the roster is already an env-set pathsep list — but not zero build configuration: the image must become client-selectable, below.

Not chosen: one shared multi-tenant app with per-user grants. OGUR-65's authorization model (ungranted ≡ unserved 404) is correct within an app, but it makes client isolation a function of grant hygiene: one mis-granted landscape serves one client's competitive brief to the other, with valid auth, and nothing structural prevents it. Per-client apps remove that exposure class structurally — the other client's pack is not in the image, not on the volume, and not in the roster, so there is nothing to mis-grant.

The structural claim requires client-selectable packaging, and today's image does not provide it — at either layer. Dockerfile:56-58 unconditionally copies both client packs into the one image, and Dockerfile:41-47 unconditionally seeds the baked content DB from the GNS561 pack — which first boot materializes as every app's /data/ogur.db. The DB layer matters because DB-backed routes are grant-bound, not roster-bound: /api/signals gates on require_landscape_access (ogur/api/routes/signals.py:31-40), so removing a pack file from the image does nothing for the signal rows already seeded — an accidental grant on the wrong app would serve the other client's DB-backed evidence with valid auth. That is precisely the hygiene-shaped guarantee this decision rejects.

The mechanism: a CLIENT_PACKS Docker build argument naming the client's pack files, driving both consumers — the runtime pack copy and the content-DB seed stage. The seed rule is ReplaySpec-keyed, not pack-keyed: for each selected pack, replay it via the landscape-parameterized scripts/seed/pack_replay.py iff its landscape_id has a REPLAY_SPECS entry (today: GNS561 and obesity — Polygon has none), and skip it with a build-log line otherwise. A spec-less client pack is not a build error: Polygon's report and figures are served entirely from the pack JSON — the Dockerfile documents this ("Polygon has no seeder yet"; Dockerfile:25-27) — so the ogur-polygon image bakes a content DB with no evidence rows, the zero-rows validation below holds trivially, and Polygon's DB layer arrives when its MVP-2 rebuild (T5.3) lands a ReplaySpec. Never fail the build on a spec-less pack, and never seed one landscape's rows from another landscape's spec — the GNS561-pinned seed_gns561_autophagy.py is a wrapper for the shared app's build, not the per-client mechanism. Default = the current full set, so the existing shared ogur app and make fly-deploy are unchanged; a per-client deploy builds with only that client's pack(s) (fly deploy --build-arg CLIENT_PACKS=...) and sets the roster to match. T6.2's validation therefore checks both layers, not just the roster: on each client app, the other client's pack file is absent from the image filesystem (fly ssh console -C "ls /app/packs"), and the other client's landscape has zero rows in the app's DB (signal count by landscape_id), in addition to the roster-isolation and cross-client-404 probes.

Two consequences of the split, stated so T6.2 does not re-derive them:

Benchmark packs (obesity-rnai-001, cardiometabolic-rnai-001) never ship to a client app. They are eval infrastructure; they live in the repo and on eval machines only.
Per-client spend ceilings come for free. LlmSpend rolls up per identity per UTC day in each app's own DB, so a client app's ceiling is naturally scoped to that client — no shared-pool accounting to design.

The existing shared ogur app is not deleted by this decision; T6.2 carries the decommission plan. Cost: the ~$5–10/mo envelope (mvp-scope.md §8) becomes roughly that much per client. Accepted.

Decision 2 — A machine that hosts discovery runs gets 2GB; a machine that only serves packs stays at 512MB¶

The current 512MB budget is already accounted for: uvicorn + SQLModel + two ~5MB lru-cached packs, plus up to 256MiB of transient scrypt during concurrent logins — 128MiB × _SCRYPT_CONCURRENCY (a BoundedSemaphore(2), ogur/security.py:41); the [[vm]] comment in fly.toml documents that lowering memory or raising the semaphore without the other is the OOM path. A discovery run adds source-harvest buffers, per-round result caches, corpus matching structures, and the Anthropic client on top of that. The plan's risk register states the conclusion directly (R5): "512MB machine can't host a run — resize first (T6.3)."

Chosen: memory = "2gb" — the top of the task contract's 1–2GB band — on any machine that hosts a discovery run. Machines that only serve packs keep 512MB. Three reasons:

No measured run-memory number exists. No task in the window produces a memory profile (T5.4 measures cost, not RSS). The band is planning-grade, so the choice inside it should be the one whose failure mode is cheap.
The failure cost is asymmetric. An OOM kills a metered run mid-flight: real LLM spend (Q5: $10 default cap) is consumed and no artifact survives. The 1GB→2GB price delta is a few dollars a month.
No CPU-class change is needed. shared-cpu-1x supports 2048MB, so this is a memory edit in fly.toml, not a machine-family migration.

Not chosen: resizing every machine unconditionally (pays for run capacity on serving-only apps), or adding swap instead (swap_size_mb turns an OOM into unbounded latency in the middle of a metered run; a run should fit or fail fast).

For Sept 7, execution is operator-side (V2), so "the machine that hosts runs" is the one T6.3 resizes and runs the first live spend-ceiling test on (cent-level ceiling → real metered call → refusal → restore). If V2's conditional in-app-execution stretch lands, it inherits this sizing decision unchanged.

Decision 3 — `/data` layout: the DB at the root, packs and runs in their own trees¶

Chosen:

/data/ogur.db          — the live DB (unchanged; DATABASE_URL already points here)
/data/packs/           — run-emitted packs (T4.4 writes here; servable after promotion)
/data/runs/<run_id>/   — one directory per DiscoveryRun: Gate C manifest, gate
                         reports, emitted artifacts (CSV/JSON)

Rules that make the layout mean something:

Image packs and volume packs coexist, and the roster spans both. Frozen packs stay baked read-only at /app/packs/*, versioned with the image; /data/packs/ holds packs produced on the machine. EXPLORE_REPORT_PACK_PATH is a pathsep list, so one roster string may name entries from both trees. Nothing moves the frozen packs onto the volume: an image-baked pack is reproducible from a git SHA, a volume file is not.
<run_id> is the DiscoveryRun primary key (ADR-0007). The run row is the source of truth; the directory holds that run's file artifacts. A directory with no surviving row is garbage, never data — cleanup may delete it, and nothing reconstructs rows from directories. This is the same direction as the repo's FK discipline: the DB does not enforce foreign keys, so ownership is a convention with a stated deletion order, and here the order is row first, directory second.
Pack writes are atomic and explicitly chmodded. mkstemp + os.replace, then an explicit chmod to 0644. os.replace preserves mkstemp's 0600 mode — this repo has hit that exact silent-permission drop before, and a served pack that only root can read fails at request time, not write time.
Volume growth is operational, not architectural. The volume is 1GB today; packs are ~5MB each and run artifacts are MB-scale. fly volumes extend grows a volume online, so T6.2 sizes each client's volume at creation and extension is one command later, not a redesign.

Decision 4 — Pack roster: env-var + restart stands for Sept 7; the dynamic replacement is specified here so nobody builds it early (V5)¶

Chosen for the window: registration = edit EXPLORE_REPORT_PACK_PATH + restart. The promotion path (DC7, recorded in the plan): rebuilt pack → gates → diff vs served → founder approval → roster update + restart. A rebuilt client pack never silently replaces the served frozen pack (mvp-scope.md §2.4). Promotion is founder-gated and diff-reviewed anyway, so the restart adds one command to a process that already has a human in it; generalizing cache invalidation inside the gate window is risk without gate value.

The roster update replaces the superseded entry — never appends alongside it. Duplicate landscape_ids resolve first-wins: report_pack._index_for keeps the first configured path carrying an id (ogur/api/report_pack.py:650). An operator who appends the promoted /data/packs path after the frozen /app/packs entry gets a restart that succeeds and a promotion that silently did not happen — the frozen pack stays served, and the next promotion's "diff vs served" step diffs against the wrong baseline. So the promotion edit removes (or precedes) the entry it supersedes, and the post-restart check is the served digest, not the roster string: fetch the landscape's served pack digest and compare it to the promoted file's — the same mechanism as the smoke gate's content check (docs/deployment.md §Verifying).

The replacement, described so T4.4 and T6.2 know the seam they must NOT build: a DB-backed roster — a table mapping landscape_id → pack path + digest + served flag, read per-request or behind an explicit invalidation hook, making promotion a row write with no restart. Two things gate it, and both are structural rather than effort:

The pack loader cache is path-keyed (lru_cache on the pack path), so a restartless roster needs a cache-invalidation design, not just a table.
A restartless promotion only pays off when runs execute in-app — that is the first moment a promotion happens without an operator already at a terminal.

Hence its vehicle: the DB-backed roster rides the in-app-execution milestone (V2's conditional stretch / the post-gate backlog). Until then, "dynamic roster" appearing in any in-window PR is scope creep against a founder decision.

Decision 5 — Backups extend to packs and run artifacts (T6.4)¶

Layer 1 — volume snapshots — is extended by Decision 3 itself. Snapshots cover the whole volume, so putting /data/packs/ and /data/runs/ on the volume puts them in every daily snapshot by construction. No additional work.

Layer 2 — the offsite copy — must be extended, because today it copies only the DB. scripts/ops/backup_db.sh pulls a WAL-consistent /data/ogur.db and clamps it to 0600. Chosen: extend the offsite flow to also copy /data/packs/ and /data/runs/ — plain-file copies (both are write-once after emission; no WAL machinery applies), under the same 0600 offsite discipline as the DB copy. Run artifacts and packs are client-scoped competitive intelligence; the backup script's own rationale ("treat it like a credential file, not a build artifact") applies to them unchanged.

The DB copy and the file copies are one snapshot, not two independent copies. They happen at different instants, and run rows and run directories reference each other: a delete_run() between the DB copy and the file copy leaves the restored DB holding a DiscoveryRun row whose /data/runs/<id> — and therefore whose unregenerable Gate C manifest — is gone; the reverse copy order has the symmetric race with run creation. The consistency boundary for Sept 7 is operational quiesce, defined as zero active runs: while the offsite backup runs, nothing is created or deleted and no run is in a non-terminal state — an executing run keeps writing to both sides of the snapshot even when nothing is created or deleted (heartbeats and per-call cost settles into the DB, artifacts and manifest state into /data/runs/<id>/ — ADR-0007), so blocking only creation and deletion is not a boundary. Free to hold, because under V2 every run mutation is operator-side and the backup is an operator command on the same machine: the operator backs up only when their own runs are done. The copy order is fixed at files-then-DB as defense in depth: an infraction that slips through then degrades to a row-less directory, which Decision 3 already classifies as harmless garbage, rather than a dangling row. And the backup manifest asserts coherence instead of listing paths: every run row in the copied DB that references artifacts — non-null artifacts_dir / manifest_path, both nullable by ADR-0007 precisely because a draft or queued run legitimately owns no directory yet — has its directory and named files in the copied set, with digests where the manifest records them; artifact-less rows pass. Checked at backup time and again in the restore drill. When run mutation stops being operator-only — in-app execution or a worker machine (Decision 6's trigger) — the quiesce convention stops holding by construction, and the boundary must become mechanical (a point-in-time volume snapshot as the export source); that lands with the same milestone.

The backup manifest lists the new paths and carries the coherence assertion above; T6.4's validation is one restore-drill transcript including that check.

Why these belong in the offsite set at all — mvp-scope.md §8's line that "client feedback + identities are the only unregenerable data" gains two qualifications:

An emitted pack is regenerable in principle (re-run the engine) but not for free: a rerun re-spends metered LLM budget and lands on drifted upstream sources.
A run's Gate C manifest is the audit record of that run. A rerun does not regenerate it — it produces a different run with a different manifest. The manifest is unregenerable in exactly the sense the feedback rows are.

Decision 6 — Postgres stays OUT; the trigger is a dedicated worker machine (Q4 — decided 2026-08-23; recorded here)¶

SQLite on a volume permits exactly one writer machine, and every topology in this ADR — including per-client apps, each with one machine — preserves that invariant. The invariant, not preference, is the decision boundary: the day a topology needs a second writer — a dedicated worker machine executing runs while the serving machine writes feedback and sessions — the one-writer premise is broken, and that is the moment the Postgres ADR gets written. Until then WAL mode on one warm machine is proven in production, and a database migration inside a 12-day gate window is risk with no gate value.

Not chosen: adopting Postgres "while we're at it" during the per-client split (two migrations at once inside the window), or a hosted variant (Supabase et al. — same decision; founder-decided OUT, mvp2-plan-final.md §9.1).

Consequences¶

T6.2 stands up ogur-genfit + ogur-polygon per Decisions 1 and 3: per-client images (CLIENT_PACKS build arg), own volumes (sized at creation, fly volumes extend later), secrets, users, packs, ceilings; runbook updated; shared-app decommission plan. Validation: per-client roster isolation, the other client's pack absent from /app/packs on each app, the other client's landscape at zero rows in each app's DB, cross-client probe → no-leak 404, backup/restore smoke per app.
T6.3 resizes the run-capable machine to 2GB (Decision 2), sets ANTHROPIC_API_KEY, and runs the first live spend-ceiling test — closing the proven-by-unit-tests-only gap.
T6.4 implements Decision 5's offsite extension and the restore drill.
T4.4 writes emitted packs to /data/packs/ (Decision 3), registers them via env-var + restart under the DC7 promotion path (Decision 4), and does not build the roster table.
docs/deployment.md gains the per-client runbook when T6.2 lands; until then it correctly describes the single ogur app and needs no edit from this ADR.

ADR-0008: MVP-2 deployment topology — per-client Fly apps, run-capable sizing, /data layout, pack roster, backups¶