ADR-0007: Discovery-run persistence and projection into the Monitor entity spine¶
Status: Accepted — the decision is ratified; the code is not written. T1.2 builds these tables and their store helpers, and nothing described here exists on main today.
Date: 2026-08-24
Driver: Khalil
Related: ADR-0005 (specifies the harvester whose in-memory value objects this ADR promotes to tables), ADR-0006 (DiscoveryRunContext — the run identity these rows persist; its fail-closed rules are reused here), docs/product/mvp-scope.md §3.4 (the no-second-spine constraint, which says an ADR is required before it ships), MVP-2 task T1.1 — blocks T1.2 (models + store helpers), T1.4 (status machine), T1.5 (per-run budget), T1.7 (manifest). The plan of record (docs/product/mvp2-plan-final.md, docs/product/mvp2-linear-tasks.md) is on branch docs/mvp2-plan-final, not yet on main.
Context¶
MVP-2 is the Explore discovery backend. Its blocking gap is that nothing persists what a discovery run finds. DiscoveredEntity and DiscoveredHit are dataclasses in ogur/engine/discovery_modality/types.py (:153-231), the file whose own docstring says "No DB / HTTP / LLM here — pure data". There is no run row, no run-level spend, and no job model.
The four client landscapes were built by scripts that write CSVs and markdown into archived_data/ and never write to the database. scripts/eval/sirna_autonomous_discovery.py:807-826 is the whole persistence layer: three files, stamped with a date, from a discover() whose return value — (scope, known, backing_signals, rounds, run_meta) — is the entire run result and is discarded when the process exits. The only DB write path in the discovery area today is scripts/seed/pack_replay.py, and it is a replay of a frozen artifact, not a harvest.
Four things follow from that, and each is a requirement on this schema:
- A run is not reproducible from anything the system holds. The
run_metadict (sirna_autonomous_discovery.py:567-580) is a proto-manifest with no config digest and no artifact hashes. Gate C — every run writes a manifest — has nowhere to write it. - Spend is metered per identity per day (
ogur/store/spend.py), never per run. A run cannot be given a budget because there is no run to attach one to. - Gold-deck replays and real runs are indistinguishable. Today that is safe only because no product run exists. It stops being safe the moment one does, and the pollution shows up as recall going up.
- Serving discovered entities has no defined destination.
docs/product/mvp-scope.md:337-345names the hazard precisely: serving entities from a pack "would otherwise stand up a second entity spine besideTarget/DrugTarget/CompanyProfile/DrugProfile, and the next entity feature would have to guess which spine to read."
This ADR decides the three tables, the status machine, the eval | product honesty flag, the projection rule into Monitor's existing spine, Landscape-row semantics, and the deletion discipline that unenforced foreign keys require. It is a proposal that T1.2 implements, not a ratification of shipped code — nothing described here exists yet.
One framing governs all of it, and Decision 8 states it in full: Monitor's schema is a projection target, not a design constraint. Monitor's backend was built for a demo and is expected to be reworked. Explore's tables are designed for reproducible runs on their own terms; projection is an adapter at the boundary, so a Monitor rework changes the adapter and leaves these tables alone. Where this ADR touches Monitor at all, it compensates at the adapter rather than reshaping shared demo-grade code.
Decision¶
Decision 1 — Three tables, and one mechanical rule for what is a column¶
DiscoveryRun, DiscoveredEntity, DiscoveredHit, in a new module ogur/models/discovery.py. Full field blocks are in Architecture; the decisions below say why they are shaped that way.
The column-vs-JSON rule, applied mechanically: a field is a column iff a WHERE, an ORDER BY, an atomic UPDATE … = … + ?, or a gate reads it. Everything else is a str column holding json.dumps output with a trailing comment naming its shape — the Landscape.targets / Signal.raw_data / Briefing.signal_analyses shape.
Chosen: str columns holding JSON text.
Not chosen: a real JSON/JSONB column type. This repo has none, and introducing the first one here would make these three tables the only ones whose read path differs from every other model. The cost is real and is accepted: no SQL-level query into a JSON payload.
The module must be added to create_tables()'s import list (ogur/store/database.py:223-241). SQLModel.metadata only knows about model modules that something imports; a module absent from that list produces no table and no error. Any column added to these tables after first deploy needs its own _ensure_*_columns() migrator in the _LANDSCAPE_ADDITIVE_COLUMNS shape (database.py:45-50), because create_all never alters an existing table — that is the entire reason _ensure_landscape_columns() and _ensure_signal_uniqueness() exist. T1.2 wires the migrator hook even while its column list is empty: an empty list you extend beats a mechanism you have to remember exists.
Decision 2 — mode: eval | product is required, has no default, and gates two things today¶
Chosen: mode is a required column on DiscoveryRun with no default value. create_run() raises on anything outside {"eval", "product"}.
Not chosen: a default of "product", or of "eval". This is the same fail-closed shape as DiscoveryRunContext.__post_init__ rejecting an empty landscape_id (classes/base.py:979) and ADR-0006 Decision 2's "there is no default_class()". A defaulted honesty flag defaults to dishonest the first time someone forgets it, and the whole value of the field is that it cannot be forgotten.
What it gates, stated without inflation:
| Consumer | Rule | Live at this milestone? |
|---|---|---|
Gate A scorers (/coverage-eval, the T2.5 harness) |
a recall denominator is built from runs of exactly one mode, and the mode is printed in the scorer's header line | yes |
| Projection into the Monitor spine (Decision 8) | mode == "eval" never projects |
yes |
| Served roster / packs | an eval run is never eligible for a served surface | no — the roster is env-var + restart at this milestone |
confirm_hypothesis (see Reserved) |
product-mode only | no — reserved, not built |
Only the first two are live. Claiming four consumers when two are hypothetical is the same dishonesty the flag exists to prevent.
The eval-projection rule is the one with teeth today. An eval run replays a gold deck. Projecting it writes deck knowledge into the spine, and the next Gate A run harvests a corpus that now contains its own answers. That is answer-key leakage, the same defect class the query planner already refuses in types.py:100-107 — cde_per_code "recovers a sponsor + CTR for an asset code DISCOVERED from another live source in an earlier round (never a deck code — that would be a leak)".
Mixing is prevented by making it hard to write, not by a constraint. Every store read helper that returns more than one run, entity, or hit takes mode as a required keyword-only argument with no default:
A caller wanting both modes asks twice, and has therefore written the mixing down in its own source where a reviewer sees it. This is the rationale _merge_hit already gives for its klass parameter (sirna_autonomous_discovery.py:255-257): "required, not defaulted: … a default here would silently apply one class's denylist to another's run".
Not chosen: separate tables per mode (doubles every store helper and the deletion order, for two things that are the same shape); a DB CHECK constraint or partial index (create_all never alters an existing table, so one added later would simply not exist on any deployed database).
The failure mode if this field is stored but unread, concretely: Gate A's floors are recall against gold decks (siRNA 16/32 assets · 10/12 deals; obesity 12/14 core · 11/11 companies). Once a product-mode run writes entities into the same tables, a scorer that selects by landscape_id alone — the obvious path, because every existing scorer keys on landscape_id — picks up product-run entities and recall goes up for free. The gate then passes a change that broke discovery. It is undetectable by inspection because it moves the number in the good direction.
The probe that catches it. min_target_count has been stored-but-unread twice in this repo, and both times a single committed artifact plus a benign default made the omission invisible. The standing answer is to probe a new consumer with a synthetic second config; the mode analogue is:
Create two runs on the same
landscape_id, oneevaland oneproduct. Write disjoint, individually identifiable entity sets into each. Invoke the consumer. Assert the returned entity set equals the eval set — as a set, never as a count.
The set assertion is load-bearing. A count assertion passes whenever the product run happens to contribute zero in-deck names, which is exactly the plausible-number shape that hid min_target_count. Make the probe non-optional by listing mode-aware readers in a module-level tuple that a parameterised test iterates, so adding a reader without adding its probe fails a roster test rather than merging quietly.
Decision 3 — DiscoveredEntity is keyed (run_id, entity_key), not (run_id, canonical_name)¶
Chosen: a composite natural primary key (run_id, entity_key), following SignalDrug (ogur/models/drug.py:20-36, whose docstring states the property directly: "The composite primary key makes the backfill idempotent — re-running the enricher cannot create duplicates"). That is exactly the idempotency T1.2's upsert test wants, obtained from the schema rather than from helper code. A surrogate uuid would be dereferenced by nobody: hits join by (run_id, entity_key), and no API path needs an entity id.
This corrects the key stated in T1.2's task contract. T1.2 says "upsert idempotency on (run_id, canonical_name)". That key is wrong, and it fails in a way its own test would not catch. From _merge_hit (sirna_autonomous_discovery.py:266-292):
- company —
canon = normalise_entity(hit.entity_raw);key = canon. Key and canonical name coincide. - asset code —
canon = hit.entity_raw.strip();key = f"code::{normalise_code(canon)}"; the entity is constructed withcanonical_name=canon, i.e. whichever raw surface form arrived first.
So "GNS-561" and "GNS 561" are one entity (code::gns561) whose canonical_name depends on hit arrival order. An upsert keyed on canonical_name is order-dependent: it passes on every company row and fails silently on codes, splitting one asset into two. It also collides across kinds — a company and an asset code sharing a display string would be two entities with one key. entity_key is not derivable from (kind, canonical_name); it needs its own column, and the ADR names it so T1.2 does not discover this at test time.
Collections are stored as sorted JSON lists. Sorting at write time is not cosmetic: T1.3's validation is A/B parity of a refactored loop against pre-refactor artifacts, and an unordered serialization makes a byte-diff of two runs' rows meaningless. co_entity_groups stays raw and pre-normalisation — _accumulate_context (sirna_autonomous_discovery.py:241-250) keeps those names unnormalised on purpose as deal hints, and normalising them at persist time destroys the signal.
The entity gains record_targets / record_indications unions the dataclass does not have. DiscoveredEntity today unions only linked_* (types.py:215-217). The projector (Decision 8) may read only record_*, so that union has to exist somewhere; computing it per-projection instead would let the served roster and the projector disagree about what one entity evidenced.
Three rollups are denormalised columns with exactly one writer. source_diversity is the trust signal ADR-0005 Decision 4 filters the roster on (>= 2 by default) and the assembler ranks by — it must be a WHERE and an ORDER BY, not a JSON parse per row. unique_evidence is the corroboration denominator and the projection gate (Decision 8), and must be a SQL count of distinct documents — in the specific form given below, because the obvious one does not run on SQLite. most_recent_date is the recency ranker for honorable-mention rows (types.py:224-231). Both committed run artifacts already carry all three as columns (archived_data/polygon_plg101_explore/discovered_companies_20260710.csv, archived_data/gns561_explore/discovered_companies_v3.csv), so this is materialising a field the pipeline already computes, not inventing one.
unique_evidence needs a SQLite-compatible query, and the obvious one does not run. SQLite rejects COUNT(DISTINCT (a, b)) with row value misused, and COUNT(DISTINCT a, b) with wrong number of arguments to function count() — both verified directly against sqlite3. Because append_hits() recomputes the rollup inside the same transaction that inserts the hits, a literal reading would fail on the first insert. The form that works, and that skips seed hits by construction:
SELECT COUNT(*) FROM (
SELECT DISTINCT document_source, document_id
FROM discoveredhit
WHERE run_id = ? AND entity_key = ?
AND document_source IS NOT NULL AND document_id IS NOT NULL
);
The IS NOT NULL pair is load-bearing rather than defensive: SELECT DISTINCT treats two NULLs as equal, so without it every entity carrying seed hits would contribute one phantom document — turning a seed into exactly the self-corroboration Decision 4 refuses.
The drift risk is real and is answered structurally: append_hits() is the only writer, and it recomputes all three in the same transaction that inserts the hits, alongside a verify_entity_rollups(session, run_id) helper the T1.2 tests call. A caller setting these fields directly is a bug. This is the min_target_count shape — a stored value whose only guarantee is that somebody remembered — and it has bitten twice, so the guarantee is placed in one function rather than in a convention.
Decision 4 — DiscoveredHit is the document substrate, and carries a citable document identifier¶
The founder ruling is that the document is the substrate, not the company: the unit every gate measures and every claim cites is the document identifier — NCT, publication number, PMID/OpenAlex id, filing. DiscoveredHit is that layer, and every DiscoveredEntity is an aggregation of it.
Chosen: uuid primary key plus UniqueConstraint("run_id", "hit_hash") — the Signal shape (ogur/models/signal.py:43-46), not SignalDrug's composite PK. The natural key here is a six-part tuple containing free-text query_text, and an unbounded string does not belong in every index. compute_hit_hash(...) materialises the composite key, mirroring compute_evidence_content_hash (ogur/models/evidence.py:9-24).
run_id is deliberately excluded from the hash, and uniqueness is on the pair. A globally unique hash would recreate the Signal bug verbatim. That bug is documented at ogur/models/signal.py:74-90: "A globally unique hash meant two landscapes projecting the same record collided and upsert_signals dropped the second, so whichever landscape ran first silently owned the evidence and the other lost it." Here the collision is between runs, and it is not a corner case — two runs re-finding the same document via the same query is what a re-run is. The second insert would be dropped, under-counting the re-run's evidence and reading as a recall regression. Keeping run_id outside the hash has a second payoff: the hash becomes the cross-run diff key, which is what T1.3's A/B parity check and Gate R comparability need.
The hit carries (document_source, document_id), not signal_ref. types.py:169 describes signal_ref as "NCT id / accession / lens_id / OpenAlex id / drug name", but that comment is stale: every engine call site passes Signal.content_hash (ogur/engine/discovery_modality/sources.py:426, :483, :631) and SeedEntity passes the literal string "seed" (classes/base.py:890, :914). Persisting it verbatim would put a 16-hex opaque string in the one table whose entire premise is that the document identifier is what gets cited — the mis-binding hazard a self-checking citation token exists to prevent.
The hit therefore carries document_source and document_id from Signal.(source, source_id), plus signal_content_hash for the join back into Monitor. This is the anchor Feedback already uses, for the same reason (ogur/models/feedback.py:8-13): it "NEVER references Signal.id — that column is a fresh uuid on every reseed, so a reseed would orphan every row". T1.3 must add an additive, defaulted field to the frozen DiscoveredHit dataclass to carry it; naming that here means T1.3 does not discover it mid-refactor.
The document triple is nullable, because one legitimate hit has no document — the seed. SeedEntity.materialise() (classes/base.py:902-921) constructs a real DiscoveredHit with source="seed" and signal_ref="seed", and there is no Signal behind it: the Polygon run depends on that path because its subject's own patents were indexed only by Lens's corpus and became un-re-findable when the key expired. A required non-null triple would leave exactly two options, and both are wrong — refuse to persist any seeded run, or manufacture a citable identity for a document that does not exist. The second is worse, and it is the failure this whole decision exists to prevent.
Chosen: document_source, document_id and signal_content_hash are nullable, and the store enforces the pairing in both directions:
source == "seed"→ the triple must be NULL. A seed that carries a document identifier is a manufactured citation.source != "seed"→ the triple must be fully populated. A harvested hit without a document is an un-citable claim, which is the case the non-null column was protecting against.
Neither direction may be defaulted or coerced; both raise at the store boundary, in the ADR-0006 Decision 2 shape.
Three consequences follow, and each one is a tightening rather than a relaxation:
- A seed hit never counts toward
unique_evidence, which counts distinct(document_source, document_id)and therefore skips NULLs by construction. That is the honest arithmetic:SeedEntity's own contract says seeding is legitimate only because "the uniqueness claim then rests on the ABSENCE of competitors, never on re-finding the subject." A seed that corroborated itself would invert that. - A seeded entity cannot reach the spine on seed evidence. Decision 8's projection gate is
unique_evidence >= 2, which counts only non-NULL document pairs — so a seed-only entity scores0, and a seeded entity found by one real channel scores1. Both fail, with no special case. Note thatsource_diversitycannot carry this gate:sourcesincludes"seed"(classes/base.py:906), so a seed plus one channel scores2there — which is exactly why Decision 8 gates on documents instead. The brief's own subject stays in the run's ledger and in the pack, where its provenance says "seeded, and here is the gap" — and out ofCompanyProfile, where it would read as independently corroborated. compute_hit_hashtakes the seed'ssignal_refin thedocument_idslot —"seed"by default, or whatever the seed declares — so two distinct seeds in one run do not collide, and a seeded run's hashes stay stable across re-runs like any other.
The reserved confirm_hypothesis gate is unaffected and in fact sharpened: "a result that cannot be written as (document_source, document_id) fails the gate by construction" now has a schema-level meaning, since a NULL triple is exactly what an un-citable claim looks like in this table.
The three linkage families stay three separate columns, and their trailing comments carry the distinction verbatim. Collapsing them is the single most expensive mistake available in this schema, and Decision 8 is where the cost lands. types.py:174-186 states it: linked_* "deliberately includes the query string … which is right for harvest attribution and wrong as EVIDENCE: an acute-MI-scoped query would otherwise confer that disease on every record it surfaced."
record_evidence is stored as an opaque JSON list with no validation of its members. The constant base.VALID_RECORD_EVIDENCE named at types.py:189 does not exist — that comment is stale, and classes/base.py:319-325 says the opposite explicitly: "The flag VOCABULARY is deliberately not defined here. It is class-specific … naming those constants in this module would put one class's classifier state in the shared layer." Shape validation already happens at the engine boundary in coerce_record_evidence (classes/base.py:337-356); the store does not re-litigate it. What identifies the vocabulary is record_evidence_version on the run row — the run's config is frozen at approval, so repeating it per hit is dead bytes.
Volume. Grounded in the two committed run artifacts: Polygon produced 387 company + 19 asset entities carrying 663 entity→document assertions; GNS561 produced 472 company entities (38 core / 355 honorable-mention / 79 off-thesis) carrying 935. With query multiplicity — one document surfaced by several axes and rounds — a run lands at roughly 10³ hit rows, worst case low 10⁴. That is an order of magnitude below what signal already holds, on a WAL-configured SQLite with busy_timeout=5000 (ogur/store/database.py:10-31), with at most one operator-executed run per day. Per-hit persistence is not a volume question, and the ADR records the numbers so nobody re-opens it on intuition.
Caps fail loudly rather than truncate, because a silent truncation is the same defect class as a degraded-empty source: query_text over 1000 chars raises at the store boundary; support_text is already capped at 300 by _support() and is asserted, not re-truncated; exceeding MAX_HITS_PER_RUN transitions the run to failed(hit_cap_exceeded) rather than writing a partial set that reads as a complete one.
Decision 5 — kind is a closed set of exactly two; document and target are refused, with reasons¶
Chosen: enforce closure over the two kinds that exist, and record the refusals. The contract asks for "kind typing"; the deliverable of that typing is enforcement plus a written refusal list, not new members. Adding a kind nothing produces is the min_target_count shape in its purest form — a field that is stored, never written by any producer, and read by a consumer that therefore never fires.
Enforced in exactly two places, both raising rather than defaulting:
ogur/store/discovery.py::upsert_entity— rejects a kind outside the set, and rejects a hit whoseentity_kinddisagrees with its entity'skind.- The engine merge. Today
_merge_hitbranchesif hit.entity_kind == KIND_COMPANY: … else: <treat as an asset code>(sirna_autonomous_discovery.py:266-286), so an unrecognised kind is silently treated as a code. T1.3 fixes that when it promotes the function.
Not chosen: a DB CHECK constraint (create_all never alters an existing table, so one added after first deploy would not exist where it matters), or a str, Enum column (it validates at serialization, but the failure to prevent is a class emitting a kind nobody merges, which is a merge-time fact, not a serialization one).
document is refused. The document layer is the DiscoveredHit table: hits are the documents, and entities are always derived. Three reasons:
- It would create a third identity for one document, beside
Signal.content_hashand(document_source, document_id). - It puts the substrate in the same table as its own aggregations, so
SELECT COUNT(*) FROM discoveredentity— a number a report prints — would mix documents with companies derived from them. The standing rule from the GNS561 engagement is that derived-entity aggregations ("N companies") are the weakest, most impressive-looking output; a schema that makes them indistinguishable from a document count is that failure mode made structural. - An entity's invariant is "aggregation of hits". A document-entity's hits would be itself.
What refusing costs, stated: the document roster is SELECT DISTINCT document_source, document_id FROM discoveredhit WHERE run_id = ? rather than a kind filter. One query, and it is the honest one.
target is refused. Targets are not discovered; they are scope input plus expansion output. The committed artifact shows it: every row of archived_data/polygon_plg101_explore/discovered_targets_20260710.csv carries provenance of either thesis or mechanism-expanded, and the file has no source, no support text and no scope-fit column — because a target has no hits. A kind="target" row would be an entity with zero hits and source_diversity = 0, which ADR-0005 Decision 4's >= 2 roster filter hides anyway. The resolved target set belongs in the run manifest (T1.7 already records it) and in Target, which has its own writer (upsert_target, ogur/store/targets.py:78-91).
indication and deal are refused on the same test: no producer emits them.
The standing rule: a kind is added when a harvester emits it, in the same change that emits it — never in advance.
Decision 6 — The status machine, and the honest form of "no stuck running"¶
draft → { scope_approved, cancelled }
scope_approved → { running, cancelled }
running → { converged, failed, cancelled }
converged → { synthesizing, failed, cancelled }
synthesizing → { packaged, conditional, refused, failed, cancelled }
packaged | conditional | refused | failed | cancelled → { }
draft → runningis illegal. The never-cut invariant "analyst scope approval before any spend" becomes an edge in the machine rather than a check somewhere that a refactor can drop.- Progress is not a transition. Round ticks update
progress,rounds_completedandheartbeat_atwhilestatusstaysrunning. There is norunning → runningedge, sotransition_status()raises on it and a caller that means "record progress" cannot accidentally mean "transition". - All five terminal states are terminal,
conditionalincluded.conditionalis the middle rung of the emission trichotomy: the gate report names the failing rows, and the analyst fixes config. Chosen: the fix produces a new run row, chained bysupersedes_run_id, not a transition back intosynthesizing. Why: fixing aconditionalmeans changing config; a config change re-keys caches and changesconfig_digest; a run row whoseconfig_digestchanged mid-life is un-replayable from its own manifest, which is Gate C's entire property. One nullable column buys "show me this landscape's run history" as a query instead of a reconstruction. failure_reasonis a closed vocabulary column,failure_detailis separate free text. T1.5's validation assertsfailed(budget_exhausted); an assertion on a substring of a prose message is a test that passes on a typo. Budget exhaustion transitions after the current round's persistence, so the partial entity set, hit set and manifest survive.
"Crash leaves failed, never a stuck running" — what is actually guaranteed. Execution at this milestone is an operator-run CLI, with no supervisor and no worker. Nothing can guarantee the column is never momentarily wrong. The invariant worth stating is the one that can be kept:
No reader is ever told a run is live. A reader is told how recently it reported, and staleness is bounded by
STALE_AFTER.A timestamp is a lease, not a proof of life — see part 2 for why the stronger wording was retracted.
Three parts, none of which needs new infrastructure:
heartbeat_at, written at every metered call — not merely every source call. The naive placement isCostMeter.add_source_timing(types.py:334-337) alone, and it is wrong: the sweep also failsconvergedandsynthesizing, and synthesis makes no source calls. A healthy synthesis running pastSTALE_AFTERwould be swept tofailed(stale_heartbeat)by the next API restart or CLI invocation, and itsrunner_tokenwould then stop the live process from completing. A false positive that permanently kills a working run is strictly worse than the stuckrunningthis mechanism exists to prevent.
The heartbeat is therefore written at every metered call in every swept stage: source calls during harvest (add_source_timing) and LLM settles during synthesis, which T1.5 already persists per call. Both are frequent, both already touch the run row, and together they cover running, converged and synthesizing. The cost is hundreds of small writes per run against a WAL database configured for exactly this.
heartbeat_at is also set on entry to every swept state, not only by metered calls. The column defaults to NULL and add_source_timing fires only after an awaited source call returns, so a freshly started run has no timestamp until its first call completes — and a first call slower than STALE_AFTER (a cold patents scan, a rate-limited registry) would be reported stale, or swept, before it ever had a chance to heartbeat. transition_status() therefore stamps heartbeat_at = now atomically on entry to running, converged and synthesizing.
That makes the bootstrap an invariant rather than a tolerance: NULL heartbeat_at on a swept state is an invariant violation, not a reading. Neither the sweep nor the derived heartbeat field has to decide what a NULL means, and T1.4's tests cover a fresh run before its first metered call completes.
The residual rule, written down so it is not rediscovered: a stage that can legitimately run longer than STALE_AFTER without any metered call must raise its own window rather than rely on the sweep. The sweep may only fail a stage that actually heartbeats.
2. Liveness is derived on read, never returned raw — and it is a bounded lease, not a proof. T1.6's GET /api/explore/runs/{id} returns status plus heartbeat_age_seconds and a computed heartbeat: "fresh" | "stale" — deliberately not live | dead. A timestamp cannot establish that a process is alive: a runner that writes a heartbeat and dies one millisecond later reads fresh for the rest of the window. Naming the field live would encode a guarantee no implementation without a supervisor can meet, and T1.4 would then be asked to test it.
So the invariant is restated to what the mechanism actually delivers: no reader is told a run is live; a reader is told how recently it reported, and staleness is bounded by STALE_AFTER. A caller that genuinely needs liveness rather than recency needs a supervisor or a lease service, and neither is in this milestone.
3. A sweep plus a claim token. sweep_stale_runs() transitions stale running/converged/synthesizing rows to failed(stale_heartbeat), called at the two entry points that already call create_tables() — the API lifespan and the head of the operator CLI. A runner_token minted on scope_approved → running is what makes the sweep safe: every run-loop write presents its token, a re-claim mints a new one, and a zombie process waking after its run was swept fails its next write instead of resurrecting a dead run.
Not chosen: a pid column (the CLI runs on an operator's machine or a client volume; a recycled pid on a rebooted host reads as alive, and cross-host pids are meaningless), or trusting the CLI's finally block (it does not run on SIGKILL, OOM, or reboot — precisely T1.4's kill-mid-run case).
The named trade-off: a run that genuinely stalls for longer than the window inside one source call is swept to failed(stale_heartbeat) and the operator re-runs. Given per-source-call heartbeating, a silence that long is a hang, not a slow source.
The machine records what was decided, not who decided it. There is no approved_by and no per-checkpoint actor field. created_by exists, but it is the run's owner — required by T1.6's session auth and T1.5's per-identity daily ceiling — not the decider of any checkpoint. Each checkpoint records a timestamp and the decision's content (scope_approved_at plus the frozen scope as approved). The consequence is deliberate: an agent that later fills POST …/approve authenticates as a User like any other caller and needs no schema change. The honest negative: you cannot audit who approved a scope from these tables. If that becomes a requirement it is an append-only RunEvent table, not a column — and it is listed as an open question, not reserved.
Decision 7 — Landscape rows: the caller supplies the slug, and a run never rewrites the row¶
landscape_idis NOT NULL fromdraftonward.DiscoveryRunContext.__post_init__already rejects an empty one (classes/base.py:979), so any run reachingrunningmust have it, and making it nullable only indraftcreates a state every reader must handle. It is also the grant key — a draft with no landscape id is readable either by everyone or by no one.- The caller supplies the slug; the API validates it and never generates one. Slugs are human (
gns561-cca-autophagy-001,cardiometabolic-rnai-001) and they appear inUserLandscapeAccess.landscape_id, inSignal.landscape_idstamps and in URLs. A uuid would be correct and unusable; a generated slug would collide with the human ones. Validate^[a-z0-9]+(-[a-z0-9]+)*$at ≤64 characters — which also makes a slug safe as a path component under the run-artifacts root. - A missing
Landscaperow does not mean the slug is free — creation checks three namespaces, not one.UserLandscapeAccess's own docstring says it outright (ogur/models/user.py:26-32): "a grant can name a landscape that only exists as a served report pack, which is exactly the MVP-1 case." Sogns561-cca-autophagy-001can be a live, granted, served landscape with noLandscaperow at all. A rule of "create iff no row exists" plus grant-based authorization is then an authorization bypass: an authenticated user submits another client's served slug, creation sees no row, and the grant that creation must issue for the run to be readable hands them access to a report they were never granted. Refusing to grant does not fix it either — the creator would immediately fail the grant check on the run they just made.
Chosen: one atomic claim step inside the run-creation transaction, resolving the slug against all three namespaces:
| Slug appears in | Outcome |
|---|---|
a Landscape row |
require_landscape_access; attach. No new grant is issued. |
served_landscape_ids() (ogur/api/report_pack.py:696) or any UserLandscapeAccess row |
claimed. Attach only if the caller already holds the grant; otherwise refuse. A run never mints a grant for a slug someone else's namespace already owns. |
| none of the three | genuinely new — create the Landscape row and the caller's grant, in the same transaction as the run. |
The residual disclosure, stated rather than papered over. An earlier draft of this decision claimed the refusal "reuses OGUR-65's no-leak shape". That was false, and the table above disproves it: an absent slug succeeds by being created while a claimed slug the caller does not hold is refused, so an authenticated caller can distinguish the two and learn which slugs exist. The escalation is closed — no grant is ever minted for someone else's slug — but a namespace existence oracle remains.
It cannot be closed while a create endpoint is keyed on a caller-supplied name: either the caller learns the name is taken, or they silently receive a landscape they did not name. So the honest position is that run creation does not carry the no-leak property, and this milestone does not expose it — execution and landscape creation are operator-side, so no client user reaches the endpoint. If self-service creation ever ships, it must choose explicitly: pre-grant slugs so creation only ever succeeds for names the caller already holds (which removes the oracle by removing the branch), or accept and document the disclosure. What leaks is slug existence, never contents; slugs already appear in the URLs a granted client uses.
Not chosen: checking only the Landscape table, which is the bypass above; and generating a fresh slug on collision, which silently gives the caller a landscape other than the one they named.
Run creation and the seed scripts remain the only two creators of Landscape rows; there is no POST /api/landscapes.
- "From the approved scope" does not define a valid row, so the mapping is enumerated here. Landscape requires five non-defaulted columns — name, indication, therapeutic_area, targets, companies (ogur/models/landscape.py:8-12) — and ScopeSpec supplies none of the first three. Left to prose, two callers would invent two different sets of defaults for the same landscape. The create request therefore carries name and therapeutic_area explicitly, and the rest derives:
Landscape column |
Value on creation |
|---|---|
id |
the caller's validated slug |
name |
from the create request — required, non-empty; not derivable from ScopeSpec |
therapeutic_area |
from the create request — required, non-empty; not derivable from ScopeSpec |
indication |
scope.indications[0], or "" when the scope names none. The legacy singular field ADR-0001 Phase 4 retires; it is written for compatibility and never read by a discovery run |
indications |
json.dumps(scope.indications) — the field that actually carries the boundary |
targets |
json.dumps(scope.targets) |
companies |
"[]" — the column's own comment says "populated after first ingestion", and a run has ingested nothing at draft |
modalities |
json.dumps(scope.modalities) |
horizon |
scope.horizon |
scope_type |
"modality_target" |
min_target_count, source_filters |
from the frozen config |
Requiring name and therapeutic_area on the request is the honest resolution: they are analyst-facing labels, not scope facts, and inferring them from a thesis string would put an LLM guess into a row every Monitor path reads. This adds two fields to T1.6's POST /api/explore/runs schema.
- A re-run attaches and does not rewrite the row's scope fields. Landscape.targets / modalities are read by every Monitor path and by class resolution; a re-run silently widening them would change class resolution for every other consumer. The Landscape row is the pool identity; the run row is the scope of one execution. The single exception is last_discovered_at, set on packaged, which is already what that field means (ogur/models/landscape.py:31).
- landscape_class is resolved once at draft and frozen on the run row. This is ADR-0006 Decision 3's rule ("the caller declares the class; the endpoint never infers it") applied one layer down: otherwise a run whose scope modalities drift from the Landscape row's would resolve one class at creation and a different one at execution.
- Once a run leaves draft, its frozen config is the only source of run policy. ClassificationPolicy reads min_target_count off the run context's landscape config (classes/base.py:246-259). If the CLI rebuilds that config from the Landscape row instead of from DiscoveryRun.config, that is a third instance of the stored-but-unread bug, in the field that has already caused it twice.
Decision 8 — Projection into the Monitor spine is one-way, post-packaged, two writes, and reads record_* only¶
First, the framing this decision must not violate — Monitor's schema is a projection target, not a design constraint (founder, 2026-08-25).
"The monitor backend is not fixed. It's something we implemented for a demo, so we should not use those abstractions as really the backbone of everything in Explore: we should really make sure that we have the right logic, the right abstractions in Explore to be able to make reproducible runs end to end. Here, we should start keeping in mind that it needs to be projected into monitor, but let's not over-index on the current monitor structure."
This is a constraint on how the no-second-spine rule is satisfied, and the two are easy to confuse. "No second spine" says an entity question must have one answer; it does not say Monitor's current columns are the right shape for anything. CompanyProfile, DrugProfile and Target were built for a demo, and MVP-2 is not the milestone that reworks them.
Three rules follow, and the rest of this decision is written to obey them:
- Explore's own tables are designed for reproducible runs on their own terms — run identity, frozen config, per-hit provenance, cross-run diff keys. Not one column in Decision 1–4 exists because Monitor wanted it. If Monitor's schema were deleted tomorrow,
DiscoveryRun/DiscoveredEntity/DiscoveredHitwould be unchanged. - Projection is an adapter at the boundary, and all the impedance-matching lives in it — the lowercasing, the precedence rules, the tier and mode gates. That is why it is a separate
project_run()step (below) rather than something packaging does: when Monitor is reworked, the adapter is what changes, and Explore's tables do not. - Monitor's current column semantics are not adopted as Explore's semantics. Where Monitor's write path is wrong today (it is — see the precedence rules below), the fix belongs to whoever reworks Monitor. Explore compensates at the adapter and records the defect; it does not perform surgery on a shared demo-grade helper to make its own projection safe.
The practical consequence is the reverse of what an earlier draft chose, and it is recorded in the precedence-rule section below.
The test that matters is not "are there two tables holding company rows" — of course there are. It is: can a feature asking "what companies do we know about?" get two different answers depending on which table it reads?
No, and it is enforced by one line of store contract: every read helper over discoveredentity and discoveredhit takes run_id as a required argument. There is no list_all_entities(). These tables are one run's evidence ledger, never queryable as entity truth. CompanyProfile / DrugProfile / Target / DrugTarget remain the only tables anyone can ask "what do we know" of. That is the structural no-second-spine guarantee, and it costs nothing.
Direction: one-way, discovery → spine, never back. The spine is never read to populate discovery tables. A bidirectional sync means both are authoritative and neither is.
Timing: a separate explicit step after packaged — project_run(session, run_id), its own function, its own CLI verb, its own tests — not folded into packaging. Two arguments:
- The two gates differ. Packaging's gate is the pack contract plus the citation gate. Projection's gate is Monitor's quality bar. The committed runs are the case: Polygon is 1 core / 274 honorable-mention / 112 off-thesis by the run's own tiering, and GNS561's v3 company file is 38 core / 355 honorable-mention / 79 off-thesis. The pack legitimately reports all of them, tiered — that is the honest output. Monitor must receive a small fraction. Fusing the two makes pack honesty and spine cleanliness the same decision, and one of them has to lose.
scripts/seed/build_target_graph.py's two-pass rule is the standing precedent: nodes keyed by canonical HGNC symbol and weighted edges built only from high-confidence derived rows, never bootstrapped from raw signal strings.
Projection has TWO layers with different rules, and an earlier draft of this ADR collapsed them (founder, 2026-08-25: "there is no existing database in monitor mode, so we would have to actually recreate it. The projection is going to be the recreation of that database, so porting this covered hit to a signal and so on… like merging the entity spine.").
The structural fact that makes them different, verified: Signal carries a landscape_id; CompanyProfile, DrugProfile, Target, DrugTarget and DrugSynonym carry none. The document layer is per-landscape. The entity spine is global — one row is visible to every landscape in the deployment.
| Layer 1 — documents | Layer 2 — entity spine | |
|---|---|---|
| Tables | Signal |
CompanyProfile, DrugProfile, DrugSynonym |
| Scope | per-landscape (landscape_id) |
global, shared by every landscape |
| Rule | breadth — the whole document layer, no tier gate | product-mode + core-tier + unique_evidence >= 2 |
| Why | it is the landscape's Monitor database; nothing else can see it | a bad row is visible everywhere and cannot be un-merged |
| Precedent | scripts/seed/pack_replay.py — already does exactly this from a frozen pack |
build_target_graph.py's two-pass rule |
The earlier draft applied Layer 2's bar to everything, and that was wrong. For a rebuilt landscape there is no pre-existing Monitor database — Polygon has 1 core company out of 387, so a core-only projection would have produced an essentially empty workspace and called it a success. The whole document layer is what makes the workspace real; the tier gate exists to protect the shared spine, and applying it to a landscape's own private corpus protects nothing.
Layer 1 is not a new mechanism. pack_replay.py already replays a frozen pack into Signal rows for one landscape, keyed (content_hash, landscape_id), with an ownership marker and a refusal to touch any other landscape's rows. Projection's document layer is that same operation sourced from a live run instead of a committed pack, and T1.2 should reuse its shape rather than invent a second one.
| Discovery row | Spine target | Rule |
|---|---|---|
DiscoveredHit |
Signal |
Layer 1 — projected, in full, with no tier gate. The run's backing Signal rows are persisted under its landscape_id; this is the recreation of the landscape's Monitor database. Idempotent on (content_hash, landscape_id). |
DiscoveredEntity(kind="company") |
CompanyProfile |
Projected iff mode == "product" and scope_fit_label == "core" and unique_evidence >= 2 (see the gate note below). normalized_name = entity_key.strip().lower() — the spine's normalization, not the discovery one — display_name fill-if-missing only, last_signal_at = most_recent_date, therapeutic_areas unioned with the existing value (never assigned). |
DiscoveredEntity(kind="asset_code") |
DrugProfile + DrugSynonym |
Same gate, plus the code must resolve to a generic name — an unresolved code does not project at all. See the resolution gate below. |
| any entity | any spine row | The projection gate is unique_evidence >= 2, not source_diversity >= 2 — see immediately below. |
| anything | Target |
Not projected — targets are not discovered (Decision 5), and Target has its own writer. |
record_targets |
DrugTarget |
Not projected — see below. |
The projection gate counts documents, not sources — because "seed" is a source. source_diversity is len(sources), and SeedEntity.materialise() does entity.sources.add(self.source) with source="seed" (classes/base.py:906). So an entity that was seeded and then found by exactly one real channel carries sources == {"seed", "clinicaltrials"}, scores source_diversity == 2, and clears a >= 2 source gate on the strength of a single citable document. Excluding seeds from unique_evidence alone does not close that — the gate has to read the field that excludes them.
Chosen: the projection gate is unique_evidence >= 2 counted over distinct document families, not over distinct (document_source, document_id) pairs. Seeds are NULL and cannot contribute, so a seeded entity needs two real documents and a seed-only entity scores 0. This is the reading of founder ruling N4 that the substrate rule implies: the document is the unit, so the corroboration bar belongs on documents.
Why families rather than raw pairs, and why this could not stay deferred. One physical paper reaches the corpus through more than one channel — the same PMID via PubMed and via OpenAlex — and a raw pair count scores that as two documents. The gate would then authorise an irreversible spine write on the strength of a single paper, which is exactly what "two distinct citable documents" was meant to exclude. Open question 2 below flagged the ambiguity but left it unresolved, and that was tenable only while nothing consumed the number; making it the projection gate consumed it, so the gate resolves it.
The collapsing rule is not invented here — it is the company-grade evidence gate's existing one: {openalex, pubmed} is a single literature family, and two hits differing only by channel count once. Where two documents share a canonical external identifier (the same PMID, the same DOI, the same publication number) they are one document regardless of source. Where no canonical identifier exists, the (document_source, document_id) pair stands as its own family.
If family collapsing cannot be implemented reliably, projection does not ship — and the reason is investigated, not shrugged at (founder-accepted, 2026-08-25). Keeping projection disabled is strictly preferable to a gate one paper can satisfy: projection is not on this milestone's critical path, and a wrong permanent spine row is not recoverable.
But "we could not collapse them" is a finding about the corpus, not a scheduling outcome, and it must not be allowed to pass as one. If collapsing fails, the question to answer is why the same physical document is not recognisable across channels — is the canonical identifier absent from one source's records, is it present but unparsed, or is the identity genuinely ambiguous? Each answer implies different work, and all three are worth knowing regardless of projection: the same question decides whether a document count anywhere in the product is a count of documents or a count of records about documents. Tracked as open question 2.
unique_evidence as a column keeps its raw distinct-pair meaning for the roster and the reproducibility line, where "how many source records backed this" is the honest question. The gate reads the family-collapsed count. Both are named at every call site, because this is the third number in this ADR whose meaning depends on who is asking.
Not chosen: keeping source_diversity >= 2 and subtracting "seed" from it at the gate. One number would then mean "sources for the roster" in ADR-0005 Decision 4 and "corroborating sources for the spine" here, and the next reader has no way to tell which is meant at a call site. source_diversity keeps its ADR-0005 meaning unchanged and stays the roster filter; it is simply not the projection gate.
The projection key is the spine's normalization, not discovery's — the two are different and must be composed. normalise_entity("AstraZeneca") returns "AstraZeneca", a display form with case preserved (verified: "ASTRAZENECA, INC." → "AstraZeneca", "Eli Lilly and Company" → "Eli Lilly"). Monitor keys CompanyProfile on name.strip().lower() (scripts/seed/build_company_profiles.py:81), and consumers lowercase before lookup. Writing normalized_name = entity_key verbatim would therefore insert AstraZeneca beside the existing astrazeneca row rather than merging into it — two spine identities for one company, with the therapeutic_areas union landing on the wrong one, and irreversible under Decision 9.
So the projector lowercases: normalized_name = entity_key.strip().lower(), and supplies canonical_name as the human form. Stated as a rule, because it generalises past this one field: entity_key is discovery's identity and is never a spine key as-is. Any future projection target applies that target's own normalization on the way in.
display_name is fill-if-missing, not assign — the same defect as therapeutic_areas, in the field next to it. upsert_company_profile assigns any non-empty incoming display_name unconditionally (ogur/store/companies.py:32-33), so projecting an entity whose canonical_name is AstraZeneca would overwrite a Monitor-curated AstraZeneca PLC — a silent downgrade on an irreversible write. Unlike therapeutic_areas there is nothing to union: two display strings for one company are alternatives, not a set. Chosen: discovery fills display_name only when the existing row has none. An existing label always wins, because it may be curated and discovery's is derived. Like therapeutic_areas, this is enforced in the projector, which reads the current row and simply does not supply a display_name when one is already set — not by changing the shared helper.
The wider lesson, since this is the second field in one function to be found this way: every column the projector writes needs an explicit precedence rule — assign, union, or fill-if-missing — and "the helper already handles it" is not one. The remaining fields the projector touches are last_signal_at, which upsert_company_profile already guards with a newer-wins comparison, and nothing else.
The asset-code resolution gate — an unresolved code does not project. DrugSynonym maps a synonym to a normalized generic name (ogur/models/drug.py:6-11: "MK-3475" → "pembrolizumab"), and DrugProfile.normalized_name is that same generic name and is unique=True. A DiscoveredEntity(kind="asset_code") carries only its code key, a display form and raw aliases — there is no generic-name field on it and no resolver in this schema. Committed artifacts are full of bare codes with no generic name attached, so writing DrugProfile.normalized_name = canonical_name.lower() would treat the code as the generic name — precisely what the previous wording claimed to avoid — and would mint a duplicate profile the moment the real generic name is learned, under a unique column that then has two rows for one drug.
Chosen: resolve, or refuse — with normalise_code applied to both sides of the comparison. An exact-string lookup does not work, because the two normalizations are genuinely incompatible: normalise_code strips punctuation (verified: "MK-3475" and "MK 3475" both → "mk3475") while existing DrugSynonym writers lowercase and retain it, storing "mk-3475". Matching either the entity's key or its first-arrived raw form against the stored string therefore misses a synonym that is genuinely present, and refuses projection over a formatting difference.
Resolution compares normalise_code of the entity's key and of each alias against normalise_code(DrugSynonym.synonym). On a hit, project_run writes the DrugProfile under the resolved normalized_name and adds the code as a synonym if absent. On a miss the entity does not project. It stays in discoveredentity, tiered, and is served through the pack.
DrugSynonym has no normalized column, so T1.2 normalises in Python over candidate rows rather than in SQL. A persisted synonym_normalized column would make this an index lookup and is the obvious follow-up — but it changes a shared Monitor table for a discovery-side need, so it is open question 7 rather than a decision here.
Not chosen: projecting under the code as a provisional profile to be renamed later. DrugProfile.normalized_name is unique=True and is the FK target for DrugTarget.drug_name, so a rename is a multi-table migration, and the provisional row is indistinguishable in the UI from a real one in the meantime. This is the same bar as the honorable-mention refusal: a row we cannot name is a row we cannot show. Refusing to project loses nothing that is not already in the pack.
therapeutic_areas must be unioned, and upsert_company_profile does not do that today. The helper is a non-null field filter, not a value merge: at ogur/store/companies.py:42-43 a non-empty incoming value is assigned — existing.therapeutic_areas = profile.therapeutic_areas — while recent_deals three lines below it is genuinely merged ((new_deals + existing_deals)[:20]). Projecting one core company through the helper as-is would therefore erase therapeutic areas learned from Monitor ingestion or from another landscape, and because projection is deliberately irreversible (Decision 9) there is nothing to restore them from. That is silent cross-landscape data loss on the shared spine.
Chosen: the projector composes the merged value and the shared helper is not modified. project_run reads the current row, unions therapeutic_areas, applies fill-if-missing to display_name, and passes values that are already correct — so the helper's assign writes the right thing without its behaviour changing for anyone else.
Why, and this reverses an earlier draft of this ADR. The earlier version extended upsert_company_profile itself, on the reasoning that it fixes the clobber for every writer including Monitor's own ingestion. That reasoning is sound in isolation and wrong under the framing above: it treats a demo-grade helper as the place where Explore's correctness lives, and it spends this milestone's changes on a write path that is expected to be reworked. Explore compensates at the adapter; Monitor's defect stays Monitor's to fix. The mechanical cost is one extra read inside project_run, in a step that already runs once per core entity.
Not chosen: extending the shared helper (above — it couples Explore's correctness to code scheduled for rework, and changes Monitor's behaviour from a discovery-side ADR); and passing a bare record_indications value into the assigning field, which is the data-loss bug itself.
The Monitor-side defect is recorded, not silently absorbed. upsert_company_profile overwrites therapeutic_areas (:42-43) and display_name (:32-33) for every caller, so Monitor's own ingestion discards data today, independently of anything discovery does. That is a real bug with a real owner, and it belongs to the Monitor rework rather than to T1.2. It is listed under "What may not be claimed yet" so closing this ADR does not read as having fixed it.
Why DrugTarget edges do not project, even though it is the tempting one. record_targets is a literal scope-entry match over record text — a co-occurrence. DrugTarget.evidence_count is what crowding queries rank on, and upsert_drug_target increments it on every re-assert (ogur/store/targets.py:94-110), so edges minted from co-occurrence inflate the busiest targets first. Projecting them would violate the very two-pass precedent this decision cites.
Only record_* crosses the boundary, and the projector asserts it rather than commenting it. The obvious mapping linked_indications → CompanyProfile.therapeutic_areas writes the query's indication onto every company that query touched. types.py:174-186 says exactly this would happen: "an acute-MI-scoped query would otherwise confer that disease on every record it surfaced." It is Monitor corruption by construction, invisible until a client opens a company profile and finds a firm tagged with a disease nobody claimed.
Monitor is temporal, so a dated backing Signal is a necessary condition for projection (founder question, 2026-08-25 — verified against the code).
The premise is correct, and the mechanism is stricter than "Monitor likes dates". ChangeDetector.detect() windows on event_date and drops anything without one (ogur/engine/detector.py:55-63, comment verbatim): "Cycle window must use event_date (when the event happened in the world), not detected_at (when we ingested it). On a fresh seed every detected_at clusters in a few minutes… Signals without event_date are dropped from the window: detected_at fallback would defeat the whole point."
Two consequences, and the second is a correction to this ADR rather than a confirmation of it.
1. The date exists on the hit, and it is the right one. DiscoveredHit.event_date is inherited from Signal.event_date at every projection site (sources.py:428, :484, :632), and DiscoveredEntity.most_recent_date is its max. So the field the question asks about is already there and already carries world-time rather than ingest-time. It is nullable because sources genuinely fail to supply it (_parse_iso_date and friends return None), and an undated hit is invisible to change detection by the rule above — so most_recent_date IS NULL on a projected entity is a defect to surface, not a blank to render.
2. A live run persists no Signal rows at all, and that breaks the projection. Verified: the discovery loop calls neither upsert_signals nor session.add anywhere (sirna_autonomous_discovery.py and discovery_modality/ — zero hits). backing_by_hash is accumulated (:385, :431), returned (:583), used for deck scoring, and discarded at process exit.
That is survivable on the --corpus db path, where the Signals already exist in the database and the hits merely index them — which is the only case the earlier "not projected" wording considered. It is not survivable for a live run: the Signals never existed, so a company projected into CompanyProfile has no Signal rows behind it, ChangeDetector has nothing to detect, and the entity is a name in a list that can never generate a briefing line. It enters Monitor's entity layer and is absent from its temporal layer.
Chosen: persisting the run's backing Signal rows is Layer 1 of projection itself, owned by the run (T1.3). upsert_signals dedupes on (content_hash, landscape_id), so it is idempotent and a no-op on the --corpus db path. project_run then refuses a Layer 2 entity whose hits resolve to no persisted Signal, and one whose most_recent_date is NULL, for the same reason it refuses an unresolved asset code: a row Monitor cannot keep up to date is a row Monitor should not show.
The temporal contract, stated rather than deferred¶
Monitor is a loop — harvest, compare against what is already stored for the landscape, surface what is new. A discovery run is a snapshot. Three properties make the snapshot compose with the loop, and each is a consequence of a choice already made rather than new machinery.
1. Backfilling a landscape's whole document layer does not manufacture a briefing. This is the property that makes Layer 1 safe to run at full breadth. ChangeDetector windows on event_date, and its own comment says why (detector.py:55-63): "On a fresh seed every detected_at clusters in a few minutes, which would otherwise" produce label artifacts. So persisting a corpus spanning years surfaces only the genuinely recent documents; the rest are stored, queryable and cited, without appearing as this week's changes. Had the detector windowed on detected_at, projection would announce every historical document as new, and the first briefing after a rebuild would be worthless.
2. A second discovery run IS a refresh cycle — that is what keeps a projected landscape current. Re-running on the same landscape persists its Signals; (content_hash, landscape_id) drops everything already stored; what remains is exactly "new stuff out there that needs to be added". It is the same dedup key, the same detector and the same briefing path Monitor's own ingestion uses. Explore's re-run and Monitor's refresh are not two mechanisms that need reconciling; they are one mechanism reached by two entry points.
3. The loop closes through the Landscape row. Decision 7 writes the run's scope into it — targets, modalities, indications, horizon, scope_type. Monitor's seeders read select(Landscape) to decide what to harvest. So a landscape a discovery run created is harvestable by Monitor's ordinary ingestion afterwards, with no discovery involvement: the run bootstraps the landscape, Monitor's cadence maintains it, and a later run adds whatever the cadence's fixed source list cannot reach. This is what "portability to the monitor workspace" means concretely, and it is why Decision 7 refuses to let a re-run rewrite those scope fields — they are the harvest instruction, not a record of what one run happened to search.
CompanyProfile.last_signal_at is monotonic (companies.py:48-50 keeps the newer value), so re-projection advances recency without losing history.
What this does not give us, stated because it is the honest residual: nothing schedules anything. There is no watcher and no cron in this milestone — Monitor's refresh is a command someone runs. The composition above is a property of the data model, not a running system, and a projected landscape goes stale exactly as fast as every other landscape does today.
Not chosen: having project_run mint Signal rows itself. Those rows would not have gone through ingestion, and the run — which holds the harvest, the landscape stamp and the manifest — is the honest owner. Nor: projecting undated entities on the theory that a date can be backfilled; nothing backfills it, and the detector would silently skip them forever.
Eval never projects — the answer-key leakage of Decision 2. Honorable-mention never projects — Polygon's 274 HM companies would become 274 low-confidence CompanyProfile rows in a spine whose rows the UI renders as fact. mvp-scope.md §3.4 already makes this exact call for the trial-inspection pane: company chips "would mostly open panes on rows we ourselves flagged marginal — a client-facing quality problem that building the pane does not solve." The symmetry is the argument: if a row is too marginal to open a pane on, it is too marginal to become a profile. HM rows live in discoveredentity, tiered, and are served through the pack. That is what these tables are for.
Net: Layer 1 is the landscape's whole document corpus; Layer 2 is two writes (CompanyProfile, DrugProfile + synonyms) gated hard. Breadth where it is private, a high bar where it is shared.
Decision 9 — Deletion order under unenforced foreign keys, and why the spine is untouched¶
Foreign keys in this repo are documentation, not enforcement — there is no PRAGMA foreign_keys=ON, and ogur/models/feedback.py:23-24 states the rule for every FK: "Like every FK in this repo it is documentation, not enforced — resolve it explicitly." Deletion order is therefore the caller's job, dependents first, parent last.
delete_run(session, run_id). The three DB deletions are one transaction; the filesystem step is not in it, and pretending otherwise is how a retained run ends up pointing at artifacts that are already gone:
- DB transaction —
DELETE FROM discoveredhit WHERE run_id = ?, thenDELETE FROM discoveredentity WHERE run_id = ?, thenDELETE FROM discoveryrun WHERE id = ?. Dependents first, parent last. - Filesystem, strictly after the commit — once the DB transaction has committed, delete
artifacts_dirbest-effort. A startup reconciliation purges any artifacts directory whoserun_idhas no row.
The ordering is commit-first, and rename-before-commit is wrong. An earlier draft renamed to a tombstone before committing and restored "on rollback" — but a crash between the rename and the commit runs no Python rollback handler at all: SQLite rolls the uncommitted deletes back by itself, the run row survives, and its artifacts directory is already gone. That is precisely the retained-run-pointing-at-nothing case the ordering was supposed to prevent, reintroduced in the window the draft did not consider. Commit-first has no such window: a crash after the commit leaves an orphan directory with no run row pointing at it — recoverable garbage that the startup sweep collects — and a crash before the commit leaves everything untouched.
artifacts_dir is its own column, distinct from manifest_path. T1.7 writes one manifest JSON inside a run-artifacts directory that also holds the run's other artifacts, while manifest_path names that one file. Deleting only the named path would leave every sibling behind. Both are relative to the configured artifacts root.
Containment is checked against the operation that actually runs — the recursive delete. An earlier draft attached the guard to a rename, and when the rename was removed the sentence stayed behind, leaving the only remaining filesystem operation — a recursive directory removal driven by a persisted, mutable column — unguarded. A stored value of ../../other-data would then delete unrelated data outside the artifacts root. delete_run therefore resolves artifacts_dir and asserts it is a strict descendant of the configured artifacts root before removing anything, refusing and failing loudly otherwise. The same assertion guards the startup reconciliation sweep, which walks the same column.
Stated as the general rule, because the draft history shows how it goes wrong: a containment check belongs to the destructive call, not to whichever operation happened to be specified when it was written. Change the operation, re-derive the guard.
run_id is the ownership marker for these three tables — no other producer ever writes a row carrying one. That is precisely what landscape_id is not, and why scripts/seed/pack_replay.py:34-38 refuses to delete by it: "the pack is authoritative for the rows it owns (the raw_data.seed_pack marker), and ONLY those: deletion is scoped to marker-carrying rows, never to landscape_id." The rule here:
Delete by
run_id. Never bylandscape_id. There is nodelete_runs_for_landscape()helper — deleting a landscape's runs means listing them and deleting each by id, so the caller has enumerated what it is destroying.
A Landscape row may not be deleted while runs reference it. FKs are off, so that is an explicit query in whatever deletes landscapes.
Deleting a run does nothing to rows it projected into the spine. Four reasons:
- The row is not solely the run's.
upsert_company_profilewritesnormalized_namekeys that Monitor's own ingestion also writes, field by field; deleting the row would remove another writer's data. - The merge is lossy. There is no per-field provenance on
CompanyProfile, so "un-merging" is not a defined operation. - The
pack_replaymarker pattern is structurally unavailable. That pattern works becauseSignalhas araw_datablob to carry a marker.CompanyProfile,DrugProfileandTargethave no such column. There is nowhere to write ownership, so ownership cannot be claimed, so it cannot be revoked. - The categories differ. A
DiscoveredHitrecords that this run saw this document — meaningless once the run is gone. ACompanyProfilerow is a claim about the world; deleting the run does not make the company stop existing. This repo already models assertions as accumulate-only:DrugTarget.evidence_countincrements on re-assert and never decrements.
The cost, stated: a bad run's projected rows survive its deletion, and there is no unproject_run(). The mitigation is the projection gate in Decision 8 — product-mode, core-tier, unique_evidence >= 2 — not the delete. Removing a bad projection is a Monitor-side data fix. See open question 4.
Architecture¶
The three tables, in house style. ogur/models/discovery.py, imported in create_tables().
DiscoveryRun¶
class DiscoveryRun(SQLModel, table=True):
"""One bounded, metered discovery execution. See ADR-0007."""
id: str = Field(default_factory=lambda: str(uuid.uuid4()), primary_key=True)
# --- Identity / scoping -------------------------------------------------
landscape_id: str = Field(index=True) # human slug; FK by documentation only
mode: str = Field(index=True) # "eval" | "product" — REQUIRED, no default
landscape_class: str # registry name, frozen at draft (ADR-0006 D3)
created_by: str = Field(index=True) # User.id — the run's OWNER, not "who decided"
supersedes_run_id: str | None = Field(default=None, index=True) # conditional re-run chain
# --- Status machine (Decision 6) ----------------------------------------
status: str = Field(index=True, default=STATUS_DRAFT)
status_history: str = Field(default="[]") # JSON list[{status, at}] — what, never who
failure_reason: str | None = Field(default=None, index=True) # closed set: FAILURE_*
failure_detail: str | None = None # free text, <=500 chars; never asserted on
gate_report: str | None = None # JSON object; set iff conditional | refused
# --- Scope + config, frozen at approval ---------------------------------
scope: str # JSON object — ScopeSpec.to_dict()
scope_digest: str = Field(index=True) # sha256(canonical scope json)[:16]
config: str = Field(default="{}") # JSON object — the frozen RunConfig record
config_digest: str = Field(index=True) # sha256 of the canonical `config` blob —
# the RUN's frozen policy, not the class's
class_digest: str = Field(default="", index=True) # LandscapeClass.config_digest() —
# adapter identity only; NOT run-distinguishing
alias_dataset_version: str = Field(default="") # pins entity_key stability
record_evidence_version: str = Field(default="") # pins the record_evidence vocabulary
run_fingerprint: str = Field(default="", index=True) # DiscoveryRunContext.fingerprint()
# --- Budget + cost (T1.5) -----------------------------------------------
budget_usd: float = 0.0 # per-run ceiling, fixed at scope_approved
spend_usd: float = 0.0 # reservations INCLUDED; replaced by actuals at settle
cost: str = Field(default="{}") # JSON object — CostMeter snapshot incl. per-source
# timings and source_failures()
# --- Progress (Decision 6: not transitions) -----------------------------
progress: str = Field(default="[]") # JSON list — one RoundResult per round
rounds_completed: int = 0
entities_found: int = 0
hits_found: int = 0
# --- Artifacts (T1.7) ---------------------------------------------------
artifacts_dir: str | None = None # RELATIVE to the artifacts root — the DIRECTORY;
# what delete_run deletes AFTER the commit (Decision 9)
manifest_path: str | None = None # RELATIVE — the manifest FILE inside artifacts_dir
manifest_digest: str | None = Field(default=None, index=True)
# --- Liveness + timestamps (naive UTC, house convention) ----------------
runner_token: str | None = None # minted on -> running; a re-claim invalidates it
heartbeat_at: datetime | None = None
created_at: datetime = Field(default_factory=datetime.utcnow)
scope_approved_at: datetime | None = None
started_at: datetime | None = None
finished_at: datetime | None = None
last_updated: datetime = Field(default_factory=datetime.utcnow)
No __table_args__. There is deliberately no uniqueness constraint on a run. A re-run against the same landscape with the same config is exactly what terminal conditional exists to permit; uniqueness on (landscape_id, config_digest, scope_digest) would forbid it.
Two column choices worth their space:
spend_usdis a float column, not a key insidecost. T1.5 admits every LLM call againstmin(run budget remaining, identity daily ceiling remaining). Computing "remaining" by parsing a JSON blob per call is slow and, decisively, non-atomic — the exact defectogur/store/spend.py's reserve-then-settle exists to avoid, whose docstring explains that concurrent handlers "could each read an under-ceiling total and all pass a plain check". One float column, incremented in SQL, mirroringLlmSpend.usdsemantics.configstays a blob. T2.1 owns the RunConfig schema. Giving its variation points columns now would pin a schema a later ADR has not written.config_digestis a column because caches and Gate C comparability key on it.config_digesthashes the frozenconfigblob, notLandscapeClass.config_digest()— and the two are separate columns. The class digest covers adapter identity only, and per-landscape policy does not enter it:min_target_countandsource_filterslive on the run's config, so two runs with genuinely different frozen policy would share one class digest. ADR-0006 open question 4 records the worked case — RNAI'sconfig_digestis byte-identical at46776eaaf6e3across both its landscapes, which is correct for a class digest and wrong for a run digest. Keying Gate C comparability or any cache on it would treat two non-equivalent runs as the same run, which is the exact property Gate C exists to provide.class_digestkeeps the adapter identity under its own name for anyone who needs it.artifacts_dirandmanifest_pathare both relative. The artifacts root is a setting; an absolute path breaks the moment a client volume mounts elsewhere, and it turns the deletion step into a traversal risk. They are separate columns because the manifest is one file inside the directory — see Decision 9.
A standing rule attaches to cost: a pack emitted from a run whose CostMeter.source_failures() is non-empty is invalid unless those failures appear in its reproducibility section. types.py:315-321 records why — a degraded-empty source is indistinguishable from an honest zero, and that gap once hid an OpenTargets schema break behind a clean zero while "the run reported it as a finding about the biology."
DiscoveredEntity¶
class DiscoveredEntity(SQLModel, table=True):
"""A merged, alias-normalised entity — an AGGREGATION of DiscoveredHit
document edges, never a primary record. See ADR-0007 Decision 3."""
run_id: str = Field(foreign_key="discoveryrun.id", primary_key=True)
entity_key: str = Field(primary_key=True) # the merge key: normalise_entity(raw) for a
# company, "code::<normalise_code(raw)>" for an
# asset code. NOT canonical_name — see D3.
kind: str = Field(index=True) # closed set: VALID_ENTITY_KINDS
canonical_name: str = Field(index=True) # display form; NOT the identity
aliases: str = Field(default="[]") # JSON list[str], SORTED — raw surface forms
sources: str = Field(default="[]") # JSON list[str], SORTED
first_round: int = 0
# Harvest attribution — INCLUDES the concept query. Never evidence, never projected.
linked_targets: str = Field(default="[]") # JSON list[str], sorted
linked_indications: str = Field(default="[]") # JSON list[str], sorted
# Record-only union — the query excluded. Evidence-grade; the ONLY fields the
# Monitor projector may read (Decision 8).
record_targets: str = Field(default="[]") # JSON list[str], sorted
record_indications: str = Field(default="[]") # JSON list[str], sorted
co_entity_groups: str = Field(default="[]") # JSON list[list[str]] — RAW, pre-normalised
scope_fit_label: str | None = Field(default=None, index=True) # core | honorable-mention |
# off-thesis
scope_fit: str | None = None # JSON object — the class's full scope-fit detail
# Rollups. append_hits() is the ONLY writer; a caller setting these is a bug.
source_diversity: int = 0 # len(sources) — the roster filter and sort key
unique_evidence: int = 0 # COUNT(DISTINCT (document_source, document_id));
# seed hits are NULL there and do not count
most_recent_date: datetime | None = None
projected_at: datetime | None = None # set iff this row graduated into the spine
created_at: datetime = Field(default_factory=datetime.utcnow)
last_updated: datetime = Field(default_factory=datetime.utcnow)
DiscoveredHit¶
def compute_hit_hash(source: str, document_id: str, entity_key: str,
query_axis: str, query_text: str, round_num: int) -> str:
"""Deterministic identity of one harvest event. Mirrors
compute_evidence_content_hash (ogur/models/evidence.py:9-24).
run_id is deliberately NOT an input: the hash is the CROSS-RUN diff key
(ADR-0007 Decision 4). Uniqueness is on (run_id, hit_hash).
document_id is the seed's signal_ref on a seed hit, where the document
triple is NULL — so two distinct seeds in one run cannot collide."""
parts = [source, document_id, entity_key, query_axis, query_text, str(round_num)]
return hashlib.sha256("|".join(parts).encode()).hexdigest()[:16]
class DiscoveredHit(SQLModel, table=True):
"""One (document, query) edge asserting an entity — THE substrate layer."""
__table_args__ = (
UniqueConstraint("run_id", "hit_hash", name="uq_discoveredhit_run_hash"),
)
id: str = Field(default_factory=lambda: str(uuid.uuid4()), primary_key=True)
run_id: str = Field(foreign_key="discoveryrun.id", index=True)
entity_key: str = Field(index=True) # -> discoveredentity.entity_key, SAME run
hit_hash: str = Field(index=True) # compute_hit_hash(...); run_id NOT included
# --- The document: the citable anchor (Decision 4) ----------------------
# NULL on a seed hit (source == "seed") and ONLY there; fully populated on
# every harvested hit. The store raises in BOTH directions. A NULL triple
# never counts toward unique_evidence and can never project.
document_source: str | None = Field(default=None, index=True) # Signal.source — BACKEND label
document_id: str | None = Field(default=None, index=True) # Signal.source_id — NCT/PMID/pub no
signal_content_hash: str | None = Field(default=None, index=True) # Signal.content_hash — the join
# --- The claim ----------------------------------------------------------
entity_raw: str # PRE-normalisation surface form
entity_kind: str # must equal its entity's kind
source: str = Field(index=True) # harvest channel: clinicaltrials|sec|openalex|lens|epo|...
query_text: str # <=1000 chars — validated, never truncated
query_axis: str = Field(index=True)
round: int = Field(index=True)
support_text: str # <=300 chars — a DISPLAY snippet, already cut by _support()
event_date: datetime | None = None
# The three linkage families are deliberately NOT collapsed (types.py:174-199):
# linked_* — INCLUDES the query string. Harvest attribution; wrong as evidence.
# record_* — the full record text ALONE, matching literal scope entries.
# record_evidence — the class's own judgement, in ITS OWN vocabulary. Stored
# opaque; identified by DiscoveryRun.record_evidence_version.
linked_targets: str = Field(default="[]") # JSON list[str]
linked_indications: str = Field(default="[]") # JSON list[str]
record_targets: str = Field(default="[]") # JSON list[str]
record_indications: str = Field(default="[]") # JSON list[str]
record_evidence: str = Field(default="[]") # JSON list[str] — opaque, class-scoped
co_entities: str = Field(default="[]") # JSON list[str] — RAW co-applicant names
created_at: datetime = Field(default_factory=datetime.utcnow)
Naming: the table classes keep the contract names¶
DiscoveredEntity and DiscoveredHit already exist as frozen dataclasses in ogur/engine/discovery_modality/types.py:153-231. Table classes with the same names shadow on import in any module that holds both — which is exactly the store conversion layer.
Chosen: keep the contract's names, and mandate that ogur/store/discovery.py is the only module that imports both, aliasing the value objects (from ogur.engine.discovery_modality.types import DiscoveredHit as HitValue). One documented import rule, in one file.
Not chosen: renaming the tables DiscoveryEntity / DiscoveryHit. It is cleaner and prefix-consistent with DiscoveryRun, and it would let any module import both freely — but it silently renames the objects the task contract, the plan and the Linear issues all name verbatim, and a schema whose names disagree with its own decision record is a worse trade than one import rule.
Reserved — the post-gate confirm_hypothesis slot¶
Not implemented in this milestone. No code, no column, and no state exists for it today.
The design it reserves: an LLM enumerates the programs a domain expert would expect in the landscape (the prior), and a deterministic gate decides which of them may be named — entity-anchored search → citable URL → two-factor name check. Source disposes; the model only proposes. Product-mode only.
What is reserved: a named section, one status-machine non-change, and zero columns. Not hypothesis: str | None, not confirmed: bool, not a confirming status.
Two arguments, the second being the real one. First, a nullable column left NULL by every row for an entire milestone is dead data, and this repo has the min_target_count history to show what dead data costs. Second — the migration cost does not justify a placeholder, because the shape is not yet known. The additive-column mechanism (_LANDSCAPE_ADDITIVE_COLUMNS, ogur/store/database.py:45-50) makes adding a nullable column later cost about six lines, so "avoiding a migration" is not a reason. And the likely correct shape is its own table — a run proposes many programs, so a column on the run row would be the wrong shape. Reserving the wrong shape is worse than reserving none.
The reservation is made instead by demonstrating that the tables decided above already express the gate's output, and by naming the one thing that would not fit:
modeis the guard, and it exists and is enforced on day one. A6 needs no new guard. This is the concrete return on deciding theeval | productfield now rather than later.- A confirmed program produces ordinary rows. It becomes a
DiscoveredEntity(kind="asset_code")with itsDiscoveredHitrows, or it does not exist. No new kind, noprovisionalflag — the closed set in Decision 5 is already the extension point, under its own rule that a kind arrives with its producer. - The "entity-anchored search → citable URL" leg consumes
(document_source, document_id). The gate's output is a document; a result that cannot be written as that pair fails the gate by construction. This is exactly why Decision 4 puts the citable pair on the hit rather than storingsignal_refverbatim. - The two-factor name check is already a query over these tables. Factor one: the proposed name appears in the record text of a hit whose document is citable. Factor two: a second, independent
(document_source, document_id)asserts the sameentity_key— that isunique_evidence >= 2computed over distinct documents, which Decision 3 already makes a column. Writing the query shape down is what makes this reservation checkable rather than aspirational. - The one thing that would genuinely need new schema is the refused hypotheses — programs the model proposed that the gate would not name. Those are, by construction, entities with no hits, which these tables cannot represent, since an entity is defined as an aggregation of hits. The future table is
HypothesisProposal; it is post-gate; it is not created here.
No new status. confirm_hypothesis runs inside synthesizing — it decides what may be named in the pack. A confirming state would change a machine T1.4 is about to test through every transition, for a feature nobody is building at this milestone.
What may not be claimed yet¶
- Nothing here is implemented. Unlike ADR-0006, this is a forward decision, not a ratification. T1.2 builds the tables and store helpers; until then no run has ever been persisted.
- Monitor's crowding view will not see discovery-only programs. Decision 8 refuses to project
DrugTargetedges fromrecord_targets, so a target–asset relationship found only by discovery stays invisible toget_drugs_by_targetuntil a higher-grade source confirms it. That is a deliberate loss, taken to protectevidence_countfrom co-occurrence inflation. - Projection is not reversible. There is no
unproject_run(), and Decision 9 explains why one cannot be written honestly against spine tables that carry no provenance marker. modehas two live consumers, not four. The served-roster andconfirm_hypothesisrows in Decision 2's table are future work.- There is no liveness guarantee, only a bounded-staleness one.
heartbeat_atis a lease: a runner that heartbeats and dies immediately readsfreshuntilSTALE_AFTERelapses. The API reports recency (heartbeat_age_seconds,fresh | stale), neverlive. Asserting liveness needs a supervisor, which this milestone does not have. - This ADR does not decide the RunConfig schema.
configis a blob andconfig_digestis its key; T2.1's ADR owns what goes inside. upsert_company_profile's overwrite bug is NOT fixed by this ADR. It discardstherapeutic_areas(ogur/store/companies.py:42-43) anddisplay_name(:32-33) for every caller, so Monitor's own ingestion loses data today, independently of discovery. The projector compensates for its own writes only. The fix belongs to the Monitor rework; closing this ADR does not close that.- Projection may not ship. It depends on collapsing one physical document arriving through several channels. If that cannot be done reliably, the decided outcome is projection disabled — and the reason is an open investigation, not a closed question.
- A live run does not persist its backing
Signalrows today, so nothing projected from one would be visible toChangeDetector. Making the run persist them is T1.3 work and is Layer 1 of projection; until it lands, projection is only meaningful on the--corpus dbpath. - Nothing here schedules anything. The temporal contract in Decision 8 is a property of the data model, not a running system: there is no watcher and no cron in this milestone, and Monitor's refresh is a command someone runs. A projected landscape goes stale exactly as fast as every other landscape does today.
Consequences¶
- T1.2 is mechanical. The field blocks in Architecture are the implementation; the store helpers it needs are
create_run,transition_status,upsert_entity,append_hits,persist_cost_progress,delete_run,sweep_stale_runs,verify_entity_rollups, plus mode-required read helpers. - T1.2's stated upsert key changes from
(run_id, canonical_name)to(run_id, entity_key), and its test asserts idempotency on an asset-code entity presented under two surface forms — the case the original key fails. - T1.3 must add a defaulted field to the frozen
DiscoveredHitdataclass to carry(document_source, document_id), and must stop treating an unrecognisedentity_kindas an asset code. - T1.3 must persist the run's backing
Signalrows viaupsert_signals— idempotent on(content_hash, landscape_id), a no-op on the--corpus dbpath. Todaybacking_by_hashis built and discarded at process exit (sirna_autonomous_discovery.py:385, :431, :583), which leaves anything projected from a live run invisible to Monitor's change detection. project_runrefuses an entity whose hits resolve to no persistedSignal, and one withmost_recent_date IS NULL.ChangeDetectorwindows onevent_dateand drops undated rows (detector.py:55-63), so an undated or Signal-less entity can never change in Monitor's eyes.- T1.2 does NOT modify
upsert_company_profile.project_runreads the current row and passes already-merged values — union fortherapeutic_areas, fill-if-missing fordisplay_name. Explore compensates at the adapter; the shared helper's overwrite bug is Monitor's to fix during its rework (Decision 8's framing). - T1.6's
POST /api/explore/runsschema gainsnameandtherapeutic_area, becauseScopeSpeccannot supply either andLandscaperequires both. - T1.6 resolves a submitted slug against three namespaces, not one —
Landscaperows,served_landscape_ids(), and existingUserLandscapeAccessgrants — and mints a grant only for a slug new to all three. Checking only the table is an authorization bypass, because a granted, served landscape can have noLandscaperow. - T1.4 stamps
heartbeat_aton entry to every swept state, so NULL in a swept state is an invariant violation rather than a reading, and a fresh run is testable before its first metered call returns. - T1.6 reports
heartbeat_age_secondsandfresh | stale, neverlive. The weaker word is the accurate one; the stronger one would put an untestable guarantee in the API contract. - T1.2 computes
config_digestfrom the frozenconfigblob, withclass_digestholdingLandscapeClass.config_digest()separately. Keying either Gate C or a cache on the class digest would treat two runs with different frozen policy as the same run. delete_runcommits the DB transaction before touching the filesystem, and a startup reconciliation purges artifact directories whose run row is gone. The reverse order has a crash window that leaves a live run pointing at deleted artifacts.- Run creation is not no-leak. It discloses slug existence to an authenticated caller. Not exposed at this milestone (creation is operator-side); self-service creation must pre-grant slugs or accept the disclosure.
- The projection gate needs family collapsing, and without it Layer 2 does not ship. A raw
(document_source, document_id)count lets one paper reaching the corpus through two channels satisfy a two-document bar. Disabling the spine write is the correct fallback — it is off the critical path, and a wrong permanent spine row is not. Layer 1 is unaffected: the document corpus is per-landscape and needs no corroboration gate, so a rebuilt landscape still gets a real Monitor workspace even if Layer 2 is disabled. - T1.2 reuses
pack_replay.py's shape for Layer 1, rather than inventing a second landscape-scoped Signal writer. Same(content_hash, landscape_id)identity, same ownership marker, same refusal to touch another landscape's rows. - Every column the projector writes carries an explicit precedence rule — assign, union, or fill-if-missing — and the projector applies it before calling the shared helper. Two fields in
upsert_company_profilewere found by review to silently overwrite curated Monitor data; "the helper already handles it" is not a precedence rule. - A Monitor rework changes the adapter, not these tables. No column in Decisions 1–4 exists because Monitor wanted it; every impedance-matching rule lives in
project_run. That is the point of making projection a separate step. delete_runresolvesartifacts_dirand asserts strict descendance of the artifacts root before removing anything, and so does the startup reconciliation sweep. The column is persisted and mutable, so an unguarded recursive delete driven by it reaches outside the root.- Asset-code entities that do not resolve to a generic name do not project, so the projected drug set is a subset of the packed one. That is intended, not a coverage bug.
entity_keyis never a spine key as-is. Discovery's normalization and the spine's are different functions —normalise_entitypreserves display case, Monitor keys on.lower()— so every projection target applies its own normalization on the way in. Skipping it forks the company into two spine rows.append_hits()must use the subquery form of the distinct-document count. The row-valued form does not execute on SQLite, so a literal reading of the earlier wording would have failed on the first insert.- T1.3 should also correct two stale comments in
types.pythat this ADR had to work around:signal_ref's description (:169), which no call site matches, and the reference tobase.VALID_RECORD_EVIDENCE(:189), which does not exist. - A new mode-aware read helper without a probe fails a test, by design, so the
min_target_countfailure cannot recur silently in this field. - Adding an entity
kindis a schema decision with a written bar: a producer must emit it in the same change. - Deleting a landscape becomes an explicit multi-step operation, since FKs are off and there is no delete-by-landscape helper.
Open questions (deferred)¶
- Cross-run entity identity.
entity_keyfor a company isnormalise_entity(raw), which reads the alias layer, so two runs under differentalias_dataset_versioncan key the same firm differently. "The same company across two runs" is only answerable within one alias version. Whether a run-independent company identity is needed is unresolved — and answering it carelessly is precisely how a second spine gets built. - Which
unique_evidencedenominator — resolved for the gate, open for the rest; and why collapsing might fail is a live investigation. Decision 8 settles the projection gate: it counts document families, so one PMID arriving via PubMed and OpenAlex is one document. The investigation the founder asked for (2026-08-25): if the four committed corpora turn out not to support reliable collapsing, establish why rather than recording it as a scheduling outcome — canonical identifier absent from a source's records, present but unparsed, or genuinely ambiguous identity. Each implies different work, and the answer decides something wider than projection: whether any document count in the product is a count of documents or a count of records about documents. What also stays open is the rest of the surface — the refinement layer counts distinct(source, signal_ref)and the assembler counts distinctsignal_ref, and they coincide today only becausesignal_refis a content hash that folds the source in. Which denominator the coverage numbers use, and whether the two should be unified, is unresolved. It was tenable to defer this only while nothing consumed the number; anything else that starts consuming it must resolve it the same way, in the same PR. - Whether
DiscoveredHitshould referenceSignalat all. Every hit is built from aSignaltoday, sosignal_content_hashalways resolves. A future source producing a document with noSignal— a fetched full text, a press-release body — breaks that. Whether the hit table becomes the primary document record or stays an index overSignalis deferred. - Projection reversibility.
CompanyProfilehas noraw_data, so thepack_replaymarker pattern is unavailable and there is no way to un-project. Whether the spine tables should grow a provenance marker is deferred. - Retention. Nothing deletes runs automatically. Roughly 10³–10⁴ hit rows per run, accumulating on a client volume that also holds the live database, is a growth curve nobody has priced. Named here so the first person to notice does not invent a policy on the spot.
- Auditing who decided. Decision 6 records what was decided, not who. If per-checkpoint attribution becomes a requirement, the shape is an append-only
RunEventtable, not columns on the run row. - Should
DrugSynonymcarry a normalized key? Resolution currently normalises in Python over candidate rows because the table stores punctuation-bearing lowercase synonyms andnormalise_codestrips punctuation. Asynonym_normalizedcolumn would make it an index lookup, but it changes a shared Monitor table to serve a discovery-side need — deferred rather than decided here. - Sub-runs. A later task contemplates two scope axes sharing one entity table, which the
(run_id, entity_key)composite key forbids. Either sub-runs collapse into one run row with an axis discriminator on the entity, or the key widens. Cheap to name now, expensive to discover during a migration.