/coverage-eval skill — contract

Project-local Claude Code skill at .claude/skills/coverage-eval/SKILL.md. Invoked with /coverage-eval [landscape-id] [--mode seed|discover|drug].

Why this exists

Coverage measurement is a cadence, not a one-off. archived_data/ already carries three generations of the same dump — sirna_coverage_dump_20260615 / 0619 / 0625.csv (+ the matching _deals_ dumps) — run so the numbers can be diffed over time. Running the harness is the easy part; the repeated mistakes are:

1. Reporting a number without provenance. A coverage figure is only meaningful with its benchmark, corpus (db vs live), and matcher state attached. A run once read 26/32 assets and was corrected to 18/32 — 8 were phantom matcher binds (company-token matches with no target backstop). The fix wasn't a better harness; it was attaching provenance and a phantom-bind guard to the number.
2. Snapshot instead of delta. The dated dumps exist to be compared. The interesting output is "which assets newly recovered / regressed since the last run," and that diff was done by hand each time.
3. Moving the number by touching the matcher. Coverage is only comparable across runs if scripts/utils/sirna_match.py / scripts/eval/sirna_discovery/scoring.py are unchanged. "Improving coverage" by loosening the matcher manufactures phantom binds — the same /new-source boundary (feed the corpus, never fork the matcher), applied to the eval side.

/coverage-eval makes the cadence reproducible: pick the right harness, attach provenance, diff against the prior dated artifact, and refuse to call a number a delta when the matcher moved.

The contract — three modes, one credibility gate

Mode	Harness (stable, on main)	Reads	Dated artifacts
seed	scripts/utils/inspect_sirna_seed.py	ogur.db + deck CSVs	sirna_coverage_dump_<date>.csv, sirna_deals_coverage_dump_<date>.csv
discover	scripts/eval/sirna_autonomous_discovery.py "<thesis>" --corpus db\|live	deck CSVs + ogur.db / live APIs	sirna_discovered_{assets,companies}_<date>.csv, sirna_gap_analysis_<date>.md
drug	scripts/eval/signal_drug_coverage.py	ogur.db	stdout + optional evals/signal_drug_coverage.json

The credibility gate runs in every mode: a number is reportable only with (benchmark + denominator, corpus, matcher-unchanged?, date) attached, and is a delta only when the matcher matches main.

Baked-in invariants

Invariant	Why	Enforced where
Every number carries provenance (benchmark/denominator, corpus, matcher state, date)	A bare 26/32 was wrong and undetectable without it	report format
Matcher unchanged ⇒ comparable	Loosening the matcher manufactures phantom binds	git diff main -- sirna_match.py scoring.py
Never overwrite a prior dated artifact	The time series is the asset being built	UTC %Y%m%d stamp, artifacts.py
Resolve the full gold deck or stop	A partial deck gives a wrong recall denominator	OGUR_GAP_ASSETS_CSV / _DEALS_CSV resolution
Phantom-bind guard	Company-token match w/o target/code backstop ≠ recovery	sirna_match.py target backstop

Scope

• In scope: running the coverage/discovery harnesses, attaching provenance, diffing dated artifacts, and the gap partition (Pattern A/B/D) read-out.
• Out of scope: extractor-quality evals (entity/outcomes F1 vs gold.jsonl — a different family), editing the matcher to change a number, and building the benchmark/landscape the eval runs against.

Stability note (read before relying on internals)

The scripts/eval/sirna_discovery/ package is under active multi-workstream development (WS1–WS4: reproducible-fetch, context-capture, CDE/patents-by-target feeds). This skill deliberately encodes only the stable, committed-on-main entry points and artifact contracts (inspect_sirna_seed.py, signal_drug_coverage.py, artifacts.py, scoring.py) and invokes the discovery harness through its CLI, never its in-flight projector internals. The discovery CLI is additive-only (new optional flags like --no-cde), so the CLI surface is safe; the internals are not. Revisit once WS1–WS4 land.

How it differs from running the script

	Running the harness	/coverage-eval
Output	A fresh dated dump	The dump + a delta vs the prior dump
Trust	A bare number	A number with provenance + a phantom-bind guard
Comparability	Implicit	Explicit: delta only if the matcher matches main

When to invoke

• On the coverage cadence (after a reseed, after a new source lands, before/after a matcher-or-source PR) — to get the tracked delta, not just a number.
• Right after /new-source ships a source: /coverage-eval is how you prove the named assets it claimed to recover actually moved the benchmark.

Future versions

• v2 — emit the delta itself as a committed archived_data/coverage_delta_<date>.md so the time series includes the diffs, not just the snapshots.
• v2 — a --landscape generalization once a second closed-world benchmark exists (the deck resolution is currently siRNA-specific).
• Pairs with /new-source: that skill proves recovery once at onboarding; coverage-eval re-measures it on the cadence.

Implementing rule of thumb

Before reporting a coverage number, ask: could I hand this fraction to someone next month and have them reproduce and compare it? If it doesn't carry its benchmark, corpus, matcher state, and date, the answer is no — and the number isn't done.