The first end-to-end run of the MODS personalization machinery on real inputs: where we actually are, the decisions to fix first, the critical-path issues, the runbook, and the checks that decide whether the machinery works.
34 min read
Smoke run #1 — real PRS → clean foundation → personalized drug ranking
Tracking issue:#439. Label:critical-path (every issue that blocks this run carries it, together with p0). Written: 2026-09-27 with Claude Opus 5.5, from a code, DB and artifact audit of work built mostly on Opus 5. Every number below was measured that day; re-verify any of them before you rely on it.
Outcome (2026-09-29). The run finished on 2026-09-29. All ten machinery checks (§6) were measured and pass; two of them, M6 and M9, were borderline, and Richard accepted both. Gate F released 590 scores over 109 diseases as smoke_v0. With her RGCC drugs held out of training, the base ranker shows no RGCC signal (graded-only ρ +0.03 and +0.05 per draw, intervals spanning zero), and the released criterion does not help. The rest of this page is the plan as written on 2026-09-27, with dated updates. Results: the run’s writeup and The smoke run lands.
The one-line version. We have never pushed Maureen’s real data through the whole machine. We have also never done it without her validation labels leaking into the training graph. This run does it once, deliberately, and checks every joint. Proving that personalization helps is not the goal. Proving that the machinery does what we think it does is.
0. What this run is, and what it is not
It is a machinery test on real inputs. The chain is:
PRS released and normalized onto a disease axis, plus her expression values.
A PDR2 patient pickle built from them.
A criterion fine-tune on a foundation trained without her drug-response labels.
Raw and personalized drug scores, stored side by side.
Per-draw RGCC metrics.
It passes when every check in §6 passes.
It is not a test of whether personalization works. The released PDR2 criterion is degenerate, its sign is fixed, and it is blind to edge types (§1.4). A null or negative lift is the expected result. Report it; it is not a failure of this run. The science run is run #2 (§9).
It is pre-registered. Record the decisions in §3 as ticked boxes in #439 before any drug is scored. Look at no metric until the whole grid in §5.4 is computed.
1. Where we actually are (verified 2026-09-27)
Component
State
What that means for this run
PRS
3,752 scores computed, all quarantined, 0 released. z_score, percentile and disease_cui are NULL on every row.
Gate D and F must run, and scores need a disease key (§1.3).
Gate D
D1, D2 and D3 are built but have never run. Nothing computes ancestry PCs.
Needs a worker-image rebuild (#339).
Gate F
No code anywhere writes validity='released'.
A v0 release writer is on the path (#296).
Disease axis
The registered schema v1 has 6 diseases: AD, a coeliac score filed as RA, breast, T2D, prostate and ischaemic stroke. Colorectal cancer is not on it.
Axis v2 is on the path (#244, #430).
Foundation
916bdd2c, 2026-05-18, maureen_focus_v1. Trained on her RGCC and medication edges and on a split with a symmetry leak.
Must retrain (§1.1, §1.2).
Criterion
Expression enters unscaled. The weights are closed-form. Eval scales by raw weights where training uses normalized ones.
Fix the inputs (#435); measure the rest.
Personalized scores
Nothing produces them for a drug list. The worker has never completed a personalized eval.
#436
RGCC metrics (PDR2 path)
Still averages the two draws.
#437
Container
:latest is issue29-c2bdc44, built 2026-05-26, apparently from a working tree with uncommitted changes, so its exact contents are unknown. Five lib/pdr2 commits since then are probably not in it.
Rebuild from HEAD, record the digest.
pdr_status “critical path”
Seven May tasks: Phase 0 AD closure, PDR1 Track A CSV, embedder pickoff, …
Stale. Do not steer by it. This playbook and the label replace it.
1.1 The foundation memorized the validation labels
The foundation’s own split files are on disk at C:\Maple\runs\2026-05-18_maureen_focus_v1\data\. Her patient edges by split (query in Appendix A.1):
Predicate
train
valid
test
MAPLE:sensitive_to_drug
5
1
2
MAPLE:partially_sensitive_to_drug
35
18
14
MAPLE:resistant_to_drug
34
19
7
medication (past / current / tried)
45 / 9 / 8
18 / 4 / 4
16 / 2 / 2
74 of the 135 RGCC edges were in train.
Four of the five Tier-2 positives were memorized.
Oxaliplatin, cisplatin and mitomycin carried sensitive_to_drug in train.
Pemetrexed’s CHEBI edge was in test, but its PharmGKB twin, pharmgkb_variants:pharmgkb_drug:pemetrexed, was in train.
Carboplatin was the only held-out positive. That is why it sat at rank 260, the “outlier” MODS_MISSION_SPEC §5.3 blamed for suppressing Tier-1.
Re-scoring the published v2 scores per draw (Appendix A.2):
Draw
All drugs
Labels never in train
Labels in train
2025-02-15
ρ +0.115 (n=105)
ρ −0.051 (n=49, p 0.73)
ρ +0.363 (n=56, p 0.006)
2025-09-19
ρ +0.149 (n=105)
ρ −0.045 (n=49, p 0.76)
ρ +0.407 (n=56, p 0.002)
The PDR2 v2 RGCC signal was memorization. On the clean subset, graded-only, ρ is +0.22 / +0.28 (n=26), which is not significant.
Other drug-related leaks:
Medication history is circular for the response-episode surface. All 14 labelled response episodes are for drugs she has a tried_drug edge to, and 7 of those 8 drugs have a patient→drug edge in train.
The same drug can appear under two CURIEs. 22 held-out RGCC edges had the same agent in train under a sibling CURIE (oxaliplatin as CHEBI and as PharmGKB). A holdout must group by drug identity, not by CURIE.
The overlay erases the draw inversion. RGCC edges carry no draw date, and writes use ON CONFLICT DO NOTHING. All 12 drugs that moved 25 points or more between draws land in partially_sensitive.
1.2 The graph has a symmetry leak
PrimeKG stores most edges in both directions under one predicate, and the export splits per directed triple. 539,594 of the 890,384 test triples (60.6%) have their exact reverse in train (Appendix A.3). Training does not remove the mirror (remove_one_hop defaults to False), so the foundation learns a lookup, and every PrimeKG foundation MRR (0.56–0.59) is inflated by it. This is the FB15k / WN18 inverse-leakage pattern. The split is also not reproducible. Set order depends on an unset PYTHONHASHSEED, and the audit found two same-seed runs sharing only ~21% of their test sets. On top of that, every downstream handler re-exports and re-splits, and re-adds the full patient overlay while doing it (#433).
1.3 Real PRS cannot reach the model yet
resolve_prs_vector keeps only score rows with a disease_cui. Every row has none, so a released, normalized fleet still exports an empty PRS vector and the run silently becomes Track A (#430). The raw material for a disease axis is better than expected:
1,620 of the 3,753 eligible scores carry a MONDO trait, covering 245 diseases, and all 245 anchor in the maureen_focus_v1 slice. (Corrected 2026-09-28: not all. Basal cell carcinoma, MONDO_0020804, has no node in the composition under either spelling; smoke_v0 disease mapping, J10.)
173 of those diseases have more than one eligible score.
PrimeKG writes MONDO unpadded (MONDO:5148) while CIViC, ClinVar, OncoKB and RTX-KG2 write it padded. 14,504 unpadded disease entities have a padded twin.
In today’s export, the colorectal node that carries drug edges is MONDO:0005575 “Colorectal Cancer”: 10 treats, 2 contraindicated_for, 2 has_off_label_use(export lines, with each drug–disease pair written in both directions: 5 / 1 / 1 distinct drugs; smoke_v0 disease mapping §5, 2026-09-28). PrimeKG’s MONDO:24331 carries only gene associations.
1.4 The criterion, as released
The released loss (losses.py, paper_code) is base + λ₁·tanh(ŵ·prs) + λ₂·tanh(v̂·expr), where ŵ and v̂ are L2-normalized. Neither term sees the batch, and the forward pass runs under no_grad, so:
The weights are closed-form. The optimum is ŵ = −PRS/‖PRS‖ and v̂ = −expr/‖expr‖. The fine-tune reads the input vector back out; it learns nothing from the graph.
The sign is fixed. Every disease term w_d·p_d ≤ 0 and grows with p_d². Drugs adjacent to her most extreme-risk diseases, high or low, get pushed down.
Expression never trained. It arrives as Caris TPM percentiles (4–100), unscaled, so tanh saturates from step 0. All 5 past fine-tunes ended with protein weights at their random-init scale. The “top protein weights” in earlier writeups are initialization.
Eval and training disagree about scale. Eval multiplies raw weights where training normalizes them, and it counts each adjacency 2–4 times depending on mirror and inverse structure.
The adjustment is edge-blind. An indication edge and a contraindication edge push the same way. That is Phase 12 (#420).
This run fixes the inputs and the eval/train mismatch (#435). It deliberately keeps the released loss, and measures the degeneracy (check M6 in §6). Phase 12 is the science fix.
1.5 Silent-failure modes still in the code
A failed checkpoint load continues on a random model (lib/pdr2/code/script/model.py, #434). A smoke run could “pass” on a random foundation. Fixed 2026-09-27 (#434): a failed load now raises.
Bare debug_log calls in the PDR handlers raise NameError. One of them sits between the Vertex submit and saving the job id, so a retry can double-submit an A100 job (#398). Fixed 2026-09-27 (#398).
Rebuilding the schema without --version overwrites v1 in place.Fixed 2026-09-28 (#497): --version is required, and an existing version is refused without --overwrite.
Releasing is one-way today.store_score_result never supersedes a released row, so a later re-score leaves old released values being served (#296’s writer needs a demotion path). Fixed 2026-09-28 (#296): the v0 release writer can demote a release (release_gate_f.py --demote).
The worker reloader used to hard-kill in-flight tasks on any .py change under C:\Maple, a git pull included (#361). It now drains instead: on a change to a module the worker imported (tests are ignored) it finishes the current task, then restarts. A task still running after WORKER_RELOAD_DRAIN_TIMEOUT_SECONDS (default 6 h; 0 = no deadline) is still killed. It has been off by default since 2026-09-28 (WORKER_AUTO_RELOAD=0). Don’t turn it back on during a pass.
1.6 What is in good shape
Gate A, B and C are sound and verified: every E2 score reconciles, and none sits outside its theoretical bounds.
D1, D2 and D3 exist with fixture tests.
The HGDP+1kGP panel is staged and md5-verified.
The export reads normalized PRS and refuses raw sums (#297).
The worker → Vertex path exists for every mode.
The foundation retrain costs ~21 A100-hours, roughly $120.
2. The run at a glance
flowchart LR
subgraph P["Lane P — PRS"]
P1["#339 rebuild workers"] --> P2["E3 default policy<br/>‖ D1 panel freqs"]
P2 --> P3["D2 on v0 subset (#431)"] --> P4["D3 EUR reference (#431)"] --> P5["Gate F v0 release (#296)"]
end
subgraph A["Lane A — disease axis"]
A1["mapping approved (#430)"] --> A2["axis v2 registered (#244)"] --> A3["pickle, Track B"]
end
subgraph G["Lane G — clean graph"]
G1["#432 export fixes<br/>#433 handler reuse<br/>#434 load must fail"] --> G2["container rebuild + digest"] --> G3["leak-free export + manifest"] --> G4["foundation retrain (~21 h A100)"]
end
subgraph R["Lane R — scoring & metrics"]
R0["#398 #435 #436 #437"] --> R1["fine-tunes λ grid"] --> R2["raw + personalized scores"] --> R3["per-draw metric grid"] --> R4["checks M1–M10, annotate top 30"]
end
P5 --> A3
A2 --> A3
A3 --> R1
G4 --> R1
G2 --> R1
Lanes P, A, G and R’s code work can all proceed in parallel. They converge at the fine-tune.
3. Decisions to fix before running (pre-registration)
Tick each one in #439 before anything is scored. The recommendations are mine, and each is argued from the audit.
#
Decision
Recommendation
Why
D1
What the base ranking queries
Disease query: score (drug, biolink:treats, <CRC anchor>) for every candidate drug, then add the personalization adjustment.
A foundation trained without her RGCC edges cannot score MAPLE:sensitive_to_drug at all: the relation is absent from the vocabulary, so score_edges returns missing_predicate. The disease query also isolates the thing we want to test, whether her genetics re-ranks medicines, on a base that holds no patient data. Patient-node queries (Mechanism A) need in-context K-fold scoring and move to run #2.
D2
What the foundation export excludes
RGCC MAPLE:*_to_drug (required), plus medication history (tried_drug, current_medication, past_medication). Keep expression, variant and phenotype edges.
This matches the design of record: train on her orthogonal biology and hold drug responses out. Medication edges make the response-episode surface circular (§1.1).
D3
Graph for the retrain
Retrain on the current canonical graph, with #432’s export fixes. Settle #438 (drug–protein roles, wrong drug merges, MONDO twins, PharmGKB synthetic drugs) before run #2.
Those fixes need re-ingest and mapping decisions under the CLAUDE.md hard rules. The smoke run should not wait on them, but the run report must list them as known limitations.
D4
PRS scoring policy
E3 re-score under the current default policy.
v0 releases only scores with zero palindromic rows and no strand-flip candidacy, so its values do not depend on the pending palindrome, EAF-route or strand-flip decisions (#422, which follows). E3 is still needed: the E2 rows store a policy dict that ScoringPolicy.from_dict refuses, so D2 cannot rebuild it.
She is essentially entirely European. Pooled all-ancestry normalization would bias every z by the EUR-vs-world difference. Ancestry PCs (#429) are the better long-term fix.
D6
Gate F v0 release rules
Adopt #296’s R1–R8 (additive weight scale; inside bounds; ≥1 scored locus; structural overlap ≥ 0.90; zero palindromic rows; complement-only ≤ 5%; callability ≥ 0.90; MONDO anchor in the slice), plus the post-D3 requirements. Tag smoke_v0, record the waivers, build a demotion path.
Leaves 738 scores over 149 diseases, all verified-scale and orientation-safe. (2026-09-28: R8 measured 736 / 148; D7’s exclusions then left 594 / 109, and Gate F released 590.) Waived and recorded: the pgsc_calc witness, callability-floor sign-off, the NR-scale policy (excluded, not decided), and the ad-hoc 0.90 thresholds.
D7
Disease axis v2
MONDO traits only, keyed by the anchor in the foundation’s vocabulary. For v0, exclude PrimeKG grouped anchors, HP-only traits, EFO disease-like traits and multi-trait scores. One score per disease by a recorded rule (most scored loci, today’s reader rule). Required index disease = the CRC anchor that carries drug edges. Register as v2; never overwrite v1.
Every choice here is an ontology mapping, so propose the table and get approval (CLAUDE.md). Excluding grouped and multi-trait cases keeps v0 to unambiguous mappings.
D8
Expression channel
Keep v1’s 84-gene axis. Probit z = Φ⁻¹(p/100), clipped to ±4. Use the most recent colorectal specimen (2025-09-13, today’s resolver behaviour).
The WTS gene axis (~36k genes, #212) is a schema change and a separate experiment.
D9
Criterion and λ
Released paper_code loss, edge-blind adjustment, target domain drug. Track B at λ ∈ {0, 0.01, 0.1, 1.0}, plus one Track A control.
λ=0 proves personalized equals raw. The Track A v2 sweep showed λ ≤ 0.1 is below the noise floor and λ ≥ 1 is destructive, so the grid brackets both. Run each λ as its own fine-tune, because the pipeline treats λ as a training-time setting. Under the released loss, ŵ should come out identical for every λ > 0, and that is itself a check (M6).
D10
Tier-2 definition
RGCC’s categorical high_sensitivity (5 per draw) against < 35%. Report n and bootstrap CIs on every cell.
Five positives is fragile; print n every time.
Two further choices are the implementer’s and must be recorded rather than decided silently: the CRC anchor, confirmed against the new export’s vocabulary, and the candidate drug set (all RGCC-resolvable drugs plus every drug entity in the vocabulary).
4. The critical path (label critical-path)
The label is the list. Each issue says whether it is agent-fixable or human-led, and human-led issues carry the phrase the fixer’s eligibility predicate reads, so the unattended fixer leaves them alone.
Lane
Issue
What
Depends on
Who
Size
P
#339
Rebuild the worker image (pgenlib, zstandard, scipy); verify all 10 workers
—
human (ops)
S
P
#431
Gate D smoke options: EUR-only reference; explicit D2 score list
—
agent
S
P
#295
Gate D umbrella: run D1 → D2 → D3
#339, #431, E3
human (runs)
M (compute)
P
#296
Gate F v0 release writer + release
#295, #430
human
M
A
#430
PRS rows need a disease key; preflight refuses silent Track A
Handlers reuse the foundation’s exact split; never re-add the overlay
#432
human
M
G
#434
A failed checkpoint load must raise
—
agent
S
G
#214
Leak-free validation design (umbrella)
#432, #433
human
—
R
#398
Bare debug_log NameErrors in the PDR handlers
—
agent
S
R
#435
Criterion inputs: expression z, normalized eval weights, adjacency once
—
human
S–M
R
#436
Raw + adjustment + personalized scores for any candidate set
#435
human
M
R
#437
Per-draw RGCC metric grid, stored
—
agent
S–M
R
#114
The run itself
everything above
human
—
—
#439
Tracking issue (decisions, runbook checklist)
—
human
—
Deliberately not on this path (they come right after it; §9):
#420 Phase 12 signed criterion.
#422 final-policy E3: the palindrome decision, strand-flip set, EAF gate, and knob plumbing.
#429 ancestry PCs.
#428 D2 knob tripwire, which is needed before #422’s first new knob.
#414 genome-wide palindromes.
#438 graph defects.
#324 restatement of the old numbers, including memorization.
#212 WTS gene axis.
#46 Cloud Build retag automation; a manual retag suffices here.
#361 reloader: off by default since 2026-09-28 (§5.0). #360 resume blindness: handled by procedure (--no-resume, §5.1).
5. Runbook
Every job goes through the worker queue or the research API, never raw gcloud, so each run lands in mods_research_run and the Training Cockpit.
5.0 Pre-flight and guardrails (before any long pass)
Worker reload is off.docker exec maple-worker-1 printenv WORKER_AUTO_RELOAD prints 0, the compose default since 2026-09-28. Then merges and pulls can go on during E3, D1, D2 or a Vertex handler; nothing needs freezing. A worker keeps the code it started with, so load merged code the way .claude/rules/worker.md describes. If it prints 1, recreate the workers from a C:\Maple that has the new default before starting.
Unattended fixer: pausing is optional. Stopping it (python scripts/fixer_agent.py stop, then python scripts/fixer_agent.py nightly off, from C:\Maple) was a workaround for #361: its nightly run starts with git pull in C:\Maple, which used to restart every worker. With reload off, that pull no longer restarts them.
Queue is clear. Nothing live in task_queue except the known stale rows: a pdr_checkpoint_eval claimed since 2026-05-27 (#313), and two March thread_embed rows.
SELECT id, task_type, status, created_at FROM task_queueWHERE status IN ('running','claimed','pending') ORDER BY created_at;
Disk. At least 20 GB free on C:. It had 49 GB on 2026-09-27; nothing on the path needs more than a few GB. Don’t stage the 13.2 GB .pgen container-locally unless space allows.
Memory. The Docker VM has 30.7 GB. E3 (2 × 8.45 GB) and D1 (~6 GB) fit together. D2 runs after both.
Credentials. The database credential is set in the shell that runs any docker-compose -f COMPOSE_FILE command, and gcloud auth application-default login is current.
5.1 Lane P — PRS to released, normalized scores
The Gate D and E drivers enqueue worker tasks, and they default to container paths (/app/backend/data/...). Run them inside a worker container the way E2 was: docker exec maple-worker-1 python /app/backend/scripts/prs/<driver>.py …. Never edit a .py in C:\Maple to launch something. Watch progress with a detached poller, not a foreground loop.
Flags marked (#431) do not exist until that issue lands.
Rebuild the workers (#339). Use the compose file that created them:
E3 re-score under the default policy (D4), in parallel with D1. E3 is estimated at ~4 h (E2 took 3 h 51 m) and D1 at ~2–2.5 h, which is also an estimate.
64a90788 is the v4-union callset (19,215,298 loci, 16,964,243 records, sha d76e1c65…) that all 3,752 live rows were scored against. The other Gate A runs are 6-locus sentinel builds; don’t pick one of those. Use --no-resume, because resume keys only on scorer version (#360).
Verify: 3,752 new quarantined rows under the new run_id; their stored policy includes palindrome_forward_scores and strand_flip_recovery_scores; E2’s rows are superseded; one gate_a.n_records value across live rows.
Compute the v0 candidate list. Take #296’s R1–R8 funnel over the E3 rows. R8 needs the approved disease mapping (§5.2 step 1, #430), so Lane A’s decision gates this step. The funnel should reproduce 594 scores over 109 diseases; if it doesn’t, stop and find out why. Write the ids to a file. (Updated 2026-09-28: D7’s mapping, approved in PR #460, applies its own exclusions after R8. R8 alone gives 736/148 on the E2 rows, not ~738/149: basal cell carcinoma has no node. Dropping multi-trait scores and grouped anchors then leaves 594/109.)
D2 on the subset (#431), then merge. Estimated ~4 h. Bound concurrency with a pipeline_config row for mods_prs_panel_score (enabled=1, max_concurrent=2; two D2 tasks fit the 30.7 GB VM), and delete the row afterwards, or the task type stays throttled forever.
no PolicyMismatch (the default --policy gate-c rebuilds her E3 policy);
panel coverage is recorded per score;
the merge wrote /app/backend/data/prs_panel/panel_scores.jsonl;
her missingness matches the panel’s: per score, compare d2.patient.dropped with d2.panel_dropped_mean (both as fractions of outcomes.matched). (Added 2026-09-28: on the first run they did not match (#499). She dropped a median of 2.3 points more, which moves her z on scores whose panel mean is far from zero: PGS005253 by about −5.5 SD.)
D2’s patient column is on by default (panel-aligned placement), which is what we want.
D3 against the EUR reference (D5, #431). Dry-run first:
Verify:percentile_v1 names the subset, with score_basis == panel_aligned, and z-scores are finite. Remember that percentile stays the pooled rank; the reader prefers z_score.
Gate F v0 release (#296, D6). The writer releases the v0 set, writes the disease anchor (#430), tags smoke_v0 with the rule version and waivers, and can demote.
Verify:mods_prs_manifest_stats(section='lifecycle') shows released equal to the v0 count, and resolve_prs_vector returns that many anchors.
5.2 Lane A — disease axis v2 and the patient pickle
The mapping table (#430, D7) is approved before any code writes it.
Register axis v2 (#244). Use python backend/scripts/build_kgia_feature_schema.py --version v2 --dry-run, then run it for real. --version is required, and an existing version is refused without --overwrite (#497). v2 registers inactive unless --activate is passed, so every pickle build names --schema-version v2. It is built from the released gate_f.axis_pick rows, so run it after the Gate F release. The CRC anchor (MONDO:0005575) must be on the axis; the build refuses otherwise, and so does every pickle build (#337, #497).
Pass --patient-id: the default lookup returns nothing when two patients exist. Never pass --allow-raw-prs-fallback.
Verify (M3): measured PRS positions > 0, all values finite, the .meta.json records value_field = z_score and the expression transform, and .resolution.json is present. Record the sha256 values.
Build the Track A control the same way, with --source synthetic_zeros.
Upload the pickles and sidecars to gs://BUCKET/patient_data/<date>_smoke1/.
5.3 Lane G — the clean graph and a new foundation
Merge the code first: #432, #433, #434, #435, #436 and #398. Then rebuild the PDR2 container from HEAD and record the digest:
The handlers read VERTEX_TRAINING_IMAGE=…:latest, so record the digest you retagged, and the one it replaced, in #439. Pass --ignore-file explicitly, or the upload balloons.
Leak-free export (#432, D2), reusing an existing dataset class name so no extra rebuild is needed. It is excluded by predicate: the three MAPLE:*_to_drug predicates plus, per D2, the three medication predicates.
Verify (M4):
grep -c "_to_drug\|_medication\|tried_drug" is 0 on train, valid and test;
the mirror check (A.3) returns 0%;
the manifest md5s reproduce on a second export.
Train the foundation through POST /api/mods/research/runs/submit. The schema forbids unknown fields; set these:
exclude_predicates is added by #432; its final field name is whatever that PR ships. The patient overlay stays in, so she exists as a node with her biology edges, but without drug edges.
Expect ~20–21 h on A100-80GB. Train-mode weights only reach GCS at job end, so keep save_every_n_epochs.
The train-mode post-training eval (entrypoint §6) calls eval_maple.pywithout--n_test_sample. The default is 0, which means the full test set. 916bdd2c’s 20.64 h appears to include that full eval, which is why timeout_seconds must stay well above 86,400. Plumbing N_TEST_SAMPLE into train mode would be a cheap follow-up.
Verify (M5):
the log shows Successfully loaded wherever a checkpoint is read;
the loss curve is sane;
the filtered MRR is recorded.
Expect it to be lower than 0.56, because the mirror shortcut is gone.
5.4 Lane R — fine-tune, score, measure
Fine-tunes (D9), one per λ ∈ {0, 0.01, 0.1, 1.0} on the Track B pickle, plus a Track A control at λ=0.01. Each goes through POST /runs/{foundation}/submit_finetune, with ckpt_uri set to the new foundation’s checkpoint and gpu_type: A100_80GB. Make sure FINE_TUNE_MAX_BATCH=128 reaches the job; the request schema has no field for it, so check the handler’s env. The pre-flight probe runs under no_grad and undersizes memory ~6×.
Verify (M6):
for every Track B λ > 0, cos(normalize(w_disease), −prs) > 0.5 with sign agreement ≥ 0.90, and the same for protein against expression; report the raw cosines. This proves the values reached the right positions. (Revised 2026-09-28 from > 0.99; see the M6 note in §6.)
ŵ agrees across λ > 0;
the Track A disease weights move only by AdamW weight decay (they get no gradient);
#433 reused the foundation’s split (manifest md5 on the child row).
Score the candidates (#436, D1) for each fine-tune: (drug, biolink:treats, <CRC anchor>) for every drug entity in the vocabulary. That includes every RGCC-resolvable drug, grouped by drug identity across CURIEs. Output columns: raw_score, adj, pers_score, raw_rank, pers_rank.
Verify (M7):
at λ=0, pers_score == raw_score on every row;
adj for 20 sampled drugs equals an independent recomputation from the pickle, the normalized weights and the export adjacency;
at λ ≥ 0.1, Kendall τ(raw, pers) < 1;
report which relation types carried the adjustment. How much of it flowed through contraindicated_for edges is the edge-blindness, measured.
Build the metric grid (#437, D10):
Axis
Values
Draw
2025-02-15, 2025-09-19
Panel
all-panel (n=112), graded-only (n=59)
Score
raw, personalized (per λ)
Drug set
all drugs, drugs unseen in training
Every drug should be unseen, since the RGCC edges were excluded; the check proves it. Report Tier-1 Spearman and Tier-2 AUROC, with n and bootstrap CIs in every cell, and print the test-retest figures.
Verify (M8): results are stored as mods_research_validation_result rows, one per draw × variant.
Annotate the top 30 (#114), personalized at the chosen λ, next to raw. Give each drug its known CRC indication status, its RGCC draw values, and the diseases whose PRS moved it. Label the whole page “smoke run: machinery validation, not a recommendation”.
Reasoning paths for the top 10(added 2026-09-29, Richard). For each of the ten, record the graph paths that drove its raw score. This is the evidence a reviewer would have to accept before taking the drug seriously. It is also the direct test of whether the ranking does more than recall known CRC drugs.
Method.
Use BaseNBFNet.visualize() in our fork (lib/pdr2/code/ultra/base_nbfnet.py). It takes the gradient of the score with respect to every edge, then beam-searches the highest-weighted paths (NBFNet’s interpretation method; defaults num_beam=10, path_topk=10).
Nothing calls it today. EntityNBFNet only gets its query and each layer’s relation inside Ultra.forward, so a thin wrapper must set both from the relation model first, exactly as forward does.
Running it needs a container mode, e.g. MODE=explain_paths. That is one more change to batch into the Lane R rebuild (§10). Submit it through the worker queue (M10).
Explain the exact score.
Use the same checkpoint, the same fact graph and the same query as score_edges.
visualize() skips negative_sample_to_tail, so pass the triple already turned: (CRC anchor, inverse of biolink:treats, drug).
Positive control: the score computed inside visualize() must equal the drug’s raw_score to float tolerance. If it doesn’t, the paths explain some other number: stop and fix the query.
Record per drug the top 5 paths with their weights, each hop as name, CURIE and predicate. Classify each drug’s strongest path:
direct: the drug’s own edge to CRC (expected for a known indication);
mechanistic: drug → target → gene or pathway → CRC;
analogy: drug → shared target or class → another drug → CRC;
personal: through her own node, via her expression, variant or phenotype edges (legitimate, since D2 keeps them);
hub: through a generic, very high-degree node.
Report the count in each class. If every drug’s support is direct or analogy, the ranking is recalling known CRC drugs. Mechanistic paths are what repurposing needs.
Limits.
The paths explain the raw score only. The personalization adjustment is added after message passing and has no path; the “diseases whose PRS moved it” note covers it.
The foundation is frozen, so the paths are the same at every λ. Compute them once.
Gradient paths show what drove the score, not whether the biology holds.
No path may use one of the six predicates D2 excludes. One that does means M4 missed a leak.
5.5 Closing the run
Record in #439 every run id, the image digest, the manifest md5, the release tag, the schema hash and the pickle sha256 (M9).
Write the results up in the house format (Purpose / Methods / Results / Discussion), including the null or negative lift and the measured degeneracy, and save a pdr_note_save decision.
Turn the fixer back on.
6. Machinery acceptance checks (the pass/fail)
#
Check
Passes when
M1
PRS released
released rows carry z_score, a disease anchor and the smoke_v0 tag. The count equals the v0 funnel. No released score sits outside its bounds.
M2
Axis
v2 is registered with a hash, v1 is untouched, and the CRC anchor is on the axis.
M3
Pickle
Measured PRS positions > 0 and equal to the axis diseases with a released score. All values are finite. Expression is z-scored. The sidecars are present.
M4
Export
0 lines with an excluded predicate. 0% of test triples have their reverse in train. The manifest md5s reproduce.
M5
Foundation
Every load logs Successfully loaded (#434 makes a failure fatal). Loss converges. The MRR is recorded, and is lower than before.
M6
Fine-tune
For λ > 0: cos(ŵ, −PRS) > 0.5 and cos(v̂, −expr) > 0.5, each with sign agreement ≥ 0.90, and ŵ agrees across λ. The raw cosines are reported. The Track A disease weights move only by weight decay.
M7
Personalized scores
At λ=0, personalized equals raw. The sampled adj equals an independent recomputation. At λ ≥ 0.1, τ < 1. The edge-type breakdown of the adjustment is reported.
M8
Metrics
Per-draw grid with n per cell. Every RGCC drug is flagged unseen. Test-retest is printed.
M9
Provenance
Every artifact links run ids, the image digest, the manifest md5, the release tag, the schema hash and the pickle sha256. Re-running one scoring job reproduces its scores exactly.
M10
Venue
Every job went through the worker queue and appears in the Training Cockpit. There were no raw gcloud submissions.
All ten pass → the machinery works. Then report the lift, whatever it is.
M6 revised 2026-09-28 (Richard), from cos > 0.99. The released loss cannot reach its own closed-form optimum ŵ = −p/‖p‖ at realistic scale. tanh saturates within about 50 steps, and the weights stall near −sign(p)/√D. Measured over 2,020 steps (PR #461):
Input
cos(ŵ, −p)
Sign agreement
Synthetic, D = 149, ‖p‖ = 12.2
0.657
0.926
Her 84-gene expression after probit
0.665
0.94
Scrambled null
−0.005
0.517
Synthetic, D = 20, ‖p‖ = 4.2
0.9997
1.0
The gate is now cos > 0.5 and sign agreement ≥ 0.90, both far outside the null. The shortfall from 0.99 is itself a finding about the released criterion, to report beside its fixed sign.
7. The fine details: every trap we know of, and the check that now catches it
Trap
Where it bit (or would have)
The guard
RGCC labels in the training graph
v2 foundation: 74 of 135 edges; 4 of 5 Tier-2 positives memorized
D2 exclusion + M4 grep + M8 unseen flag
Medication edges make the response-episode surface circular
62 medication edges in train
D2
Same drug under two CURIEs (CHEBI vs pharmgkb_variants:pharmgkb_drug:*)
22 held-out edges had a sibling in train
Group by drug identity (#437)
Mirror pairs straddle splits
60.6% of test triples
#432 mirror-grouped split + M4
Split depends on PYTHONHASHSEED; seed 42 vs 41504
Every past re-export
#432 deterministic split + manifest
Handlers re-export and re-add the overlay
Every fine-tune, eval and score job
#433 + manifest md5 on child rows
Two RGCC draws pooled
PDR2 metrics path; the overlay itself
#437 + per-draw grid
Overlay erases the draw inversion (no draw date on edges)
All 12 movers land in partially
Excluded from this run; fix before RGCC edges are ever used as context (run #2)
RGCC zeros rendered as NULL (falsy coercion)
The MCP tool before 2026-09-24
Query the table; #437 asserts 112 per draw
Phantom KRAS
Issue text (#104, #51), not data
She has no KRAS mutation; check mods_list_alterations
Melanoma WTS rows (the 2026 specimen)
Latent: NULL report_date sorts last today
Most recent colorectal specimen (D8); #212’s scoping fix
Random model after a failed checkpoint load
model.py fallback
#434 + M5
Stale :latest image of unknown content
May image, uncommitted code
Rebuild from HEAD, record the digest (§5.3)
Double-submitted A100 job
#398 site between submit and saving the job id
#398
Silent Track A (no disease key)
Every PRS row
#430 preflight + M3
Raw PRS sums saturate tanh
Would have, without #297
resolve_prs_vector refuses them; never pass --allow-raw-prs-fallback
Expression percentiles saturate tanh
All 5 past fine-tunes
#435 + M6
Eval uses raw weights where training normalizes; adjacency counted 2–4×
evaluate_personalized_maple.py
#435 + M7 recomputation
Criterion sign fixed and closed-form
The released loss
Measured by M6; fixed in #420 (run #2)
Edge-blind adjustment
indication = contraindication
Measured by M7’s edge-type breakdown; fixed in #420
Schema rebuild overwrites v1
build_kgia_feature_schema.py default, before #497
Fixed: --version required, existing versions refused without --overwrite (#497)
Her no-calls deflate her score but not the panel’s
D2 before #499
Compare patient.dropped with panel_dropped_mean per score (§5.1 step 4); #499
Released rows never superseded by a later re-score
store_score_result
Demotion path in #296’s writer
Unscored disease reads as exactly average risk (0.0)
Reference fill; the loss ignores the measured mask
The axis only includes diseases with a released score
Worker reloader kills long passes
Gate E task 1041499, 2026-09-24
WORKER_AUTO_RELOAD=0, the compose default since 2026-09-28 (§5.0)
Resume keyed only on scorer version
Would have blended E1 and E2
--no-resume (#360)
Checkpoints reach GCS only at job end
Train mode
save_every_n_epochs
eval_maple.py auto-runs a full 1-vs-all test (11–30 h)
Post-train eval
--n_test_sample 50000
Pre-flight probe undersizes memory ~6×
Fine-tune OOM at batch 256
FINE_TUNE_MAX_BATCH=128
BPE is a step count; batch changes change the training volume
PrimeKG drug–protein roles collapsed; wrong drug merges; MONDO twins; PharmGKB synthetic drugs
The canonical graph
Known limitation (D3); #438 before run #2
8. What we will be able to say afterwards, and what we won’t
If M1–M10 pass, we can say:
The machinery runs end to end on real inputs with no drug-response leakage, and every artifact is traceable.
Her PRS reached the model at the right positions.
The honest base-ranker Tier-1 and Tier-2 numbers per draw. These are the first clean RGCC numbers the project has had.
How much the released criterion moves the ranking, in which direction, and through which edge types.
For each of the top 10, which graph evidence drove its raw score, and how much of the list rests on recall (direct or analogy paths) rather than mechanistic evidence.
We will not be able to say:
That personalization helps. The criterion is degenerate and edge-blind; that question belongs to run #2.
Anything clinical. Single patient; RGCC is in vitro, fixed-concentration CTC data; five Tier-2 positives.
That a path’s biology is true. The paths show what drove a score, so they are hypotheses to review, not findings.
Anything about patient-node queries (Mechanism A). This run does not use them.
9. After the smoke run (run #2 and the rest)
Phase 12 (#420): a signed per-relation adjustment, plus a fine-tune whose signal depends on outcomes. Its parameterization, decision (ii), is Richard’s.
Final PRS policy:
#422: the palindrome forward set, strand-flip set and EAF gate, with the knob plumbing its comment describes; land #428 first;
#429: ancestry PCs;
the pgsc_calc witness;
then a release v1 that retires smoke_v0 through the demotion path. This is what “release most of the scores” needs.
Graph (#438): decide on the drug–protein roles, the bad drug merges, the MONDO twins and the PharmGKB drugs. Then re-export and retrain.
Mechanism A: drug-grouped K-fold in context. Fold k is scored with the other folds’ RGCC edges, one draw at a time, added to the fact graph. No retrain per fold is needed, since ULTRA has no per-relation parameters. This needs the overlay to carry draw dates.
Restate the old numbers (#324), for memorization as well as pooling.
The larger inputs: the WTS gene axis (#212) and population supervision (#327).
Migrations: none expected; release_v0 and the axis live in existing columns and JSONB, so confirm before adding any.
MCP tools: no. The mods_pdr_export tool ignores the schema, so don’t use it for this run.
DB-stored config: yes, the pdr2_feature_schema v2 row.
Docs: yes (this file, the results writeup).
Tests: yes, in every code issue.
Container: yes, one rebuild after #434, #435 and #436, including the path-explanation mode for §5.4 step 4.
Parallel split.
Lane P code (#431) and runs (#339 → E3 ‖ D1 → D2 → D3) run independently of Lane G code (#432, #433, #434) and Lane R code (#435, #436, #437).
#430 and #244 need the mapping decision first.
Sequential joins: #296’s release waits on D3 and #430; the pickle waits on the release and axis v2; the fine-tunes wait on the pickle, the foundation and the rebuilt image.
Collision hotspots.
worker/task_handlers.py is shared by #398, #433 and #422’s plumbing: sequence them.
lib/pdr2/code/script/* is shared by #434, #435, #436 and the path-explanation mode: batch them into one rebuild.
requirements.txt (#339) is already done.
No migrations.py, component_registry.py, Terminal.jsx or main.py changes are expected.
Verify-how per surface.
Backend and PRS: pytest via scripts/run_backend_tests.ps1. The :5434 DB is broken (#421), so use CI or the agent DB.
Worker: a dry-run submission with a mismatched manifest refuses.
Container: a synthetic-pickle machinery test (M6, M7).
DB: the lifecycle counts (M1).
End to end: M1–M10.
Appendix A — Audit queries (reproduce §1)
A.1 Patient edges by split in the foundation’s training files.
cd /c/Maple/runs/2026-05-18_maureen_focus_v1/datafor f in train valid test; do echo "== $f"; grep "MAPLE_PATIENT:maureen" $f.txt | cut -f2 | sort | uniq -c; done
A.2 Per-draw Tier-1 of the v2 scores, split by whether the drug’s label was ever in train. Inputs: runs/2026-05-20_rgcc_tier1_v2/metrics.matched.csv (the sensitive_to_drug rows), edges.csv (agent → CURIEs), train.txt, and the labels:
SELECT coalesce(p.collected_date, p.reported_date) AS draw, a.agent_name, a.score_rawFROM mods_functional_test_agent_result aJOIN mods_functional_test_profile p ON p.id = a.functional_test_idWHERE p.patient_id = 'PATIENT_UUID' AND p.test_name ILIKE '%onconomics%' AND a.score_raw IS NOT NULL;
A drug is “clean” when none of its CURIEs is the object of a MAPLE_PATIENT:maureen… MAPLE:*_to_drug line in train.txt. Spearman is computed per draw, never pooled.
A.3 Mirror leak.
tr = {tuple(l.rstrip('\r\n').split('\t')) for l in open('train.txt', encoding='utf-8')}te = [tuple(l.rstrip('\r\n').split('\t')) for l in open('test.txt', encoding='utf-8')]print(sum((t, r, h) in tr for h, r, t in te) / len(te)) # 0.606 on 916bdd2c
A.4 PRS rows have no disease key.
SELECT validity, count(*), count(disease_cui), count(z_score), count(percentile)FROM mods_polygenic_score WHERE patient_id = 'PATIENT_UUID' GROUP BY 1;
A.5 MONDO padding twins.
SELECT count(*) FROM mods_kg_entity u JOIN mods_kg_entity p ON p.curie = 'MONDO:' || lpad(split_part(u.curie, ':', 2), 7, '0')WHERE u.curie ~ '^MONDO:[0-9]+$' AND u.curie !~ '^MONDO:[0-9]{7}$'; -- 14,504
A.6 Drug–protein roles.
SELECT predicate, count(*), count(*) FILTER (WHERE metadata ? 'display_relation')FROM mods_kg_relation WHERE source_id = 'prime_kg' AND source_predicate = 'drug_protein' GROUP BY 1;-- biolink:has_target | 49664 | 0
A.7 The Gate A callset behind the live scores.
SELECT metadata->'gate_a'->>'n_records', count(*) FROM mods_polygenic_scoreWHERE validity = 'quarantined' GROUP BY 1; -- 16964243 | 3752SELECT id, union_size, left(dense_callset_checksum, 12) FROM mods_prs_runWHERE gate = 'gate_a_dense_callset' ORDER BY created_at; -- 64a90788… 19215298 d76e1c65ee40
Appendix B — Time and cost (estimates unless marked measured)