Smoke run #1 — Formal Writeup

Authors: whafa, Claude (Maple project) Dates: 2026-09-28 to 2026-09-29 Tracking: #439: status, provenance log, decisions Playbook: mods_smoke_run_1_playbook.md · PM charter: mods_smoke_run_1_pm_charter.md

Machinery validation, not a recommendation. Nothing here is clinical advice. The run tests whether the pipeline’s parts connect and measure what they claim to. It was never meant to show that personalization helps.

---

Summary

• The pipeline runs end to end for the first time with her real, released PRS. Here is the chain, with each stage’s first real output:
  • PRS: E3 → D1 → D2 → D3 → Gate F v0, releasing 590 scores over 109 diseases.
  • Patient inputs: KGIA axis v2, then the Track B and Track A pickles.
  • Model: a leak-free foundation, five fine-tunes, and a personalized CRC-anchor ranking.
  • Evaluation: the per-draw RGCC grid.
• All ten machinery checks were measured, and all ten pass.
  • Eight pass outright.
  • Two were borderline, and Richard accepted both on 2026-09-29. M6: disease sign agreement 0.881 against a 0.90 bar. M9: scores reproduce to float precision, not bit for bit.
• The clean holdout shows no RGCC signal. With her RGCC drugs removed from training, every drug is unseen. The base ranker’s graded-only Spearman ρ is +0.03 and +0.05 per draw, with CIs spanning 0, and its AUROC is about 0.61 on 5 positives. That agrees with the 2026-09-27 audit’s reading that the old PDR2 v2 “signal” was memorization.
• The released personalization does not help. It is null at λ 0.01. At λ 0.1 it is slightly and borderline-measurably harmful (all-panel Δρ −0.057, CI [−0.121, −0.004]), and at λ 1.0 it trends harmful with CIs spanning 0. The expression-only control is null.
• The criterion’s degeneracy is now measured on her data:
  • its weights stall within the first epoch;
  • λ has no effect on the learned weights;
  • the adjustment can only push a drug down;
  • 73 % of it flows through contraindicated_for edges.
• The foundation generalizes weakly. Its filtered MRR is 0.021 on held-out triples, well below published ULTRA-class numbers, even though it fits its training edges almost perfectly. (2026-10-05: it also loses to a non-learned baseline, found on 2026-10-01 (#523). Ranking by how often a node ends an edge of each relation in training scores 0.031, and a drug’s edge count alone reaches a graded RGCC ρ of +0.37 against the model’s +0.03; the model’s scores correlate 0.86 with that count. Suspected cause: the mirror of each query edge stays in the training graph.)

1. Purpose

Decided 2026-09-27: catalogue and retire the issues blocking a first end-to-end MODS run with real PRS. The run is a machinery smoke test. Its acceptance criteria (M1–M10, playbook §6) ask whether each stage produces what it claims, not whether the output is clinically useful. Against the mission’s goals:

• G4 (traceability): tested directly, through M9 and the #439 provenance log.
• G2 (the ranking recovers her measured sensitivities on a clean holdout): measured for the first time, on a leak-free foundation.
• G3 (personalization improves on the raw ranking): measured with the released PDR2 criterion, which the project already suspected to be degenerate (#47, #420).

2. Materials and Methods

2.1 Patient inputs

Input	What was used
PRS	Gate F v0 release smoke_v0 (rules gate_f_v0_rules_v1, release run c3be9df0): 590 released scores over 109 MONDO anchors, one axis pick each. Placed as EUR z-scores (667 unrelated EUR panel samples, panel-aligned). D2 ran under the #499 fix (gate_d2_panel_scorer_v2), so her no-called loci are dropped from the panel too.
Expression	84 genes (v1’s gene axis, decision D8) from her most recent colorectal Caris WTS specimen, as probit z-scores of TPM percentiles (#435).
Axis	KGIA v2: 109 diseases × 84 genes, hash b95dcaff004ccf89, registered inactive (v1 untouched). The CRC anchor MONDO:0005575 is at position 8.
Pickles	Track B: 109/109 PRS positions measured, sha 2761f70d…. Track A: PRS all zero with the same expression, sha b6fa381c….

2.2 Graph and foundation

• Graph: composition maureen_focus_v1 plus the maple_patient overlay. The six MAPLE:* drug predicates are excluded (the three RGCC sensitivity predicates and the three medication predicates, decision D2).
• Split: mirror-grouped (#432): 4,549,173 triples, split 2,796,395 / 876,339 / 876,406. Split fingerprint 4518539f… was reproduced by a second export (M4). The mirror leak is 0.000, against 0.606 for the old foundation.
• Model: ULTRA / NBFNet, 6 layers, hidden size 64, from scratch, seed 41504.
• Training: 20 epochs of about 3,700 s each on one A100-80GB, image ccad24d-smoke1 (5597a993…).
• Early stop: the post-training full-test eval was cancelled at 09:40 on 2026-09-29, because it could not finish inside the job timeout (#515). The run row therefore reads failed, as the timeout would have left it.

2.3 Fine-tunes (decision D9)

• Configurations: Track B at λ ∈ {0, 0.01, 0.1, 1.0} (λ1 = λ2), plus Track A at λ 0.01.
• Loss and image: the released paper_code loss, on image 9accf69-smoke1-ft (8261b505…).
• Settings: batch 128 (FINE_TUNE_MAX_BATCH, #501); every child reused the foundation’s split (#433).
• Early stopping: every run stopped after 12 epochs and restored epoch 6. The early stop keys on the epoch’s training loss, which the frozen foundation’s prediction term dominates.

2.4 Scoring (decisions D1, #436)

• Query: the disease query (drug, biolink:treats, MONDO:0005575), over all 9,099 candidate drugs; 7,793 of them are in the drug target domain.
• Adjustment: the edge-blind adjustment adds λ·ŵ·PRS over the drug’s disease anchors and λ·v̂·expr over its gene anchors. The anchors are its neighbours in the training graph, with direction and relation ignored.

2.5 Metrics

• M5: full-ranking filtered MRR on 50,000 sampled test triples (#472).
• M8: the per-draw RGCC grid (#437), for draws 2025-02-15 and 2025-09-19. The two draws are never pooled.
  • Panels: all-panel (n = 109 matched) and graded-only (n = 59).
  • Variants: raw and personalized; all drugs and drugs unseen in training.
  • Tier 1 is Spearman ρ against % cell death. Tier 2 is AUROC with RGCC’s 5 high_sensitivity agents per draw as positives against < 35 %.
  • Every cell carries its n and a bootstrap 95 % CI.

3. Procedure

The PM session ran the playbook between 2026-09-28 10:00 and 2026-09-29 11:40. The detours, each filed and fixed or recorded:

When	What happened	Issue
09-28 18:34	D2’s first pass showed her no-calls deflating her score but not the panel’s. That biased her z by up to −5.5 SD on high mean/sd scores. Fixed, approved and re-run.	#499 (PR #505)
09-28 19:30	Three freshly restarted workers broke on their first claim: a lazily imported model module raced debug_service’s writer thread.	#507 (PR #509)
09-28 18:45	No worker handler set FINE_TUNE_MAX_BATCH, so every fine-tune would have run out of memory at batch 256.	#501 (PR #503)
09-28 21:55	GitHub Actions used the private repo’s free monthly minutes. Richard set a budget; the CI trim is #516.	#516
09-29 08:28	The local backend had stopped reloading at 11:58 the previous day, so it served 20-hour-old code.	#517
09-29 09:42	The cancelled foundation had no kg_dataset_class, so all six children failed and were resubmitted. A cancel is also overwritten by a retry.	#515
09-29 10:22	The rank endpoint parses the fine-tune’s patient_id label as a UUID and returns a 500.	#518
09-29 11:02	The M5 eval sat 78 minutes queued on A100-40GB and was moved to A100-80GB.	—

Every job went through the worker queue or the research API (M10). The only direct Vertex calls were two cancels, made through the app’s own provider.

4. Results

4.1 Machinery checks

Check	Result
M1 PRS released	✓ 590 released rows, all with z_score, a disease anchor and the smoke_v0 tag. The count equals the v0 funnel; 0 are outside their bounds. The four non-pick alternates were held back by the post-D3 panel-coverage rule.
M2 Axis	✓ v2 registered (hash b95dcaff004ccf89), v1 untouched, the CRC anchor on the axis.
M3 Pickles	✓ Track B measures 109 of 109 PRS positions; every value is finite; expression is probit z; the sidecars are present.
M4 Export	✓ 0 excluded-predicate lines; mirror leak 0.000; the manifest md5s reproduce.
M5 Foundation	✓ Loss 0.0573 → 0.00058 over 20 epochs. Every load logs its checkpoint sha. Filtered MRR 0.0209 (Hits@10 0.035), below the old leaky 0.56.
M6 Fine-tune	✓ disease cos(ŵ, −PRS) +0.649. ✗ disease sign agreement 0.881 < 0.90. ✓ protein cos +0.652, sign agreement 0.952. ✓ ŵ identical across λ > 0. ✓ Track A’s disease weights are bit-identical to the λ 0 control’s (weight decay only). ✓ split reuse on all five runs. Accepted by Richard, 2026-09-29.
M7 Personalized scores	✓ At λ 0, pers == raw on all 9,099 rows. ✓ The sampled adj matches an independent recomputation to 1.2e-7. ✓ Kendall τ 0.952 at λ 1.0. ✓ The relation breakdown is reported (§4.3).
M8 Metrics	✓ 20 rgcc_per_draw rows, n in every cell; every RGCC drug flagged unseen; test-retest printed on every run.
M9 Provenance	✓ The #439 provenance log links run ids, image digests, manifest md5s, the release tag, the schema hash and pickle shas. A re-run of one scoring job reproduces raw scores and ranks exactly, and adj to ≤ 1.9e-9 (float32 rounding). It is not bit-identical. Accepted by Richard as reproduced, 2026-09-29.
M10 Venue	✓ Every job went through the worker queue or the research API.

4.2 The foundation

The training loss fell to 0.00058, but the held-out filtered MRR is 0.0209: mean rank about 3,000 of about 103K entities, and Hits@1 is essentially zero. The model separates its training edges from sampled negatives almost perfectly, but ranks held-out edges only moderately. The old foundation’s 0.56 came from the mirror shortcut, which #432 removed. (2026-10-05: #432 removed the split-level shortcut. A training-time one, where each query edge’s mirror stays in the message-passing graph, is the suspected cause of #523.) Published ULTRA-class numbers on dense biomedical graphs are 0.13–0.4, so this foundation is weak for link prediction as well as for RGCC.

4.3 The released criterion’s behaviour

• The weights stall almost immediately. Sign agreement is already 0.881 after the first epoch (100 steps), and between epochs 0 and 6 each weight moves by about 1e-4.

  • The tanh term saturates within about 50 steps, and AdamW’s second-moment memory then shrinks every step to almost nothing.
  • Against the λ 0 run, which shows the seed-42 starting point, every weight moved toward −sign(input): 109 of 109 disease weights and 84 of 84 gene weights. 38 disease signs flipped from wrong to right and none the other way.
  • The 13 still wrong are leftovers from the random start, spread across the |PRS| range.

• λ does nothing at fine-tune time. The λ 0.01, 0.1 and 1.0 runs learned identical weights (cos 1.000000, max difference 2e-6), because Adam cancels the gradient’s λ scale. λ acts only as a scale on the adjustment at scoring, so the D9 fine-tune sweep was redundant.

• The adjustment can only lower a score. With ŵ ≈ −PRS/‖PRS‖, each anchor term is about −z²/‖z‖, negative whatever the sign of her z. At λ 0.01, 1,254 of the 1,314 adjusted drugs go down.

• It is edge-blind. Share of the adjustment’s absolute mass, by relation:

  relation	share
  disease contraindicated_for	72.7 %
  disease treats	14.5 %
  disease has_off_label_use	3.3 %
  gene has_target	9.5 %

  A drug contraindicated for one of her diseases is pushed the same way as a drug that treats it.

4.4 RGCC (M8)

Tier 1 is Spearman ρ; Tier 2 is AUROC on 5 positives. Each cell gives the 2025-02-15 draw, then the 2025-09-19 draw, as point [95 % CI].

variant	graded ρ	all-panel ρ	all-panel AUROC
raw	+0.029 [−0.24, +0.32] / +0.046 [−0.23, +0.32]	−0.193 [−0.37, −0.02] / −0.187 [−0.37, −0.00]	0.525 / 0.521
pers λ 0.01	Δ −0.002 / −0.003	Δ −0.010 / −0.009	Δ −0.006 / −0.007
pers λ 0.1	Δ −0.063 / −0.056	Δ −0.057 [−0.121, −0.004] / −0.054 [−0.117, +0.001]	Δ −0.054 / −0.058
pers λ 1.0	Δ −0.110 [−0.41, +0.16] / −0.15	Δ −0.034 / −0.047	Δ −0.082 / −0.083
Track A λ 0.01	Δ +0.001	Δ +0.000	Δ +0.002

The all-panel ρ below zero is driven by the 53 MoAb/SMW agents that RGCC grades only as sensitive or not, all tied at 0. The graded-only panel is the one to read. The test-retest ceiling is ρ 0.9616 all-panel and 0.7635 graded.

4.5 The top 30

The annotated top 30, raw and at λ 1.0, is on #114.

• Ranks 1–2 are the only two drugs whose treats-CRC edge is in the training split: tipiracil and trifluridine, at raw scores near +7.8.
• Most of ranks 3–25 are CIViC / OncoKB therapy and regimen entities outside the drug domain. They include her own cetuximab/encorafenib regimen, FOLFOX, an anti-EGFR antibody and RAF inhibitors. The model recovers her actual treatment lines through evidence edges, but the personalization cannot reach them.
• Cetuximab’s treats-CRC edge is held out, and it ranks 26 of 9,099.
• At λ 1.0, personalization only removes drugs from the top 30: erlotinib and crizotinib.

5. Discussion

What the run shows. The machinery works and is traceable: every stage produced its artifact, and every artifact carries its provenance. The science is an honest null.

• The base ranker has no signal on her RGCC sensitivities once the leak is closed.
• The released criterion cannot help, for four reasons:
  • it stalls near its start;
  • it ignores λ;
  • it can only demote;
  • it cannot tell treats from contraindicated_for.

These are the defects #420 (a signed per-relation criterion) was designed around. This run quantifies them on her own data.

Limitations:

• 8 of the RGCC agents have no in-domain node, so the adjustment cannot reach them (#470).
• One axis pick (PGS005253) carries a predicted +0.67 SD residual from the panel’s own missing calls (#513).
• The fine-tune’s early stop keys on a training loss the criterion weights cannot move.
• Tier 2 rests on 5 positives per draw.
• The v0 release waived the pgsc_calc witness and the NR-scale policy.
• The foundation’s held-out MRR is low.

What to do next, in the order the evidence suggests:

1. #420, the signed per-relation criterion, so the adjustment can promote as well as demote and can tell treats from contraindication.
2. Start the criterion weights at zero, and early-stop on the criterion’s own loss, not the training loss.
3. Use λ as a score-time knob, running one fine-tune rather than one per λ.
4. Revisit the foundation’s generalization (MRR 0.021) before relying on its ranking: depth, and more epochs past the stall, or a composition change.
5. The larger levers already on the roadmap: population supervision and variant-aware edges.

(2026-10-05: the Q4 2026 plan, written 2026-10-01, reorders this list. A no-retrain test of the #523 shortcut, an evaluation rebuilt around non-learned baselines, and a retrain that drops each query edge’s mirror come first. That retrain, Arm A, became the checkpoint of record and failed gates S5 and S6 on 2026-10-04. #420 runs only if a conditioned baseline beats drug-mean on a cell-line proxy benchmark.)

Operational lessons are in §3’s table. The durable ones:

• a fix to data doesn’t repair data already derived (#499);
• a reloader that watches 177 GB can silently stop (#517);
• cancels and stops need first-class paths (#515);
• timed wakeups can silently not fire, so waits hang on completion signals.

Provenance

Every run id, image digest, manifest md5, release tag, schema hash and pickle sha is in the #439 provenance comment. The large local artifacts (panel matrices, pickles, fine-tune weights, train.txt) are under backend/data/pdr2_pickles/2026-09-28_smoke1/ and backend/data/prs_panel/, host-only and gitignored.