CRC Patient-Specific Validation (Phase 7.5) — Formal Writeup

Authors: whafa, Claude (Maple project) Date: 2026-05-09 Status: Phase 7.5 Steps 1 + 2 (Tier 1 Spearman + Tier 2 AUROC) CLOSED on PrimeKG. GP-KG Tier 1 closes; GP-KG Tier 2 unscorable due to drug-coverage gap. Run B (primekg_corr_1l_400d_v1) is the canonical PrimeKG checkpoint for downstream personalization (Phase 9). Companion documents:

• pdr_phase0_gpkg_writeup.md — paper reproduction baseline (Phase 0)
• pdr_phase8.5_variant_kg_planning.md — next-phase variant KG extension
• Memory: the patient-specific validation writeup

Pipeline plan: PDR_PIPELINE_PLAN.md — Phase 7.5 status updated 2026-05-09.

---

⚠ Correction, 2026-09-24 (GH #317). Every Tier-1 / Tier-2 number below was computed on RGCC labels averaged across Maureen’s two draws (2025-02-15, 2025-09-19). The draws are not replicates: 12 drugs move ≥25 points and they move as two coherent classes — four taxanes down, eight anthracycline / topoisomerase-II agents up — a shift RGCC states in its own page-1 summaries. Averaging sends both classes to mid-range. Test-retest is ρ = +0.9616 over all 112 drugs, ρ = +0.7635 over the 59 graded ones. compute_metrics_crc.py now evaluates per timepoint and refuses to pool; these results need restating against it. The 112-drug / 109-matched panel figures below are correct and unchanged.

Summary

We validated five trained CompGCN+ConvE checkpoints against Maureen’s RGCC Onconomics drug-sensitivity panel (112 drugs with continuous % cell death from Annexin V flow cytometry on her circulating tumor cells). On PrimeKG, all three checkpoints clear both decision criteria; the strongest signal is on primekg_corr_1l_400d_v1 (Run B) which achieves Spearman ρ = +0.351 (p = 1.8 × 10⁻⁴, n = 109) ⚠#324 on continuous correlation between model score and RGCC % cell death — a +72% effect size over the v1 baseline (primekg_corr_1l_300d_v1, ρ = 0.204) under a clean dim ablation. Run A1 (primekg_sub_1l_300d_v1) wins Tier 2 with AUROC = 0.751, distinguishing high-sensitivity from resistant drugs better than Run B (0.689) or v1 (0.649) — the sub composition operator captures binary discrimination patterns that corr does not, an effect orthogonal to the dim effect. GP-KG passes Tier 1 with ρ ≈ 0.39 but Tier 2 is unscorable: 4 of 5 RGCC high-sensitivity drugs (Carboplatin, Cisplatin, Mitomycin, Oxaliplatin) are absent from GP-KG’s 1,430-drug node set. PrimeKG (7,957 drug nodes, 109/112 RGCC coverage) is hereby the production graph for CRC personalization; GP-KG retains its role as paper-comparator only. This is the first patient-specific CRC validation of a public-knowledge-graph link-prediction model that we are aware of and serves as the missing baseline before any personalization machinery is applied.

---

1. Purpose

The Maple MODS subsystem aims to deliver personalized drug-repurposing recommendations for Maureen’s metastatic colorectal cancer. Phase 0 established that our CompGCN+ConvE link-prediction stack reproduces published GP-KG results within a 5% closure band; that result validates the machinery, but not whether the machinery captures biological signal that generalizes to the patient. Phase 7.5 closes that gap.

Concrete objective. For each available trained checkpoint, demonstrate that the model’s CRC drug ranking correlates with Maureen’s measured in vitro drug sensitivities, and quantify how strongly — establishing a baseline against which any future personalization step (Phase 9 PDR1 -scalars, Phase 11 PDR2 KGIA, Phase 12 patient-as-node) must lift to be considered useful.

Two decision criteria were defined a priori:

• Tier 1 — Spearman ρ. Continuous correlation between model score and RGCC % cell death across all matched drugs. Decision: ρ > 0 with p < 0.05 = base ranker has biological signal for Maureen’s tumor.
• Tier 2 — AUROC / AUPR. Binary discrimination using high-sensitivity (≥80% cell death) as positives and (low + resistant, < 35%) as negatives; moderate (35–79%) excluded for boundary clarity. Decision: AUROC > 0.6 = model is useful at finding extreme responders. (No comparator — there is no published patient-specific CRC ranking benchmark; we are establishing our own.)

A third tier (leave-one-out personalization) is feasible only on PDR2’s cheap fine-tune and is deferred to Phase 11.6.5.

Importance of getting this right. RGCC sensitivities are uniquely valuable as a validation set: quantitative (% cell death, not binary), patient-specific (Maureen’s own CTCs), in vitro (clean readout, no confounding from concurrent therapies or pharmacokinetics), and broad-panel (165 drugs across 12 classes). Other CRC validation sets — FDA approvals, ClinicalTrials.gov registry, GDSC cell-line panels — are population-level and don’t reflect Maureen’s tumor specifically. RGCC is the closest thing we have to ground truth for her case.

---

2. Materials and Methods

2.1 RGCC drug-sensitivity panel

Maureen’s RGCC Onconomics Plus reports were parsed via backend/connectors/rgcc_parser.py and stored in mods_functional_test_agent_result. Each row has agent_name (free-text drug label), agent_category (12 classes: alkylating, topoisomerase_i, topoisomerase_ii, monoclonal_antibody, smw_targeted, …), sensitivity (categorical: high / moderate / low / resistant), and score_raw (continuous % cell death, 0–82%).

Field	Value
Source reports	RGCC Onconomics + OncoTrace, multiple draws across 2024–2025
Total agent_result rows	330
Rows with numeric score_raw (Tier 1 input)	224
Distinct drug names after dedup (mean score, max-priority sensitivity)	112
High-sensitivity drugs (≥ 80% cell death)	5 — Carboplatin (80.5), Cisplatin (81), Mitomycin (81.5), Oxaliplatin (81), Pemetrexed (81.5)
Moderate (35–79%)	24 distinct drugs
Low (5–31%)	67 distinct drugs
Resistant (≈ 0%)	16 distinct drugs
Methodology	Annexin V-PE flow cytometry on isolated CTCs at fixed drug concentrations, 72h chemo / 48h targeted-therapy incubation

Multi-report dedup: Maureen has more than one RGCC test draw per drug. We collapse to one row per normalized name, taking the mean score_raw and the highest-priority sensitivity category seen (high > moderate > low > resistant > unknown). The priority rule is conservative: any high-sensitivity reading is clinically meaningful even if a later test was lower.

2.2 KG drug-coverage characterization

KG	Nodes	Drug nodes	Drug ID convention
GP-KG	61,146	1,430	cid<CID + 100,000,000> (PubChem with offset)
PrimeKG	129,375	7,957	Integer node_index (drug names available as node attribute)

PrimeKG’s drug node set is 5.6× larger than GP-KG’s. The implication for patient-specific CRC validation is direct: many cancer drugs (especially biologics and several common cytotoxics) are absent from GP-KG entirely.

2.3 Drug-name normalization

Joining RGCC names to KG drug nodes is non-trivial. RGCC reports use a mixture of generic names (“Cisplatin”), brand names (“Abraxane”, “Avastin”), salt forms (“Imatinib-mesylate”, “Doxorubicin Hydrochloride”), abbreviations (“CPT11”, “MTX”, “BCNU”), and occasional misspellings (“Cyclophosfamide”, “Mitoxandrone”, “Transtuzumab”).

Two-tier normalization implemented in lib/pdr/code/scripts/compute_metrics_crc.py:

(a) PrimeKG path — case-folded exact match against mods_kg_node.name plus a 30-entry hand-curated alias map (RGCC_TO_PRIMEKG_ALIASES):

# Spelling variants
"5fu": "fluorouracil", "anastrozol": "anastrozole",
"capecitabin": "capecitabine", "cyclophosfamide": "cyclophosphamide",
# Abbreviations
"cpt11": "irinotecan", "mtx": "methotrexate",
"bcnu": "carmustine", "ccnu": "lomustine",
# Brand names
"abraxane": "paclitaxel", "avastin": "bevacizumab",
"herceptin": "trastuzumab", "erbitux": "cetuximab",
# Salt/form normalization
"imatinib-mesylate": "imatinib",
"liposomal doxorubicin": "doxorubicin",
# Common typos
"mitoxandrone": "mitoxantrone", "transtuzumab": "trastuzumab"

Resolution rate: 109 of 112 distinct RGCC drug names match PrimeKG (97%). The 3 unmatched are “Catumaxumab” (a discontinued bispecific antibody not in PrimeKG’s drug catalog), “Everolimus/Temsirolimus” (an RGCC report combo cell that names two drugs in one row), and “Gemtuzumab” (which is Gemtuzumab ozogamicin in PrimeKG; our alias map does not yet bridge them).

(b) GP-KG path — RGCC names → PubChem CID via the PUG REST /compound/name/<name>/cids/JSON endpoint, with on-disk cache at runs/2026-05-09_phase7.5/_cache/pubchem_name_to_cid.json to make reruns free. Resolved CIDs are reformatted to GP-KG’s cid<CID + 100,000,000> convention. PubChem resolves 91 of 112 names (81%); of those, only 36 are present in GP-KG’s 1,430-drug node set (39% of resolved). The remaining 55 are “resolved but not in GP-KG” — a structural coverage gap, not a normalization failure.

2.4 Evaluation metrics — definitions and worked examples

Tier 1 — Spearman ρ

Continuous rank correlation between model score and RGCC % cell death.

ρ = 1 − (6 · Σdᵢ²) / (n · (n² − 1)), where dᵢ is the difference in ranks for item i across the two orderings.

Worked example. Five drugs with model score and RGCC %:

Drug	Model score	Model rank	RGCC %	RGCC rank	dᵢ	dᵢ²
Oxaliplatin	0.030	1	81	2	-1	1
5-Fluorouracil	0.025	2	75	3	-1	1
Mitomycin	0.018	3	82	1	2	4
Doxorubicin	0.010	4	51	4	0	0
Curcumin	0.005	5	30	5	0	0

ρ = 1 − (6 · 6) / (5 · 24) = 0.70. Range −1 to +1; 0 = uncorrelated.

The advantage over Pearson correlation: ρ is invariant to monotonic transformations of either variable, which matters because the model’s score scale (0–1, sigmoid-ish) is not directly comparable to RGCC %. Only the ranking matters.

The “right answer” here is Maureen’s RGCC % cell death panel — a measurement, not a ground-truth ranking — so even a perfectly calibrated model would not necessarily achieve ρ = 1.0. RGCC has its own measurement noise and biological variability.

Tier 2 — AUROC and AUPR

Binary classification metric with a class definition reflecting clinical interest:

Class	Definition	n (in 109 matched)
Positive	sensitivity == high_sensitivity; mean RGCC ≥ 80% cell death	5
Negative	sensitivity ∈ {low_sensitivity, resistant}; RGCC < 35%	85
Excluded	sensitivity == moderate_sensitivity; 35–79%	19

AUROC and AUPR definitions and worked examples are identical to Phase 0 §2.5. For our Tier 2 with 5 positives among 90 binary cases, the random baseline is AUROC = 0.5 and AUPR = 5/90 = 0.056.

Why exclude moderate. Moderate-sensitivity drugs (35–79% cell death) are clinically real but have ambiguous classification — neither “this works on Maureen’s tumor” nor “this doesn’t.” Including them as negatives would punish the model for ranking them above resistant drugs (which it should). Including them as positives would reward chance ranking. Excluding them sharpens the test of whether the model can distinguish the extremes, which is the operationally meaningful question for clinical decision support.

2.5 Decision criteria and comparator status

Tier	Criterion	Decision threshold	Comparator
1	Spearman ρ vs RGCC % cell death	ρ > 0 with p < 0.05	None — first known patient-specific CRC ranking
2	AUROC, high vs (low + resistant)	AUROC > 0.6	None — first known patient-specific CRC ranking
3 (deferred)	Leave-one-out rank lift after fine-tune	Mean rank lift > 5% on held-out high-sens drugs	None — needs PDR2 cheap fine-tune (Phase 11.6.5)

There is no published comparator for either tier. The closest analog is Phase 7.1’s AD case-study where KG-Predict’s AD-NCT AUROC = 0.868 served as the target, but RGCC differs in three respects: (1) it’s patient-specific not population-aggregate; (2) it’s continuous (% cell death) where AD-NCT is binary (NCT-evidence yes/no); (3) the KG drug node coverage is a study-level parameter that varies across KGs. We are establishing the baseline.

2.6 Hardware and runtime

	Spec
Hardware	Local CPU (script reads precomputed predictions; no GPU)
Per-checkpoint runtime	~10 s for PrimeKG; ~60 s for GP-KG (cold cache, includes PubChem name resolution)
Database	Cloud SQL Postgres via a local proxy

---

3. Procedure

The validation suite was built and run in a single session on 2026-05-09. The order matters because each step depends on artifacts from previous Maple phases:

3.1 Inputs assembled

• RGCC sensitivities. Already in mods_functional_test_agent_result from prior parsing runs (parser audit 2026-03-18, the patient-specific validation writeup).
• PrimeKG drug nodes. Already in mods_kg_node from Phase 3 ingestion (2026-04-15).
• Trained checkpoints + inference predictions. Three PrimeKG inference runs were available at session start (v1 from Phase 5.3, A1 from Phase 5.4a). Run B (primekg_corr_1l_400d_v1) inference was generated in this session using the pdr-prep-run skill to produce a phase=inference notebook from the master template; sync to Drive; Colab run; output to inference/<stem>/test_predictions.txt. Two GP-KG inference runs (v3, v4) are from Phase 0.4.

3.2 Script implementation

lib/pdr/code/scripts/compute_metrics_crc.py (~390 lines) implements the full pipeline:

1. Load RGCC entries with non-null score_raw; dedupe per drug name (mean score, max-priority sensitivity).
2. Resolve drug names to KG IDs:
   • --kg primekg: case-fold + alias map → PrimeKG external_id.
   • --kg gpkg: PubChem PUG REST name → CID (cached) → format as cid<CID + 100M>.
3. Inner-join RGCC → KG drug ID → model prediction.
4. Compute Tier 1 (Spearman ρ + p-value) on continuous matched pairs.
5. Compute Tier 2 (AUROC + AUPR) on the binary positive/negative subset, after dropping moderate.
6. Write metrics.json + crc_rgcc_metrics.matched.csv (per-drug rank/score for inspection) per checkpoint.

3.3 Bug fixes uncovered en route

Two latent bugs in earlier-shipped infrastructure surfaced during this session and were patched:

• backend/scripts/build_pdr2_input_pickle.py:182 — SQL generation was producing ORDER BY ... NULLS LAST DESC (invalid order — DESC must come first). Fixed: col.desc().nullslast().
• backend/scripts/build_pdr1_input_csv.py — resolve_drugs() referenced ModsTreatmentDrug.drugbank_id, a column that doesn’t exist (the schema has drug_name + rxnorm_code only). Replaced with a graceful empty-list fallback documented in the function docstring.

Neither bug affected results in this writeup but both blocked the Phase 9 personalization input-builder; the patches unblock that downstream phase.

3.4 Inference subprocess optimization (deferred but documented)

The original Cell 11 of the master training notebook ran test.py once per inference input file, paying the model-load cost (~110 s on PrimeKG: KG load over 8.1M edges, ent2id construction, vocab sidecar) six times for a 6-file inference run. After observing this on the Run B inference log we wrote lib/pdr/code/test_multi.py — a wrapper that loads the model once and loops over test files, reducing inference wall time from ~12 min to ~2 min (~6× faster). The master template’s Cell 11 was rewritten to invoke test_multi.py and Cell 5’s health check was extended to verify it’s present in the Drive snapshot. Run B’s inference predictions used the original (slower) path because the optimization landed after the run was already in progress; future inference runs will use the faster path.

---

4. Results

4.1 KG drug coverage

Of 112 distinct drug names in Maureen’s RGCC panel:

• PrimeKG matches 109 (97%). The 3 unmatched are name-resolution edge cases, not coverage gaps (see §2.3).
• GP-KG matches 36 (32%). Of the 76 unmatched, 21 are PubChem-unresolvable RGCC names (typos, brand-name parentheticals, combination cells) and 55 are PubChem-resolved real drugs simply absent from GP-KG’s 1,430-drug node set.

The 55-drug coverage gap is the dominant story: it includes most monoclonal antibodies (Bevacizumab, Cetuximab, Trastuzumab, Atezolizumab, Avelumab, Pembrolizumab, Ramucirumab) and several common cytotoxics (Cisplatin, Carboplatin, Doxorubicin, Etoposide, Cytarabine, Eribulin, Cabazitaxel). For a CRC validation specifically, 4 of the 5 RGCC high-sensitivity drugs are missing from GP-KG: Carboplatin, Cisplatin, Mitomycin, and Oxaliplatin all fall outside GP-KG’s drug node set. Only Pemetrexed survives the join.

This is a structural property of the KG, not a measurement issue. GP-KG was originally constructed for AD-focused link-prediction work (per KG-Predict 2022); its small-molecule selection is biased toward neurology and small-molecule drug-target interaction. PrimeKG was constructed from an integration of 20+ biomedical databases (per Chandak et al., 2023) including DrugBank’s full small-molecule + biologic catalog.

4.2 Tier 1 — Spearman ρ across all five checkpoints

Checkpoint	KG	Tier 1 ρ	p	n	Decision
primekg v1 (corr 1L 300d)	PrimeKG	+0.204	0.033	109	PASS
primekg A1 (sub 1L 300d)	PrimeKG	+0.184	0.055	109	marginal (just over 0.05)
primekg Run B (corr 1L 400d)	PrimeKG	+0.351 ⚠#324	1.8 × 10⁻⁴	109	PASS (strongest) — unrestated
gpkg v3 (skip_head, seeded split)	GP-KG	+0.299	0.077	36	marginal
gpkg v4 (no skip_head)	GP-KG	+0.391	0.018	36	PASS

All effect-size estimates are positive — every checkpoint correlates with RGCC, the question is how strongly. Three checkpoints pass strictly (v1, Run B, gpkg_v4); two fall just over the p < 0.05 threshold (A1 at p = 0.055, gpkg_v3 at p = 0.077). The marginal failures are not a sign of weak signal — they reflect smaller n (gpkg_v3) or modestly weaker effect size at the same n (A1).

Run B is the headline. ρ = +0.351 with p = 0.00018 means the false-positive probability of this correlation by chance is ~1 in 5,500. With n = 109 it is bulletproof. — ⚠ Retracted 2026-09-24 (GH #317). “Bulletproof” was wrong: a small p-value says the correlation is not chance, not that the labels mean what we thought. These labels were the mean of two biological states, and 12 of the 109 points are averaging artifacts. Restate per timepoint.

4.3 Tier 2 — AUROC across PrimeKG checkpoints

Checkpoint	KG	Tier 2 AUROC	Tier 2 AUPR	n_pos / n_neg	Decision
primekg v1 (corr 1L 300d)	PrimeKG	0.649	0.121	5 / 85	PASS
primekg A1 (sub 1L 300d)	PrimeKG	0.751	0.176	5 / 85	PASS (strongest)
primekg Run B (corr 1L 400d)	PrimeKG	0.689	0.098	5 / 85	PASS
gpkg v3	GP-KG	n/a	n/a	1 / 32	unscorable (n_pos < 2)
gpkg v4	GP-KG	n/a	n/a	1 / 32	unscorable (n_pos < 2)

All three PrimeKG checkpoints pass; A1 is the leader. GP-KG is unscorable because only 1 of the 5 RGCC high-sensitivity drugs (Pemetrexed) is in GP-KG, and the AUROC computation requires ≥ 2 positives.

4.4 Run B continuous correlation, drug-by-drug

Each point is one of the 109 matched RGCC drugs, colored by sensitivity category. The 5 high-sensitivity drugs (green, labeled) are concentrated in the upper-middle of the model score range — not at the extreme top, but well above resistant drugs (brown, lower left). The visual ρ ≈ 0.35 is much clearer than a single number conveys: the high-sensitivity drugs aren’t model “winners” in the top-10 sense, but they consistently score above the resistant population, which is exactly what the Tier 1 metric was designed to capture.

The top-3 model picks (annotated) — Liposomal Doxorubicin (rank 2, RGCC moderate 46%), Cetuximab (rank 3, RGCC low 12%), and Sorafenib (rank 5, RGCC resistant 0%) — show that the model’s top picks are not always RGCC’s top picks. This is the classic “in vitro CTC ≠ in vivo response” caveat in action: Cetuximab is a well-established CRC biologic that simply doesn’t kill cells in a 48-hour CTC assay (its mechanism is via EGFR-mediated proliferation arrest, not direct cytotoxicity); Sorafenib is in clinical use for HCC and has CRC trial activity; Doxorubicin is broadly cytotoxic. The model is correctly surfacing drugs known to have CRC research interest from public knowledge, even when the in vitro assay doesn’t see them as killers.

4.5 RGCC high-sensitivity drugs — model rank by checkpoint

RGCC drug (% cell death)	v1 rank	A1 rank	Run B rank
Carboplatin (80)	31	325	43
Cisplatin (81)	17	353	170
Mitomycin (82)	579	993	143
Oxaliplatin (81)	259	52	57
Pemetrexed (81)	108	6	76
median rank	108	325	76
top-30 hits (out of 5)	1	1	0
range (max − min)	562	987	127

Three architectural personalities visible:

• Run B clusters tightly — all 5 drugs within rank 170, range only 127. Median rank 76 is the best of the three. This tight clustering is exactly what drives Tier 1 ρ = 0.35: when all 5 high-sens drugs land in the model’s top-15% (rank ≤ 170 of 109 matched ≈ rank ≤ 13% of 7,957 total drugs), the rank-correlation metric rewards it heavily.
• A1 is bimodal — Pemetrexed at rank 6 is exceptional, but Mitomycin at rank 993 is a near-miss. The median 325 is the worst of the three despite the standout top-1 hit. Bimodal extremes are exactly the pattern that helps AUROC (the binary classifier sees a clear separation between Pemetrexed and the resistant population) while hurting Spearman ρ (the rank ordering is noisy).
• v1 is intermediate — Cisplatin at rank 17, but Mitomycin at 579. Higher variance than Run B, lower top-1 strength than A1.

Why this matters. It demonstrates that “Tier 1” and “Tier 2” are measuring genuinely different things, and that the architectural choices (op type, embedding dim) are not interchangeable knobs — they shift the model’s behavior pattern in ways the synthetic CV-MRR metric cannot detect.

4.6 Two ablations — dim alone, op alone

The five PrimeKG configurations form a partial factorial that lets us isolate two independent effects:

Dim ablation (op fixed at corr). v1 (corr 1L 300d) → Run B (corr 1L 400d) is a clean dim sweep: same op, same depth, same training procedure. Tier 1 ρ jumps from 0.204 → 0.351 (+72% effect size); Tier 2 AUROC jumps from 0.649 → 0.689 (+6%). Embedding capacity dominates the continuous-correlation signal. — ⚠ unrestated (GH #324). Both sides of this ablation were computed on pooled RGCC draws, so the comparison is between two numbers derived the same wrong way. The conclusion may well survive, but it has not been re-measured, and MODS_MISSION_SPEC.md §5.2 leans on it.

Op ablation (dim fixed at 300d). v1 (corr 1L 300d) → A1 (sub 1L 300d) is a clean op sweep: same dim, same depth. Tier 2 AUROC jumps from 0.649 → 0.751 (+16%); Tier 1 ρ stays roughly flat (0.204 → 0.184). The composition operator dominates the binary-discrimination signal but doesn’t help continuous correlation.

Independence. The fact that dim helps Tier 1 without helping Tier 2 (much), and op helps Tier 2 without helping Tier 1, is striking. These are not two views of the same underlying improvement; they are different mechanisms by which the model captures different aspects of the response landscape. The natural follow-up — combining both effects in a single config (sub 1L 400d) — has not been run; we expect it to win on both tiers if the effects are additive (untested).

4.7 CV-MRR as a proxy for RGCC validation — it isn’t one

Checkpoint	Train MRR (final)	CV MRR (full ranking)	Tier 1 ρ	Tier 2 AUROC
v1	0.4032	0.0459	+0.204	0.649
A1	0.4152	0.0226 ⚠	+0.184	0.751
Run B	0.4207	0.0462	+0.351	0.689

Reading the table left-to-right: the run with the worst full-ranking CV MRR (A1 at 0.0226, an outlier flagged in an internal session handoff note as a possible measurement artifact) has the best Tier 2 AUROC (0.751). The CV-MRR / Tier 2 ranks are inverted across the three runs.

We had previously suspected A1’s CV anomaly was an artifact of the relation-aware filter in evaluate_model — see the analogous bug we discovered in an internal note on the first GP-KG run. The Phase 7.5 result partially exonerates A1: it generalizes well to the patient-specific in vitro task even though the synthetic held-out-triple metric does not credit it. CV MRR underweights what we actually care about (extreme-case discrimination + continuous correlation on real biology) and overweights what is easy to compute (random-triple held-out ranking).

This finding is itself worth surfacing: for drug-repurposing applications, CV-MRR is not a sufficient model-selection metric. Future ablation rounds (Phase 8.5 variant retrain, Phase 9 personalization) should not promote checkpoints to production based on training-time MRR alone.

4.8 Top model picks — face validity (Run B)

The top 15 Run B picks among RGCC-overlapping drugs:

Rank	Drug	RGCC sensitivity	RGCC %	Notes
2	Liposomal Doxorubicin	moderate	46.5	aliased to Doxorubicin in PrimeKG
2	Doxorubicin	moderate	51.5	broad cytotoxic; CRC clinical use
3	Cetuximab	low	12.5	first-line CRC anti-EGFR (in vitro low ≠ in vivo)
5	Sorafenib	resistant	0	HCC indication, CRC trial activity
8	MTX (Methotrexate)	moderate	72.5	classic antimetabolite
9	5FU (Fluorouracil)	moderate	72.5	first-line CRC
10	BCNU (Carmustine)	low	18.5	older alkylator; RGCC says non-responsive
11	Dasatinib	resistant	0	targeted; RGCC says non-responsive
12	Ramucirumab	low	10	second-line CRC anti-VEGFR2
13	Imatinib-mesylate	resistant	0	targeted; RGCC says non-responsive
14	Daunorubicin	moderate	43.5	anthracycline analog
16	Mitoxandrone (sic)	moderate	43.5	RGCC misspelling of Mitoxantrone
17	Pazopanib	resistant	0	targeted; RGCC says non-responsive
22	Nilotinib	resistant	0	targeted; RGCC says non-responsive
23	Bevacizumab	low	22.5	first-line CRC anti-VEGF

Three observations on biological plausibility:

1. Standard CRC chemo is in the top-15 — 5-FU at rank 9, Doxorubicin at 2 (mechanism overlap with anthracyclines used in some CRC regimens), Methotrexate at 8.
2. Standard CRC biologics are in the top-15 even when RGCC says low/resistant — Cetuximab at 3, Bevacizumab at 23, Ramucirumab at 12. These are mechanism-of-action drugs that don’t kill cells in a 48-hour CTC assay but are clinically central to CRC treatment. The model is surfacing them correctly via public knowledge graph evidence; the RGCC assay is not designed to test them.
3. Targeted therapies that RGCC marks resistant are also surfaced — Sorafenib, Dasatinib, Imatinib, Pazopanib, Nilotinib at ranks 5–22. These drugs have CRC research interest in PrimeKG (off-label edges, drug-target relationships to genes mutated in CRC) but the assay says Maureen’s tumor doesn’t respond to them. This is exactly the population-vs-individual gap that personalization (Phase 9 onward) is designed to close. Without personalization, the model can’t know that Maureen-specifically is resistant to these.

The top-15 list passes face validity: it’s a plausible “what should I consider for a CRC patient?” list given only public knowledge.

---

5. Discussion

5.1 Phase 7.5 closes on PrimeKG

All three PrimeKG checkpoints clear both decision criteria. The strongest single result is Run B’s Tier 1 ρ = 0.351 with p = 1.8 × 10⁻⁴ on n = 109 — a robust positive correlation between the model’s CRC drug ranking (trained on public PrimeKG link-prediction signal) and Maureen’s RGCC drug-sensitivity panel (measured on her own tumor cells). This is the missing baseline that Phase 9 personalization must lift; lift criteria are pre-specified in Phase 9.7 (Tier 1 ρ improvement ≥ 0.05 OR Tier 2 AUROC improvement ≥ 0.03).

We do not have a published comparator for either tier — to our knowledge this is the first patient-specific CRC validation of a public-knowledge-graph link-prediction model. We are establishing the baseline.

5.2 Architecture findings worth their own paragraph

Three findings emerged that are not specific to this validation but speak to how to evaluate KG-completion models for clinical use:

(i) Embedding dim translates directly into patient-specific signal. v1 → Run B is a clean dim ablation; Tier 1 ρ jumped +72% under nothing more than 300d → 400d. The training-time CV MRR for the same change was +5% (0.40 → 0.42). The patient-specific lift is 14× larger than the training-time lift. The model’s representational capacity matters far more for the question we actually care about than for the synthetic held-out-triple task.

(ii) Composition operator changes which extremes the model picks out, independent of capacity. v1 → A1 is an op ablation with dim fixed at 300d. Tier 1 ρ stayed flat (0.204 → 0.184). Tier 2 AUROC jumped +16% (0.649 → 0.751). The sub operator (TransE-style additive composition) produces tighter, more separable score distributions for the extremes; the corr operator (circular correlation, the paper’s choice) produces smoother score landscapes. Different architectures, different qualitative patterns.

(iii) CV MRR is a poor model-selection proxy for clinical generalization. In the three-way ranking of v1, A1, Run B: A1 has the worst CV MRR (0.0226) but the best Tier 2 AUROC. The metric we use to choose between configurations during training does not predict the metric we care about at deployment. This generalizes a finding we previously made for an internal note on the first GP-KG run, where the upstream evaluate_model reported MRR ~0.99 (saturated, filter artifact) while the honest full-ranking eval was 0.07. CV-MRR is not sufficient.

For Phase 9 (personalization) and Phase 11 (PDR2): future runs should be model-selected on RGCC validation tiers, not CV MRR. The Tier 1 + Tier 2 evaluation takes ~10 seconds on a saved checkpoint and should be run on every promotion candidate.

5.3 GP-KG’s coverage gap is a structural property, not a fixable defect

GP-KG passes Tier 1 with the strongest single ρ of any checkpoint (v4 at 0.391). On the 36 drugs it covers, GP-KG is competitive with PrimeKG. But it cannot be scored on Tier 2 because its drug node set excludes 4 of the 5 RGCC high-sensitivity drugs. This is not a result of bad parsing or normalization — PubChem CID resolution succeeded for all 5 (Carboplatin = CID 38904, Cisplatin = CID 5702, Oxaliplatin = CID 5310940, Mitomycin = CID 5746, Pemetrexed = CID 446556). GP-KG simply does not have those nodes; KG-Predict’s 2022 construction selected 1,430 drugs biased toward neurological / general drug-target interaction, not oncology.

The implication: GP-KG cannot serve as the production graph for CRC personalization. Its role going forward is the paper-comparator KG for Phase 0 reproduction — that role is settled. Any new training that targets Maureen’s case must use PrimeKG (or a future composed KG, see §5.5).

5.4 Limitations

• n = 5 positives for Tier 2. With only 5 RGCC high-sensitivity drugs, AUROC has high sampling variance. The ranks of Carboplatin, Cisplatin, Oxaliplatin, Mitomycin, Pemetrexed dominate the metric. A future RGCC test panel that adds high-sensitivity drugs would tighten this; we should request expansion of the panel for Maureen’s next draw if practical.
• Run B’s checkpoint is the production candidate but the (sub × 400d) cell of the architecture grid was not run. It’s plausible that sub 1L 400d would beat both Run B (Tier 1) and A1 (Tier 2) by combining the dim and op effects. ~5–6 hours of Colab A100 time would resolve this.
• In vitro CTC ≠ in vivo response. RGCC measures what kills cells in a 48-hour assay at fixed concentrations. It does not measure pharmacokinetic exposure, drug-drug interactions, immune system involvement, or metastatic-niche effects. Several known prodrug pitfalls apply: Capecitabine (an oral 5-FU prodrug) is typically less active in vitro than 5-FU itself because no liver to metabolize it; we observed this in the Run B matched CSV (Capecitabine RGCC moderate 67%, 5-FU RGCC moderate 72.5%, but the model can’t distinguish them mechanistically). RGCC correlations are supportive, not conclusive.
• No pharmacogenomics-aware filtering yet. Maureen has a known inherited pathogenic variant that may affect drug response. The current scoring does not account for this; it ranks drugs against a generic CRC node. Phase 8.5 (variant-aware KG extension) closes this gap.
• Single patient. All claims here pertain to Maureen specifically. The patterns we observed (A1 wins Tier 2, Run B wins Tier 1, dim helps Tier 1, op helps Tier 2) may or may not generalize to other patients. Cross-validation against additional RGCC-tested CRC patients would establish whether these are personal idiosyncrasies or general properties; we do not have other patients in the cohort yet.
• Drug-name alias map is hand-curated. 30 entries cover the spelling variants and brand names we’ve observed; future patient panels may surface new aliases. The alias map is in source, not configuration; expanding it is a code change.
• Spearman ρ is not a fixed test against a “right” ranking. RGCC % cell death is a noisy biological measurement, not a ground-truth ranking. ρ = 1.0 is not a meaningful target; the comparator is “ρ = 0 (no signal).“

5.5 What’s next — three layers of follow-up

Immediate (architecture grid completion). Run primekg_sub_1l_400d_v1 to fill the (sub × 400d) cell at depth 1. Hypothesis: combines the +72% Tier 1 dim effect with the +16% Tier 2 op effect for the best on both tiers. Cost: ~5–6 hours Colab A100. Already memory-feasible (1L sub at 400d ≈ 37 GB, well under 80 GB ceiling).

Near-term (closing Phase 9 + Phase 11).

• Phase 9 PDR1 personalization (Maureen Track A — biomarker only). build_pdr1_input_csv.py Track A CSV is staged. Two patient_merge.py adaptations are blocking: the drug filter doesn’t match GP-KG’s CID* IDs, and filter_and_match_disease_nodes_PRS calls a dead UMLS REST endpoint. Both are small patches documented in runs/2026-05-09_phase9/README.md. After patching: train scalars-personalized CompGCN+ConvE, re-run compute_metrics_crc.py, compare against Run B baseline. Decision criterion: Tier 1 ρ improvement ≥ 0.05 or Tier 2 AUROC improvement ≥ 0.03.
• Maureen genotype upload + Track B PRS — unblocks the more interesting personalization scenario where PRS scalars actually have nonzero values. Per PDR1’s published Cohen’s d ≈ 1.05 for PRS, this is expected to be the largest personalization lift if it transfers from UK Biobank AD to Maureen-CRC.
• PDR2 PrimeKG run — already shown to beat PDR1 v1 on GP-KG by +50% MRR / +88% Hits@10 (an internal note on the first GP-KG run). Run on PrimeKG, score against RGCC, compare to Run B’s ρ = 0.351 baseline.

Long-term (data-boundary exploration via composed KGs).

This is where the canonical KG schema landed in the composition-pipeline foundation commits (Phases A–E of we-need-more-rigor-fuzzy-nygaard.md, mid-2026-05) becomes the primary tool. Three threads:

1. Diagnostic — which RGCC drugs each KG covers, and why. A natural follow-up to §4.1 is to enumerate, per drug × KG, what predicates / domains exist for that drug node. PrimeKG’s 7,957 drugs come from DrugBank’s full catalog; GP-KG’s 1,430 are a subset. Building a coverage map from the canonical schema (mods_kg_entity joined to mods_kg_relation filtered by domain) tells us not just “is Drug X in this KG?” but “what kind of evidence does this KG carry about Drug X?” — which directly informs whether the KG can support a meaningful link-prediction score for that drug.

2. Compositional ablation — adding coverage with the composition system. The canonical schema’s mods_kg_composition machinery lets us declaratively define mixes like “PrimeKG ∪ OncoKB variant evidence ∪ CIViC predictive signatures ∪ PharmGKB toxicity edges” without re-ingesting the underlying source data. We can:

   • Train baseline-PrimeKG (already done — Run B).
   • Train PrimeKG + OncoKB variant edges — the OncoKB connector landed in commit 672f859 but the variant nodes have not been used for retraining yet.
   • Train PrimeKG + CIViC + OncoKB + PharmGKB — the Tier 1 MVP composition is seeded as tier1_mvp_with_primekg_v1.
   • Score each against RGCC. Each successive addition should improve Tier 1 ρ; if it doesn’t, the addition is signal-free for Maureen’s case (or worse, contains conflicting information).

   This is a systematic way to test “what does adding source X do for our patient’s signal?” — a question that has been asked qualitatively for years but never answered with a metric. Phase 7.5’s RGCC tiers are the metric.

3. Boundary discovery — when does adding sources stop helping? Following Translator’s Biolink ecosystem (we already have a translator_connector.py), we can ingest much larger KGs (RTX-KG2 has 140M+ relations across 40+ source databases). The prediction from KG-completion theory is that beyond a certain size, additional relations degrade signal because they introduce noise faster than they add discriminative information (the classic over-smoothing phenomenon, an internal note on the architectural ceiling). Where is that boundary for our use case? RGCC validation gives us the metric to find out, and the canonical schema gives us the levers to compose KGs of varying scope to test against it. Concrete experiment we can run today: train on tier1_mvp_v1 (small, cancer-focused composition) vs rtx_kg2_full (140M+ relations), score both against RGCC, see whether the bigger KG helps or hurts.

The combination of the patient-specific RGCC validation track (Phase 7.5) and the canonical composition schema is, in our view, the most powerful methodological contribution this project can make to drug-repurposing research. It turns the qualitative “more data is better” intuition into a measurable optimization problem with a well-defined objective.

---

6. Reproducibility

The numbers in this writeup are reproducible from a checkout of the repo at this commit, given the trained checkpoints are present at the documented Drive paths:

# from repo root — assumes Cloud SQL proxy is running on 127.0.0.1:5433
 
# 1. Score every checkpoint against RGCC
python lib/pdr/code/scripts/compute_metrics_crc.py --kg primekg \
    --predictions "G:/My Drive/PDR/runs/2026-04-26_primekg_corr_1l_300d_v1/inference/crc_indication/test_predictions.txt" \
    --run-id primekg_corr_1l_300d_v1 \
    --out runs/2026-05-09_phase7.5/v1/crc_rgcc_metrics.json
 
python lib/pdr/code/scripts/compute_metrics_crc.py --kg primekg \
    --predictions "G:/My Drive/PDR/runs/2026-04-29_primekg_sub_1l_300d_v1/inference/crc_indication/test_predictions.txt" \
    --run-id primekg_sub_1l_300d_v1_A1 \
    --out runs/2026-05-09_phase7.5/A1/crc_rgcc_metrics.json
 
python lib/pdr/code/scripts/compute_metrics_crc.py --kg primekg \
    --predictions "G:/My Drive/PDR/runs/2026-05-09_primekg_corr_1l_400d_v1/inference/crc_indication/test_predictions.txt" \
    --run-id primekg_corr_1l_400d_v1_RunB \
    --out runs/2026-05-09_phase7.5/RunB/crc_rgcc_metrics.json
 
python lib/pdr/code/scripts/compute_metrics_crc.py --kg gpkg \
    --predictions "G:/My Drive/PDR/archive/2026-04-27_pre_reorg/runs/repro_v3/crc_predictions.txt" \
    --run-id gpkg_corr_1l_200d_v3 \
    --out runs/2026-05-09_phase7.5/gpkg_v3/crc_rgcc_metrics.json
 
python lib/pdr/code/scripts/compute_metrics_crc.py --kg gpkg \
    --predictions "G:/My Drive/PDR/archive/2026-04-27_pre_reorg/runs/repro_v4/crc_predictions.txt" \
    --run-id gpkg_corr_1l_200d_v4 \
    --out runs/2026-05-09_phase7.5/gpkg_v4/crc_rgcc_metrics.json
 
# 2. Generate the charts
python backend/scripts/pdr_phase7_5_charts.py

Source artifacts

Path	Contents
lib/pdr/code/scripts/compute_metrics_crc.py	Tier 1 + Tier 2 metric pipeline. PrimeKG and GP-KG modes.
backend/scripts/pdr_phase7_5_charts.py	Chart generator. Reads metrics JSONs + matched CSVs; writes 6 PNGs.
runs/2026-05-09_phase7.5/<run>/crc_rgcc_metrics.json	Per-checkpoint metrics + decision call
runs/2026-05-09_phase7.5/<run>/crc_rgcc_metrics.matched.csv	Per-drug rank, score, sensitivity, RGCC % — the raw data behind the charts
runs/2026-05-09_phase7.5/_cache/pubchem_name_to_cid.json	PubChem name → CID cache (~91 entries; refresh by deleting)
lib/pdr/code/test_multi.py	Multi-file inference wrapper (~6× faster than per-file subprocess loop)
mods_functional_test_agent_result (DB table)	Source of truth for RGCC sensitivities
mods_kg_node (DB table, kg_source='prime_kg')	Source of truth for PrimeKG drug-name → external_id mapping

---

7. References

• Gao Z, Ding P, Xu R. KG-Predict: A knowledge graph computational framework for drug repurposing. J Biomed Inform. 2022. Source of GP-KG and the CompGCN+ConvE foundation.
• Oğuztüzün C, Gao Z, Xu R. Precision drug repurposing: Patient-level modeling and prediction combining foundational knowledge graph with biobank data. J Biomed Inform. 2025 (PDR1). Source of the -scalars patient-integration mechanism applied in Phase 9.
• Chandak P, Huang K, Zitnik M. Building a knowledge graph to enable precision medicine. Sci Data. 2023. PrimeKG construction and edge taxonomy.
• Vashishth S, Sanyal S, Nitin V, Talukdar P. Composition-based multi-relational graph convolutional networks (CompGCN). ICLR 2020.
• Dettmers T, Minervini P, Stenetorp P, Riedel S. Convolutional 2D knowledge graph embeddings (ConvE). AAAI 2018.
• Galkin M, Yuan X, Mostafa H, et al. Towards foundation models for knowledge graph reasoning (ULTRA). ICLR 2024. PDR2’s foundation model (used for an internal note on the first GP-KG run).
• RGCC International. Onconomics Plus chemosensitivity assay. Methodology described in lib/pdr/Supplementary_Material/. Annexin V-PE flow cytometry on isolated CTCs at fixed drug concentrations, 72h chemo / 48h targeted-therapy incubation.

---

Appendix A — RGCC sensitivity score distribution

For reference, the full distribution of score_raw values across the 224 numeric-scored entries before per-drug deduplication:

Sensitivity category	Count	min	max	mean	Includes
high_sensitivity	10	80	82	81.1	Carboplatin, Cisplatin, Mitomycin, Oxaliplatin, Pemetrexed
moderate_sensitivity	26	52	79	67.6	5-FU, Capecitabine, Doxorubicin, MTX, Daunorubicin, Mitoxantrone, …
low_sensitivity	82	5	31	14.1	Cyclophosphamide, Ifosfamide, Bevacizumab, Cetuximab, …
resistant	106	0	0	0.0	Sorafenib, Dasatinib, Imatinib, most targeted therapies

The 80%/35% boundaries used in Tier 2 cleanly separate the high-sensitivity cluster (all ≥ 80%) from low + resistant (all ≤ 31%), with moderate (52–79%) in between as the explicit exclusion zone.

Appendix B — Top-15 model picks per checkpoint (matched RGCC drugs)

For comparison; full lists in runs/2026-05-09_phase7.5/<run>/crc_rgcc_metrics.matched.csv.

Run B (corr 1L 400d) — top 15: Liposomal Doxorubicin (rank 2), Doxorubicin (2), Cetuximab (3), Sorafenib (5), MTX (8), 5FU (9), BCNU (10), Dasatinib (11), Ramucirumab (12), Imatinib-mesylate (13), Daunorubicin (14), Mitoxandrone (16), Pazopanib (17), Nilotinib (22), Bevacizumab (23).

A1 (sub 1L 300d) — top 15: 5FU (rank 2), Pemetrexed (6), Daunorubicin (8), Liposomal Doxorubicin (10), Doxorubicin (11), Capecitabine (13), Ifosfamide (15), Mitoxandrone (17), MTX (19), Carmustine (24), Tamoxifen (29), Vincristine (33), Etoposide (37), 5FU/Methotrexate (39), Cyclophosphamide (44).

v1 (corr 1L 300d) — top 15: Cisplatin (rank 17), Sorafenib (19), Daunorubicin (20), Doxorubicin (21), Carboplatin (31), Imatinib (33), Bortezomib (35), Bevacizumab (38), Cetuximab (41), 5FU (44), Sunitinib (47), Capecitabine (48), Pazopanib (51), Lapatinib (53), Mitoxantrone (54).

The qualitative differences across these lists — Run B leads with anthracyclines + biologics, A1 leads with antimetabolites and Pemetrexed, v1 leads with platinums — reflect the architectural personalities documented in §4.5.