Superseded

This page is kept for the record. The current version is PDR2 Track A v2 — Foundation Retrain. The v2 retrain’s reported 224% lift on the patient’s own edges is unrestated, because both foundations trained on her drug-response edges (GH #214, #439). The architectural findings do carry forward. Treat every number below as historical.

PDR2 Track A — First Personalized Inference, Mechanism Reproduction, and Architectural Findings

Authors: whafa, Claude (Maple project) Date: 2026-05-18 Status: Reproduction of PDR2’s personalization mechanism on Maureen’s KG achieved. Foundation training + patient-conditional fine-tune + drug-only evaluation pipeline is end-to-end runnable on Vertex AI A100-80GB. Pipeline reproduces paper-faithful behavior; synthetic positive-control test confirms wiring is correct. Cross-predicate architectural limitation surfaced.

Companion documents:

• docs/pdr_personalization_writeup.md — PDR1 vs PDR2 personalization mechanism reference (this writeup operationalizes §4 of that doc).
• docs/PDR_PIPELINE_PLAN.md — master pipeline plan; this run executes Phase 11 (PDR2 upgrade) for the first time end-to-end.
• docs/pdr_phase7_5_rgcc_writeup.md — RGCC validation framework (used as the future per-patient validation hook).
• Memory: the personalization mechanism reference.

⚠ Correction, 2026-09-27 (GH #435). The “interpretable criterion weights” in the Summary and §4.1 are initialization values, not learned ones. Expression entered the loss as raw 0–100 percentiles, so tanh(Σ normalize(v)·expr) was saturated from the first step and the protein weights received essentially no gradient. Across all five fine-tunes run so far, protein-weight RMS ended at 0.009–0.016 against 0.01 at initialization. The ranking of ADORA2A, TSC2 and the rest is therefore noise. The mechanism-reproduction and architectural findings below are unaffected.

---

Summary

On 2026-05-17/18 we built Maple’s Vertex AI orchestration for PDR2 patient-conditional fine-tune and evaluation, ran the first real foundation-model fine-tune for Maureen (Track A: zero PRS + real Caris MI Profile expression on the PrimeKGCIViCPharmGKBMaureen graph), and confirmed via synthetic positive-control test that our pipeline faithfully reproduces the paper’s apply_personalization mechanism. The fine-tune produced interpretable criterion weights (top protein-side weights on ADORA2A, TSC2, SSTR2, NRG1, CCNE1 — biologically reasonable for Maureen’s tumor) but no MRR lift on her held-out edges at λ₂=0.01 (Δ_MRR = −0.41% vs foundation baseline). A synthetic test with extreme parameters (TP53 anchor, w=1.0, expr=100, λ=1.0) produced the predicted dramatic score shifts, confirming the pipeline math is correct. The same synthetic test surfaced an architectural limitation of the paper’s design: the drug-only adjustment mask isolates receivers but not non-drug-tail queries, because rankings are over all 139K entities — boosted drugs pollute the top of every ranking regardless of the query’s true tail type. This is the paper’s design as faithfully reproduced; addressing it for general KG completion (Maureen’s heterogeneous tail-type queries) would require a per-relation criterion architecture, deferred. With OncoKB now license-approved as of this morning, the next concrete move is to land Phase 8.5 (variant-aware KG with OncoKB chains) and retrain the foundation against a richer Maureen overlay — making the fine-tune meaningful at normal λ, not just synthetic extremes.

---

1. Purpose

PDR2’s personalization mechanism had three open questions when we started:

1. Does it run end-to-end on Vertex AI? Phase 1 of the PDR2 training automation plan (an internal note on the training-automation status) had reached aiplatform.CustomJob.submit() but was blocked on quota. We confirmed quotas, ran the first real training, and added two new container modes for fine-tune and personalized eval.
2. Does the mechanism produce signal on Maureen’s data? The 2026-05-09 writeup (docs/pdr_personalization_writeup.md) documented the loss formula and apply_personalization algorithm. This run was the first end-to-end empirical test.
3. Does our implementation reproduce the paper faithfully? We could not distinguish “real null result” from “wiring bug” without a positive control. A synthetic-data sanity test resolved this.

Out of scope for this run: variant-aware KG (Phase 8.5 — unblocked this morning by OncoKB license approval; deferred to a follow-up session), Track B (requires Maureen’s genotype upload), λ sweep on real data, real-data eval at non-default lambdas.

---

2. Pipeline We Built

2.1 Vertex AI orchestration

Layer	Artifact	What it does
Container	gcr.io/PROJECT/maple-pdr2-training:c8a3826-oomprobe → :949839f-vocabfix → :4a85ec4-classifyfix (final)	PyTorch 2.1.0 + CUDA 11.8 base, PDR2 code at /opt/pdr2/code, entrypoint dispatches on MODE env. Built via cloudbuild-pdr2-training.yaml.
Entrypoint	lib/pdr2/docker/entrypoint.sh	3 modes: train, fine_tune, eval_patient, evaluate_personalized. Downloads training data + checkpoints from GCS, runs the matching Python entry, uploads outputs to GCS.
Submit script (foundation)	backend/scripts/submit_pdr_run.py	POSTs to /api/mods/research/runs/submit → worker → Vertex CustomJob. Existed before this run; used implicitly for the original foundation training in Colab.
Submit specs (this run)	backend/scripts/vertex_finetune_job.json, vertex_eval_patient_job.json, vertex_evaluate_personalized_job.json, vertex_eval_synth_tp53_job.json	One-off CustomJob spec JSONs, invoked via gcloud beta ai custom-jobs create --config=…. Deliberately not auto-generated; we want submit-by-hand for now (Phase 2 of automation is deferred).
Service account	the training service account	Used by Vertex jobs to read/write GCS.
GCS bucket	gs://BUCKET/	Holds container input data, foundation checkpoints, patient pickles, fine-tune outputs, eval JSONs.

2.2 Container modes (new)

Two new modes added this run, alongside the existing train and eval_patient:

MODE=fine_tune (entrypoint.sh L155+) — invokes fine_tune_maple.py, our Maple-authored wrapper around upstream fine_tune_patient_specific. Required env: RUN_ID, DATASET_CLASS, EPOCHS, BPE, PATIENT_ID, DATA_URI, OUTPUT_URI, CKPT_URI (foundation), PATIENT_PICKLE_URI. Optional: LAMBDA1_PRS, LAMBDA2_BIOMARKER (default 0.01), FINE_TUNE_MAX_BATCH (default 256). Outputs pmaureen_pc_full_state_dict.pth (bundle: {"model", "criterion", "patient_specific_only"}) + metrics.json + manifest.json.

MODE=evaluate_personalized (entrypoint.sh L213+) — invokes evaluate_personalized_maple.py, Maple’s paper-faithful eval. Required env: RUN_ID, DATASET_CLASS, DATA_URI, OUTPUT_URI, CKPT_URI (fine-tune bundle), PATIENT_CURIE, PATIENT_PICKLE_URI, RESOLUTION_SIDECAR_URI. Optional: LAMBDA1_PRS, LAMBDA2_BIOMARKER, ADJUSTMENT_TARGET_DOMAIN (default any; set drug for paper-faithful). Outputs eval_personalized.json with raw + personalized + per-predicate metrics.

MODE=eval_patient (pre-existing) — Maple-original tool that scores Maureen’s outbound edges using foundation-only model scoring (no criterion). Used as the baseline comparator.

2.3 Patient input artifacts

Pickle builder. backend/scripts/build_pdr2_input_pickle.py generates the PDR2-expected dict {prs_values, expr_values, disease_names, protein_names} from Maureen’s DB rows. Three source modes:

• synthetic_zeros (Track A): zero PRS + real expression
• maureen (Track B): real PRS + real expression — blocked until genotypes upload
• ukb_sample: reserved placeholder

Resolution sidecar (new this run). The builder now also emits <pickle>.resolution.json mapping each criterion entry (disease CUI or HGNC gene symbol) to its canonical KG CURIE. resolve_disease_kg_curies (UMLS direct match + xref) and resolve_gene_kg_curies (name = sym AND domain = 'gene', lowest NCBIGene ID = human canonical). For Maureen: 6/13 diseases resolved (CRC, CAD, HT, AF, PD, AST, MS are missing — they exist in PrimeKG as MONDO:* but aren’t UMLS-xrefed in mods_kg_entity_source); 84/84 proteins resolved cleanly via NCBIGene.

Numpy-portable pickle (durable patch). The original pickle saved np.ndarray directly. Local Maple env runs NumPy 2.x; Vertex container runs NumPy 1.x. Cross-version pickle of np.ndarray fails with ModuleNotFoundError: No module named 'numpy._core'. Fixed in as_dict(): serialize as Python lists (numpy-version-portable; container’s torch.tensor(...) accepts lists identically).

2.4 PDR Input Inspector (frontend)

A new tab in Research Dashboard for inspecting the assembled artifact pre-training:

• Backend: GET /api/mods/prs/preview/pdr-input?format={pdr1|pdr2}&source={synthetic_zeros|maureen|ukb_sample} (mods_prs_api.py:466+)
• Tests: backend/tests/test_pdr_input_preview.py — 8/8 pass.
• MCP tool: mods_preview_pdr_input in mods_server.py.
• Frontend: PDRInputInspector.jsx mounted as a tab in ResearchDashboard.jsx:33 after “Patient Profile.”

The inspector renders the disease × PRS table, gene × expression table, PDR1 trailing list columns (medications, CUIs, AD column), raw response JSON (copyable), and reuses the existing /export/pdr1 and /export/pdr2 endpoints for downloads. Track A vs Track B toggle, format toggle, optional z-scored preview for PDR2.

---

3. Materials and Methods

3.1 Foundation model source

The foundation was not trained on Vertex in this session — it was the existing Colab run 2026-05-16_pdr2_primekg_civic_pharmgkb_v1_with_maple_patient_scratch_full_v1, copied pristine from Drive to GCS at:

gs://BUCKET/runs/2026-05-17_track_a_pharmgkb_finetune_v1/foundation_pristine_copy.pth

(691 KB; raw model.state_dict(), no wrapping dict.)

Foundation architecture (per the Colab run’s train/manifest.json): hidden_dim=64, n_layers=6, message_func=distmult, aggregate_func=sum, short_cut=True, layer_norm=True. 168,705 parameters. Dataset class PrimeKGCIViCPharmGKBMaureen (composition primekg_civic_pharmgkb_v1 + maple_patient overlay). 139,226 entities, ~9.2M relations, Maureen’s 248 outbound canonical edges materialized via mods_patient_kg_service.materialize_patient_kg.

3.2 Fine-tune (Track A)

	Value
Run id	2026-05-17_track_a_pharmgkb_finetune_v1
Vertex CustomJob	1153812916479197184
GPU	A100-80GB
Container	c8a3826-oomprobe
Epochs	20
Batches per epoch	500
Batch size	Probe-discovered = 128 (halved once from 256 starting point)
λ₁ (PRS)	0.01
λ₂ (biomarker)	0.01
Patient pickle	gs://BUCKET/patient_data/maureen_track_a_synthetic_zeros.pkl
Trainable parameters	97 (13 disease + 84 protein weights)
Frozen parameters	168,705 (entire foundation)
Wallclock	24m 30s

3.3 Synthetic positive control

To distinguish “real null result” from “wiring bug,” we constructed a synthetic patient + criterion bundle designed to produce a predictable and dramatic score shift if the pipeline is wired correctly:

• Anchor: TP53 (NCBIGene:7157)
• protein_weights = tensor([1.0]) (hand-set, not gradient-trained)
• expr_values = [100.0] (hand-set)
• disease_weights = tensor([]) / prs_values = [] (no disease channel)
• λ₂ = 1.0 (100× default)
• Predicted per-entity adjustment for 1-hop TP53 neighbors: 1.0 × 1.0 × 100.0 = +100.0

Raw foundation scores observed on Maureen’s queries are in the [10⁻⁴, 10⁻²] range; a +100 additive offset must catastrophically dominate. The exact predicted observable: any entity 1-hop-adjacent to TP53 should rank near the top of every Maureen-anchored query, regardless of the query’s actual semantics.

Built via backend/scripts/build_synth_finetune_bundle.py.

3.4 Evaluation methodology

We score Maureen’s outbound edges in valid+test splits (138 edges across 11 predicates) by ranking all 139,226 candidate tails per query. Metrics:

• Filtered rank: number of entities with score strictly greater than the true tail’s score, excluding other known-true tails for the same (head, relation) pair, +1.
• MRR: mean(1/rank) across edges; bounded [0, 1]; random baseline = 1/139,226 ≈ 7.2×10⁻⁶.
• Hits@k: fraction of edges with rank ≤ k; bounded [0, 1].

The eval is over the entire entity vocabulary (139K), not over a candidate set restricted to drugs. This is the same protocol our baseline eval_patient uses; it’s stricter than the paper’s protocol (which scores only drug-tail queries against drug candidates), but it lets us evaluate Maureen’s heterogeneous tail-type predicates honestly.

---

4. Results

4.1 Fine-tune output

Wall-clock: 24m 30s. Fine-tune converged (pre-flight probe settled at batch_size=128 on the first iteration; A100-80GB had ~12 GiB headroom on top of the trained batch).

Top criterion weights after 20 epochs (metrics.json):

Rank	Protein (gene)	Weight	Sign	Note
1	ADORA2A	+0.0255	+	Adenosine A2A receptor (immune-checkpoint target, increasingly relevant in cancer immunotherapy)
2	TSC2	−0.0254	−	TSC2 (mTOR-pathway suppressor)
3	SSTR2	+0.0193	+	Somatostatin receptor 2 (neuroendocrine target)
4	NRG1	+0.0182	+	Neuregulin 1 (fusion-positive cancer target)
5	CCNE1	−0.0176	−	Cyclin E1 (cell-cycle marker)

(⚠ 2026-09-27: these are initialization values, not learned weights. See the correction at the top.)

Disease weights all stayed at random-init scale (~0.017) — expected for Track A. The gradient on disease_weights[i] is λ₁ × ... × prs_values[i]; with prs_values = 0 everywhere, the gradient is zero and the weights don’t move.

Per the gradient analysis in the personalization mechanism reference, the loss is gradient-degenerate (criterion gradient is batch-independent), so the protein weights also converge to a closed-form optimum — Σ w·v → −∞, i.e., weights aligned with −sign(expr_values). The signs above reflect this: TP53 weight is positive if Maureen has low TP53 expression (or weight is whatever direction makes the dot product more negative).

4.2 Real fine-tune eval — no measurable lift

	Foundation baseline	Personalized (λ=0.01, target=any)	Δ
Filtered MRR	0.005152	0.005131	−0.41%
Filtered Hits@10	0.0000	0.0000	0
Filtered Mean Rank	4524.1	4540.2	+0.4% (worse)

By predicate (top-3 movers):

Predicate	N	Δ_MRR
MAPLE:partially_sensitive_to_drug	29	+0.000161 (+1.8%)
MAPLE:has_genetic_alteration_in	3	−0.001617 (−6.4%)
All other 9 predicates	106	within noise (±0.05% of baseline)

Verdict: at λ=0.01 with the protein weights this fine-tune produced (max magnitude 0.025), the additive adjustment is comparable to raw model-score noise. No real signal.

4.3 Synthetic test — pipeline reproduces paper-faithful behavior

We ran three synthetic eval iterations to isolate the wiring:

Synth v1 — adjustment_target_domain=any, fixed classifier. Hand-set TP53 anchor, w=1.0, expr=100, λ=1.0. Predicted: every 1-hop neighbor of TP53 (regardless of type) gets +100. Observed:

Predicate	N	Raw MRR	Pers MRR	Δ
MAPLE:has_genetic_alteration_in	3	0.0253	0.0559	+121%
MAPLE:partially_sensitive_to_drug	29	0.0089	0.0004	−95%
MAPLE:tried_drug	6	0.0086	0.0005	−94%
(8 other predicates)	100	(raw small)	(smaller)	−88% to −95%
Overall filtered MRR	138	0.005152	0.001888	−63%

The huge gene-tail MRR jump is has_genetic_alteration_in working perfectly — Maureen’s true gene tails (BRAF, ATM, APC) are biologically close to TP53 (all tumor-suppressor / DNA-damage related), so they ARE TP53-neighbors in PrimeKG, so they get the +100 boost. The drug-tail MRR craters: Maureen’s actual drugs (e.g., Cisplatin, Cetuximab) are NOT TP53-adjacent, but ~thousands of TP53-adjacent entities (other genes, pathways, etc.) all get +100 and rank above her drugs. Predicted behavior, exactly observed.

Synth v2 — adjustment_target_domain=drug, but classifier broken. Same inputs, drug-only mask applied. Observed: zero change across all predicates (Δ exactly 0.000000 for every predicate). Reason: adjustment_target_entity_count: 0 — drug_inv_vocab was empty.

Diagnosis: TransductiveDataset.classify_entity uses the paper’s naked-identifier regexes (^C\d{7}$ for disease, ^DB\d{5}$ for drug) which don’t match our Biolink-style CURIEs (UMLS:C0006142, DrugBank:DB00303). Every entity falls through to the “gene” catch-all bucket, leaving drug_inv_vocab empty.

Fix: Override classify_entity to match Biolink prefixes (DrugBank:, CHEBI:, PUBCHEM.COMPOUND:, UNII: for drugs; MONDO:, UMLS:, DOID:, OMIM:, EFO: for diseases; HPO:, HP:, MP: for phenotypes; NCBIGene:, HGNC:, etc. for genes; everything else stays “gene” catch-all). See datasets.py:260.

Synth v3 — drug-only with fixed classifier. Same inputs. Observed:

Predicate	N	Raw MRR	Pers MRR	Δ
biolink:has_variant	1	0.000072	0.000072	0 (exact)
biolink:has_phenotypic_feature	4	0.000114	0.000114	0 (exact)
MAPLE:resistant_to_drug	23	0.0028	0.0027	−3%
MAPLE:sensitive_to_drug	4	0.0044	0.0042	−4%
MAPLE:current_medication	8	0.0038	0.0036	−6%
MAPLE:past_medication	29	0.0032	0.0029	−11%
MAPLE:over_expresses	24	0.0028	0.0024	−16% (gene-tail)
MAPLE:under_expresses	7	0.0073	0.0064	−13% (gene-tail)
MAPLE:tried_drug	6	0.0086	0.0075	−13%
MAPLE:partially_sensitive_to_drug	29	0.0089	0.0071	−20%
MAPLE:has_genetic_alteration_in	3	0.0253	0.0174	−31%

adjustment_target_entity_count: 7,782 (drug entities in the KG). Mask is working.

4.4 Architectural finding: cross-predicate noise

Synth v3 surfaces something the paper’s eval scope hid. Even with the paper-faithful drug-only restriction, gene-tail predicates moved (some by 13–31%). Why?

The ranking is over all 139,226 entities by final_score = raw_score + adj. With +100 boost on ~7,782 TP53-adjacent drugs, every TP53-adjacent drug that previously had raw_score below the true gene-tail now ranks above it. The gene-tail’s rank gets worse by that count.

For predicates where the true tail’s raw rank is already very high (e.g., has_variant, has_phenotypic_feature — raw MRR < 10⁻³, true tail ranks below most things anyway), few TP53-adjacent drugs are below them in the first place, so nothing changes. For predicates where the true tail had a good raw rank (has_genetic_alteration_in, raw MRR 0.025 ≈ rank 40), many TP53-adjacent drugs were below it, and now they’re all above.

This means: the drug-only mask isolates the receivers, but doesn’t isolate non-drug queries from the boosted-drug pollution. The paper avoided this because they only evaluate drug-tail queries against drug candidates. For Maureen’s heterogeneous tail-type ground truth (genes, drugs, variants, phenotypes mixed), the paper’s mechanism has an inherent cross-predicate noise floor.

This is a real architectural limitation of the paper’s design for general KG completion. Phase 12 (specialist per-relation criteria — see paan_planning.md) is the natural extension that would eliminate it.

---

5. Quantitative Comparison: v1 vs v3 on the Same Synthetic Inputs

Predicate	N	Raw MRR	v1 (any) Δ	v3 (drug-only) Δ
MAPLE:has_genetic_alteration_in	3	0.0253	+121%	−31%
MAPLE:partially_sensitive_to_drug	29	0.0089	−95%	−20%
MAPLE:tried_drug	6	0.0086	−94%	−13%
biolink:has_variant	1	0.000072	−7%	0
biolink:has_phenotypic_feature	4	0.000114	−17%	0

The v1→v3 sign flip on has_genetic_alteration_in is the cleanest demonstration of the mask working: in v1 the gene tails received +100 directly (because gene neighbors of TP53 anchor were also boosted), so they ranked first. In v3 the gene tails don’t receive +100 (mask excludes non-drugs), but drug-boost pollution still hurts them — though much less than in v1.

The v1→v3 magnitude reduction on drug predicates (−95% → −20%) shows the mask also helping drug-tail queries: in v1, ALL entities adjacent to TP53 (including many genes/diseases above Maureen’s actual drugs) got the boost, swamping drug rankings; in v3 only drugs swam to the top, and some of Maureen’s true drugs do benefit from the boost while others suffer relative drop.

---

6. What Now Works

End-to-end Vertex pipeline:

1. Local builder: build_pdr2_input_pickle.py --source synthetic_zeros produces pickle + resolution sidecar.
2. Local builder: build_synth_finetune_bundle.py produces synthetic test artifacts (no fine-tune needed).
3. GCS upload via gcloud storage cp.
4. Foundation pristine-copy from Drive (manual one-time per source run).
5. Submit Vertex CustomJob via gcloud beta ai custom-jobs create --config=<json> — supports modes fine_tune, eval_patient, evaluate_personalized.
6. Outputs land in GCS; manual inspection / download.

Verified to work:

• Foundation pristine-copy + fine-tune produces interpretable criterion weights (Section 4.1).
• Baseline eval_patient scores Maureen’s edges against foundation only.
• evaluate_personalized scores Maureen’s edges against foundation + criterion adjustment.
• adjustment_target_domain=drug faithfully reproduces the paper’s apply_personalization.
• adjustment_target_domain=any extends to all entities (broader receivers, but cross-predicate noise).
• OOM auto-probe finds the largest fitting batch (e.g., 128 on A100-80GB for this graph).
• NumPy 2.x ↔ 1.x pickle portability (lists, not arrays).

Still requires manual orchestration:

• Job submission is by hand (not auto via the worker handler).
• Foundation training itself happens in Colab; we copy artifacts to GCS.
• Phase 2 of the PDR2 training automation (frontend dashboard) is deferred.

---

7. Architectural Findings — Things to Remember

7.1 The classify_entity-Biolink gotcha

The upstream TransductiveDataset.classify_entity (lib/pdr2/code/ultra/datasets.py:260) assumes the paper’s naked-identifier convention. Our KG uses Biolink-style CURIEs. Without the patch, every entity falls into gene_inv_vocab, breaking the paper-faithful drug-only restriction in apply_personalization. Any future PDR2 fork using our data will hit this — the symptom is adjustment_target_entity_count: 0 (or whatever count = “no drugs found”). Saved in memory.

7.2 The personalized loss is gradient-degenerate

As coded (losses.py:60-86), the criterion’s gradient depends only on (criterion_weights, fixed_patient_values) — not on batch contents. 20 epochs and 1 epoch converge to the same optimum. The paper’s TEXT suggests a per-triple score multiplier in the loss; the CODE doesn’t have one. We faithfully reproduce the code. Saved in memory.

7.3 Drug-only mask isolates receivers, not queries

Already documented in §4.4. Important enough to call out separately: even with the paper-faithful restriction, Maureen’s gene-tail and variant-tail queries are noisily affected by drug-receiver boosts, because rankings span the whole entity vocabulary. Saved in memory.

7.4 Vertex AI ↔ Maple-data prefix mismatch is a recurring failure mode

Beyond classify_entity, two other code points needed Maple-side patches:

• eval_patient.py:179-184 and evaluate_personalized_maple.py:328-339 looked for vocab keys inv_entity_vocab / inv_rel_vocab; the dataset actually saves them as entity_dict_forward / relation_dict_forward. Fixed with a fallback chain.
• apply_personalization itself (training.py:198+) has a hardcoded author-local vocab path (/usr/homes/cxo147/...) that we do not currently invoke directly; our evaluate_personalized_maple.py reimplements the adjustment logic to avoid needing the typed-vocab pickle at that path. If anyone ever wants to call upstream evaluate_model_personalized directly, this path will need to be parameterized.

7.5 NumPy 2.x / 1.x pickle cross-version incompatibility

Local Maple env is NumPy 2.x; the PyTorch 2.1 base image in our container is NumPy 1.x. pickle.dump(np.ndarray, ...) from one side fails to load on the other (ModuleNotFoundError: 'numpy._core'). Fix: serialize as Python lists in the pickle. Saved in memory.

7.6 L4 capacity is unreliable in us-central1 right now

We had quota for L4 granted but the Vertex jobs sat in PENDING for hours waiting for actual hardware availability. A100-40GB and A100-80GB capacity was reliable. For eval jobs (which don’t actually need 80GB), switching to A100-40GB resolved provisioning. Saved in memory.

---

8. Open Questions & Next Steps

8.1 The Phase 8.5 OncoKB unblock (this morning)

OncoKB license approval came through this morning. The PDR_PIPELINE_PLAN.md Phase 8.5.4 (“OncoKB connector ingest”) is now license-cleared. The next concrete move (per user direction, in another session) is to:

1. Ingest OncoKB actionable variants → canonical KG via backend/connectors/oncokb_connector.py
2. Re-materialize Maureen’s patient overlay with variant-level edges (biolink:has_variant to her actual variants, not gene-level placeholders) — the resolver fix already shipped per the variant-aware KG extension.
3. Rebuild the composition primekg_civic_oncokb_pharmgkb_v1 (or similar) with Maureen included.
4. Retrain the foundation from scratch on this richer graph.
5. Re-run the Track A pipeline (this session’s outputs become the comparator).

Expected payoff: Maureen’s patient overlay grows from ~248 edges (mostly gene-level) to a much larger and more biologically specific set with variant chains (BRAF V600E predicts sensitivity to vemurafenib; the second-hit question predicts PARP-inhibitor benefit; APC variants pattern…). The fine-tune criterion will have richer features to anchor against, and apply_personalization will reach drugs through clinically meaningful evidence chains rather than generic gene-drug edges.

8.2 What’s not addressed by Phase 8.5

OncoKB doesn’t fix:

• The gradient-degeneracy of the loss (§7.2) — still need Phase 12 or paper-equivalent loss rework if we want criterion gradient to actually use batch information.
• The cross-predicate noise (§4.4, §7.3) — Phase 12 per-relation criteria is the architectural answer.
• The lack of a true validation gold standard for personalization on Maureen — RGCC (pdr_phase7_5_rgcc_writeup.md) is the closest, but it’s drug-tail-only. For variant or expression tails, we’d need clinical follow-up or separate ground-truth construction.

8.3 Items intentionally deferred

• Visualization of the receiver-set (Tier 1 PNG vs Tier 2 Explorer overlay) — deferred. Would be useful once we have a Phase-8.5-retrain checkpoint to visualize against.
• λ sweep on real Maureen data — deferred. Without a Phase 8.5 retrain, the protein weights’ magnitude is too small for any λ that’s not synthetically extreme to register.
• Phase 2 of the PDR2 training automation (frontend integration of submit/monitor flow) — deferred. Manual gcloud submission is adequate for now.
• Per-relation criterion architecture (Design A from earlier discussion) — deferred behind Phase 8.5.

---

9. Reference: GCS Artifacts from this Run

Artifact	Path
Track A patient pickle	gs://BUCKET/patient_data/maureen_track_a_synthetic_zeros.pkl
Track A resolution sidecar	gs://BUCKET/patient_data/maureen_track_a_synthetic_zeros.resolution.json
Foundation pristine copy	gs://BUCKET/runs/2026-05-17_track_a_pharmgkb_finetune_v1/foundation_pristine_copy.pth
Fine-tune output bundle	gs://BUCKET/runs/2026-05-17_track_a_pharmgkb_finetune_v1/finetune/pmaureen_pc_full_state_dict.pth
Fine-tune metrics	gs://BUCKET/runs/2026-05-17_track_a_pharmgkb_finetune_v1/finetune/metrics.json
Fine-tune manifest	gs://BUCKET/runs/2026-05-17_track_a_pharmgkb_finetune_v1/finetune/manifest.json
Baseline eval_patient v3 (vocabfix container)	gs://BUCKET/runs/2026-05-17_track_a_pharmgkb_eval_patient_v1/eval_patient/eval_patient.json
Personalized eval v3	gs://BUCKET/runs/2026-05-17_track_a_pharmgkb_eval_personalized_v1/eval_personalized/eval_personalized.json
Synth v1 (any-domain)	gs://BUCKET/runs/2026-05-18_synth_tp53_w1_v100_eval_v1/eval_personalized/eval_personalized.json
Synth v2 (drug-only, vacuous)	gs://BUCKET/runs/2026-05-18_synth_tp53_w1_v100_eval_v2_drugonly/eval_personalized/eval_personalized.json
Synth v3 (drug-only, fixed classifier)	gs://BUCKET/runs/2026-05-18_synth_tp53_w1_v100_eval_v3_classifyfix/eval_personalized/eval_personalized.json
Synth patient pickle	gs://BUCKET/runs/2026-05-18_synth_tp53_w1_v100/synth_patient.pkl
Synth resolution sidecar	gs://BUCKET/runs/2026-05-18_synth_tp53_w1_v100/synth_patient.resolution.json
Synth finetune bundle	gs://BUCKET/runs/2026-05-18_synth_tp53_w1_v100/finetune/pmaureen_pc_full_state_dict.pth

10. Reference: Container Image Tags

Tag	Adds
9d4f1ef-finetune	MODE=fine_tune branch in entrypoint, fine_tune_maple.py
8438209-personalized	MODE=evaluate_personalized branch, evaluate_personalized_maple.py
8438209-batchfix	FINE_TUNE_MAX_BATCH env var override
c8a3826-oomprobe	Pre-flight VRAM probe in fine_tune
949839f-vocabfix	Vocab-key fallback (entity_dict_forward)
4a85ec4-drugonly	--adjustment_target_domain flag in evaluate_personalized
4a85ec4-classifyfix	Biolink CURIE patterns in classify_entity (current production)

11. Cross-References

Code touched this run

• NEW: lib/pdr2/code/script/fine_tune_maple.py
• NEW: lib/pdr2/code/script/evaluate_personalized_maple.py
• NEW: backend/scripts/build_synth_finetune_bundle.py
• NEW: frontend/src/components/PDRInputInspector.jsx
• NEW: backend/scripts/vertex_*.json (4 spec files)
• NEW: backend/tests/test_pdr_input_preview.py
• PATCHED: lib/pdr2/code/script/training.py — FINE_TUNE_MAX_BATCH + pre-flight probe
• PATCHED: lib/pdr2/code/script/eval_patient.py — vocab-key fallback
• PATCHED: lib/pdr2/code/ultra/datasets.py — classify_entity Biolink CURIEs
• PATCHED: lib/pdr2/docker/entrypoint.sh — two new modes
• PATCHED: backend/scripts/build_pdr2_input_pickle.py — resolution sidecar + portable pickle
• PATCHED: backend/mods_prs_api.py — /preview/pdr-input endpoint
• PATCHED: backend/mcp_servers/mods_server.py — mods_preview_pdr_input MCP tool
• PATCHED: frontend/src/components/ResearchDashboard.jsx — PDR Input tab

Memory pointers (saved this run)

• the personalization mechanism reference — Mechanism A + B + apply_personalization, with code snippets and the gradient-degeneracy observation.
• an internal note on the training-automation status — Phase 1 status updated (quota granted, first run done, container modes documented).
• an internal note on Biolink entity classification — Classifier gotcha.
• an internal note on NumPy pickle portability — NumPy 1.x↔2.x pickle gotcha.
• an internal note on Vertex L4 capacity — L4 PENDING vs A100 reliable.

Things NOT captured anywhere (so they only exist in this writeup)

• The Vertex CustomJob IDs from this run (operational ephemera; can be looked up by display_name pattern).
• The exact build-iteration sequence (3 container builds before drug-only worked; recoverable from gcloud builds list).
• The full prefix survey output for our train.txt (only the top-N counts are above).
• The interplay between PDR Input Inspector frontend and ResearchDashboard wiring (sufficient code references above, but no narrative).
• The reasoning behind the OOM probe halving strategy vs pre-flight estimation (mentioned briefly; not standalone).
• The Cloud Build duration vs Vertex provisioning duration ratios (operational; not load-bearing).
• The chronology of L4 quota grant → L4 capacity failure → A100 fallback (in memory as the gotcha but no full timeline).

If you need any of these durable later, mention and I’ll spin them into a follow-up note.