PDR2 alt-loss A/B: paper-text formula is destructive; paper-code is inert

Date: 2026-05-22 Patient: Maureen (Track A, synthetic_zeros PRS + Caris MI Profile 84 protein TPM percentiles) Foundation: v1 PrimeKGCIViCPharmGKBMaureen (2026-05-16 Colab pretrain; identical to the 2026-05-17 baseline fine-tune) Code: lib/pdr2/code/script/losses.py (added PaperTextPersonalizedRepurposingLoss + MinimalPatchPersonalizedRepurposingLoss), container gcr.io/PROJECT/maple-pdr2-training:2f1e268-altloss Cost: ~$8fine-tune+$2 eval = ~$10 Run IDs: runs/2026-05-22_alt_loss_{paper_text,minimal_patch}_l{0p01,0p1}_v1 (4 fine-tunes) + _eval_v1 siblings (4 evals)

Correction, 2026-10-09. Several passages below assume that apply_personalization adds each adjustment to its own candidate’s score: in What this implies about the paper, in Suggested next directions for the authors, and twice in the draft letter. That is how our own evaluation code applies them. The authors’ released apply_personalization applies them differently. Every number on this page was measured with our evaluation.

Motivation

The audit in docs/pdr_personalization_writeup.md showed that the released PDR2 loss in losses.py (byte-identical to upstream at github.com/ceragoguztuzun/PDR-II) differs from the formula in §“Patient-Specific Loss function” of the paper text in three places:

	Paper text	Released code
Sign on λ terms	minus	plus
s_pred coupling	per-triple multiplication	none (per-patient scalar)
Output transform	none	tanh(·)
Weight normalization	none	F.normalize(·, p=2)

Maureen Track A on this foundation with the released loss showed null lift (Δ MRR = −0.4%). The question we wanted to answer: is the paper-text formula a better personalizer that the released-code regression obscures, or does the released code’s tanh/normalize actually do necessary work?

We built two alt-loss classes:

• paper_text — strict paper-text formula: minus signs, per-triple s_pred coupling (multiplied by mean(pos_scores) per batch for batch-size invariance), no tanh, no F.normalize. Reference for expr_j − ref_j is set to 0.5 (TPM-percentile midpoint).
• minimal_patch — keeps tanh + F.normalize (the shape of the released code) but applies the two paper-text corrections: flip signs, multiply by mean(pos_scores). Tests the hypothesis that tanh/normalize are stability tricks the paper text omits for brevity.

Default in our pipeline remains paper_code (the released loss); the alt classes are opt-in via LOSS_VARIANT env / --loss_variant CLI.

Headline result

Filtered MRR on Maureen’s 138 outbound edges (valid+test); higher is better.

Loss variant	λ	Baseline MRR	Personalized MRR	Δ %
paper_code	0.01 (2026-05-17)	0.005152	0.005131	−0.4% (null)
paper_text	0.01	0.005152	0.000391	−92.4% (destructive)
paper_text	0.1	0.005152	0.000288	−94.4% (destructive)
minimal_patch	0.01	0.005152	0.005215	+1.2% (marginal)
minimal_patch	0.1	0.005152	0.005136	−0.3% (null)

The paper-text formula is catastrophically destructive at both λ values tested. The minimal-patch variant (which preserves tanh + F.normalize) reproduces the paper-code result’s null-lift behavior almost exactly.

Why paper_text blows up: weight saturation, not learning

The criterion weights after fine-tuning tell the mechanism cleanly:

Variant	λ	Top-5 by |w|	Wall-clock	Epochs
paper_code (2026-05-17)	0.01	±0.025 (ADORA2A, TSC2, SSTR2, NRG1, CCNE1)	25 min	20 (no early-stop signal in writeup)
paper_text	0.01	all ≈ +4.07 (MUC1, EREG, MYC, NTRK2, EZH2)	67 min	20
paper_text	0.1	all ≈ +4.07 (CTLA4, NRG1, CLDN6, TOP1, TNFRSF9)	68 min	20
minimal_patch	0.01	±0.024 (TNFRSF9, ERBB3, MUC16, MAPRE1, TACSTD2)	28 min	early-stopped @ 8
minimal_patch	0.1	±0.022 (VEGFA, TF, BRCA1, MDM2, SSTR3)	28 min	early-stopped @ 8

The fine-tune stdout.log confirms:

[paper_text λ=0.01]
Epoch 5  - Loss: -186.1827  → New best model! Saving checkpoint
Epoch 9  - Loss: -318.8240  → New best model! Saving checkpoint
Epoch 14 - Loss: -491.1436  → New best model! Saving checkpoint
Epoch 19 - Loss: -666.1769  → New best model! Saving checkpoint
(every epoch is a new best; loss unbounded negative)

[minimal_patch λ=0.01]
Epoch 0 - Loss: -0.0071  → New best model! Saving checkpoint
Epoch 2 - Loss: -0.0075  → New best model! Saving checkpoint
Epoch 7 - Loss: -0.0059
Early stopping triggered after 8 epochs
(loss saturates in random-noise regime)

The paper-text loss is unbounded below: with PRS = 0 (Track A) and no regularizer, the optimizer can drive L_expr = (Σⱼ vⱼ · (exprⱼ − 0.5)) · mean(s_pred) arbitrarily large simply by scaling vⱼ in whichever direction matches the sign of (exprⱼ − 0.5). There is nothing in the loss to stop it.

The fact that every top-5 weight lands at ~+4.07 is an AdamW signature, not a learned representation:

• AdamW normalizes gradients to roughly unit RMS per parameter
• Effective step size per update ≈ lr = 2e-3
• Steps per fine-tune ≈ 20 epochs × 100 batches = 2,000
• Walk distance from random init ≈ 2,000 × 2e-3 = 4.0 ✓

The weights have walked the maximum AdamW distance in the gradient’s sign direction. There is no biological structure in this — just optimizer saturation. The released code’s tanh squashes the loss into [−1, 1], which bounds gradients and triggers early-stopping at ~8 epochs.

Per-predicate breakdown

Where does the personalization land? Sorted by |Δ%| within each run; n = number of edges:

paper_text λ=0.01 — uniform destruction

Predicate	n	Baseline	Personalized	Δ %
biolink:has_variant	1	0.000072	0.000182	+151% (n=1, noise)
MAPLE:partially_sensitive_to_drug	29	0.008861	0.000274	−97%
MAPLE:tried_drug	6	0.008615	0.000310	−96%
MAPLE:current_medication	8	0.003836	0.000186	−95%
MAPLE:sensitive_to_drug	4	0.004371	0.000219	−95%
MAPLE:past_medication	29	0.003214	0.000178	−94%
MAPLE:has_genetic_alteration_in	3	0.025343	0.002384	−91%
MAPLE:under_expresses	7	0.007336	0.000710	−90%
MAPLE:resistant_to_drug	23	0.002821	0.000407	−86%
MAPLE:over_expresses	24	0.002842	0.000597	−79%
biolink:has_phenotypic_feature	4	0.000114	0.000163	+42% (n=4)

Every meaningful-n drug predicate loses 86-97% of MRR. The criterion adjustment uniformly biases drug scores by ~+4 × (per-drug aggregate of patient-aligned expression neighbors), which is far larger than the foundation’s signal — so the foundation’s ranking gets washed out.

paper_text λ=0.1 — same pattern, more severe

Drug predicates collapse to −91% to −98%. The single biolink:has_variant edge still moves +107% (the only edge the criterion’s bulk-boost happens to align with).

minimal_patch λ=0.01 — near-null, one bright spot

Predicate	n	Baseline	Personalized	Δ %
MAPLE:over_expresses	24	0.002842	0.003352	+18%
MAPLE:tried_drug	6	0.008615	0.008307	−3.6%
biolink:has_phenotypic_feature	4	0.000114	0.000117	+2.7%
(other 8 predicates)	—	—	—	all |Δ| < 2%

Most predicates barely move. over_expresses gets a real +18% lift, which is consistent with the protein-criterion path actually mattering for over-expression queries (the criterion weights for proteins are exactly what should boost edges to those proteins). This is the only signal-bearing result across all 4 alt-loss runs.

minimal_patch λ=0.1 — mixed, bigger swings

Predicate	n	Baseline	Personalized	Δ %
MAPLE:over_expresses	24	0.002842	0.004700	+65%
MAPLE:has_genetic_alteration_in	3	0.025343	0.011806	−53%
MAPLE:tried_drug	6	0.008615	0.006075	−30%
biolink:has_phenotypic_feature	4	0.000114	0.000132	+16%
MAPLE:resistant_to_drug	23	0.002821	0.002935	+4%
(other 6 predicates)	—	—	—	all |Δ| < 4%

The over_expresses lift compounds (+65%), but at the cost of has_genetic_alteration_in (−53%) and tried_drug (−30%). Net overall is still null because the negatives offset the positives.

What this implies about the paper

Three findings worth communicating to the authors:

1. The paper-text formula is not viable as written. Without tanh or another saturating nonlinearity, the loss is unbounded below for any patient with non-zero (expr − ref) and a trained foundation (mean(s_pred) ≠ 0). Fine-tuning collapses to the AdamW walk distance, and the resulting criterion adjustment uniformly biases drug scores by an amount that overwhelms the foundation’s signal.

2. The released code’s tanh + F.normalize are necessary stability tricks, not regressions. Removing them (even while keeping the paper-text minus signs and s_pred coupling — the paper_text variant does both) destroys ranking quality across every drug-related predicate.

3. The released code’s loss is essentially inert for default λ. The minimal-patch variant — which keeps tanh + F.normalize and applies the sign/coupling corrections — produces near-identical null lift to the released code. Whatever directional value comes from the corrections gets squashed back to noise by the tanh. The lone bright spot is MAPLE:over_expresses (+18% at λ=0.01, +65% at λ=0.1) — protein expression queries that directly align with the criterion’s protein weights. Drug queries do not benefit.

The combination of (1) + (2) + (3) means the published mechanism does not personalize drug rankings for real patient data, in either formulation. The synthetic TP53 positive control that we ran on 2026-05-17 (and that’s documented in project_pdr2_mechanism.md) only confirms the inference-time apply_personalization plumbing — not that the loss-trained weights carry biological signal.

Suggested next directions for the authors

Three architectural changes might rescue this:

1. Bound the criterion loss. A simple tanh is destructive at default λ (as our minimal_patch confirms — early-stop at noise scale). But a bounded-but-saturating form like log(1 + exp(prs_impact)) (softplus) or an L2 weight regularizer on the criterion would let the loss stop driving the optimizer past meaningful magnitudes.

2. Per-predicate criterion. The current criterion adjusts every entity’s score by the same Σ λ·w·value·strength term. For Maureen, this lifts over_expresses MRR (where the adjustment direction aligns with the query intent) at the expense of has_genetic_alteration_in and tried_drug (where it does not). A predicate-conditional criterion — different weights for different query relations — would let the model express that biomarker-derived signal helps over_expresses queries but not history queries.

3. Anchor the criterion to query type at inference. The current apply_personalization adjusts all candidate entities regardless of what predicate is being scored. Restricting the adjustment to drug-tail predicates (the ADJUSTMENT_TARGET_DOMAIN=drug option we already have) might cleanly separate the “boost drugs adjacent to high-PRS diseases” signal from the cross-predicate damage we see in paper_text.

Reproduction

# Generate the 4 fine-tune + 4 eval Vertex job specs
python backend/scripts/alt_loss_2026_05_22/generate_specs.py
 
# Submit fine-tunes
for spec in backend/scripts/alt_loss_2026_05_22/finetune_*.json; do
    gcloud beta ai custom-jobs create --region=us-central1 --project=PROJECT \
        --config="$spec" --display-name="$(basename $spec .json)"
done
 
# Wait for completion
bash backend/scripts/alt_loss_2026_05_22/wait_for_finetunes.sh
 
# Submit evals
for spec in backend/scripts/alt_loss_2026_05_22/eval_personalized_*.json; do
    gcloud beta ai custom-jobs create --region=us-central1 --project=PROJECT \
        --config="$spec" --display-name="$(basename $spec .json)"
done
 
# Wait + compile
bash backend/scripts/alt_loss_2026_05_22/wait_for_evals.sh
python backend/scripts/alt_loss_2026_05_22/compile_comparison.py

Container tag built off working-tree state at SHA 2f1e268 with uncommitted edits to:

• lib/pdr2/code/script/losses.py (added two alt classes)
• lib/pdr2/code/script/training.py (loss_variant routing)
• lib/pdr2/code/script/fine_tune_maple.py (—loss_variant CLI)
• lib/pdr2/docker/entrypoint.sh (LOSS_VARIANT env)

Draft message to the authors (Cerag, et al.)

Never sent. We drafted this in May 2026 and hesitated over it. See the correction at the top of the page.

Hi Cerag,

Maple’s been working through PDR2 reproducing your patient-specific framework on a single oncology patient (Maureen — Track A, biomarker- only, no PRS), foundation pretrained on PrimeKG+CIViC+PharmGKB. We noticed that the released losses.py differs from the formula in your paper §“Patient-Specific Loss function” in a few places: the released code uses + λ·tanh(F.normalize(w)·PRS) (per-patient scalar, plus sign, tanh-squashed), while the text describes − λ·Σ wᵢ·PRSᵢ·s_pred (per-triple coupling, minus signs, no nonlinearity).

We wanted to understand which version reflects intent. We built two alt variants — a strict paper-text implementation and a “minimal patch” that keeps the released code’s tanh + F.normalize but applies the sign + s_pred corrections — and fine-tuned each at λ=0.01 and λ=0.1 on the same patient + foundation as our baseline. Results:

• Released code: −0.4% MRR (null lift, as we’d seen before)
• Strict paper-text: −92% to −94% MRR (catastrophic)
• Minimal patch (tanh + F.normalize retained): +1.2% / −0.3% (near-null, mirrors released code)

The strict paper-text loss is unbounded below: with PRS=0 the L_expr term is λ·(Σⱼ vⱼ·(exprⱼ−0.5))·mean(s_pred), which the optimizer drives to −∞ by scaling each vⱼ in the sign direction of (exprⱼ−0.5). After 20 epochs all top criterion weights land at ≈ +4.07 — exactly the AdamW saturation walk distance (2000 steps × lr=2e-3). When apply_personalization then adds + λ·w·value·strength ≈ +4 × per-drug-aggregate to each candidate score, the foundation’s signal gets washed out and drug-query MRR collapses uniformly across all 138 of the patient’s outbound edges.

The minimal-patch variant tells us the released code’s tanh + F.normalize aren’t bugs — they’re what’s keeping the loss in a stable regime. Without them the loss has no fixed point. With them the loss saturates in noise and the criterion weights stay at random- init magnitudes (early-stop at epoch 8, loss ≈ −0.005). Effectively equivalent to the released code.

One promising thread: in the over_expresses predicate alone (n=24 edges), minimal_patch at λ=0.1 produces +65% MRR — i.e., the protein-criterion path does carry signal for protein-targeted queries, just not for drug queries. The apply_personalization mechanism applies criterion adjustments to every candidate entity regardless of query type, which seems to be why the per-predicate lift gets offset by per-predicate damage elsewhere.

Our hypothesis: a per-predicate (or query-type-conditional) criterion would separate the two signals cleanly. We’re considering a few follow-ups along that axis and would value your read on whether the paper-text formulation is what you intended (in which case the loss may need a regularizer the text doesn’t mention), or whether the released code with tanh + F.normalize is canonical and we should treat the paper text as illustrative.

Full per-predicate breakdown, mechanism walk-through, criterion weight signatures, and reproduction recipe attached.

Thanks much, [name] Maple, [date]

---

Generated artifacts:

• backend/scripts/alt_loss_2026_05_22/results/comparison.md — full per-predicate tables
• backend/scripts/alt_loss_2026_05_22/results/comparison.json — machine-readable
• backend/scripts/alt_loss_2026_05_22/results/eval_*.json — raw eval outputs from each Vertex job
• backend/scripts/alt_loss_2026_05_22/generate_specs.py — job spec generator
• backend/scripts/alt_loss_2026_05_22/wait_for_{finetunes,evals}.sh — polling scripts
• lib/pdr2/code/script/losses.py — PaperTextPersonalizedRepurposingLoss + MinimalPatchPersonalizedRepurposingLoss