PRS Validation — Audit Handoff

Date: 2026-07-12 Subject: Validation of 5,368 polygenic risk scores (PRS) computed for the patient “Maureen” from her germline whole-genome sequence. Purpose of this doc: Independent review. Please challenge methods, numbers, assumptions, and the plan. Author is a non-specialist; explicitly assume nothing is obviously correct.

---

1. Context & objective

• Patient: Maureen. Stage IV colorectal adenocarcinoma, microsatellite-stable (MSS), TMB 3 (low). One pathogenic inherited variant on file (gene, variant and genotype withheld).
• Input genome: 30x germline WGS, GRCh38.p13, from sequencing.com (kit id withheld). File: a 30x WGS VCF under backend/data/genomes/, named for the ordering account rather than the patient. Note it arrived under her husband’s account, hence the provenance concern below.
• PRS output: 5,368 scores, one per PGS Catalog scoring file, in DB table ModsPolygenicScore. Computed by backend/mods_prs_service.py (compute_prs_pure), genotypes imported by backend/connectors/genotype_importer.py. Genotype store = 4,419,571 variants.
• Objective: determine whether these scores are trustworthy, and validate them against what is independently known about Maureen.

2. Key artifacts (for reproduction)

Item	Path / handle
Germline WGS (GRCh38)	a 30x WGS VCF under backend/data/genomes/
Tumor DNA+RNA BAMs/VCFs (hg38)	backend/data/caris/{the 2024 colorectal specimen, the 2025 colorectal specimen [colorectal], the melanoma specimen [melanoma]}/
Tumor transcriptome (percentile TSV)	backend/data/caris/WTS_tab/the 2025 colorectal specimen_allgenes_percentile.tsv
PGS scoring files (cache)	worker/data/pgs_catalog/<PGSID>/<PGSID>_hmPOS_GRCh38.txt.gz
PRS compute + matching logic	backend/mods_prs_service.py, backend/connectors/genotype_importer.py
Identity-check script + result	<scratchpad>/idcheck/concordance.py, concordance_results.json
PRS status snapshot (5,368 scores)	via MCP mods_prs_status

3. Validation steps completed this session

3.1 Inventory of the PRS output — found the core blocker

• All 5,368 scores have z_score = null and ukb_trait_code = null. → No normalization/percentiles exist; scores are raw only.
• raw_score ranges −3.2M to +837K; 40 of 5,368 have |raw| > 1,000. Different PGS use different weight conventions, so raw scores are not comparable across PGS and not interpretable individually. A per-PGS reference distribution is a hard precondition for any interpretation (including the “is her colorectal score high?” question).

3.2 Clinical grounding (shapes what validation is even possible)

• n=1 disease status is a weak PRS validator (population-level tool, individual outcome).
• MSS/TMB-low ⇒ not Lynch-driven ⇒ a common-variant colorectal PRS is more relevant for her, but her inherited variant is a monogenic contributor common-variant PRS does not capture ⇒ a merely-average colorectal PRS would NOT falsify the pipeline.

3.3 Multi-omic sample-identity check (provenance) — PASSED

• Why: all 5,368 scores derive from one germline file that arrived under a different-named account; the link to Maureen must be verified, not assumed.
• Method (pure Python, no external tools; concordance.py): built a shared common-SNP panel of 923 biallelic-SNV rsID sites from the three Caris tumor DNA VCFs (all hg38, matching the germline build), streamed the germline gVCF once, extracted genotypes at those sites (888 had germline genotypes), computed pairwise genotype concordance. Logic mirrors somalier/NGSCheckMate.
• Result: germline WGS vs each tumor = 99.7–99.9% exact GT concordance, 100% allele-share at identity-informative (non-ref) sites; tumor-vs-tumor = 99.9–100%. ⇒ The germline is Maureen’s; all three specimens (2 colorectal + 1 melanoma) are the same individual.
• Independent corroboration: the germline WGS independently carries the same pathogenic variant the clinical panel found, on a different platform.
• Caveats: panel is ~888 common SNPs from Caris panel regions (unrelated individuals would score near chance, so this margin is unambiguous). The “100% het retention” LOH readout is only over these panel sites, not a genome-wide LOH claim.

3.4 Coverage (“~50% of scoring SNPs matched”) — diagnosed as BENIGN

• Concern: snps_used / snps_in_scoring_file ≈ 50% looked like half the data missing.
• Evidence it is expected, not a bug:
  1. Code. mods_prs_service.py (~L102-107): an unmatched scoring SNP is missing += 1; continue (skipped), which is numerically identical to dosage 0 (score = Σ dosage·weight). Matching is rsID → (chrom,pos) fallback with strand-flip and palindrome handling. genotype_importer.py (~L175) skips gVCF reference blocks (<NON_REF>), storing only ALT-bearing sites ⇒ store is variant-only (explains the 4.4M count) ⇒ a truly hom-ref scoring SNP is legitimately absent ⇒ correctly contributes 0.
  2. Distribution. Coverage mean 44.9%, median 48.8%, unimodal, no near-zero spike (a build/identifier bug would spike near 0). Large PGS (≥100k SNPs) median 48.3% ≈ small PGS (<1k) median 47.4% ⇒ coverage is independent of file size = a population constant (“fraction of common SNPs where a person is non-reference”), not an artifact. The only <5%-coverage cases (60 PGS) are 1–2 SNP scores (granularity).
  3. Single-SNP reconstruction (PGS000019, 192-SNP CAD score, 187 with GRCh38 positions). Classified every scoring SNP against her WGS: 49 carry the effect allele (should be scored); pipeline reported snps_used = 50 — match. Full decomposition: 100 hom-ref in reference block, 16 variant-but-for-the-other-allele (dosage 0), 12 palindromic (skipped), 8 explicit 0/0, 2 no-call, 49 scored.
• Conclusion: raw scores are not invalidated by ~50% coverage.
• One real (minor, unbiased) finding: ~6% palindromic (A/T, C/G) SNPs are conservatively dropped; recoverable since her WGS and the scoring files are both GRCh38/forward-strand.
• Caveat: the per-SNP proof is n=1 PGS; the code + distribution generalize it, but the definitive fleet-wide confirmation is the pgsc_calc cross-check (still pending, see §5).

3.5 Ancestry — self-report EUROPEAN, genetically corroborated (spot-check only)

• Self-report (patient + mother): solidly European (German/“Euro-mutt”), possible trace Middle Eastern from a distant relative.
• Genetic spot-check (6 loci read from WGS): SLC45A2 rs16891982 G/G, LCT rs4988235 A/A (Northern-European lactase persistence), SLC24A5 / EDAR / DARC all hom-reference (= European pigment allele / non-East-Asian / non-African). ⇒ Consistent with European; rules out substantial African or East Asian ancestry. Cannot resolve the trace Middle Eastern component (pigment loci are shared across West Eurasia).
• Consequence: reference panel = 1000G EUR / continuous-PCA; PRS portability is best-case for a European individual (most PGS were trained/validated in Europeans), so no per-score portability flagging is needed.
• Caveat: this is a crude few-SNP sanity check, not a formal ancestry estimate. Genome-wide PCA is still required (see §5) for precise placement and to quantify admixture.

3.6 Transcriptome usage boundary (important scope limit)

• The Caris WTS is tumor RNA, not germline. It validates identity, genotyping QC, and single-variant mechanism — NOT germline quantitative-trait risk.
• Consistency signals observed: MLH1 at 99th (MMR intact ⇒ consistent with MSS).
• Explicit trap to avoid: tumor LDLR at 96th percentile does NOT validate an LDL PRS (tumor cholesterol-uptake demand, not serum LDL or germline lipid genetics).

3.7 Normalization feasibility — both shortcuts ruled out

• No reference panel on disk (only the PGP hu52B7E5 sample).
• Only 7% (17/250 sampled) of scoring files embed allelefrequency_effect ⇒ the analytic route needs external European allele frequencies (gnomAD NFE or 1000G EUR).

4. Interim conclusions (what we believe is now established)

1. The germline substrate is Maureen’s DNA (99.7–100% concordance).
2. The ~50% coverage is expected/benign; raw scores are not corrupted by it.
3. Her ancestry is European (spot-check; formal PCA pending) ⇒ EUR reference, best-case portability.
4. The only blocker to interpretation is the absence of percentiles (no reference distribution yet).

5. What is still needed (gaps)

1. Percentiles (the deliverable). Convert raw → z/percentile against an ancestry-matched reference. Blocks ALL biological validation, including the original colorectal question.
2. Formal ancestry placement. Genome-wide PCA vs 1000G+HGDP (not the 6-SNP sanity check) to confirm/place precisely and quantify the trace, and to drive continuous ancestry adjustment.
3. Fleet-wide coverage/matching audit. Reproduce our raw scores with an independent implementation (pgsc_calc) across all 5,368 to confirm matching/dosage beyond the single-PGS proof; recover palindromes.
4. Quantitative-trait biological validation (Tier 2, strongest). Compare her PRS percentiles for measurable traits (height [sentinel], LDL/HDL/triglycerides, fasting glucose/HbA1c, BMI, blood pressure, CBC indices) against her actual medical-record values. Requires percentiles (#1) + sourcing measured values. Blocker: two MODS tools are currently broken — trt_list_bloodwork (BloodworkResult has no attribute 'test_type') and mods_search_documents (SQL parameter-binding error) — and must be fixed or bypassed to reach lab values. Control for treatment confounders (e.g., statins→LDL, cancer therapy→CBC): use untreated values where possible.
5. Disease-concordance checks (Tier 3, weak). Colorectal, ATM-linked breast/pancreatic, once percentiles exist. Frame as consistency, not confirmation.
6. Label normalization. 1,760 disease_label values with fragmentation (e.g., “Breast cancer” vs “Breast cancer (female)”; three T2D labels). Normalize to EFO/MONDO before internal-consistency grouping.

6. Plan

• Task 1 — Ancestry PCA. 1000G+HGDP reference; project her WGS. Output: ancestry coordinates + trace quantification. Async/worker.
• Task 2 — Empirical percentiles (gold standard). Run pgsc_calc (PGS Catalog’s official Nextflow calculator) scoring Maureen + the 1000G EUR reference, with continuous PCA ancestry adjustment. Output: ancestry-adjusted percentiles for the PGS set; also serves as the fleet-wide coverage/matching audit (#5.3) and recovers palindromes. Async/cloud/worker (multi-GB reference + Nextflow).
• Task 3 — Analytic first-pass. gnomAD non-Finnish-European allele frequencies → per-PGS μ = Σ 2·p·w, σ² = Σ 2·p·(1−p)·w²; z = (raw − μ)/σ → percentile across all 5,368. Fast rough numbers + independent cross-check on Task 2. Caveats: ignores LD (σ approximate), μ/σ over all scoring SNPs vs raw over used-only (small inconsistency), uses training-population AF.
• Order: Task 3 (analytic) first for speed (rough percentiles on all 5,368, incl. colorectal, ~1 day), then Task 2 (empirical) to confirm/adjust. Task 1 feeds both.
• Then: Tier 2 quantitative validation and Tier 3 disease concordance once percentiles land.
• Execution: async cloud/worker jobs (not local Windows Docker), consistent with project infra.

7. Questions we specifically want the auditor to scrutinize

1. Identity check rigor: is a 888-common-SNP pure-Python concordance sufficient, or should we re-run with a standard tool (somalier / NGSCheckMate) for defensibility?
2. Palindromic SNPs: is dropping ~6% acceptable, or should we recover them (both sides GRCh38/forward)? Any risk in recovering?
3. Analytic μ/σ approximation: is the HWE + no-LD + all-SNPs-vs-used approximation acceptable for a first pass, or misleading enough to skip? Should σ include an LD correction?
4. Ancestry adjustment specification: is continuous PCA adjustment (regress PRS mean and variance on PCs, then standardize) the right method vs a EUR-bucket? Are we handling the trace admixture correctly?
5. Coverage proof generality: the per-SNP reconstruction is n=1 PGS. Is that acceptable pending pgsc_calc, or should we reconstruct several PGS (incl. a large genome-wide one) now?
6. Build consistency: confirm no GRCh37 leakage anywhere (germline GRCh38.p13; scoring files hmPOS_GRCh38; tumor VCFs hg38). Any silent liftover risk?
7. PRS best-practices we may be missing: duplicate/overlapping PGS per trait, effect-allele orientation edge cases, indel handling, multi-allelic sites, and whether to weight by PGS Catalog per-ancestry performance metrics.
8. Reference choice: 1000G EUR (~500 samples) vs 1000G+HGDP combined vs gnomAD — which for the empirical distribution, and does N matter for tail percentiles?

8. Known limitations / what we did NOT do

• Did not compute any percentiles yet (no reference distribution).
• Did not run formal genome-wide ancestry PCA (only a 6-SNP sanity check).
• Coverage correctness proven on one small PGS only.
• Did not verify the genotype importer captured all true variant sites (only that used≈effect-allele-carrying for one PGS); pgsc_calc cross-check is intended to close this.
• Did not yet source any measured clinical lab values (tooling broken).