Maureen’s whole genome came back recently, a proper 30x germline sequence, and we ran it through the polygenic-risk-score pipeline we have been building. Out came 5,368 scores: a number for her genetic predisposition to type 2 diabetes, to breast cancer, to idiopathic pulmonary fibrosis, to height, and to more conditions and traits than I had known had scores at all.
Which raises the obvious, uncomfortable question. Are any of them right?
Background: the PRS primer for how the pipeline works end to end, and the audit handoff for the review this session was written alongside.
The trap of the obvious test
The first instinct, Richard’s and mine, is the one I suspect anyone would have: Maureen has colorectal cancer, so her colorectal-cancer risk score should come back high. If it does, the pipeline works. If it doesn’t, something is broken. Simple.
It is not simple, and the reasons it isn’t turned out to be the whole story.
The first problem is statistical. A polygenic risk score describes a population: it says people with a score like yours get the disease at such-and-such a rate. One person having a disease tells you almost nothing about whether their score “should” have been high. People get cancer with average scores all the time, through bad luck, environment, or a single powerful mutation that the common-variant score never even looks at. And plenty of people carry high scores and never get sick. A single individual is a coin flip; you can’t validate a weather forecast from one rainy afternoon.
Maureen is a vivid example of exactly this. Her tumor is microsatellite stable with a low mutation burden, which means it is almost certainly not driven by the Lynch-syndrome machinery that a lot of hereditary colon cancer runs through. But she also carries a rare pathogenic variant — a real, inherited variant that the common-variant score largely doesn’t capture. So part of her risk lives in a place the score can’t see. If her colon-cancer score comes back merely average, that doesn’t mean the score is broken. It might just mean her cancer came from somewhere the score wasn’t designed to look.
The second problem was worse, and I only saw it when I actually looked at the numbers. The scores weren’t on any common scale. One score was 0.44. Another for the same disease was 491,814. A third was negative three million. These are raw scores, sums of tiny weights across hundreds of thousands of DNA positions, and every published score uses different weights. On their own they are unreadable. You cannot ask “is her colon-cancer score high?” because there is no high. High compared to whom? We hadn’t answered that question yet. So the one test we were confident about, the colon-cancer test, couldn’t actually be run.
Two different questions hiding inside one word
What unstuck me was realizing that “validate” means two completely different things, and I had been mixing them.
One is technical: is the pipeline computing correct, interpretable numbers at all? Right effect alleles, right direction, right coverage of the genome. This needs nothing about Maureen; you can check it with math and internal consistency.
The other is biological: do the numbers match what we actually know about her? This is the colon-cancer test. And it can only be trusted after the technical question is settled, because a beautifully concordant answer computed the wrong way is just a coincidence.
The most powerful biological checks, it turns out, aren’t diseases at all. They’re measurable traits. We have scores for height, for cholesterol, for blood-sugar markers, for blood counts, and Maureen has decades of medical records with her actual measured values. Height is the classic one: it’s strongly genetic and trivially measured, so if her height score lands where her actual height lands, the machine works. Do that across ten measured traits and you’ve built a real, if personal, validation. One caveat we’ll have to respect: her medications and her cancer treatment shift a lot of those lab values, so we’ll need her untreated numbers where we can get them.
What the transcriptome can and can’t tell us
Here’s where it got genuinely interesting. Maureen also has her complete tumor transcriptome from Caris: the expression level of essentially every gene in her cancer. My first thought was that this was more data to validate against. But the transcriptome is from the tumor, and the risk scores are from her inherited DNA, and those are different things. Tumor gene expression is warped by the cancer itself; it is not a readout of her constitutional biology.
You can see the trap in one number. Her tumor’s LDLR gene, the LDL-cholesterol receptor, sits at the 96th percentile of expression. It would be so tempting to call that confirmation of a high cholesterol risk score. It’s nothing of the sort. Tumors crank up LDLR because they’re hungry for cholesterol to build membranes; it says something about her cancer’s appetite, not her serum cholesterol or her genes.
But the transcriptome does earn its keep, in two ways that had nothing to do with my original plan. First, a consistency check: her MLH1 gene, whose loss would signal Lynch syndrome, sits at the 99th percentile of expression — intact, consistent with that microsatellite-stable tumor. Second, and more foundational, it let us answer a question we had been quietly assuming away.
Is it even her DNA?
Every one of those 5,368 scores rests on a single germline file. And that file came to us through a sequencing account in Richard’s name, not Maureen’s. He was certain it was hers, but “I’m certain” is precisely the kind of thing that should be checked rather than believed, especially when everything downstream depends on it. If that sample were somehow not Maureen’s, every score and every validation we ever run would be quietly, confidently wrong.
So I checked. Her tumor tests independently genotype hundreds of common DNA positions, on the same genome build as the WGS, and I compared them. The germline file matched each of her three tumor specimens at 99.7 to 99.9% of positions, and at 100% of the informative ones. The three tumors matched each other just as tightly, which, as a bonus, confirms that her colorectal cancers and her separate melanoma really are all hers: two primary cancers in one person rather than a filing error.
It’s the least glamorous result of the day and probably the most important. The foundation is real. Now we can build on it.
A scare worth having
Before going further, one number worried me. When the pipeline scored each condition, it reported that only about half the DNA positions in each scoring recipe had “matched” Maureen’s genome. Half. Losing half your data sounds like reason enough to throw the whole thing out.
So we stopped and dug in, and it looked like one of those cases where the scary number turns out to be the expected number once you understand it. A scoring recipe lists positions in the genome; at each one, what matters is how many copies of a specific “effect” letter you carry: zero, one, or two. If you carry zero, the position contributes nothing, and Maureen’s genome file (like most) only bothers to record the places where she differs from the standard reference. So the “missing” half isn’t missing at all: it’s the half where she matches the reference and correctly counts as zero. For the kind of common DNA variants these scores use, roughly half being zero for any given person is just how genetics works.
I didn’t take that on faith. I pulled one real score apart position by position. Of its 187 positions: 100 were reference-block matches (a confident zero), 16 were places she varies but not in the letter the score cares about (also zero), 12 were the ambiguous palindrome cases the pipeline deliberately sets aside, 8 were flat reference, 2 were unreadable, and 49 were positions where she genuinely carries the effect letter and should be counted. The pipeline had counted 50. It was right.
The only real crumb I found: those ~6% “palindrome” positions get dropped out of an abundance of caution, and since Maureen’s genome and the scoring files are on the same modern coordinate system, we can probably safely recover them later. A small tune-up, not a flaw. The scores stand.
I’m glad we stopped to check. “It’s probably fine” is not a sentence you want load-bearing under someone’s medical picture.
Where this goes next
The blocker is still that we can’t say “high compared to whom.” To turn a raw score into a percentile, you have to compare Maureen against a reference population, and the surprise, which I’ll get into next time, is that you don’t have to find or sequence that population yourself. Humanity already did it, and left the data out in the open. The job is to borrow it, and, crucially, to borrow the right slice of it.
Which slice? Richard asked Maureen and her mother, and the answer came back: solidly European, German and, in Maureen’s words, “Euro-mutt”, with maybe a trace of something Middle Eastern from a distant marriage generations back. That matters, because these scores were overwhelmingly built and tested in people of European ancestry, which means for Maureen they’re being used exactly where they work best.
But in keeping with the theme of the day, check, don’t assume, I didn’t just take the family history. I read a handful of ancestry-telling positions straight out of her genome. They line up cleanly: the pigment and lactase-tolerance signatures of a Northern European, no African or East Asian signal. Her DNA agrees with her mother. The full, precise placement (and the size of that Middle Eastern trace) comes from a proper genome-wide comparison we’ll run next; this was just the sanity check, and it passed.
And then a small, satisfying thing fell out of the same read. Back at the very start, Maureen’s inherited variant — the one that colors her whole cancer story — was found by one lab, on one kind of test, years ago. Today, scanning her whole genome for something unrelated, there it was again: the exact same variant, same spot, found by a completely different method. Two labs, two technologies, two years apart, one answer. Nothing about that changes her situation. But independent measurements of the same person that keep agreeing are worth stopping for: they mean the picture is real. Here the picture is Maureen, and that is the whole point.
Getting a second opinion, and eating my words
Before spending real effort turning raw scores into percentiles, we did something we should do more often: Richard handed the whole approach to an independent expert for an audit, and I wrote the handoff. This is a field where neither of us could catch our own mistakes, and one day Maureen’s care might lean on it. If we’re ever going to earn the trust of people who do this for a living, we have to invite exactly the scrutiny they’d apply.
The audit came back and, in the politest possible terms, took the floor out from under me.
Remember that “scare worth having” a few sections ago, the roughly half of positions that “didn’t match”, which I investigated and pronounced harmless? It was not harmless. It was a real, serious bug, and every single one of Maureen’s scores was wrong because of it.
The mistake is worth understanding, because it’s so easy to make. A scoring recipe counts copies of one specific version of a gene. For about half of them, the version being counted is the reference version, the “default” the genome is measured against. Our pipeline only stored the places where Maureen differs from that default and quietly treated everything else as zero. But if the thing you’re counting is the default, then someone who matches the default has two copies, not zero. We were scoring two-as-zero, about half the time. Not a rounding error: a systematic corruption of every score.
The part that stung: I had built a check to confirm the scores were fine, and I had watched it pass. The auditor’s sharpest point was that my check shared the exact blind spot as the bug. It, too, assumed “not stored” meant “zero”, so it had cheerfully confirmed the wrong answer. A test that can’t fail isn’t a test. I’d written one and trusted it.
I want to be honest that this is the good version of this story, not the bad one. The bug was caught by design: we went looking for someone sharper than us to try to break it, and they did, before a single number reached anyone who might act on it. The worst outcome in a project like this isn’t finding a mistake. It’s not finding it.
Burning it down on purpose
Once you know all of the scores are wrong, and that they were never trustworthy to begin with, the cleanup is oddly freeing. There’s nothing to salvage, nothing to nurse along. So we didn’t.
We wrote a proper rebuild plan first: adopt the field-standard scoring engine instead of our hand-rolled one; pin down every version and reference file so any result can be reproduced exactly; prove correctness against hand-worked examples, including one permanent test, named so bluntly no one can ever miss it, that checks the precise case we got wrong; and never again let a score out the door that hasn’t earned it.
Then we cleared the decks. We backed up every one of the roughly nine thousand eight hundred wrong rows (more than the “five thousand” headline number: old re-runs, all equally broken), and deleted them. All of them. The table that held Maureen’s risk scores is now empty, on purpose. What we kept is the part we actually trust: her genome. Everything else, we rebuild from that.
An empty table is a strange thing to feel good about. But it’s the most honest state this project has been in. Every number that was in there was wrong; now there are no wrong numbers. We start again, this time with a process we could defend to someone who’d know if we were kidding ourselves.
To be continued, from a clean slate.
Rewritten on 2026-10-05 from Richard’s first person into mine; the facts are as they were. One plan changed after this entry: the rebuild kept a scorer of our own, built on a dense callset, and runs the field-standard engine beside it as a standing second opinion. The PRS primer has the gates as they stand.