When “It Ran Without Errors” Isn’t the Same as “It Worked”
Scoring a whole genome against the whole catalog — and the quiet bug that would have thrown away half of it.
A patient’s whole genome came back from the sequencer this week — a real 30× whole-genome sequence, about five million variants. The goal was simple to state: compute polygenic scores for this person. Not a handful — all of them. The public PGS Catalog has ~5,385 scores harmonized to the current human reference, and we wanted the whole shelf.
A polygenic score is, underneath, almost embarrassingly simple. For each of thousands of spots in the genome, you ask: how many copies of the risk version does this person carry — zero, one, or two? You multiply that by how much that version shifts risk (a weight from a big association study), and you add it all up. Thousands of tiny multiplications, one sum. That’s it.
The hard part isn’t the arithmetic. It’s the lining up. To score a variant, you have to match “this person’s genotype at this spot” to “the scoring file’s weight for this spot.” Get the match wrong and you either score the wrong number or — worse — score nothing and never notice.
The join key that used to be enough
For years the way you matched was the rsID: a stable name for a known
variant, like rs199706086, assigned by the reference database of human
variation. Same rs number on both sides, same variant. Clean. Our pipeline did
exactly this, and on the small, older scores it worked perfectly — we checked,
and the coverage was identical whether we matched cleverly or naively.
Then we pointed it at the big modern scores. And the coverage came back zero.
Not an error. Not a crash. A calm, confident, wrong answer: this six-million-variant genome-wide score matched nothing in the patient’s genome, contributed a flat zero, and moved on. If you weren’t looking closely, you’d conclude the person had an unremarkable score. You’d be throwing away the single most informative model in the catalog and calling it a result.
Why the biggest scores go dark
Here’s the thing nobody warns you about: the newer, larger scoring files often
don’t carry rsIDs at all. They identify their variants purely by position —
chromosome and coordinate, 1:10250 — and leave the rsID column empty. It makes
sense from their side; positions are unambiguous and rsIDs drift over time. But if
your pipeline’s only join key is the rsID, every one of those position-only
variants is invisible to you.
And it’s not a rare edge case. In the batch we tested, the scores split into two clean camps: the small, older ones were ~0% position-only (rsIDs everywhere), and the big genome-wide ones were 100% position-only. The richest half of the catalog was exactly the half we were silently dropping.
The fix is boring; the effect is not
The fix is a fallback: match on rsID first, and when there’s no rsID, match on
chromosome:position instead. Because we’d been careful to keep both the
patient’s genome and the scoring files on the same reference build, the
coordinates line up directly — no translation needed.
The numbers tell the story better than I can. One score, six-point-seven-million variants, position-only:
- rsID matching: 0 variants used.
- rsID + position matching: 2,594,593 variants used.
Zero to two-and-a-half million, from one if branch. Several of the biggest
scores told the same story — hundreds of thousands to millions of variants that
went from invisible to counted. The small rsID-based scores didn’t budge, which
is exactly what you want: the fix adds signal without disturbing what already
worked.
The unglamorous part: getting five million variants off disk
None of that matters if you can’t get the genome into the system in the first place, and a whole-genome file is not a spreadsheet. The variant file is millions of lines; the raw version is a gzipped block-format monster where most of the content is just “the patient matches the reference here, and here, and here” for long stretches.
The first version of the loader did the obvious thing — read the file, hold it in memory, process it. On a small consumer DNA file that’s fine. On a whole genome it tries to hold multiple gigabytes at once and falls over. So we rewrote it to stream: pull one record, hand it off, forget it, pull the next. Peak memory to walk the entire five-million-variant genome: 0.9 megabytes. The file never has to fit; only one record ever does.
And then the lesson you only learn by getting burned: don’t write four million rows in a single database transaction. The first full load ran for over an hour accumulating everything and planning to commit at the very end — and something (a timeout, a reload, take your pick) killed it a few minutes short. All of it, gone, because none of it had been made durable yet. The rewrite commits in chunks: a few hundred thousand rows at a time, each its own small transaction, each one that finishes staying finished. Now a hiccup costs you one chunk and a resume, not the whole afternoon.
The actual lesson
The bug that would have quietly cost us half the catalog wasn’t in the math and wasn’t in the model. It was in the join key — the humble question of how you decide that two records describe the same thing. It threw no error. The pipeline reported success. The scores came out looking plausible.
In this kind of work, “it ran without errors” and “it did the right thing” are two completely different claims, and the gap between them is where the interesting bugs live. The only defense is to check the thing you’re most sure of. We were sure the scores were fine. We looked anyway. Half of them were zero.
The genome is loaded, the scoring path is validated end-to-end, and the full sweep across all ~5,385 scores is next. There will, of course, be more gotchas. There always are.
The gotchas arrived on schedule
We kicked off the full sweep — the whole shelf, thousands of scoring files — and within the hour the machine fell over. Not one service: all of them. Frontend, backend, and every worker went down together, the workers spitting the same terse complaint on the way out — Unexpected EOF — and the backend logging the least helpful exit code in computing: 137. That’s the number a process gets when the operating system runs out of patience and kills it outright. Classic out-of-memory.
Except the machine had thirty gigabytes of memory, and almost none of it was in use.
The call was coming from inside the house
When a crash blames memory but memory is fine, look at disk. And the disk told the story immediately: the container images for our workers had swollen to twenty gigabytes each. A worker image should be a few hundred megabytes — a little Python runtime and some libraries. Ours were the size of a video game.
Here’s what happened, and it’s almost poetic. To score against the catalog, the pipeline downloads each scoring file and caches it locally so it never has to fetch the same one twice. Do that across the whole catalog and you accumulate thirty-seven gigabytes of cached scoring files. Good — that’s the working set, and caching it is exactly right.
But those files landed in a folder that, through an accident of configuration, was being swept into every container build. The build system has an ignore list — a file that says “don’t ship these directories into the image” — and it did exclude the big data folders. It just excluded them by their exact top-level names. The scoring-file cache lived one level deeper, under a different folder, and the ignore rule — anchored to the root — slid right past it. So every build packed all thirty-seven gigabytes into the image, every image carried it, the disk filled, and the operating system started shooting containers.
The thing that broke the build was the data the build had so diligently produced. The pipeline’s success was the pipeline’s undoing.
One line, thirty-eight gigabytes
The fix was a single line added to the ignore list — the nested cache folder, named explicitly this time. Rebuilt images dropped from twenty gigabytes back to about four hundred megabytes. We cleared out the bloated images and the stale build cache, and the machine handed back roughly thirty-eight gigabytes of disk it never should have been holding.
And the reassuring part: nothing about running the sweep needed those files inside the image at all. At runtime the workers read the cache from the host disk directly, mounted in live. We’d been paying — in disk, in build time, in stability — to bake in a copy that was never used. Excluding it cost us nothing and bought back the whole machine.
The second bill: paying to re-read a genome that never changes
With the machine stable, a subtler cost came into focus — the kind that doesn’t crash anything, it just quietly runs up a tab.
Every scoring task began the same way: load the patient’s genotypes. All of them — about 4.4 million rows — streamed out of the cloud database into the worker. Which is fine once. But the sweep is thousands of tasks, and each one re-streamed the entire genome from the cloud. The database lives in a data center; reading from it is billed by the byte. We were paying, over and over, to re-download a genome that had not changed since the last time we’d downloaded it thirty seconds earlier.
The genome is the most static thing in the entire system. It was the one thing we were re-fetching most.
Pull once, read forever
The fix follows from the observation. Pull the genotypes from the cloud exactly once, reduce them to the compact form the scorer actually needs, and write that to a single file on the local disk — one file every worker on the machine can share. From then on, each task reads the genome from local disk in a second or two and touches the cloud database zero times.
We were careful about the boundary: this local shortcut only switches on for workers running on our own machine. Anything running in the cloud keeps reading the database the normal way, because up there the “local disk” is ephemeral and the shortcut would be a lie. And the cache is strictly an optimization — if the file is missing, unreadable, or a version behind, the code shrugs and falls back to the database. Correctness never depends on it; only the bill does.
Then we proved it three ways, because “it should work” is not a load-bearing statement. We compiled it. We ran a round-trip test with the database access rigged to throw an exception — so that if the code ever fell through to the database on what should have been a cache hit, the test would explode. It didn’t. And we warmed the real thing: pulled all 4,419,571 of the patient’s variants once, wrote the 245-megabyte cache file, and watched a fresh worker read it back and report, in its own logs, zero cloud egress.
Fewer workers, finished faster
Resuming was its own small lesson. Eleven tasks had died in the crash — marked dead after exhausting their retries while the machine was busy immolating itself. We reset them to a clean state and started a handful of workers, detached this time so they’d survive a closed terminal or a sleeping laptop.
Five workers. They promptly ran the machine out of memory again — but for a brand-new reason. Each worker, on each task, loads that genome cache into a roughly two-gigabyte structure in memory. Five of them at once is the same genome held five times over, and the peaks collided. So we did the counterintuitive thing and ran fewer workers — two. Two workers, no crashes, steady progress. Fewer workers finished faster, because the only thing faster than a fast worker is a worker that never has to be restarted.
The tail was slow — the last few tasks were downloading scoring files we hadn’t cached yet, which is network-bound and simply takes as long as it takes — but it was stable, which is the only speed that matters when what you’re trying to do is finish.
Done, for real this time
All thirty-six tasks complete. 5,368 polygenic scores computed for one patient — very nearly the entire catalog. (The handful that didn’t produce a score weren’t failures; they were scores whose variants simply don’t overlap this particular genome, correctly skipped.) And from the moment the cache went in, the genome left the cloud database exactly once — the warm — while every task that followed read it from local disk.
Two halves, two kinds of failure. The first was a bug that insisted everything was fine while silently zeroing out half the catalog — a wrong answer wearing the costume of a right one. The second was the opposite: loud, crashing, impossible-to-ignore failures — whose causes were quiet and a little embarrassing. A cache that outgrew its container. An ignore rule that missed by one folder. A bill for re-reading a file that never changed.
Neither kind announces itself as what it is. The silent bug looks like success; the loud crash looks like a memory problem when it’s really a disk problem, which is really a config file that was almost right. The work — the actual work — is the part after “it ran,” where you go and check whether the thing that happened is the thing you meant.
The catalog is scored. The machine is thirty-eight gigabytes lighter and no longer tips over. And the genome only had to leave the database once.