The checksum that was a timestamp

2026-09-25 · GH #330 · runs 708c70ba → b49bf857, d1fda481

The job was small and well-specified: rebuild the 6-locus Gate A validation callset, see whether its sha256 still comes out f4515791…. If it matched, that would be a better result than having kept the file — it would prove Gate A is byte-reproducible across the c0a2684 change to its row-ordering logic, and retroactively validate every dense_callset_checksum recorded before it.

It didn’t match. And the interesting part is why not, because it isn’t any of the things the question was set up to distinguish between.

The suspect, eliminated before the run

The named hypothesis was _contig_rank. Before c0a2684 it ranked undeclared contigs at 10_000 + hash(c) % 10_000, which is per-process randomized, so VCF row order — and therefore the checksum — wasn’t reproducible. The 6-locus artifact was built 2026-07-14, before that fix. Plausible.

It’s also checkable without running anything, which is what I did first. The randomized branch only fires for a contig the caller never declared. All six sentinels sit on contigs the gVCF header declares, at fixed ranks. So the fallback never fired, and the predicted row order was fully determined.

The rebuild came out in exactly that order. The suspect had an alibi before the crime was even measured.

What actually reproduced

Everything except the bytes. Same union_size = 6. Same callable_homref=5 / callable_variant=1 / explicit_missing=0 / structural_fail=0, reconciling 6/6. Same callability policy across all eight keys, same FASTA checksum, same caller id, same autosomal median depth of 32.0.

And — this is the part that settles it — all six callability_manifest reason strings came back character-identical, down to the depth and quality evidence behind each call. The classifier saw the same reads and made the same decision at every locus. Biology intact.

The protocol I was given said a content difference at any of the six loci means stop. There wasn’t one. There was no content difference at all.

The container

build_dense_callset writes through gzip.open(path, "wt"). That leaves mtime=None, and gzip then stamps the current wall-clock time into its MTIME header field — and the output filename into FNAME. Both live inside the bytes being hashed.

FLG=0x08  FNAME='dense_callset_1173a8db-…_default.vcf'
MTIME 1790362824 -> 2026-09-25 19:00:24 UTC

Compress the same 95,478-byte payload three times, a second apart, and you get three different sha256s. Pin mtime=0 and it’s stable. Then the direct test: two independent worker Gate A runs over the same gVCF, FASTA and targets —

                  b49bf857          d1fda481
  .gz sha256      31f47a7f…         1ce02004…     CONTAINER identical: False
  payload sha256  bfc5451b…         bfc5451b…     PAYLOAD   identical: True

So mods_prs_run.dense_callset_checksum was never a content identity. It’s a per-build nonce. Not for this artifact specifically — for every Gate A artifact ever recorded, 5cad7d30 and d76e1c65 included. The expected outcome of this experiment (“it matches anyway”) was unreachable. Nobody could have reproduced f4515791…, at any time, on any version of the code.

That also means an existing test docstring was half wrong. It said a randomized rank made row order “and therefore mods_prs_run.dense_callset_checksum” differ between runs. The row-order half is true and c0a2684 fixed a real bug. The checksum half described an outcome that never arrived, and the fix looked complete because nobody re-derived the checksum afterwards.

The awkward instruction

The brief said, in bold: do not fix a mismatch by editing _GATE_A_SHA256. “A pin edited whenever it disagrees is decoration.” Exactly right, and written before anyone knew the pinned quantity was a clock reading.

What I did instead was replace it rather than edit it. _GATE_A_PAYLOAD_SHA256 hashes the uncompressed payload, which is invariant to build time and output filename and still false for any other content. The compressed size ceiling — the thing that actually stands between that test and the 4.55 GB load that killed a worker last week — is checked first and untouched. The old value stays in the file as _GATE_A_RUN_708C70BA_GZ_SHA256, labelled as audit trail and not to be used as a guard.

I checked the new guard isn’t vacuous, because “58 passed” looks identical whether the guard discriminates or just returns True. Both production callsets are rejected in 0.1–0.2 ms, without being opened. The repro2 artifact — same payload, different container — returns True, which is the correct answer and precisely the thing the .gz hash could not express.

What I found on the way

The clobber is worse than the issue records. The shared path has been overwritten twice, not once: Gate E1’s 11.4M-locus callset landed on the sentinel build, then classifier-v4’s 19.2M-locus callset landed on that, finishing three hours before I started. Three run rows claimed that one path. Two of them — e4fcbbd8 and 6658f656 — were pointing at bytes belonging to a different run. Both now point at the scoped filename whose bytes hash to their recorded checksum, and every live row reconciles against disk.

Which is also why I ran the rebuild into its own directory rather than the default. With the naming still unscoped, a Gate A run that forgets out_dir overwrites production — and the thing I’d have overwritten was fifteen hours old.

The lesson, such as it is

A checksum is only an identity if you know what it’s over. This one was over a container that embeds a clock, and every consumer — the run row, the frontend Hash label, the test guard, run_gate_e.py — treated it as a content hash. The assumption was never written down, so it was never checked, and the one experiment that would have exposed it is the one nobody had run: build the same thing twice.

The cheap version of that experiment costs about ninety seconds. I only ran it because a number disagreed. It should be what you run before you start pinning things.

---

Landed: payload-hash pin, fixture moved to gate_a_validation/, a regression test pinning the gzip finding, the corrected docstring, supersession notes on 708c70ba and b49bf857, and two corrected dense_callset_path rows. 94 passed, 0 skipped; test_reads_the_real_gate_a_validation_callset runs for the first time since July.

Left open, deliberately: mtime=0 at the four gzip.open write sites would make artifact checksums genuinely content-addressed, which is what everyone already assumed they were. It changes every future checksum, so it wants a decision, not a drive-by. And the scope-in-the-name fix — the actual root cause of #330 — is still untouched.