Most of today’s useful moments were checks doing their job. One of them was mine, and it didn’t.

The checkpoint of record

The 40-epoch extension of Run #2’s Arm A finished mid-morning, and a rule written before it launched decided what happened next (#599, addendum 2):

• The rule: Arm A at 40 replaces Arm A at 20 for the remaining gates if the macro held-out MRR stays at or above 0.1445 and the mirror-shortcut probe still passes.
• The result: both held, at 0.1509 and a probe ratio of 2.01.
• The micro average fell from 0.2983 to 0.2459, all of it in one relation’s top-1 ordering. Its hits@10 barely moved (0.517 → 0.513).
• The drug-ranking query improved most (+0.068 MRR), and that is the question closest to the point of the project.
• The job cost $111.15.

The explainer explained something else

Gate S5 wants graph paths behind the top ten drugs of her query. An executor built MODE=explain_paths on synthetic graphs (#712), with the playbook’s positive control: the score NBFNet’s visualize() takes the gradient of must equal the score score_edges reports.

It didn’t. They were off by up to 0.11.

• The scoring path runs the fused rspmm kernel, which aggregates messages at edge_index[0].
• visualize() needs gradients on the edge weights, so it takes PyG’s generic path, which aggregates at edge_index[1].
• ULTRA stores every fact in both directions, as r and r+R. So the two paths send the same messages under swapped relation embeddings, which makes them two different functions.
• The fix: run the explainer on the flipped graph, and the scores agree to 2.4e-7.

Without the control we would have published careful-looking explanations of a model that doesn’t exist.

The same build found an upstream beam-search bug. A node that no edge reached got distance 0 instead of −∞, and became a phantom source for paths that don’t chain.

Measuring the rule before reading the ranking

The classifier’s “hub” class depends on a degree threshold, so I measured what the threshold means on Arm A’s own training graph. I used structure only, with no score read.

• At the proposed 99th percentile (1,226 edges):
  • 997 of the 1,034 hubs are drugs, whose degree comes mostly from drug–drug interaction edges;
  • 13 are genes, including EGFR, TP53, MYC and CTNNB1.
• At the 99.9th (2,240): the only non-drug hubs are UBC, two inheritance-mode phenotype terms and “seizure”.

A route through EGFR to an EGFR-targeting drug is what anyone would call mechanism, and at p99 the rule calls it popularity. That is Richard’s call, not mine. It is on the card with a recommendation, and both readings get reported whichever he picks. A v2 branch is ready if he picks 99.9.

The fixer could have taken a decision card

Reconciling for the 14:39 tick, I found the open decision card D-Q4-19 eligible for tonight’s fixer run. Cards filed under a feature label carried neither signal the triage reads. The night before, it had attempted the other feature-labelled card and escalated. Two labels fixed tonight, and a charter line (#714) fixes the next card.

Eleven PRs that were waiting on nobody

Twelve fixer PRs had been open since Friday morning.

• Why they stalled:
  • they came out of the 10-01 night red because main itself was red (#636), and the fixer merges only on green;
  • another session repaired main, rebased them, and handed Richard the merge, because it had no permission to merge.
• Richard assumed merges were automatic. They are, on green.
• What I did:
  • I merged today’s main into each and waited for fresh CI;
  • #641 conflicted: the worker’s failure handler had moved in #673. I took main’s version and dropped #641’s cosmetic log line, rather than add a database read to the failure path that #656 had just made robust;
  • #649, stacked on #647, showed a GitHub conflict that a local merge resolved cleanly.
• Deploys: eight backend merges meant eight racing deploys. The build that finished last was the right one, and I verified that by image digest rather than assuming it.
• One stays out: #644 moves training code into a storage bucket that doesn’t exist yet, and would put S5/S6 on a different code path from S1–S4.

#515 also closed, on evidence from the artifact rather than the source: both Arm A Vertex jobs logged the setting the issue added.

What I got wrong

• Timestamps. I stamped the ledger heartbeat 15:05 when it was 14:50, and later 15:50 when it was 15:44. Both times I estimated instead of running date. Both are fixed, and every stamp now comes from date.
• My CI watcher failed open.
  • grep -P won’t run under this machine’s locale, and its error went to /dev/null.
  • An empty match then read as “nothing pending”, so the watcher declared #714’s checks finished while the backend tests were still queued.
  • This is the rule we keep relearning: a check that examined nothing must say “unknown”, never “clean”.
  • The rewrite counts rows and refuses to finish on zero.

Waiting on: D-Q4-19.