MODS Notebook

Properties5
authormaple
modelclaude-opus-5-5
last-verified2026-10-04
statuslog
descriptionGates S5 and S6 ran for the first time on the model of record, and both failed. For the patient's query the model still ranks drugs mostly by popularity, and on her ex vivo panel it adds nothing measurable over popularity. Every number needed its own positive control.

4 min read

At 19:34 Richard answered the card that held up the model lane’s next two gates, and by 20:40 both had run. Both said no.

What ran

The decision fixed two things before anyone looked at the ranking:

  • S6 excludes one class of targeted agents that the panel can’t judge fairly: it tests single agents, and these work only in combination.
  • S5 counts a node as a hub only above the 99.9th percentile of degree.

Then:

  • The new rule merged, a PDR2 image was built from that commit, and :latest moved to it once nothing was in flight.
  • Two jobs went to Vertex: the path explainer on an A100 and the scorer on a T4. Each ran about four minutes, under a dollar together.

The first S5 attempt failed before it reached Vertex. The explain handler was written before #515 gave the other handlers a shared way to read a run’s dataset type, and it kept the old logic, which can’t read the newer runs. I reran it with the type passed explicitly, the same value the scorer had resolved for itself, and filed #764. The nightly fixer merged the fix 20 minutes later.

Gate S5 asks two things of the patient’s top ten drugs.

At least three must come with a mechanistic or personal path. None did.

  • The strongest paths were 8 “other” and 2 “analogy”. At the lower p99 threshold, the two analogy paths became hubs; neither moved into a counting class.
  • Most paths share one shape: the disease, then a gene associated with both it and another cancer, then that cancer, then that cancer’s drug. That is analogy at the level of diseases. The rule has no class for it, so it lands in “other”. I’ve noted that, not proposed a change: changing a rule after seeing its result is how goalposts move.

The ranking must not simply follow degree: Spearman(score, degree) over all drug candidates at most 0.6. Neither job measured this. The held-out eval had run without an anchor, and the explainer doesn’t record it. So I rebuilt the measurement from the eval’s own definition:

  • The checks before trusting it:
    • my vocabulary had to come to 102,914 entities and my drug domain to 7,793, both matching the explainer’s;
    • my ten highest scores had to be the explainer’s top ten, in the same order.
  • All three held. The answer was +0.839, against 0.86 for the model before the shortcut fix.

S6: nothing beyond popularity

On the graded ex vivo panel, per draw, the partial correlation between the model’s score and measured cell death, holding degree fixed, came to +0.028 and +0.164. Both confidence intervals cross zero. On that panel, degree alone does about as well as the model does.

One set of rows did pass. The all-panel rows, which include the binary-graded agents and keep the excluded class, have partial correlations with intervals above zero on both draws. They are not the gate’s population. Citing the row that passes is exactly what writing the gate down in advance exists to prevent, so they are reported and they decide nothing.

What it means

  • The mirror-shortcut fix did what it was for. Held-out recovery improved (S3, S4).
  • It didn’t change what the model believes about this patient’s disease. Its ranking there is still close to a popularity list, and its explanations are disease analogies, not mechanisms.
  • Foundation work continues. The pre-registered stop needs S4 and S6 to fail, and S4 passes.

The next real test is the fresh pair of runs Richard asked for on a cleaned graph (D-Q4-20). That is where a better graph, rather than a better shortcut fix, gets its chance.

Controls that earned their keep tonight

  • The new time lint for the ledger. Its first positive control caught a hole of its own: “at 18:45:”, a time followed by a colon, slipped through. Fixed before it was relied on.
  • The ledger lookup in the steward’s new write script. Its first live run picked the wrong issue: “Options Ledger” matches “ledger”. It refused cleanly, because that issue had no status comment.
  • The two count checks and the top-ten order check on the Spearman measurement. A number rebuilt outside the pipeline is only as good as the evidence that it is the pipeline’s number.

Graph View