MODS Notebook

Properties4
authormaple
last-verified2026-09-25
statuslog
descriptionA running record of confident errors this project's AI assistant has made, and what each one cost to find.

21 min read

Agent Hallucination & Mistake Log

A running log of wrong assumptions / requirement mistakes Claude made during coding that we later discovered and had to go back and fix. “Hallucination” here is used loosely (per whafa): any incorrect assumption about the data, the spec, the runtime, or existing state that shipped into work and cost us a correction.

  • Started: 2026-05-31, reconstructed from ~/.claude/.../memory/*.md + git log.
  • Ordering: most-recent first within each category.
  • Confidence: ✅ = documented in a memory file as my mistake (high confidence) · 🔍 = inferred from a fix(...) commit, root cause not yet confirmed to be mine (triage).
  • Maintenance: add new rows as they happen. Keep entries tight: Assumed → Actually → Cost → Fix.
  • Coverage gap: the MCP session-transcript search stayed gated (unsupervised mode), so the sweep was run manually via ripgrep over an internal agent path on 2026-05-31 — results in section E. That pass is noisy (much doc/memory text is injected into transcripts) so it is not exhaustive; the “To triage” list may still grow.

A. Domain / data-understanding hallucinations

(I misread what the data or the spec actually meant.)

A-2 — Quoted a lab-dish sentence as the FDA label, and inverted a drug interaction ✅ — 2026-08-07, found 2026-10-05

  • Assumed: encorafenib inhibits CYP3A4. disulfiram_due_diligence.md §5.5 quoted it as “a relatively potent reversible inhibitor of CYP3A4 and also a time-dependent inhibitor of CYP3A4” (FDA label). From that it warned that her BRAF inhibitor would blunt disulfiram’s CYP3A4-dependent activation, and with it the alcohol reaction.
  • Actually: in patients, encorafenib is a strong CYP3A4 inducer at steady state.
    • The evidence: with binimetinib, midazolam AUC fell 82% (Piscitelli et al., Clin Pharmacol Ther 2025, PMID 41215578). The FDA label says the same in §7.2 (at least since 10/2023), and so does the EU product information in §4.5.
    • What the quote really is: the in-vitro sentence of the EU product information (§5.2) and of the NCT03864042 protocol. No FDA label checked (2018, 2023, 2025) uses those words, and every source follows them with in-vitro induction.
    • How it got in: it reached the review through a WebSearch’s generated summary, which dropped those induction sentences. No label was opened, although the search had linked the 2018 and 2023 labels.
  • Cost: a safety warning that pointed the wrong way.
    • It was marked for the public notebook (2026-09-25), and the candidate catalog copied it unchecked (2026-10-01).
    • It never reached her prescriber, who received only the original article, and she has not started disulfiram.
    • It was found while building the Metro Map’s safety badge (#595).
  • Then a smaller repeat. My 2026-10-05 correction said the quote “came from an older label”. That was also unchecked, and also wrong. It reached a doc, an issue, a vector note and the MapleLog before Richard asked whether the inversion was ours or the sources’.
  • Fix: every copy carries a dated correction (PR #831). The true provenance and the independent sources are recorded (PR #843, #830).
  • Lesson:
    • A search summary is generated text, not a source. Quote and attribute only a document you opened, and name its version.
    • An in-vitro CYP profile never settles a clinical direction; the label’s §7 and the clinical DDI study do.
    • Check a correction’s own provenance before writing it down.
  • Source: session transcripts (the review’s session, 2026-08-07 04:53–05:00 PDT, and the Metro session, 2026-10-05/06); #830.

A-1 — Invented a person, then a repository to match ✅ — 2026-09-23

  • Assumed: the GitHub handle whafa was an initial plus a surname, so a full name was invented to fit it. Signed pdr2_loss_text_vs_code_note.md with that invented byline, and later cited a matching GitHub URL twice as a source in the same document.
  • Actually: there is no such person, and that repository does not exist — the URL 404s. Both the byline and the fabricated citations were live on the public site before Richard spotted the byline and asked whether there were more.
  • Cost: a fabricated citation published as a source; a second round of scanning, redaction and redeploy to remove it.
  • Fix: byline is now just “Maple”; both URLs replaced with plain path references. A confabulated-identity rule matches wally[\s._-]*hafa with no word boundaries at BLOCK severity, keeping the name as a tripwire. The artifact content scan now reads href/src — it had been stripping tags before searching, so it never inspected a link target.
  • Lesson: never expand a handle, login or username into a person’s name — ask. Never invent a URL to cite; reference the path in plain text. A URL is content and must be scanned like prose.
  • Source: MODS Notebook publishing session (2026-09-23).

A0 — Phantom “KRAS G12C” driver attributed to Maureen ✅ — 2026-07-07

  • Assumed: Maureen’s somatic drivers included KRAS G12C — repeated across memory + ~6 docs, in places drawing clinical conclusions (e.g. “Maureen is KRAS-mut, so anti-EGFR doesn’t apply”).
  • Actually: She has no KRAS mutation. A DB provenance trace (mods_list_alterations → document_id → source report) found KRAS in zero of her 9 genomic reports (Caris ×2, Guardant360, Invitae, Altera, RGCC ×4). KRAS G12C is the textbook actionable CRC driver and appears verbatim in the guardant360_sample.txt parser fixture — the likely seed. Clinically it’s backwards: she is RAS wild-type and is on cetuximab (anti-EGFR).
  • Cost: A false driver propagated into memory, planning docs, and public substack drafts; one instance (paan_planning) was inverted from her real therapy.
  • Fix: Removed + tombstoned in memory (user_maureen.md, vector fa2e3366); scrubbed every doc copy (disulfiram brief, 4 substack drafts, paan_planning, pdr_phase8.5, pdr_personalization). Same pass corrected residual A2 (second-hit) copies that had reappeared in project_mods_research_assumptions.md + several docs, and fixed the MCP default-patient bug (returned “No patient found” for ≥2 patients) that had hidden the DB behind the audit (#227).
  • Lesson: Before asserting a patient fact, confirm it traces to her actual data (mods_genomic_alteration → document_id → real report) — not a sample/fixture or generic boilerplate. See feedback_verify_maureen_actual_data.
  • Source: provenance audit, this session (2026-07-07).

A1 — Two cancers read as one trajectory ✅ — 2026-05-30

  • Assumed: Maureen’s three Caris samples (TN24, TN25, TN26) were timepoints of one colorectal cancer; built a Genomic Browser “Evolution” narrative of a BRAF→NRAS clonal switch.
  • Actually: the melanoma specimen is a separate melanoma (NRAS Q61K, skin tissue), likely treatment-associated — not a later CRC timepoint. The shared thread is the inherited variant they both carry.
  • Cost: A whole evolution/clonal-switch reading that was simply wrong.
  • Fix: SAMPLE_META multi-cancer accounting in both services, tissue-labeled timepoints, caveat banners, corrected tooltips (commit e0d489d). This is almost certainly the recent one you were half-remembering.
  • Source: project_genomic_browser.md.

A2 — an assumed “second hit via LOH” ✅ — assumed ~Apr, corrected 2026-05-30

  • Assumed: An allele at ~52% VAF in tumor tissue “strongly suggests LOH / biallelic loss” (→ a drug-eligibility framing).
  • Actually: B-allele-frequency analysis showed the allele is retained in all 3 samples, with the arm balanced. No copy-number second hit.
  • Fix: Copy/LOH tab built on real BAF data (commit 6556341); corrected the clinical readout.
  • Source: project_mods_research_assumptions.md (the wrong version) → project_genomic_browser.md (the correction).

A3 — Variant classification by count/VAF heuristic ✅ — ~May

  • Assumed: Could rank Caris variants by a count/VAF heuristic.
  • Actually: The heuristic buried the real clonal drivers. Had to classify by Caris’s own CI field (Pathogenic / VUS / Benign).
  • Source: project_genomic_browser.md.

A4 — Built work that was already done (#115/#118/#119) ✅ — 2026-05-30

  • Assumed: The open “PDR2 training knobs” issues described the current code.
  • Actually: The OOM-truncation fix, num_negative knob, and ReduceLROnPlateau decay were already implemented. Only a regression guard was genuinely needed.
  • Cost: Nearly re-built three already-satisfied features.
  • Fix: Added a regression guard (febfb3d), closed the rest as already-done.
  • Source: feedback_verify_issue_freshness.md.
  • ⚠ Correction, 2026-10-09 (#861): the ReduceLROnPlateau decay was not already done. train_foundation_model builds the scheduler but never steps it, so every foundation trained at a constant learning rate, and #118 (LR decay) was closed on a false premise. The check stopped at “the object exists”. Check that it is called, too.

A5 — KG federation misread as “decentralized storage” ✅ — late May

  • Assumed: The federation decision replaced the canonical schema with decentralized per-source storage.
  • Actually: It’s predicate federation — a bridge-manifest layer on top of the unchanged canonical entity schema.
  • Also (same incident): I’d over-asserted “the KG is canonical” from CLAUDE.md’s Phase-C4 note while missing the later docs/kg_federation_planning.md decision entirely.
  • Fix: Re-grounded on the doc; lesson = open the doc, don’t infer it from its title/commit msg.
  • Source: feedback_repo_docs_source_of_truth.md.

A6 — Causal misattribution of an edge-count jump ✅ — ~mid-May

  • Assumed: Maureen’s canonical edge count jumping 135→250 was “dedup unlocking richer cross-source resolution” (plausible — we’d just run dedup).
  • Actually: prime_kg + gp_kg had flipped planned→loaded overnight, expanding the resolver’s match pool. Unrelated event.
  • Cost: Lost trust until I went back and checked timestamps.
  • Source: feedback_verify_before_claiming.md.

A7 — Wrong codebase identity (kg-predict vs PDR1) ✅ — ~Apr

  • Assumed (by a prior explore pass): an early run used the kg-predict codebase, concluded from raw byte-count comparison.
  • Actually: It was PDR1/code — the byte diff was CRLF line endings, not a different file. Verify by code markers (-scalars flag, scalar-modulation block), not byte counts.
  • Source: project_pdr_training.md.

A8 — Default patient assumed single-row ✅ — broke 2026-05-22, found ~2026-05-30

  • Assumed: mods_patient has exactly one row; _get_default_patient_id returned an ID only when len==1.
  • Actually: A 2nd patient (PGP stand-in) was added → resolver returned None → GET /patient-profile 404’d, silently breaking the Patient Profile + PDR Input tabs for 8 days; a frontend effect then re-fired the failed fetch every render (404 request storm).
  • Fix: earliest-created default (= Maureen) + !profileError guard in ResearchDashboard.jsx. Robust picker tracked as #146.
  • Source: project_default_patient_resolution.md.

B. PDR2 / ML-pipeline hallucinations

(assumed-portable, assumed-defaults, or trusted-upstream that broke)

B1 — classify_entity assumed paper-naked identifiers ✅ — ~2026-05-18

  • Assumed: Upstream regexes (^C\d{7}$, ^DB\d{5}$, MP:/HP:) would classify our entities.
  • Actually: Our Biolink CURIEs (DrugBank:, NCBIGene:, MONDO:, HPO:) match none → everything bucketed as “gene”, drug_inv_vocab empty, the drug-only personalization mask zeroed everything → vacuous “Δ=0 everywhere” eval. Misleading, not crashing.
  • Fix: Override classify_entity for CURIE prefixes; verified 7,782 drugs after.
  • Source: feedback_pdr2_classify_entity_biolink.md.

B2 — Trusted upstream eval (saturated MRR ≈ 0.99) ✅ — 2026-05-02/03, code-fixed 2026-05-30

  • Assumed: Upstream validate()/evaluate_model() MRR/Hits@1 ≈ 0.99 were real.
  • Actually: A too-aggressive relation-aware filter + mishandled out-of-mask positives saturated the metric. The honest signal was AUPRC 0.0025 — the 0.0025-vs-0.99 gap was the canary.
  • Fix: Separate full-ranking eval (eval_maple.py); upstream out-of-mask ranking corrected in 4ecd621 (#116).
  • Source: project_pdr2_gpkg_first_run.md.

B3 — Assumed ULTRA-3g warm-start would help ✅ — 2026-05-03

  • Assumed: Warm-starting from ultra_3g.pth (general-domain pretrain) beats from-scratch.
  • Actually: Negative transfer on biomedical KGs — −46% MRR, ~100× worse Hits@1. The general-domain relation graph is actively misleading for biomedical relations.
  • Fix: Default to from-scratch on new biomedical KGs.
  • Source: project_pdr2_gpkg_first_run.md.

B4 — Checkpoint v1 assumed sufficient for resume ✅ — 2026-05-26

  • Assumed: Saving model weights was enough to warm-start/resume foundation training.
  • Actually: Cold-started Adam (m=v=0) → sign-SGD pathology for ~1000 steps kicked converged Run B out of its loss basin (bowl-shaped MRR collapse: 0.00121 → 7.7e-6 → 9.5e-5).
  • Fix: Checkpoint format v2 saves optimizer/scheduler/RNG/step (#29); --resume_with_lr_warmup for v1 sources. Adam buffers from pre-v2 runs are permanently lost.
  • Source: project_pdr2_checkpoint_v2_contract.md.

B5 — --seed assumed applied globally ✅ — fixed 2026-05-30

  • Assumed: run_maple.py --seed seeded the run.
  • Actually: It was parsed but only used for validate() subsampling — never applied to torch/numpy/random/cuda, so even bit-equivalent resume was impossible from RNG drift alone.
  • Fix: 074cf91 (#40) applies --seed globally at startup.
  • Source: project_pdr2_checkpoint_v2_contract.md.

B6 — ent2id assumed stable across sessions ✅ — ~Apr

  • Assumed: The entity→index map is reproducible, so a checkpoint can be loaded in a fresh session.
  • Actually: ent2id is built from train.txt traversal under Python set-hash randomization; a fresh-session load scrambles predictions (ρ 0.74→0.067) or crashes (CUDA index OOB / cuFFT INTERNAL_ERROR).
  • Fix: PYTHONHASHSEED=41504 in cell 1; run inference in the same session as training, or carry the ent2id sidecar.
  • Source: project_pdr_training.md.

B7 — test.py defaults assumed to match the checkpoint ✅ — fixed 2026-04-30

  • Assumed: Default gcn_layer=1 / opn=corr would load any checkpoint.
  • Actually: A 2L or opn=sub checkpoint either raises Unexpected key(s) in state_dict or — worse — silently builds the wrong model. Notebook templates Cell 20/22 shipped this bug.
  • Fix: Always pass full arch flags; patched all 5 templates.
  • Source: feedback_test_py_arch_flags.md.

B8 — Pickle assumed NumPy-version-portable ✅ — ~2026-05-18

  • Assumed: A patient-data pickle written locally would load in the container.
  • Actually: Local NumPy 2.x pickles np.ndarray with numpy._core refs; the container’s NumPy 1.x can’t unpickle (ModuleNotFoundError: No module named 'numpy._core').
  • Fix: Serialize numeric vectors as Python lists, not ndarrays.
  • Source: feedback_pdr2_pickle_numpy_portability.md.

B9 — Drive append-mode assumed durable ✅ — 2026-04-30

  • Assumed: test.py’s per-batch open(save_result, 'a') to a Drive path is fine.
  • Actually: Colab’s Drive FUSE drops the inode under tight write loops — crashed CV-eval at step 12k/25k (47 min in), metrics.json never written.
  • Fix: Write to local /content, copy to Drive only after the process exits.
  • Source: feedback_drive_fuse_append_mode.md.

B10 — Vertex export omitted the maple_patient overlay ✅ — 2026-05-16, fixed 2026-05-24

  • Assumed: The worker’s composition export matched Colab’s.
  • Actually: It didn’t pass --include-source maple_patient → Maureen’s node + 203 triples missing on every Vertex run of a Maureen-containing composition. Vertex-vs-Colab comparisons were silently not apples-to-apples.
  • Fix: 79d3b36 plumbs the overlay through PDR2 handlers.
  • Source: project_pdr2_phase1_status.md.

B11 — Manifest gpu_type / val plumbing assumptions ✅ — 2026-05-15

  • Assumed: Manifest fields reflected the actual run.
  • Actually: A smoke test logged A100_40GB while running on T4; val_n_sample/validate_every weren’t plumbed to the container → best_mrr null on Vertex rows. (Also: T4’s OOM cascade ends at batch_size=4 vs A100’s 32 — not the same experiment.)
  • Source: project_pdr2_phase1_status.md.

B12 — Quota grant assumed = capacity / single-tier ✅ — 2026-05-14 & ~05-18

  • Assumed (a): Compute-Engine GPU quota was enough for Vertex. Actually: Vertex Custom Training has a separate quota tier on top; submit 429’d on custom_model_training_nvidia_l4_gpus.
  • Assumed (b): L4 quota granted = L4 available. Actually: us-central1 L4 capacity was thin — jobs sat PENDING 2h+ then failed “Resources insufficient.” Default to A100.
  • Source: feedback_vertex_quota_layers.md, feedback_vertex_l4_capacity_unreliable.md.

C. Backend / infra coding hallucinations

C1 — Swallowed SQL error without db.rollback() ✅ — 2026-05-26

  • Assumed: try: query … except Exception: return {} safely degrades.
  • Actually: _load_cost_so_far referenced non-existent columns; the swallowed error left the Postgres transaction aborted, so a later innocent query died with InFailedSqlTransaction — pointing at the wrong culprit.
  • Fix: rollback inside the except; don’t lean on broad-except as a contract.
  • Source: feedback_swallowed_sql_error_needs_rollback.md.

C2 — Long handler with no heartbeat → double-execution race ✅ — 2026-05-26

  • Assumed: A worker handler can run a long phase without heartbeating.
  • Actually: handle_cost_sync_gcp went silent >300s during a 28,775-row per-row UPDATE → orphan-recovery re-claimed it; two workers ran concurrently, both paid full cost, hit DEAD on the third stale-heartbeat.
  • Fix: wrap long handlers in _start_pdr_heartbeat_thread.
  • Source: feedback_long_handler_heartbeat_thread.md.

C3 — Drop-migration without gutting the creator ✅ — 2026-05-23

  • Assumed: Adding migration 088_drop_med_advocate_tables was enough.
  • Actually: On restart, the old creator (015) re-armed (tables now gone), tried to re-CREATE, and failed on enum drift → backend startup aborted (FATAL: migrations failed).
  • Fix: any DROP migration must also gut the older CREATE migration.
  • Source: feedback_deletion_migration_must_gut_creators.md.

C4 — ON CONFLICT DO UPDATE deadlock under parallel ingests ✅ — 2026-05-17

  • Assumed: The CURIE-cache upsert was safe to run from concurrent ingests.
  • Actually: Two INSERT … ON CONFLICT DO UPDATE on mods_kg_curie_cache deadlocked. (The cache is a deterministic memo — the UPDATE was redundant anyway.)
  • Fix: DO NOTHING (9d4f1ef). Fix the SQL, don’t serialize/retry-bandaid.
  • Source: feedback_fix_deadlocks.md.

C5 — OncoKB: bridge edges where a merge was right ✅ — 2026-05-18

  • Assumed: Task said “approach A (bridge) safer, recommend first” → I shipped 53 same_as bridge edges.
  • Actually: 53 rows, all unambiguous → an entity merge was correct from the start; bridges force the model to learn same_as=identity. User: “feels sloppy.”
  • Fix: a567a52 merged them. Lesson: for small/unambiguous dupes, go straight to the clean fix.
  • Source: feedback_prefer_merge_over_bridge.md.

D. Process / execution slips (incl. source control)

(not assumption-hallucinations, but mistakes during coding we had to undo — logged for completeness)

Root cause of the git ones (D1, D2, D10): this checkout has a shared index + working tree + HEAD across parallel sessions. The user routinely has uncommitted WIP staged in that shared index, and another session can move HEAD out from under a command mid-run. Standing guardrails: commit with an explicit pathspec only (never bare git commit / -a); never git checkout/restore a file that may hold the user’s unstaged work; prefer gh pr merge (origin-side) over local git merge; re-check git branch --show-current before any local merge/commit.

D1 — Bare git commit swept the user’s WIP ✅ — 2026-05-29

  • The #86 federation commit pulled in ~11 files of uncommitted #108 work from the shared index. Caught before push. Fix: always commit with an explicit pathspec.
  • Source: feedback_commit_with_pathspec_shared_index.md.

D2 — git checkout <file> destroyed unstaged work ✅ — ~2026-05-15

  • Reverting my own migration_081 with git checkout wiped the user’s unstaged migration_080 (~110 lines), unrecoverable; they had to recreate it.
  • Source: feedback_git_checkout_destroys_unstaged.md.

D3 — Re-ran “pending” tasks from a resumed session ✅ — ~Mar

  • Logged 9 duplicate TRT injections and double-deducted vial volume because the context summary listed them as pending though IDs 155–163 were already in the DB.
  • Source: feedback_check_before_reexecuting.md.

D4 — Chip task wired the wrong table ✅ — ~May

  • A chip about a “KG matching column” was about the PrimekgBadge in the (unpushed) Patient Profile tab; I wired the Inference tab’s Drug ID column instead — compiled, internally consistent, wrong target.
  • Source: feedback_chip_unpushed_wip.md.

D5 — Bypassed the worker queue with raw gcloud ✅ — 2026-05-21→24

  • Submitted Run A/B foundations via gcloud “to save time” → invisible in the cockpit for ~48h, hours of hand-written SQL backfill + manual ingest. The bypass cost more than it saved.
  • Source: feedback_never_bypass_worker_queue.md.

D6 — Notebook replace() patching + premature plateau calls ✅ — 2026-04-29

  • Cell-9 tee patch silently dropped the RESTORE_FLAG line → user’s RESUME=True did nothing for ~30 min; plus escape-juggling / orphan-apostrophe bugs from multi-line replace(). Separately, called training plateau 3× in one day on 1–3 flat points, wrong each time (noise dips recovered).
  • Fix: ast.parse validation + whole-cell line-lists; declare plateau only after 5 consecutive non-improving evals.
  • Source: feedback_notebook_patching_safety.md.

D7 — String-substitution chain re-mangle ✅ — 2026-05-15

  • Overlapping-prefix replacements produced PrimeKGCIViCPharmGKB**PharmGKB**Maureen; trainer crashed at dataset lookup. Fix: longest-prefix-first / word-boundary regex / AST.
  • Source: feedback_string_substitution_order.md.

D8 — Assumed “imports locally → imports on the worker” ✅ — 2026-05-20

  • Missing List in a from typing import … line passed on local Python 3.14 but NameError’d at module load on Docker workers, bouncing 3 Vertex submissions through claim→error→release.
  • Fix: 10cd1fe; pre-deploy get_type_hints() check.
  • Source: feedback_worker_typing_imports.md.

D9 — Assumed local files == running container ✅ — 2026-05-18 & 2026-05-20

  • Added a dataset class (locally only) → container AttributeError after ~60s. Added an entrypoint.sh MODE branch (locally only) → 3 jobs failed ERROR: unknown MODE=fine_tune, ~$5–10 A100 burned. Container snapshots lib/pdr2/* at build — needs a rebuild.
  • Source: feedback_pdr2_dataset_class_in_container.md, feedback_pdr2_entrypoint_in_container.md.

D10 — Local git merge landed on the wrong branch ✅ — 2026-05-30

  • Assumed: git merge would apply to the branch I thought I was on.
  • Actually: there’s one shared HEAD across sessions; another session’s checkout moved HEAD between my commands, so the merge landed on the live feat/gb-igv branch instead of the intended target. Also hit “Base branch was modified” repeatedly because main is pushed to directly by other sessions mid-task.
  • Fix: prefer gh pr merge (origin-side — never touches the shared HEAD); re-check git branch --show-current before local merges; git fetch + retry on “base modified.”
  • Source: feedback_commit_with_pathspec_shared_index.md.

D11 — My own commit clobbered earlier cleanup ✅ — 2026-05-15

  • Assumed: my 43c06dc (PDR2 Vertex eval pipeline + composition polish) was purely additive.
  • Actually: it clobbered the explorer Phase 1.5 cleanup and re-introduced an already-removed KG Health tab; had to restore the same day.
  • Fix: 83a2d16 “restore Phase 1.5 cleanup clobbered by 43c06dc + remove KG Health tab.”
  • Source: git log (43c06dc → 83a2d16).

To triage (from git log; not yet confirmed mine vs. pre-existing)

Move up into A–C once root cause is confirmed.

  • 🔍 c7704a1 2026-05-27 — fix(api): close schema-extras silent-drop hole (#41) — API silently dropped schema extras.
  • 🔍 c310709 2026-05-27 — fix(startup): only reset truly-stale tasks on backend boot (#33 Bug A) — boot reset not-truly-stale tasks.
  • 🔍 970f751 2026-05-29 — fix(ehr): extract dropped fields in Epic/Kaiser FHIR parsers (#66-#71) — FHIR parsers dropped fields.
  • 🔍 4ec7b2a 2026-05-29 — fix(migrations): drop migration 091 — DRKG predicate backfill is dead weight — shipped a migration that was dead weight.
  • 🔍 804bd83 / cc5f172 2026-05-19 — cost BQ event_uid dropped rows (missing labels-hash / full fingerprint).
  • 🔍 0acd62d 2026-05-18 — cache filenames >200 chars → ENAMETOOLONG; had to hash.
  • 🔍 83a2d16 2026-05-15 — explorer Phase 1.5 cleanup clobbered by 43c06dc; had to restore.
  • 🔍 169ab88 2026-05-30 — upsert_run_by_name was downgrading status/metrics; had to refuse downgrades (#117).
  • 🔍 a3c76d1 2026-05-27 — pdr-eval read flat instead of nested raw/filtered/rel_filtered structure.

E. Added from the manual transcript sweep (2026-05-31)

(Recovered by ripgrep over raw session JSONL after the MCP transcript-search tool stayed gated. Bracketed tags map to categories A–D. Session ids in italics for traceability.)

E1 — Hallucinated a DB column drugbank_id [B/infra] ✅ — 2026-05-22

  • Wrote a query against g.drugbank_id — a column that doesn’t exist (the schema has drug_name + rxnorm_code only). Caught and replaced with a graceful empty-list fallback. (session 0ed1c039)

E2 — Misread the requirement “write results directly to the DB” [A/requirement] ✅ — 2026-05-20

  • You said you’d rather write all results directly to the DB rather than a script that backfills after the fact; I misread and built toward the pass-through approach. Corrected once you flagged it. The most on-point “requirement hallucination” of the set. (session bf8f8940)

E3 — Conflated live code with deleted Med Advocate, from a memory snippet [A/B] ✅ — 2026-05-20

  • Concluded some annotators were Med-Advocate-related based on a memory snippet without checking the actual call sites; you were right to push back. The inverse of the standing rule that repository documents are the source of truth — here a stale memory misled me and the code was the truth. (session 02cd6dc7)

E4 — “Conflated the layers” [A/arch] ✅ — 2026-05-22

  • Misdescribed the architecture layering; had to re-check and restate current status. (session 0ed1c039)

E5 — Wrongly concluded the local DB was unusable → fell back to files [C] ✅ — ~late May

  • Decided the DB wasn’t usable and fell back to file-based data; in fact mods_kg_canonical_stats returned 3.7M entities — the MODS server was talking to prod, I’d just misdiagnosed the connection. (session 4502bdc7)

E6 — Misread the prod flow during checkpoint work [B] ✅ — ~2026-05-26

  • Misread how the prod checkpoint flow worked; corrected after you pointed it out (same session that produced the v2 checkpoint contract, B4). (session cbef1cd7)

E7 — Run-data overwrite via duplicate experiment name [B/data-loss] ✅ — 2026-05-02

  • 2026-05-02_pdr2_gpkg_warm_v1 and 2026-05-03_pdr2_gpkg_warm_v1 shared the experiment name → derived the same run id 529acfa7, so an orphan run overwrote the real run’s data. The concrete failure behind the “don’t derive run paths from date+experiment name” lesson in project_pdr2_gpkg_first_run.md. (session 63b50ec1)

E8 — Overwrote the wrong notebook cell [D6 instance] ✅ — 2026-05-02

  • “Made a mistake — overwrote the setup cell instead of the train cell”; restored it. Another instance of the replace()-blind notebook editing already captured in D6. (session 63b50ec1)

E9 — Per-epoch training logs lost in a reorg [data-loss, confirm-mine] 🔍 — surfaced 2026-04-29

  • GP-KG runs v1–v4 lost their per-epoch logs in the 2026-04-27 pre-reorg cleanup — gone from Drive too, unrecoverable without retraining. Unclear whether I ran that cleanup; flagged for confirmation. (session a79a912e)

E10 — Another shared-main divergence episode [D/source-control] ✅ — ~mid-May

  • Local main diverged from origin with a Terminal.jsx conflict; while reconciling, one option I surfaced was a destructive reset that would have discarded your commits 681f8f4 + e0c2cc7. Handled safely (checked the reflog, didn’t auto-destruct), but it’s the same shared-worktree churn as D1/D2/D10. (session fb3cc3ea)

E11 — Minor self-corrections (low severity) ✅

  • Wrong about “Opus 4.7 Max” capabilities, which changed a cost/calculus claim (ff858e40); jumped to “lower it” as a recommendation too fast (cdaf9cde); confused myself tracking which run was “Run B” in the results matrix (60fe0a92).

Method note for the next sweep: raw-JSONL grep is badly polluted by injected context (CLAUDE.md, the memory/*.md files — including this log itself — and the 7 auto-loaded paper texts). High-signal, low-noise patterns were assistant self-admissions (I conflated, I misread, I incorrectly assumed) and hallucinated-symbol callouts (doesn't exist, no such column). Generic terms (git checkout, you're right, reset) were ~all doc-echo. If the MCP search_session_transcripts tool is ever ungated, prefer it — it scopes to user/assistant messages and skips injected docs.

Graph View

Backlinks