KG Predicate Federation — Plan (#86)

Decision (2026-05-29): Federation, not forced Biolink canonicalization. Each KG keeps its native predicate; cross-KG equivalence lives in a small, reviewable bridge manifest that is DB-backed and UI-editable (visualize / create / change), with a version-controlled YAML seed.

This doc is the “proper planning session” #86 asked for. Read it (and the “Data and Mapping Integrity Rules (HARD)” section of CLAUDE.md) before touching predicate_mapper.py or any KG ingest.

Status, 2026-10-01 (GH #628). For four months only the manifest module and its seed existed, and nothing read them. When they were first checked against the data, 17 of the seed’s 32 non-RTX members named no stored row, because the seed was written from predicate_mapper.py’s dictionary keys rather than from mods_kg_relation.source_predicate.

Phase 1 of #628 rebuilt the seed and made it verifiable:

the stored pairs are snapshotted, and a test fails any member that names none of them (§3);

every biolink: claim is checked against the pinned release (§4);

each bridge carries an approval status;

the training export gained an opt-in bridged relation identity, option B′ of decision card D-Q4-01 (#611).

Phase 2, the DB-backed editor, is built (§5, as built on 2026-10-02). The database is the live copy, edited in the Research dashboard’s KG Mappings tab. The training export still reads only bridges.yaml, which is exported from the database on demand and merged through a PR.

Decided 2026-10-04 (Richard, GH #628 review):

D-Q4-01 is B′: Run #2’s Arm B and later runs use the bridged export (#611).

Negative assertions are exported as relation types of their own: never merged with their positive, dropped or negated. Showing that the model reads them as negative is #726.

§6 is option A (below).

The bridges from #709’s review were approved: A1–A7, plus the three drug–protein role bridges.

biolink:has_target is renamed maple:drug-target, because Biolink 4.4.2 defines has target disease → gene (§4).

maple:drug-adverse-effect now points at biolink:has_side_effect.

GP-KG’s target stays proposed. GP-KG is the papers’ graph, kept for reproduction.

Bridge-aware querying (#735) and the DRKG ingest (#736) moved to issues of their own (§8).

1. Why federation

The old approach mapped every source predicate into one biolink:* vocabulary. Two failures came out of it:

Invented slots. biolink:has_enzyme, biolink:has_carrier, biolink:has_transporter are not defined slots in Biolink v4.4.2. The biolink: prefix made them look canonical; they were coined in May 2026.
Lossy collapse. Forcing DrugBank’s target / enzyme / transporter / carrier into one predicate erased the drug-metabolism distinction and led to a deletion-on-collision migration that dropped 15,045 rows.

Federation fixes both: no predicate is rewritten into a fake slot, and distinct source meanings stay distinct. Equivalence becomes an explicit, bounded declaration instead of an implicit, lossy transform.

2. The key enabler: bridges key on `source_predicate`

Every canonical row already stores the raw source predicate in mods_kg_relation.source_predicate (the legacy mapper only added a mapped predicate on top — it never erased the raw value). So a manifest that declares equivalence over (source, source_predicate) tuples works against the existing data with no re-ingest, and cross-KG query expansion filters on source_predicate.

Consequence: cleaning up the invented biolink:* values that currently sit in the predicate column is decoupled from making federation work — it’s a cosmetic/legacy follow-up, not a blocker (see §6).

Two qualifications (2026-10-01, GH #628, #627). “No re-ingest” holds only for what the ingest kept:

PrimeKG’s drug–protein roles were never stored. The connector drops display_relation (#438), so the enzyme, carrier and transporter bridges had nothing to point at and were removed until the re-ingest.

Relations are unique on the mapped predicate, so within one source two raw predicates mapped to the same Biolink predicate for the same pair keep only the first row (#627). A bridge can only federate what survived ingest.

Both addressed 2026-10-02.

Migration 113 (PR #696) adds source_predicate to the relation key, so distinct raw predicates are now distinct rows. A repeat of one raw predicate with other evidence still keeps only the first row’s evidence; rolling that up stays open on #627.

PrimeKG’s drug–protein rows were re-ingested with their DrugBank role as source_predicate: target, enzyme, transporter, carrier. The enzyme, carrier and transporter bridges are back, proposed (#438 item 1).

3. The bridge manifest

Shape (see backend/kg_normalize/bridge_manifest.py + bridges.yaml):

- id: biolink:treats          # real Biolink slot — verified
  label: Drug treats disease
  biolink: biolink:treats
  members:
    - { source: prime_kg, predicate: indication }
    - { source: gp_kg,    predicate: treat }
    - { source: rtx_kg2,  predicate: "biolink:treats" }
 
- id: maple:drug-metabolizing-enzyme   # no Biolink slot — named honestly
  label: Drug metabolized by enzyme
  biolink: null
  members:
    - { source: prime_kg, predicate: enzyme }

Rules:

A member’s predicate is the source predicate exactly as stored in mods_kg_relation.source_predicate (case-insensitive), never a predicate_mapper.py key. backend/kg_normalize/stored_predicates.tsv snapshots the stored pairs, and test_bridge_manifest.py fails any member that names none of them. After an ingest changes what is stored, run backend/scripts/kg_bridge_report.py --refresh-snapshot; --check compares the live table with the snapshot and the manifest.
Every bridge has a status, approved or proposed, and a member may override its bridge’s. Only approved members are applied: by equivalents() unless asked otherwise, and by the training export’s bridged mode always. Approving a merge is a mapping decision and is Richard’s. Demoting approved to proposed only splits relations, so it needs no sign-off.
An approval is a record, not a word (Phase 2). An approved bridge, and a member whose own status is approved, carries approved_by and approved_at, which validate() and the database require, and an approval_note saying where it was made. A proposal carries none of the three. The seed’s approvals cite the merges they came from: PR #137 (2026-05-30) and PR #650 (Richard’s 2026-10-02 review).
mapped_predicate restricts a member to the rows stored under one mapped predicate. It is needed when a raw predicate is stored under several: CIViC Predictive is stored under three.
id is biolink:* only when a real slot genuinely matches; otherwise maple:*. Never invent a biolink:*.
A (source, predicate) member belongs to at most one bridge (two bridges sharing a member means the bridges are themselves equivalent — merge them).
Unbridged predicates are fine — they stay queryable by their raw value and can be bridged later in the editor.
Distinctions are preserved on purpose: PPI and DDI are separate bridges even though both relate to biolink:interacts_with.

Read primitive: Manifest.equivalents(source, predicate) returns the matching predicates in the other KGs — the one call a cross-KG query needs.

4. Predicate audit (predicate_mapper.py vs Biolink v4.4.2)

predicate_mapper.py declares 35 distinct biolink:* strings. Confirmed real: treats, has_target, interacts_with, has_phenotype, subclass_of, part_of, gene_associated_with_condition, expressed_in, participates_in, enables, located_in, affects, correlated_with.

Confirmed invented (not slots in v4.4.2): has_enzyme, has_carrier, has_transporter. → now maple:* bridges.

Not yet verified against the live v4.4.2 schema (treated as maple:* with a “verify before promoting” note in the seed, so we never over-claim): has_off_label_use, contraindicated_for, predicts_sensitivity_to, predicts_resistance_to, predicts_toxicity_for, decreases_response_to, increases_response_to, affects_metabolism_of, biomarker_for, has_not_phenotype, not_expressed_in, increases_/decreases_expression_of, gene_associated_with_phenotypic_feature, binds, catalyzes, has_substrate, contributes_to, causes.

A definitive pass (fetch the v4.4.2 LinkML YAML, diff the 35 strings) is a sub-task; until then the seed errs toward maple: honesty.

Done 2026-10-01 (GH #628). backend/kg_normalize/biolink_predicates.tsv lists the 247 predicate slots of Biolink 4.4.2, derived from the release’s own schema by backend/scripts/kg_biolink_predicates.py. The bridge test checks every biolink: claim against it.

Fifteen biolink: strings stored in the predicate column are not 4.4.2 slots, about 313K rows across all sources:

Stored predicate Rows Written by
gene_associated_with_phenotypic_feature 193,964 GP-KG, PrimeKG
contraindicated_for 59,784 PrimeKG
not_expressed_in 39,774 PrimeKG
has_off_label_use 5,090 PrimeKG
predicts_sensitivity_to 3,502 CIViC, OncoKB
has_not_phenotype 2,386 PrimeKG
predicts_toxicity_for 2,170 CIViC, PharmGKB
predicts_response_to 2,029 PharmGKB
affects_metabolism_of 1,776 PharmGKB
predicts_resistance_to 1,651 CIViC
drug_regulatory_status_world_wide 402 RTX-KG2 (its own Biolink version)
has_diagnostic_implication_for 87 OncoKB
has_prognostic_implication_for 36 OncoKB
has_phenotypic_feature 6 patient overlay
has_variant 4 patient overlay

Real slots exist for several of the meanings, under different names:

contraindicated in;

associated with sensitivity to;

associated with resistance to;

associated with response to.

Biolink 4.4.2 expresses negation (not_expressed_in, has_not_phenotype) with a qualifier, not a slot. One more caveat: has target is a real slot, but 4.4.2 declares it disease → gene, not drug → protein. So the drug–target bridge, biolink:has_target until 2026-10-04, is now maple:drug-target (GH #628). The stored predicate value is unchanged (§6).

What to do with the stored values is still §6’s open decision. The per-source report is backend/scripts/kg_bridge_report.py.

Stored predicate	Rows	Written by
`gene_associated_with_phenotypic_feature`	193,964	GP-KG, PrimeKG
`contraindicated_for`	59,784	PrimeKG
`not_expressed_in`	39,774	PrimeKG
`has_off_label_use`	5,090	PrimeKG
`predicts_sensitivity_to`	3,502	CIViC, OncoKB
`has_not_phenotype`	2,386	PrimeKG
`predicts_toxicity_for`	2,170	CIViC, PharmGKB
`predicts_response_to`	2,029	PharmGKB
`affects_metabolism_of`	1,776	PharmGKB
`predicts_resistance_to`	1,651	CIViC
`drug_regulatory_status_world_wide`	402	RTX-KG2 (its own Biolink version)
`has_diagnostic_implication_for`	87	OncoKB
`has_prognostic_implication_for`	36	OncoKB
`has_phenotypic_feature`	6	patient overlay
`has_variant`	4	patient overlay

5. The editor (what the user asked for)

DB-backed so it’s live-editable; YAML stays the seed/export.

Migration (backend/migrations.py): mods_kg_bridge (id text PK, label, biolink nullable, note, is_active, timestamps)
- mods_kg_bridge_member (bridge_id FK, source, source_predicate, unique on (source, source_predicate) — enforces the one-bridge rule at the DB level).
Seed/sync script: load bridges.yaml → upsert into the tables; the module’s validate() runs first and refuses a dirty manifest.
Service: BridgeManifestService reads from the tables (falls back to YAML if the tables are empty), exposes equivalents() for query expansion.
API (backend/mods_kg_api.py or a new bridges_api.py): list bridges + members; create / update / delete bridge; add / remove member; an “unbridged predicates” report (every (source, source_predicate) in mods_kg_relation with no bridge) so the user sees what’s left to map.
MCP: mods_kg_list_bridges, mods_kg_create_bridge, mods_kg_add_bridge_member, mods_kg_unbridged_predicates.
Frontend: a “KG Mappings” tab in the Research dashboard — bridges as cards (id, label, biolink-or-maple chip, member list grouped by source); create-bridge form; drag/add unbridged predicates onto a bridge; the unbridged-predicates panel as the worklist. Register in the 3 component locations per CLAUDE.md.

5.1 As built (GH #628 Phase 2, 2026-10-02)

Richard decided three things before the build:

The database is authoritative and live; the YAML is what training reads. The tab, the API, the MCP tools and query expansion read mods_kg_bridge / mods_kg_bridge_member, and fall back to bridges.yaml while the tables are empty. The training export keeps reading the YAML and recording its sha256. So an approval reaches a model only after the YAML is exported and that PR merges, and the tab shows what the YAML has not seen.
The YAML catches up on demand. backend/scripts/kg_bridge_sync.py export writes the database as canonical YAML, and the change goes through a normal PR. The backend never commits.
The UI is §5’s, with a picker instead of drag-and-drop.
- Bridges are cards grouped by section.
- Members are grouped by source, with live row counts.
- The unbridged worklist adds a pair to a bridge through an “add to…” picker.
- Approve buttons appear only for Richard.

What landed:

Migration 110: the two tables. The rules are enforced by the database, not trusted to the editor:
- one owner per (source, lower(source_predicate)) unless split by distinct mapped_predicate: a unique index, plus a trigger for the case an index cannot express;
- approved requires approved_by and approved_at, and proposed carries neither;
- a member cannot join an approved bridge without a status of its own, which would approve it by inheritance;
- an approved bridge’s id or Biolink claim changes only with a new approval.
backend/mods_kg_bridge_service.py (BridgeManifestService). It returns the same Manifest the YAML loader builds, so relation_identity and the export accept either. It also holds the edit rules:
- a new bridge is always proposed;
- a member added to an approved bridge joins as proposed;
- demoting clears the approval record;
- only the owner can approve.
API under /api/mods/kg/bridges (mods_kg_api.py):
- Every route needs api:med:access. Signing in creates a user for any Google account, so a sign-in alone is not access.
- Approval needs the owner’s own credential and refuses the agent token and the preview mock token by name.
MCP: mods_kg_list_bridges, mods_kg_create_bridge, mods_kg_add_bridge_member and mods_kg_unbridged_predicates. There is no approve tool, and a test fails if any server grows one.
kg_bridge_sync.py: seed (a dry run unless --apply, refuses an invalid manifest or tables that already hold edits), export, diff, format.
bridges.yaml is now in canonical form, with every approval recorded. A test fails a hand edit that is not.
Row counts come from a live GROUP BY over mods_kg_relation. It takes about 20 s on prod, so it is cached for ten minutes, and the tab renders from the committed snapshot until the scan lands.

Not done here: wiring equivalents() into the canonical read path (§8).

6. Decided 2026-10-04: leave the stored `predicate` values (option A)

The predicate column holds the mapper’s legacy label, not a Biolink claim:

fifteen of its strings are not Biolink 4.4.2 slots (§4);
PrimeKG’s four drug–protein roles all sit under biolink:has_target, so the column alone cannot tell a target from an enzyme.

The options were:

Option	What	Cost
A (decided)	Leave `predicate` as is.	~0
B	Migration: set `predicate = source_predicate` wherever the mapped value was invented. Clean, but a large UPDATE and an audit of every reader of `predicate`.	medium
C	Migration: rename invented `biolink:` values to `maple:`. Honest, but still a non-source value in a column federation doesn’t use.	medium

Richard chose A on 2026-10-04 (GH #628), without the per-row metadata.predicate_is_legacy_mapped flag first proposed here.

The flag would have meant about 313K row writes to record what kg_bridge_report.py already computes on demand.
Biolink claims come only from bridges (their id and biolink fields), which are checked against the pinned release.

Two consequences:

A reader that shows only this column shows a merged meaning. Until GH #628’s display fix (2026-10-04), the canonical read service returned predicate and not source_predicate.
- So the Explorer and the KG MCP tools showed a CIViC Does Not Support row under the predicate of the claim it fails to support.
- Every edge they return now carries assertions: each (source_id, source_predicate) behind it.
- Querying by bridge id instead is #735.
Since #707 (2026-10-04), CIViC’s 24 “Reduced Sensitivity” rows are stored under predicts_resistance_to, as Richard decided on 2026-10-02.
- That is a mapper fix applied in place, not a column cleanup.
- CIViC’s Does Not Support rows keep their support predicate, so only source_predicate tells them apart (#726).

7. Parallelization Analysis (per CLAUDE.md)

Surfaces touched: backend API ✓, frontend component ✓, MCP tool ✓, migration (migrations.py) ✓, DB-stored config ✓ (the bridge tables), docs ✓, tests ✓. Not touched: worker handler, embeddings, training.
Parallel split: the manifest module + seed + tests (this PR) are standalone and already landed. The DB layer + API + MCP can proceed in one session; the frontend tab waits on the API contract (serialize after the API lands). The v4.4.2 verification pass is fully independent.
Collision hotspots: migrations.py (Phase 2 took 110 on 2026-10-02, after checking every branch; 111 and 112 went to #674 and #676, which merged first), main.py (router registration), component_registry.py, ComponentLoader.jsx, Terminal.jsx. Batch these; do not parallelize across them. Phase 2 avoided the last four: its routes joined the mounted mods_kg_api.py router, and the editor is a tab, not a new dashboard.
Verify-how per surface: module → pytest tests/test_bridge_manifest.py (10 tests, passing). Seed/sync → run script, assert table row counts == manifest. API → pytest the CRUD + unbridged-report. UI → build + walk the tab. Query expansion → a canonical-service test that a cross-KG query for biolink:treats returns rows from prime_kg AND gp_kg.

That last test can’t fail (2026-10-04). PrimeKG indication and GP-KG treat are both stored as biolink:treats. Every approved cross-source bridge’s members share one stored predicate, so merging already works through predicate. Bridge-aware queries add splitting by meaning, and #735 defines tests that can fail.

8. Status / sequencing

Tracked in GH #628 since 2026-10-01; that issue’s checklist is the live one.

References

backend/kg_normalize/bridge_manifest.py, bridges.yaml, stored_predicates.tsv
backend/mods_kg_bridge_service.py, backend/mods_kg_api.py (/bridges), backend/scripts/kg_bridge_sync.py, frontend/src/components/ModsKGMappingsTab.jsx
backend/kg_normalize/biolink_predicates.py, biolink_predicates.tsv
backend/kg_normalize/relation_identity.py (the export’s bridged tokens)
backend/scripts/kg_bridge_report.py, backend/scripts/kg_biolink_predicates.py
backend/kg_normalize/predicate_mapper.py (the legacy per-source maps)
docs/kg_predicate_mapping_report.md (the 2026-05 audit that introduced the invented slots)
CLAUDE.md → “Data and Mapping Integrity Rules (HARD)”
Biolink v4.4.2: https://github.com/biolink/biolink-model/releases/tag/v4.4.2

MODS Notebook

Explorer

KG Predicate Federation — Plan (#86)

1. Why federation

2. The key enabler: bridges key on `source_predicate`

3. The bridge manifest

4. Predicate audit (predicate_mapper.py vs Biolink v4.4.2)

5. The editor (what the user asked for)

5.1 As built (GH #628 Phase 2, 2026-10-02)

6. Decided 2026-10-04: leave the stored `predicate` values (option A)

7. Parallelization Analysis (per CLAUDE.md)

8. Status / sequencing

References

Graph View

Table of Contents

Backlinks

author	maple
last-verified	2026-10-04
status	seedling
description	Keeping native predicates alongside a bridge manifest, instead of flattening eleven sources into one vocabulary.

MODS Notebook

Explorer

KG Predicate Federation — Plan (#86)

1. Why federation

2. The key enabler: bridges key on source_predicate

3. The bridge manifest

4. Predicate audit (predicate_mapper.py vs Biolink v4.4.2)

5. The editor (what the user asked for)

5.1 As built (GH #628 Phase 2, 2026-10-02)

6. Decided 2026-10-04: leave the stored predicate values (option A)

7. Parallelization Analysis (per CLAUDE.md)

8. Status / sequencing

References

Graph View

Table of Contents

Backlinks

2. The key enabler: bridges key on `source_predicate`

6. Decided 2026-10-04: leave the stored `predicate` values (option A)