An Expert-Curated Benchmark for Frontier and Local LLMs Querying a Biomedical Knowledge Graph
- 1University of Tennessee Health Science Center
Abstract
We combine GeneNetwork's 1TB structured database with its metadata as an RDF graph and an associated literature corpus. We benchmark frontier and local LLMs querying it through four architectures: document, graph and agentic retrieval-augmented generation (RAG), and a combination of the three. We evaluate them on 40 domain specific expert-curated questions with a new metric: the satisfaction frequency psi (psi, i.e, the share of questions answered with precision and recall of at least 70% or an f1 of at least 60%). psi ranks systems the same way precision, recall and f1 -- a machine learning metric built from the harmonic mean of precision and recall -- but is easier to read and assess for humans. System architecture had a large effect: with full use of Claude Opus 4.8 the plain document RAG of our earlier GNQA system had a psi of 36% against 86% for our new combined tools. Delegating most of the work from Claude Opus 4.8 to a local model halved query costs while still reaching a psi of 59%.
Related
- A Comparative Benchmark of Biomedical Language Models for Concept Normalization from Real-World Text
- Every Cure Knowledge Graph: A Unified Biomedical Knowledge Graph for Drug Repurposing
- HypoKG: Evidence-Disciplined Biomedical Hypothesis Generation Beyond Endpoint Knowledge
- A heterogeneous biomedical knowledge network framework for rare disease drug candidate prioritization: integrating Orphadata and DisGeNET via gene-bridge harmonization
- ddkg.skill: A Compositional Agent Skill for Translating Biomedical and Bioinformatics Questions into Cypher for the Data Distillery Knowledge Graph
BibTeX
@misc{medagbe2026expertcurated,
title = {{An Expert-Curated Benchmark for Frontier and Local LLMs Querying a Biomedical Knowledge Graph}},
author = {J. F. P. Medagbe and B. M. Kilyungi and A. Isaac and A. Kabua and F. Lisso and S. Wambua and R. W. Williams and P. Prins},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.04.756532},
url = {https://www.biorxiv.org/content/10.64898/2026.10.04.756532v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - An Expert-Curated Benchmark for Frontier and Local LLMs Querying a Biomedical Knowledge Graph
AU - J. F. P. Medagbe
AU - B. M. Kilyungi
AU - A. Isaac
AU - A. Kabua
AU - F. Lisso
AU - S. Wambua
AU - R. W. Williams
AU - P. Prins
PY - 2026
DA - 2026/10/04
PB - bioRxiv
DO - 10.64898/2026.10.04.756532
UR - https://www.biorxiv.org/content/10.64898/2026.10.04.756532v1
AB - We combine GeneNetwork's 1TB structured database with its metadata as an RDF graph and an associated literature corpus. We benchmark frontier and local LLMs querying it through four architectures: document, graph and agentic retrieval-augmented generation (RAG), and a combination of the three. We evaluate them on 40 domain specific expert-curated questions with a new metric: the satisfaction frequency psi (psi, i.e, the share of questions answered with precision and recall of at least 70% or an f1 of at least 60%). psi ranks systems the same way precision, recall and f1 -- a machine learning metric built from the harmonic mean of precision and recall -- but is easier to read and assess for humans. System architecture had a large effect: with full use of Claude Opus 4.8 the plain document RAG of our earlier GNQA system had a psi of 36% against 86% for our new combined tools. Delegating most of the work from Claude Opus 4.8 to a local model halved query costs while still reaching a psi of 59%.
ER -