Skip to content
DraconiticResearch
Search Sources API
181,713 studies31 sourcesUpdated 12 min ago
PreprintCC BYbioRxiv

Turning Domain Expertise into Multi-Dimensional Evaluation of Biomedical AI with Karenina

  • F. Carli
  • P. Rusina
  • L. Ai
  • L. Kuechenhoff
  • P. K. P. To
  • E. M. McDonagh
  • S. Lobentanzer
  • F. Petroni
  • A. Dugourd
  • D. Ochoa
  • J. Saez-Rodriguez1
  1. 1European Bioinformatics Institute (EMBL-EBI), European Molecular Biology Laboratory, Hinxton, UK;Open Targets, European Molecular Biology Laboratory, Hinxton, U
Read full textPDFDOICite

Abstract

Language models and agents are increasingly used in biomedicine, but current benchmarks reward correct answers even when the underlying reasoning is flawed. Here we introduce Karenina, an open-source framework that turns expert knowledge into multi-dimensional evaluations of questions, conversations and autonomous agents. Illustrated in Question-Answer pairs, multi-turn conversations and autonomous data-analysis, these dimensions together moves evaluation beyond scoring, enabling trustworthy decision-making with AI in biomedicine.

Related

  1. TCA precipitation and ethanol/HCl single-step purification evaluation: One-dimensional gel electrophoresis, bradford assays, spectrofluorometry and Raman spectroscopy data on HSA, Rnase, lysozyme - Mascots and Skyline data
    Balkis Eddhif, Nadia Guignard, Yann Batonneau +4
    Data in BriefPeer-reviewedCC BYFull text10 citedGeneral biochemistrySystems & synthetic biologyHAL
  2. Three-dimensional heterotypic colorectal cancer spheroid models for evaluation of drug response
    Jia Ning Nicolette Yau, Giulia Adriani
    Frontiers in OncologyPeer-reviewedCC BYFull text31 citedGeneral biologyOncologyHAL
  3. Psychonautic Benchmarking: Towards Standardized Expertise Criteria in Non-Ordinary States Research
    Juliana Estevez, David Lameiras, Giancarlo Allocca +2
    PsyArXivPreprintCC BY 4.0Full textNeuroscienceOSF
  4. Specific expertise from diffuse feedback: A feedforward-gated mechanism
    Ruining Guo, An Yan, Minggui Chen +2
    Science AdvancesPeer-reviewedCC BYNeuroscienceCrossref
  5. Basophilic Erythroblast Emerges as the Key Turning Point in Polycythemia Vera
    D. Konak, P. Pir
    bioRxivPreprintAll rights reservedFull textBioinformatics & computational biologybioRxiv

Cite

BibTeXRIS
BibTeX
@misc{carli2026turning,
  title = {{Turning Domain Expertise into Multi-Dimensional Evaluation of Biomedical AI with Karenina}},
  author = {F. Carli and P. Rusina and L. Ai and L. Kuechenhoff and P. K. P. To and E. M. McDonagh and S. Lobentanzer and F. Petroni and A. Dugourd and D. Ochoa and J. Saez-Rodriguez},
  howpublished = {bioRxiv},
  year = {2026},
  month = sep,
  doi = {10.64898/2026.09.01.748513},
  url = {https://www.biorxiv.org/content/10.64898/2026.09.01.748513v1},
  note = {CC BY},
}
RIS
TY  - UNPB
TI  - Turning Domain Expertise into Multi-Dimensional Evaluation of Biomedical AI with Karenina
AU  - F. Carli
AU  - P. Rusina
AU  - L. Ai
AU  - L. Kuechenhoff
AU  - P. K. P. To
AU  - E. M. McDonagh
AU  - S. Lobentanzer
AU  - F. Petroni
AU  - A. Dugourd
AU  - D. Ochoa
AU  - J. Saez-Rodriguez
PY  - 2026
DA  - 2026/09/01
PB  - bioRxiv
DO  - 10.64898/2026.09.01.748513
UR  - https://www.biorxiv.org/content/10.64898/2026.09.01.748513v1
AB  - Language models and agents are increasingly used in biomedicine, but current benchmarks reward correct answers even when the underlying reasoning is flawed. Here we introduce Karenina, an open-source framework that turns expert knowledge into multi-dimensional evaluations of questions, conversations and autonomous agents. Illustrated in Question-Answer pairs, multi-turn conversations and autonomous data-analysis, these dimensions together moves evaluation beyond scoring, enabling trustworthy decision-making with AI in biomedicine.
ER  -