Study design indexing in transition: a focused comparison of manual NLM indexing versus transformer-based automated models
4 affiliations
- 1School of Information Sciences, University of Illinois at Urbana-Champaign, Champaign, IL 61820, United States.
- 2Information School, University of Wisconsin-Madison, Madison, WI 53706, United States.
- 3Department of Epidemiology, University of North Carolina, Chapel Hill, NC 27514, United States.
- 4Department of Psychiatry, University of Illinois at Chicago, Chicago, IL 60612, United States.
Abstract
Objectives
We sought to evaluate the accuracy and suitability of a transformer-based model (TM) for indexing clinical study designs, in comparison to National Library of Medicine (NLM) indexing.Materials and methods
For 2016 (manual NLM) and 2025 (automated NLM) published articles, we conducted a limited evaluation of the TM model, with articles that received the most confident predictions, with which NLM assignments disagreed. Ground truth was established through independent dual annotation for 4 prominent study designs: cohort, case-control, cross-sectional, and case report.Results
For 3 designs (case-control, case report, and cross-sectional), the articles having the top 100 predictive TM scores (when NLM failed to assign that design) were judged to exhibit that design in the great majority (86%-100%) of cases. Conversely, the articles having the lowest 100 predictive TM scores (when NLM did assign the study design) exhibited the design only in relatively few (0%-21%) of cases. The most confident predictions of the TM model were highly accurate and not redundant with automated NLM indexing; the exception was cohort studies articles, in which both TM and NLM labels showed high error rates of both omission and commission.Discussion
TM may have value for identifying articles exhibiting study designs, which is important for clinical decision-making and evidence syntheses.Conclusion
NLM's cohort studies indexing is unreliable as a gold standard for building automated systems, warranting efforts to create a new manually annotated corpus.
Related
- Non-invasive control of gene editing in vivo by photoswitchable Cas12f and focused ultrasound
- Image-guided focused ultrasound for non-invasive modulation of superficial and deep tissues.
- Geometric Relevance Is Not Sufficient: Evaluating Focused Post-Training Supervision in Diffusion Docking
- CBAT-Net: a unified CNN–BiLSTM–MultiHead transformer architecture for joint Q3/Q8 protein secondary structure prediction
- A novel LLM-based GPT-neo transformer with machine learning architecture for promoters prediction in Agrobacterium
BibTeX
@article{das2026study,
title = {{Study design indexing in transition: a focused comparison of manual NLM indexing versus transformer-based automated models}},
author = {Puranjani Das and Jodi Schneider and Evan Mayo-Wilson and Halil Kilicoglu and Joe D Menke and Dongin Nam and Kiran Ninan and Jean-Pierre Oberste and Ang Michael Troy and Xiangji Ying and Arthur W Holt and Neil R Smalheiser},
journal = {JAMIA open},
year = {2026},
month = oct,
doi = {10.1093/jamiaopen/ooag197},
url = {https://europepmc.org/article/PMC/PMC13718441},
note = {CC BY-NC},
}
RIS
TY - JOUR
TI - Study design indexing in transition: a focused comparison of manual NLM indexing versus transformer-based automated models
AU - Puranjani Das
AU - Jodi Schneider
AU - Evan Mayo-Wilson
AU - Halil Kilicoglu
AU - Joe D Menke
AU - Dongin Nam
AU - Kiran Ninan
AU - Jean-Pierre Oberste
AU - Ang Michael Troy
AU - Xiangji Ying
AU - Arthur W Holt
AU - Neil R Smalheiser
PY - 2026
DA - 2026/10/01
JO - JAMIA open
DO - 10.1093/jamiaopen/ooag197
UR - https://europepmc.org/article/PMC/PMC13718441
AB - Objectives We sought to evaluate the accuracy and suitability of a transformer-based model (TM) for indexing clinical study designs, in comparison to National Library of Medicine (NLM) indexing.Materials and methods For 2016 (manual NLM) and 2025 (automated NLM) published articles, we conducted a limited evaluation of the TM model, with articles that received the most confident predictions, with which NLM assignments disagreed. Ground truth was established through independent dual annotation for 4 prominent study designs: cohort, case-control, cross-sectional, and case report.Results For 3 designs (case-control, case report, and cross-sectional), the articles having the top 100 predictive TM scores (when NLM failed to assign that design) were judged to exhibit that design in the great majority (86%-100%) of cases. Conversely, the articles having the lowest 100 predictive TM scores (when NLM did assign the study design) exhibited the design only in relatively few (0%-21%) of cases. The most confident predictions of the TM model were highly accurate and not redundant with automated NLM indexing; the exception was cohort studies articles, in which both TM and NLM labels showed high error rates of both omission and commission.Discussion TM may have value for identifying articles exhibiting study designs, which is important for clinical decision-making and evidence syntheses.Conclusion NLM's cohort studies indexing is unreliable as a gold standard for building automated systems, warranting efforts to create a new manually annotated corpus.
SN - 2574-2531
ER -