Study Design Indexing in Transition: A Focused Comparison of manual NLM Indexing vs. Transformer-based Automated Models
- 1University of Illinois Chicago
Abstract
ObjectivesWe sought to evaluate the accuracy and suitability of a transformer-based model (TM) for indexing clinical study designs, in comparison to National Library of Medicine (NLM) indexing.
Materials and MethodsFor 2016 (manual NLM) and 2025 (automated NLM) published articles, we conducted a limited evaluation of the TM model, with articles that received the most confident predictions, with which NLM assignments disagreed. Ground truth was established through independent dual annotation for four prominent study designs: cohort, case-control, cross-sectional, and case report.
ResultsFor three designs (case-control, case report, cross-sectional), the articles having the top 100 predictive TM scores (when NLM failed to assign that design) were judged to exhibit that design in the great majority (86-100%) of cases. Conversely, the articles having the lowest 100 predictive TM scores (when NLM did assign the study design) exhibited the design only in relatively few (0-21%) of cases. The most confident predictions of the TM model were highly accurate and not redundant with automated NLM indexing; the exception was cohort studies articles, in which both TM and NLM labels showed high error rates of both omission and commission.
DiscussionTM may have value for identifying articles exhibiting study designs, which is important for clinical decision-making and evidence syntheses.
ConclusionNLMs cohort studies indexing is unreliable as a gold standard for building automated systems, warranting efforts to create a new manually annotated corpus.
LAY SUMMARYCollecting all relevant evidence on a given clinical question can be important. However, the existing indexing in PubMed/MEDLINE is neither accurate nor comprehensive enough for many purposes. Our group has created an automated machine learning model for finding and indexing articles using important clinical study designs. We compared our indexing against the existing indexing to see whether our approach adds value. When our model is highly confident, and disagrees with PubMed/MEDLINE indexing, our model is significantly more accurate. However, for one study design (cohort studies), both approaches frequently made errors such as missing studies or mislabeling them. For this study design, the current PubMed/MEDLINE indexing is not reliable enough to serve as a standard for training future automated indexing systems. This work advances automatic detection of important clinical study designs and complements existing PubMed/MEDLINE indexing.
Related
- Comparative analysis of CNN, Transformer, and Mamba-based models for automated eye disease classification from fundus images
- HemaViT: transformer-based deep learning for automated non-invasive anemia detection using conjunctival imaging
- Comparison of two automated PGMI-based software tools for breast positioning quality assessment in DBT screening
- An explainable vision transformer approach for automated neurological disorder classification from brain MRI scans
- Method Comparison of Three Automated D-Dimer Assays.
BibTeX
@misc{das2026study,
title = {{Study Design Indexing in Transition: A Focused Comparison of manual NLM Indexing vs. Transformer-based Automated Models}},
author = {P. Das and J. Schneider and E. Mayo-Wilson and H. Kilicoglu and J. D. Menke and D. Nam and K. Ninan and J.-P. Oberste and A. M. Troy and X. Ying and A. W. Holt and N. R. Smalheiser},
howpublished = {medRxiv},
year = {2026},
month = jun,
doi = {10.64898/2026.06.03.26354854},
url = {https://www.medrxiv.org/content/10.64898/2026.06.03.26354854v2},
note = {CC BY},
}
RIS
TY - UNPB
TI - Study Design Indexing in Transition: A Focused Comparison of manual NLM Indexing vs. Transformer-based Automated Models
AU - P. Das
AU - J. Schneider
AU - E. Mayo-Wilson
AU - H. Kilicoglu
AU - J. D. Menke
AU - D. Nam
AU - K. Ninan
AU - J.-P. Oberste
AU - A. M. Troy
AU - X. Ying
AU - A. W. Holt
AU - N. R. Smalheiser
PY - 2026
DA - 2026/06/03
PB - medRxiv
DO - 10.64898/2026.06.03.26354854
UR - https://www.medrxiv.org/content/10.64898/2026.06.03.26354854v2
AB - ObjectivesWe sought to evaluate the accuracy and suitability of a transformer-based model (TM) for indexing clinical study designs, in comparison to National Library of Medicine (NLM) indexing. Materials and MethodsFor 2016 (manual NLM) and 2025 (automated NLM) published articles, we conducted a limited evaluation of the TM model, with articles that received the most confident predictions, with which NLM assignments disagreed. Ground truth was established through independent dual annotation for four prominent study designs: cohort, case-control, cross-sectional, and case report. ResultsFor three designs (case-control, case report, cross-sectional), the articles having the top 100 predictive TM scores (when NLM failed to assign that design) were judged to exhibit that design in the great majority (86-100%) of cases. Conversely, the articles having the lowest 100 predictive TM scores (when NLM did assign the study design) exhibited the design only in relatively few (0-21%) of cases. The most confident predictions of the TM model were highly accurate and not redundant with automated NLM indexing; the exception was cohort studies articles, in which both TM and NLM labels showed high error rates of both omission and commission. DiscussionTM may have value for identifying articles exhibiting study designs, which is important for clinical decision-making and evidence syntheses. ConclusionNLMs cohort studies indexing is unreliable as a gold standard for building automated systems, warranting efforts to create a new manually annotated corpus. LAY SUMMARYCollecting all relevant evidence on a given clinical question can be important. However, the existing indexing in PubMed/MEDLINE is neither accurate nor comprehensive enough for many purposes. Our group has created an automated machine learning model for finding and indexing articles using important clinical study designs. We compared our indexing against the existing indexing to see whether our approach adds value. When our model is highly confident, and disagrees with PubMed/MEDLINE indexing, our model is significantly more accurate. However, for one study design (cohort studies), both approaches frequently made errors such as missing studies or mislabeling them. For this study design, the current PubMed/MEDLINE indexing is not reliable enough to serve as a standard for training future automated indexing systems. This work advances automatic detection of important clinical study designs and complements existing PubMed/MEDLINE indexing.
ER -