Skip to content
DraconiticResearch
Search Sources API
201,683 studies31 sourcesUpdated 42 min ago
Peer-reviewedCC BY 4.0PLOS One

AI versus human coding of NIH grant abstracts

  • Sarah A. Alkhatib1
  • Nadiya Jiwa2
  • Dallin Judd3
  • Justin M. Luningham4
  • Ginnie Sawyer-Morris5
  • Merve Ulukaya6
  • Todd Molfenter
  • Faye S. Taxman
  • Scott T. Walters
6 affiliations
  1. 1Department of Population & Community Health, College of Public Health, University of North Texas Health, Fort Worth, Texas, United States of America
  2. 2Texas College of Osteopathic Medicine, University of North Texas Health, Fort Worth, Texas, United States of America
  3. 3College of Science & Engineering, Texas Christian University, Fort Worth, Texas, United States of America
  4. 4Friends Research Institute, Baltimore, Maryland, United States of America
  5. 5George Mason University, Schar School of Policy and Government, Fairfax, Virginia, United States of America
  6. 6University of Wisconsin Center for Health Enhancement Systems Studies, Department of Industrial & Systems Engineering, Madison, Wisconsin, United States of America
Open at sourcePDFDOICite

Abstract

Large language models (LLMs) are increasingly used for qualitative analysis in substance use research, yet their performance relative to human coders remains underexplored. This study compares ChatGPT-4.0 with human coders in performing qualitative coding tasks using NIH grant abstracts as a test case, focusing on the identification and description of research innovations. Using a sample of NIH HEAL Initiative grant abstracts related to opioid overdose prevention, a total of 125 abstracts were independently coded by ChatGPT and humans to generate innovation descriptions, which were then evaluated by both human raters and ChatGPT for depth/detail and relevance/completeness using 5-point Likert scales. Identical instructions were used across all coding and evaluation stages. ChatGPT-generated descriptions were consistently rated higher than human-generated descriptions on both dimensions. Human evaluators rated ChatGPT outputs at an average of 4.47 for both depth/detail and relevance/completeness, compared to 3.33 and 3.24 for human outputs, respectively (F(1,176)=133.9, p < 0.001). These findings suggest that LLMs, when carefully prompted, can enhance the efficiency and quality of qualitative research evaluation.

Related

  1. Non-coding RNAs drive transcriptional network rewiring underlying thermal dimorphism in a human pathogenic fungus
    X. Hu, J. Wang, E. Yang
    bioRxivPreprintCC BY-NC-NDFull textMicrobiologybioRxiv
  2. Predictive coding networks capture human neural representations missing in supervised DNNs
    D. Gütlin, D. Kittelmann, R. Auksztulewicz
    bioRxivPreprintCC BY-NC-NDFull textNeurosciencebioRxiv
  3. Predictive coding of visual feedback of actions in human parieto-frontal circuits
    M. Tani, S. Ferruzzi, R. Costanzo +2
    bioRxivPreprintCC BY-NC-NDFull textNeurosciencebioRxiv
  4. Structural polymorphism and population-variable coding capacity of HERV-K(HML-2) in human pangenomes
    D. A. Murimi-Worstell, M. Freeman, A. Minkina +3
    bioRxivPreprintCC BYFull textGenetics & genomicsbioRxiv
  5. Entorhinal grid coding as a functional link between tau accumulation and episodic memory in human aging
    V. Segen, T. Belge Bernard, G. Callau Navarro +11
    bioRxivPreprintCC BYFull textNeurosciencebioRxiv

Cite

BibTeXRIS
BibTeX
@article{alkhatib2026versus,
  title = {{AI versus human coding of NIH grant abstracts}},
  author = {Sarah A. Alkhatib and Nadiya Jiwa and Dallin Judd and Justin M. Luningham and Ginnie Sawyer-Morris and Merve Ulukaya and Todd Molfenter and Faye S. Taxman and Scott T. Walters},
  journal = {PLOS One},
  year = {2026},
  month = oct,
  doi = {10.1371/journal.pone.0343857},
  url = {https://doi.org/10.1371/journal.pone.0343857},
  note = {CC BY 4.0},
}
RIS
TY  - JOUR
TI  - AI versus human coding of NIH grant abstracts
AU  - Sarah A. Alkhatib
AU  - Nadiya Jiwa
AU  - Dallin Judd
AU  - Justin M. Luningham
AU  - Ginnie Sawyer-Morris
AU  - Merve Ulukaya
AU  - Todd Molfenter
AU  - Faye S. Taxman
AU  - Scott T. Walters
PY  - 2026
DA  - 2026/10/08
JO  - PLOS One
DO  - 10.1371/journal.pone.0343857
UR  - https://doi.org/10.1371/journal.pone.0343857
AB  - Large language models (LLMs) are increasingly used for qualitative analysis in substance use research, yet their performance relative to human coders remains underexplored. This study compares ChatGPT-4.0 with human coders in performing qualitative coding tasks using NIH grant abstracts as a test case, focusing on the identification and description of research innovations. Using a sample of NIH HEAL Initiative grant abstracts related to opioid overdose prevention, a total of 125 abstracts were independently coded by ChatGPT and humans to generate innovation descriptions, which were then evaluated by both human raters and ChatGPT for depth/detail and relevance/completeness using 5-point Likert scales. Identical instructions were used across all coding and evaluation stages. ChatGPT-generated descriptions were consistently rated higher than human-generated descriptions on both dimensions. Human evaluators rated ChatGPT outputs at an average of 4.47 for both depth/detail and relevance/completeness, compared to 3.33 and 3.24 for human outputs, respectively (F(1,176)=133.9, p < 0.001). These findings suggest that LLMs, when carefully prompted, can enhance the efficiency and quality of qualitative research evaluation.
SN  - 1932-6203
ER  -