{
  "abstract": "Objectives\n\nWe sought to evaluate the accuracy and suitability of a transformer-based model (TM) for indexing clinical study designs, in comparison to National Library of Medicine (NLM) indexing.Materials and methods\n\nFor 2016 (manual NLM) and 2025 (automated NLM) published articles, we conducted a limited evaluation of the TM model, with articles that received the most confident predictions, with which NLM assignments disagreed. Ground truth was established through independent dual annotation for 4 prominent study designs: cohort, case-control, cross-sectional, and case report.Results\n\nFor 3 designs (case-control, case report, and cross-sectional), the articles having the top 100 predictive TM scores (when NLM failed to assign that design) were judged to exhibit that design in the great majority (86%-100%) of cases. Conversely, the articles having the lowest 100 predictive TM scores (when NLM did assign the study design) exhibited the design only in relatively few (0%-21%) of cases. The most confident predictions of the TM model were highly accurate and not redundant with automated NLM indexing; the exception was cohort studies articles, in which both TM and NLM labels showed high error rates of both omission and commission.Discussion\n\nTM may have value for identifying articles exhibiting study designs, which is important for clinical decision-making and evidence syntheses.Conclusion\n\nNLM's cohort studies indexing is unreliable as a gold standard for building automated systems, warranting efforts to create a new manually annotated corpus.",
  "abstract_shown": true,
  "affiliations": [
    "School of Information Sciences, University of Illinois at Urbana-Champaign, Champaign, IL 61820, United States.",
    "Information School, University of Wisconsin-Madison, Madison, WI 53706, United States.",
    "Department of Epidemiology, University of North Carolina, Chapel Hill, NC 27514, United States.",
    "Department of Psychiatry, University of Illinois at Chicago, Chicago, IL 60612, United States."
  ],
  "article_type": "research article",
  "arxiv": null,
  "author_total": 12,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Puranjani Das",
      "orcid": "0000-0003-0657-7700"
    },
    {
      "affs": [
        1
      ],
      "name": "Jodi Schneider",
      "orcid": "0000-0002-5098-5667"
    },
    {
      "affs": [
        2
      ],
      "name": "Evan Mayo-Wilson"
    },
    {
      "affs": [
        0
      ],
      "name": "Halil Kilicoglu",
      "orcid": "0000-0003-3987-9393"
    },
    {
      "affs": [
        0
      ],
      "name": "Joe D Menke"
    },
    {
      "affs": [
        0
      ],
      "name": "Dongin Nam"
    },
    {
      "affs": [
        2
      ],
      "name": "Kiran Ninan"
    },
    {
      "affs": [
        2
      ],
      "name": "Jean-Pierre Oberste"
    },
    {
      "affs": [
        3
      ],
      "name": "Ang Michael Troy"
    },
    {
      "affs": [
        2
      ],
      "name": "Xiangji Ying"
    },
    {
      "affs": [
        3
      ],
      "name": "Arthur W Holt"
    },
    {
      "affs": [
        3
      ],
      "name": "Neil R Smalheiser",
      "orcid": "0000-0003-1079-3406"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/1ms2ndtr8ak1q.bib",
  "cited_by": 0,
  "doi": "10.1093/jamiaopen/ooag197",
  "fulltext_url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13718441/?tool=EBI",
  "id": "1ms2ndtr8ak1q",
  "issns": [
    "2574-2531"
  ],
  "landing_url": "https://europepmc.org/article/PMC/PMC13718441",
  "language": "eng",
  "licence": "CC-BY-NC",
  "licence_label": "CC BY-NC",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by-nc/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "heading",
      "AcademicSubjects/MED00010"
    ],
    [
      "heading",
      "AcademicSubjects/SCI01060"
    ],
    [
      "heading",
      "AcademicSubjects/SCI01530"
    ],
    [
      "heading",
      "Research and Applications"
    ],
    [
      "openalex",
      "1312|Biomedical Text Mining and Ontologies"
    ]
  ],
  "openalex": "W7224440827",
  "pdf_url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13718441/pdf/?tool=EBI",
  "pmcid": "PMC13718441",
  "pmid": "42960168",
  "published": "2026-10-01",
  "publisher": "Oxford University Press",
  "ris": "https://research.draconitic.com/paper/1ms2ndtr8ak1q.ris",
  "sources": [
    {
      "first_seen": "2026-10-31T00:20:39Z",
      "native_id": "PMC:PMC13718441",
      "source": "europepmc",
      "url": "https://europepmc.org/article/PMC/PMC13718441"
    },
    {
      "first_seen": "2026-11-01T00:20:50Z",
      "native_id": "MED:42960168",
      "source": "europepmc",
      "url": "https://europepmc.org/article/MED/42960168"
    },
    {
      "first_seen": "2026-11-02T12:22:28Z",
      "native_id": "PMC13718441",
      "source": "pmc",
      "url": "https://doi.org/10.1093/jamiaopen/ooag197"
    }
  ],
  "subjects": [
    {
      "field": "biochemistry",
      "origin": "openalex:1312|Biomedical Text Mining and Ontologies",
      "subfield": "molecular-biology"
    }
  ],
  "title": "Study design indexing in transition: a focused comparison of manual NLM indexing versus transformer-based automated models",
  "title_text": "Study design indexing in transition: a focused comparison of manual NLM indexing versus transformer-based automated models",
  "type": "peer-reviewed",
  "updated_at": "2026-11-02T12:22:28Z",
  "url": "https://research.draconitic.com/paper/1ms2ndtr8ak1q",
  "venue": "JAMIA open",
  "versions": []
}