{
  "abstract": "BackgroundLarge language models (LLMs) are increasingly used to automate title-abstract screening in systematic reviews, but validation studies typically report sensitivity, specificity, or Cohens kappa -- metrics that are unstable under the low and unbalanced prevalence that characterises most reviews. No previous study has simultaneously combined a medical topic; a hybrid two-stage pipeline (rules + LLM); a three-way classification with a separately sampled uncertain class; stratified sampling with three independent human reviewers; and dual reporting of prevalence-robust coefficients (PABAK and Gwets AC1).\n\nObjectiveTo validate a two-stage automated screening pipeline (a rule-based Python script followed by a reproducible LLM adjudication of uncertain records) against three independent human reviewers in a medical systematic review on high on-treatment platelet reactivity (HTPR), aspirin resistance, and carotid plaque. The primary validation target is the Stage 1 automatic-exclusion decision (670/904 records), which is the workload the pipeline is designed to take off the human reviewer.\n\nMethodsFrom 904 screened records (Stage 1: 131 INCLUDE, 670 EXCLUDE, 94 UNCERTAIN; Stage 2 LLM via API, temperature = 0: 11 INCLUDE, 83 EXCLUDE), a stratified sample of 185 records (50 per Stage 1 stratum + 35 Stage 2; seed = 42) was judged independently and blind to the automated labels by three physicians: E.P. (pipeline designer), D.C. and P.P.B. (both external to the thesis project). Concordance was quantified with sensitivity, specificity, PABAK (q = 3 throughout), Gwets AC1, work saved over sampling (WSS), and number needed to read (NNR) -- AI versus each reviewer and between reviewers -- overall and by stratum. Majority labels (2 of 3) are reported as a supplementary panel; they do not replace pairwise tables as the primary reference for sensitivity.\n\nResultsOn the primary target, Stage1-EXCLUDE concordance was PABAK = AC1 = 1.000 versus E.P. (48/48) and versus D.C. (50/50), and PABAK = 0.940 / AC1 = 0.959 versus Berti (48/50 EXCLUDE, 2 UNCERTAIN). That stratum is 670/904 records (74.1%). Pairwise human agreement was highest between the two external readers (Costazza vs Berti: PABAK = 0.854, AC1 = 0.881) and lower when E.P. was one of the pair (0.703 / 0.774 vs Costazza; 0.621 / 0.702 vs Berti). AI versus each human remained poor (PABAK 0.225 / 0.205 / 0.222). Majority labels on 185 records were 146 EXCLUDE / 27 UNCER-TAIN / 4 INCLUDE / 8 NO_MAJORITY. The machine labelled INCLUDE 0 of the 4 majority-INCLUDE records (74, 75, 76, 80). Overall sensitivity was 0.800 versus E.P. (8/10; false negatives 623, 712), 0.500 versus D.C. (1/2; false negative 76), and 0.833 versus Berti (5/6; false negative 76). Versus majority the false-negative count is 1 (record 76); records 623 and 712 are majority EXCLUDE. Stage1-INCLUDE sensitivity of 1.000 is tautological. WSS = 74.1% describes the exclusion stratum only.\n\nConclusionsWhat this study validates, now with three independent physicians, is automatic exclusion at Stage 1. Fine classification of INCLUDE records is not validated: humans do not agree on which residual records are INCLUDE (triple INCLUDE intersection = record 74 only), Stage 2 missed most of the few human INCLUDE labels it received, and the machine recovered none of the four majority-INCLUDE records. Domain-familiarity as an explanation of INCLUDE divergence remains a hypothesis, not a measured variable.",
  "abstract_shown": true,
  "affiliations": [
    "Hospital of Bolzano (SABES-ASDAA)"
  ],
  "article_type": "PUBLISHAHEADOFPRINT",
  "arxiv": null,
  "author_total": 3,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "E. Pirani"
    },
    {
      "name": "D. Costazza"
    },
    {
      "name": "P. P. Berti"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/3hsmg2q470383.bib",
  "cited_by": 0,
  "doi": "10.64898/2026.10.05.26364147",
  "fulltext_url": "https://www.medrxiv.org/content/10.64898/2026.10.05.26364147v1.full",
  "id": "3hsmg2q470383",
  "issns": [],
  "landing_url": "https://www.medrxiv.org/content/10.64898/2026.10.05.26364147v1",
  "language": null,
  "licence": "CC-BY",
  "licence_label": "CC BY",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "medrxiv",
      "health informatics"
    ],
    [
      "openalex",
      "1804|Meta-analysis and systematic reviews"
    ]
  ],
  "openalex": "W7221343878",
  "pdf_url": "https://www.medrxiv.org/content/10.64898/2026.10.05.26364147v1.full.pdf",
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-05",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/3hsmg2q470383.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:20:20Z",
      "native_id": "10.64898/2026.10.05.26364147",
      "source": "medrxiv",
      "url": "https://www.medrxiv.org/content/10.64898/2026.10.05.26364147v1"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "medrxiv:health informatics",
      "subfield": "epidemiology-public-health"
    }
  ],
  "title": "Validation of a two-stage automated screening pipeline for medical systematic reviews: a stratified concordance study with three independent human reviewers, using PABAK and Gwet's AC1",
  "title_text": "Validation of a two-stage automated screening pipeline for medical systematic reviews: a stratified concordance study with three independent human reviewers, using PABAK and Gwet's AC1",
  "type": "preprint",
  "updated_at": "2026-10-30T12:20:20Z",
  "url": "https://research.draconitic.com/paper/3hsmg2q470383",
  "venue": "medRxiv",
  "versions": []
}