Cross-System Legibility of a Practitioner-Derived Workflow-Error Taxonomy for Conversational AI: A Three-Comparator Agreement Study
- 1Xavier University of Louisiana
Abstract
Practitioner-derived taxonomies of conversational artificial intelligence (AI) workflow errors show low inter-rater agreement among human coders, leaving open whether the instrument is ill-specified or the judgments are inherently difficult. We delivered a locked eight-category workflow-error taxonomy verbatim, under standardized conditions, to three frontier large language model comparators from distinct developer lineages, which coded a documented 45-incident error corpus. On the same 16 incidents coded by three human reviewer-authors, comparator category agreement was substantial (Fleiss {kappa}=0.625) against slight human agreement ({kappa}=0.155); across the full corpus it was stable ({kappa}=0.632), and a 10-category refinement did not reduce it ({kappa}=0.690). Severity and a claimed-verification flag remained only fair in both arms and on both samples. Substantial cross-system consistency provides a legibility signal consistent with recoverable category distinctions, but cannot separate instrument clarity from shared model priors, and is not a validation of any coding.
Related
- AI‐Based Wound Measurement in a Telemedicine‐Style Workflow: Agreement With In‐Person Reference Measurements Compared With Individual Physicians and Expert Consensus
- Comparator Event Rates, Baseline Activity, and Operational Selection Pressure in Relapsing Multiple Sclerosis Trials: A Meta‐Epidemiological Analysis
- Prospective associations between psychosocial work exposures and general practitioner consultations in France
- AI-enabled and comparator digital interventions for adolescent mental health (2020–2025): early detection, human–AI collaboration, and ethical governance—a PRISMA-ScR scoping review
- Evaluating the quality and safety of a nurse practitioner-led virtual clinic for secondary prevention in stable coronary artery disease: a pilot feasibility study.
BibTeX
@misc{austria2026crosssystem,
title = {{Cross-System Legibility of a Practitioner-Derived Workflow-Error Taxonomy for Conversational AI: A Three-Comparator Agreement Study}},
author = {D. Austria and B. McCollister and J. E. Lindsey and M. Arowolo and M. Okon},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.02.26362090},
url = {https://www.medrxiv.org/content/10.64898/2026.09.02.26362090v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Cross-System Legibility of a Practitioner-Derived Workflow-Error Taxonomy for Conversational AI: A Three-Comparator Agreement Study
AU - D. Austria
AU - B. McCollister
AU - J. E. Lindsey
AU - M. Arowolo
AU - M. Okon
PY - 2026
DA - 2026/09/02
PB - medRxiv
DO - 10.64898/2026.09.02.26362090
UR - https://www.medrxiv.org/content/10.64898/2026.09.02.26362090v1
AB - Practitioner-derived taxonomies of conversational artificial intelligence (AI) workflow errors show low inter-rater agreement among human coders, leaving open whether the instrument is ill-specified or the judgments are inherently difficult. We delivered a locked eight-category workflow-error taxonomy verbatim, under standardized conditions, to three frontier large language model comparators from distinct developer lineages, which coded a documented 45-incident error corpus. On the same 16 incidents coded by three human reviewer-authors, comparator category agreement was substantial (Fleiss {kappa}=0.625) against slight human agreement ({kappa}=0.155); across the full corpus it was stable ({kappa}=0.632), and a 10-category refinement did not reduce it ({kappa}=0.690). Severity and a claimed-verification flag remained only fair in both arms and on both samples. Substantial cross-system consistency provides a legibility signal consistent with recoverable category distinctions, but cannot separate instrument clarity from shared model priors, and is not a validation of any coding.
ER -