Human inter-rater reliability of retrospective neuro-type classification: a Registered Report Stage 1 protocol (stratified n=200, blinded multiple raters)
- 1Polytechnic Institute, Zhejiang University, Hangzhou, China
Abstract
Stage 1 Registered Report for a blinded human inter-rater reliability study. A stratified random sample of 200 historical figures plus 30 gold-standard items is independently coded by at least three blinded raters with the six-dimensional rubric. Primary endpoint: six-category Fleiss kappa of at least 0.70 with bootstrap 95% lower bound at least 0.60 (three-category ND-subtype kappa at least 0.80); Gwet AC1 reported as a robustness check. A first-round kappa below 0.50 triggers exactly one rubric revision; persistent failure below 0.50 constitutes person-level falsification of the coding scheme. LLM-based ratings (observed agreement up to ~0.79) are treated as auxiliary and cannot replace human reliability. The protocol is registered before raters are recruited or any results are seen; requires IRB review.
Related
- A falsifiable framework for retrospective neuro-type (ADHD/ASD/AuDHD) attribution in historical figures: a Registered Report Stage 1 protocol
- Test-retest reliability of the Manikin Task and the robotic Kinarm Approach-Avoidance Task as measures of approach and avoidance tendencies towards physical activity and sedentary behaviours: A registered report
- LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability
- Complementing Engel Classification: Postoperative Scalp EEG as an Objective Indicator in SEEG‐Guided Radiofrequency Thermocoagulation—A Seizure‐Onset Pattern‐Based Stratified Analysis—
- Prehabilitation and Early Recovery After Surgery (ERAS) in head and neck cancers: a protocol for a prospective, stratified, open-label randomised controlled trial in adults (PreERA)
BibTeX
@misc{xu2026human,
title = {{Human inter-rater reliability of retrospective neuro-type classification: a Registered Report Stage 1 protocol (stratified n=200, blinded multiple raters)}},
author = {Xin Xu},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23586785},
url = {https://zenodo.org/records/23586785},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Human inter-rater reliability of retrospective neuro-type classification: a Registered Report Stage 1 protocol (stratified n=200, blinded multiple raters)
AU - Xin Xu
PY - 2026
DA - 2026/10/24
PB - Zenodo
DO - 10.5281/zenodo.23586785
UR - https://zenodo.org/records/23586785
AB - Stage 1 Registered Report for a blinded human inter-rater reliability study. A stratified random sample of 200 historical figures plus 30 gold-standard items is independently coded by at least three blinded raters with the six-dimensional rubric. Primary endpoint: six-category Fleiss kappa of at least 0.70 with bootstrap 95% lower bound at least 0.60 (three-category ND-subtype kappa at least 0.80); Gwet AC1 reported as a robustness check. A first-round kappa below 0.50 triggers exactly one rubric revision; persistent failure below 0.50 constitutes person-level falsification of the coding scheme. LLM-based ratings (observed agreement up to ~0.79) are treated as auxiliary and cannot replace human reliability. The protocol is registered before raters are recruited or any results are seen; requires IRB review.
ER -