Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation
Abstract
Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task designs, three independent runs, and 3,000 tweets labeled for offensive language and hate speech. Repeating the same model and task design produced high agreement (median Fleiss' κ= 0.91). Agreement fell when we changed the task design for the same tweets (median Cohen's κ= 0.76). Task design and model choice increased the variance of estimated prevalence by factors of 76.7 for offensive language and 110.6 for hate speech compared with sampling variance alone. Variation across LLM task designs reached 560-572 basis points, compared with 270-331 basis points across five human instrument versions. Confidence scores did not solve this problem. They tracked repeated model outputs more closely than agreement with human labels, and grouping six tweets in one prompt lowered mean offensive-language confidence by 660 basis points. We call the variation caused by task design and model choice instrument uncertainty. Researchers can measure it only by comparing reasonable task designs. Repeating one setup or relying on confidence scores cannot replace that test.
Versions
Related
- Bayesian Uncertainty Quantification of Breast-Cancer Pathology AI Agreement under Reader- and Annotation-Condition-Dependent Reference Labels
- A Simple and Reliable Method for Estimating the Duration of an Online Questionnaire at the Design Stage
- Uncertainty Quantification Methods for Optimal Excitation Design in Parameter Identification
- Network Flexibility Design Under Endogenous Uncertainty
- Beyond χ2: Ensemble null tests for reliable scientific discovery
BibTeX
@misc{reiter2026reliable,
title = {{Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation}},
author = {Thomas Reiter and Christoph Kern and Fedor Miasnikov and Sofiia Nikolenko and Rob Chew and Stephanie Eckman and Frauke Kreuter},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.35824},
eprint = {2609.35824},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.35824},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation
AU - Thomas Reiter
AU - Christoph Kern
AU - Fedor Miasnikov
AU - Sofiia Nikolenko
AU - Rob Chew
AU - Stephanie Eckman
AU - Frauke Kreuter
PY - 2026
DA - 2026/09/22
PB - arXiv
DO - 10.48550/arxiv.2609.35824
UR - https://arxiv.org/abs/2609.35824
AB - Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task designs, three independent runs, and 3,000 tweets labeled for offensive language and hate speech. Repeating the same model and task design produced high agreement (median Fleiss' κ= 0.91). Agreement fell when we changed the task design for the same tweets (median Cohen's κ= 0.76). Task design and model choice increased the variance of estimated prevalence by factors of 76.7 for offensive language and 110.6 for hate speech compared with sampling variance alone. Variation across LLM task designs reached 560-572 basis points, compared with 270-331 basis points across five human instrument versions. Confidence scores did not solve this problem. They tracked repeated model outputs more closely than agreement with human labels, and grouping six tweets in one prompt lowered mean offensive-language confidence by 660 basis points. We call the variation caused by task design and model choice instrument uncertainty. Researchers can measure it only by comparing reasonable task designs. Repeating one setup or relying on confidence scores cannot replace that test.
ER -