Unsupervised Careless Responding Detection: A Comprehensive Comparison between Model-based and Model-Agnostic Approaches
Abstract
Careless responding (CR) is a major concern for data quality in the social sciences. It is common for researchers to attempt to detect and filter out CR at the respondent level. Though various choices are available, detection methods relying solely on response data have received much attention since they do not require the collection of additional information. Along these lines, model-based approaches to CR detection rely on specifying distinct measurement models for careful and careless respondents in a mixture model. In contrast, model-agnostic methods aim to identify thresholds of careless respondent indices, making classifications without reliance on measurement models. Although these two streams share some common assumptions for CR, these methods have been developed in parallel and have not been directly compared. To systematically compare these two streams of approaches, we conducted three large-scale simulation studies manipulating the number of items, proportion of reverse-coded items, contamination rates, carelessness types, presence of model misspecification, item properties, as well as sample sizes. To balance generalizability and realism, conditions for the first two simulations were inspired by real instruments from personality and health outcome measures, while the third simulation study explicitly manipulated item properties to de-confound naturally occurring variability induced under the first two studies. Generally, results indicate that model-based approaches are more specificity-focused and should be preferred under lower contamination rates and fewer items, while model-agnostic approaches are sensitivity-focused and outperform model-based approaches under other contamination rates and more CR types provided there are a sufficient number of items.
Related
- Comparison of two approaches to weighted topological entropy
- A Structural Deconstruction of All Known Spectral, Geometric, Motivic and Tropical Approaches to the Riemann Hypothesis: A Comprehensive No-Go Map of 25 Paradigms
- Efficient Closure Learning in Dynamical Systems: A Comparison of Data-Assimilated and Embedded Approaches
- Skewness and Kurtosis for Heavy-Tailed Distributions: A Comparison of Mean Absolute Deviation and L-Moment Approaches for Pareto and Extreme-Value Families
- Estimating heterogeneous treatment effects from randomised trials: a comparison of the risk modelling and treatment effect modelling approaches
BibTeX
@misc{li2026unsupervised,
title = {{Unsupervised Careless Responding Detection: A Comprehensive Comparison between Model-based and Model-Agnostic Approaches}},
author = {Wenshuo Li and Carl F. Falk},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/ahk4t_v2},
url = {https://osf.io/preprints/psyarxiv/ahk4t_v2/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Unsupervised Careless Responding Detection: A Comprehensive Comparison between Model-based and Model-Agnostic Approaches
AU - Wenshuo Li
AU - Carl F. Falk
PY - 2026
DA - 2026/09/10
PB - PsyArXiv
DO - 10.31234/osf.io/ahk4t_v2
UR - https://osf.io/preprints/psyarxiv/ahk4t_v2/
AB - Careless responding (CR) is a major concern for data quality in the social sciences. It is common for researchers to attempt to detect and filter out CR at the respondent level. Though various choices are available, detection methods relying solely on response data have received much attention since they do not require the collection of additional information. Along these lines, model-based approaches to CR detection rely on specifying distinct measurement models for careful and careless respondents in a mixture model. In contrast, model-agnostic methods aim to identify thresholds of careless respondent indices, making classifications without reliance on measurement models. Although these two streams share some common assumptions for CR, these methods have been developed in parallel and have not been directly compared. To systematically compare these two streams of approaches, we conducted three large-scale simulation studies manipulating the number of items, proportion of reverse-coded items, contamination rates, carelessness types, presence of model misspecification, item properties, as well as sample sizes. To balance generalizability and realism, conditions for the first two simulations were inspired by real instruments from personality and health outcome measures, while the third simulation study explicitly manipulated item properties to de-confound naturally occurring variability induced under the first two studies. Generally, results indicate that model-based approaches are more specificity-focused and should be preferred under lower contamination rates and fewer items, while model-agnostic approaches are sensitivity-focused and outperform model-based approaches under other contamination rates and more CR types provided there are a sufficient number of items.
ER -