Analyzing Binary Judgments: An Empirical and Simulation-Based Comparison of Aggregated and Trial-Level Approaches for Hypothesis Testing
Abstract
Binary judgments are widely used in psychological research to investigate cognitive processes and decision making. Statistical approaches for analyzing such data differ in whether responses are aggregated (e.g., analysis of variance, ANOVA) or modeled at the trial level (e.g., generalized linear mixed models, GLMMs), and in whether results are interpreted as choice proportions or in terms of signal detection theory (SDT) parameters for response bias and discrimination ability. We examined how these approaches differ in specification, interpretation, and results. First, we reanalyzed 20 openly available datasets on the illusory truth effect (212,555 observations), comparing by-subjects (F1) and by-items (F2) ANOVAs of choice proportions, their combination via min-F', a two-step SDT approach (ANOVA of individual SDT estimates), and seven GLMMs with random-effects structures of varying complexity. Second, a Monte Carlo simulation assessed Type I error rates, power, and parameter recovery. GLMMs with random slopes for both participants and items controlled Type I error rates best but often resulted in singular fits. GLMMs with random slopes only for participants were favored by AIC and BIC, but inflated Type I error rates for interactions by ignoring item-specific variability in the repetition effect. Random-intercept-only GLMMs, F1 and F2 ANOVA, and two-step SDT were too liberal for some effects, whereas min-F' was overly conservative. Unlike aggregated analyses, GLMMs account for both participant and item variability and estimate SDT parameters with less bias. We therefore recommend GLMMs with theoretically justified random-effects structure, simplified only if necessary and with the resulting risk of Type I errors for confirmatory hypothesis testing.
Related
- A perspective note on likelihood approximation and inference for complex simulation models using a chain of aggregated normalizing flows
- Statistical methods for analyzing change over time of continuous outcomes in observational health studies: a scoping review of suggested approaches
- A Structural Deconstruction of All Known Spectral, Geometric, Motivic and Tropical Approaches to the Riemann Hypothesis: A Comprehensive No-Go Map of 25 Paradigms
- Anytime-valid simulation-based hypothesis testing
- Comparison of two approaches to weighted topological entropy
BibTeX
@misc{aktepe2026analyzing,
title = {{Analyzing Binary Judgments: An Empirical and Simulation-Based Comparison of Aggregated and Trial-Level Approaches for Hypothesis Testing}},
author = {Semih C. Aktepe and Daniel W. Heck},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/xn397_v2},
url = {https://osf.io/preprints/psyarxiv/xn397_v2/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Analyzing Binary Judgments: An Empirical and Simulation-Based Comparison of Aggregated and Trial-Level Approaches for Hypothesis Testing
AU - Semih C. Aktepe
AU - Daniel W. Heck
PY - 2026
DA - 2026/10/29
PB - PsyArXiv
DO - 10.31234/osf.io/xn397_v2
UR - https://osf.io/preprints/psyarxiv/xn397_v2/
AB - Binary judgments are widely used in psychological research to investigate cognitive processes and decision making. Statistical approaches for analyzing such data differ in whether responses are aggregated (e.g., analysis of variance, ANOVA) or modeled at the trial level (e.g., generalized linear mixed models, GLMMs), and in whether results are interpreted as choice proportions or in terms of signal detection theory (SDT) parameters for response bias and discrimination ability. We examined how these approaches differ in specification, interpretation, and results. First, we reanalyzed 20 openly available datasets on the illusory truth effect (212,555 observations), comparing by-subjects (F1) and by-items (F2) ANOVAs of choice proportions, their combination via min-F', a two-step SDT approach (ANOVA of individual SDT estimates), and seven GLMMs with random-effects structures of varying complexity. Second, a Monte Carlo simulation assessed Type I error rates, power, and parameter recovery. GLMMs with random slopes for both participants and items controlled Type I error rates best but often resulted in singular fits. GLMMs with random slopes only for participants were favored by AIC and BIC, but inflated Type I error rates for interactions by ignoring item-specific variability in the repetition effect. Random-intercept-only GLMMs, F1 and F2 ANOVA, and two-step SDT were too liberal for some effects, whereas min-F' was overly conservative. Unlike aggregated analyses, GLMMs account for both participant and item variability and estimate SDT parameters with less bias. We therefore recommend GLMMs with theoretically justified random-effects structure, simplified only if necessary and with the resulting risk of Type I errors for confirmatory hypothesis testing.
ER -