Methodological Flexibility in the Iowa Gambling Task Undermines Interpretability: A Meta-method Review
Abstract
Methodological flexibility–researcher degrees of freedom in procedure and scoring–can contribute to poor measurement and low replicability of findings. To quantify methodological flexibility in the Iowa Gambling Task (IGT), a widely used measure of decision-making in clinical psychology, we conducted a meta-method review of a random sample of 100 studies. We found prevalent under-reporting and wide-spread modifications of procedural details. We identified 244 distinct scores used to quantify IGT outcomes, 177 of which appeared only once in our sample. To quantify the potential impact of this flexibility, we performed a multiverse analysis using 205 scores on an open IGT dataset. Correlations among these ranged from rs = -0.942 to 0.998 with a median r = 0.022, suggesting that different scoring strategies can produce extremely different study outcomes. Given these findings, without substantive information about which specific procedural and scoring choices were employed, results from the IGT are rendered uninterpretable. Supplementary Materials, Data, and Code are available at github.com/anniria/igt-methodological-flexibility.
Related
- Climate education's credibility trap: When alarm undermines action
- When Standards Protect and Pressure Undermines: A Bayesian and Person-Centred Analysis of Perfectionism, Self-Handicapping, Motivation, and Academic Well-Being in Gifted Adolescents.
- Heterogeneity in CT-Based Skeletal Muscle Mass Assessment and Sarcopenia Diagnosis Undermines Comparability in Hepatopancreatobiliary Surgery: A Scoping Review
- Association of Borderline Personality Traits and Impulsivity with Problem Gambling Severity in Gambling Disorder
- Financially focused self-concept and problem lottery gambling: the chain mediating role of gambling-related luck beliefs and lottery gambling passion
BibTeX
@misc{kulpmann2026methodological,
title = {{Methodological Flexibility in the Iowa Gambling Task Undermines Interpretability: A Meta-method Review}},
author = {Annika I. Külpmann and Jan-Paul Ries and Ian Hussey and Malte Elson},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/4g3vr_v2},
url = {https://osf.io/preprints/psyarxiv/4g3vr_v2/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Methodological Flexibility in the Iowa Gambling Task Undermines Interpretability: A Meta-method Review
AU - Annika I. Külpmann
AU - Jan-Paul Ries
AU - Ian Hussey
AU - Malte Elson
PY - 2026
DA - 2026/10/29
PB - PsyArXiv
DO - 10.31234/osf.io/4g3vr_v2
UR - https://osf.io/preprints/psyarxiv/4g3vr_v2/
AB - Methodological flexibility–researcher degrees of freedom in procedure and scoring–can contribute to poor measurement and low replicability of findings. To quantify methodological flexibility in the Iowa Gambling Task (IGT), a widely used measure of decision-making in clinical psychology, we conducted a meta-method review of a random sample of 100 studies. We found prevalent under-reporting and wide-spread modifications of procedural details. We identified 244 distinct scores used to quantify IGT outcomes, 177 of which appeared only once in our sample. To quantify the potential impact of this flexibility, we performed a multiverse analysis using 205 scores on an open IGT dataset. Correlations among these ranged from rs = -0.942 to 0.998 with a median r = 0.022, suggesting that different scoring strategies can produce extremely different study outcomes. Given these findings, without substantive information about which specific procedural and scoring choices were employed, results from the IGT are rendered uninterpretable. Supplementary Materials, Data, and Code are available at github.com/anniria/igt-methodological-flexibility.
ER -