Transparency Is Necessary but Not Sufficient for Trust: Information Leakage and Procedural Overfitting in Supervised Machine Learning for Psychology
Abstract
Min et al. (2026) provide a timely checklist for improving the transparency and trust of supervised machine learning (ML) research in psychology. I strongly endorse this goal and recognize that their focus on retrospective reporting is both valuable and the appropriate scope for an article. However, I suggest that two related issues warrant greater emphasis in their framework: information leakage and procedural overfitting. Both can produce optimistic estimates of model performance despite apparently transparent reporting. I propose two extensions to the checklist: reporting requirements that make information leakage auditable and prospective specification of key analytical decisions through preregistration. I argue that these additions would further strengthen the ability of readers and reviewers to evaluate the trustworthiness of supervised ML analyses in psychology. Moreover, comprehensive guidelines that do not sufficiently emphasize these issues may create unwarranted confidence that threats to valid model evaluation have been ruled out, allowing information leakage or procedural overfitting to remain undetected.
Related
- Bellman-sufficient Information Complexity
- Task-Sufficient Contraction: Source Selection for Machine Information Interfaces
- Scalable partial information decomposition for symptom networks via supervised embeddings
- Benign Overfitting Property of the Minimum ℓq -norm Interpolant Estimator in Regression Problem via Feature Space Decomposition
- Benign Overfitting for General Norms and Distributions
BibTeX
@misc{lavellehill2026transparency,
title = {{Transparency Is Necessary but Not Sufficient for Trust: Information Leakage and Procedural Overfitting in Supervised Machine Learning for Psychology}},
author = {Rosa Lavelle-Hill},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/cw348_v1},
url = {https://osf.io/preprints/psyarxiv/cw348_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Transparency Is Necessary but Not Sufficient for Trust: Information Leakage and Procedural Overfitting in Supervised Machine Learning for Psychology
AU - Rosa Lavelle-Hill
PY - 2026
DA - 2026/10/22
PB - PsyArXiv
DO - 10.31234/osf.io/cw348_v1
UR - https://osf.io/preprints/psyarxiv/cw348_v1/
AB - Min et al. (2026) provide a timely checklist for improving the transparency and trust of supervised machine learning (ML) research in psychology. I strongly endorse this goal and recognize that their focus on retrospective reporting is both valuable and the appropriate scope for an article. However, I suggest that two related issues warrant greater emphasis in their framework: information leakage and procedural overfitting. Both can produce optimistic estimates of model performance despite apparently transparent reporting. I propose two extensions to the checklist: reporting requirements that make information leakage auditable and prospective specification of key analytical decisions through preregistration. I argue that these additions would further strengthen the ability of readers and reviewers to evaluate the trustworthiness of supervised ML analyses in psychology. Moreover, comprehensive guidelines that do not sufficiently emphasize these issues may create unwarranted confidence that threats to valid model evaluation have been ruled out, allowing information leakage or procedural overfitting to remain undetected.
ER -