What Pilot Studies Can (and Cannot) Do for Validity in Psychological Research
Abstract
Recent debates about psychology’s credibility have renewed attention to validity threats. We argue that pilot studies can function as methodological due diligence by improving early decisions about design, measurement, manipulation, sampling, and analysis prior to a main study. We examine four types of validity: construct, internal, external, and statistical-conclusion validity. Across these domains, we show how piloting can help researchers assess whether materials and procedures function as intended, identify potential validity threats, and provide key design parameters for planning and model choice. However, we also caution that piloting can harm validity when treated as a consequence-free space in which design or analytic choices are made without disclosing the pilot evidence behind them, or when small, noisy pilots are used to justify overconfident inferences. We therefore argue that when pilots are used to justify substantive design or analytic choices, they should be treated as part of the evidential record and reported transparently. Greater transparency would enable readers to evaluate how pilot evidence shaped the final design of a study and, consequently, better assess the validity of its findings.
Related
- Measurement reliability, construct validity, and transparent reporting in original and replication psychological research
- Designing multi‐site studies for external validity: Site selection via synthetic purposive sampling
- Kernel-based External Validity of Experimental Studies
- Addressing threats to validity in large dataset research
- Piecewise modeling for longitudinal psychological research: a tutorial using R
BibTeX
@misc{seetahul2026what,
title = {{What Pilot Studies Can (and Cannot) Do for Validity in Psychological Research}},
author = {Yashvin Seetahul and Mahmoud Elsherif and Caroline Zygar-Hoffmann and Lukas Wallrich and Priya Silverstein and Bjørn Sætrevik and Ilse L. Pit and Hannah Loenneker and Neele Henriette Heiser and Isaac Handley-Miner},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/t968e_v4},
url = {https://osf.io/preprints/psyarxiv/t968e_v4/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - What Pilot Studies Can (and Cannot) Do for Validity in Psychological Research
AU - Yashvin Seetahul
AU - Mahmoud Elsherif
AU - Caroline Zygar-Hoffmann
AU - Lukas Wallrich
AU - Priya Silverstein
AU - Bjørn Sætrevik
AU - Ilse L. Pit
AU - Hannah Loenneker
AU - Neele Henriette Heiser
AU - Isaac Handley-Miner
PY - 2026
DA - 2026/09/02
PB - PsyArXiv
DO - 10.31234/osf.io/t968e_v4
UR - https://osf.io/preprints/psyarxiv/t968e_v4/
AB - Recent debates about psychology’s credibility have renewed attention to validity threats. We argue that pilot studies can function as methodological due diligence by improving early decisions about design, measurement, manipulation, sampling, and analysis prior to a main study. We examine four types of validity: construct, internal, external, and statistical-conclusion validity. Across these domains, we show how piloting can help researchers assess whether materials and procedures function as intended, identify potential validity threats, and provide key design parameters for planning and model choice. However, we also caution that piloting can harm validity when treated as a consequence-free space in which design or analytic choices are made without disclosing the pilot evidence behind them, or when small, noisy pilots are used to justify overconfident inferences. We therefore argue that when pilots are used to justify substantive design or analytic choices, they should be treated as part of the evidential record and reported transparently. Greater transparency would enable readers to evaluate how pilot evidence shaped the final design of a study and, consequently, better assess the validity of its findings.
ER -