{
  "abstract": "Recent debates about psychology’s credibility have renewed attention to validity threats. We argue that pilot studies can function as methodological due diligence by improving early decisions about design, measurement, manipulation, sampling, and analysis prior to a main study. We examine four types of validity: construct, internal, external, and statistical-conclusion validity. Across these domains, we show how piloting can help researchers assess whether materials and procedures function as intended, identify potential validity threats, and provide key design parameters for planning and model choice. However, we also caution that piloting can harm validity when treated as a consequence-free space in which design or analytic choices are made without disclosing the pilot evidence behind them, or when small, noisy pilots are used to justify overconfident inferences. We therefore argue that when pilots are used to justify substantive design or analytic choices, they should be treated as part of the evidential record and reported transparently. Greater transparency would enable readers to evaluate how pilot evidence shaped the final design of a study and, consequently, better assess the validity of its findings.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": null,
  "author_total": 10,
  "authors": [
    {
      "name": "Yashvin Seetahul"
    },
    {
      "name": "Mahmoud Elsherif"
    },
    {
      "name": "Caroline Zygar-Hoffmann"
    },
    {
      "name": "Lukas Wallrich"
    },
    {
      "name": "Priya Silverstein"
    },
    {
      "name": "Bjørn Sætrevik"
    },
    {
      "name": "Ilse L. Pit"
    },
    {
      "name": "Hannah Loenneker"
    },
    {
      "name": "Neele Henriette Heiser"
    },
    {
      "name": "Isaac Handley-Miner"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/dxpd8j08nf96p.bib",
  "cited_by": 0,
  "doi": "10.31234/osf.io/t968e_v4",
  "fulltext_url": null,
  "id": "dxpd8j08nf96p",
  "issns": [],
  "landing_url": "https://osf.io/preprints/psyarxiv/t968e_v4/",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "bepress",
      "Experimental Design and Sample Surveys"
    ],
    [
      "bepress",
      "Psychometrics"
    ],
    [
      "bepress",
      "Quantitative Methods"
    ],
    [
      "bepress",
      "Quantitative Psychology"
    ],
    [
      "bepress",
      "Social and Behavioral Sciences"
    ],
    [
      "bepress",
      "Social and Personality Psychology"
    ],
    [
      "bepress",
      "Statistical Methods"
    ],
    [
      "bepress",
      "Theory and Philosophy of Science"
    ],
    [
      "openalex",
      "3205|Psychology Research and Bibliometrics"
    ]
  ],
  "openalex": "W7223059352",
  "pdf_url": "https://osf.io/t968e/download",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-02",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/dxpd8j08nf96p.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:19:59Z",
      "native_id": "t968e",
      "source": "osf",
      "url": "https://osf.io/preprints/psyarxiv/t968e_v3/"
    }
  ],
  "subjects": [
    {
      "field": "mathematics",
      "origin": "bepress:Statistical Methods",
      "subfield": "statistics"
    }
  ],
  "title": "What Pilot Studies Can (and Cannot) Do for Validity in Psychological Research",
  "title_text": "What Pilot Studies Can (and Cannot) Do for Validity in Psychological Research",
  "type": "preprint",
  "updated_at": "2026-10-30T12:26:06Z",
  "url": "https://research.draconitic.com/paper/dxpd8j08nf96p",
  "venue": "PsyArXiv",
  "versions": []
}