Fully specified Bayes factors for hypothesis testing and sensitivity analysis in process tracing
Abstract
In process tracing, researchers ask how strongly their evidence favors their explanation, the working theory, over a rival. Fairfield and Charman (2022) compare the two theories with a Bayes factor whose probabilities are set by hand, which Zaks (2021) argues lets researchers overstate findings. We derive those probabilities from a fully specified model of the evidence a researcher could have examined, given which observations support each theory. The working theory says most of the evidence supports it. The rival says no more than half does. The model gives two Bayes factors, neither designed to favor the working theory. The second grants the rival more benefit of the doubt and is provably the smallest value the rival's claim allows. Researchers can report how much observation bias, re-coding, or a smoking-gun weight would change a conclusion. In six published studies we reanalyze, both Bayes factors exceed 20 for two studies and not for the other four.
Related
- A Model of Multiple Hypothesis Testing
- Principal-Agent Hypothesis Testing
- PDE-Based Bounds on Optimal Exponents in Active Simple Hypothesis Testing
- Binary Quantum Hypothesis Testing with a Rejection Option: The Exact Error-Exponent Trade-off
- Principal-agent hypothesis testing under adaptive experimentation
BibTeX
@misc{lopez2026fully,
title = {{Fully specified Bayes factors for hypothesis testing and sensitivity analysis in process tracing}},
author = {Matias López and Jake Bowers and Daniel Gajardo Cooper},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2606.16683},
eprint = {2606.16683},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.16683},
}
RIS
TY - UNPB
TI - Fully specified Bayes factors for hypothesis testing and sensitivity analysis in process tracing
AU - Matias López
AU - Jake Bowers
AU - Daniel Gajardo Cooper
PY - 2026
DA - 2026/08/31
PB - arXiv
DO - 10.48550/arxiv.2606.16683
UR - https://arxiv.org/abs/2606.16683
AB - In process tracing, researchers ask how strongly their evidence favors their explanation, the working theory, over a rival. Fairfield and Charman (2022) compare the two theories with a Bayes factor whose probabilities are set by hand, which Zaks (2021) argues lets researchers overstate findings. We derive those probabilities from a fully specified model of the evidence a researcher could have examined, given which observations support each theory. The working theory says most of the evidence supports it. The rival says no more than half does. The model gives two Bayes factors, neither designed to favor the working theory. The second grants the rival more benefit of the doubt and is provably the smallest value the rival's claim allows. Researchers can report how much observation bias, re-coding, or a smoking-gun weight would change a conclusion. In six published studies we reanalyze, both Bayes factors exceed 20 for two studies and not for the other four.
ER -