Skip to content
DraconiticResearch
Search Sources API
197,768 studies31 sourcesUpdated 16 min ago
Peer-reviewedCC BYBiometrical Journal

When to Adjust for Multiple Testing: A Unifying Guiding Principle

  • Sabine Hoffmann1
  • Simon Lemster2
  • Gary Collins3
  • Alexander Hapfelmeier4,5
  • Georg Heinze6
  • Andreas J. Mayr7
  • Matthias C. Schmid8
  • Juliane C. Wilcke2
  • Anne‐Laure Boulesteix2,9
9 affiliations
  1. 1Department of Statistics LMU Munich Munich Germany
  2. 2Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Medizin LMU Munich Munich Germany
  3. 3Department of Applied Health Sciences, School of Health Sciences, College of Medicine and Health University of Birmingham Birmingham UK
  4. 4Institute of AI and Informatics in Medicine, TUM School of Medicine and Health Technical University of Munich Munich Germany
  5. 5Institute of General Practice and Health Services Research, TUM School of Medicine and Health Technical University of Munich Munich Germany
  6. 6Institute of Clinical Biometrics, Center for Medical Data Science Medical University of Vienna Vienna Austria
  7. 7Institute for Medical Biometry and Statistics University of Marburg Marburg Germany
  8. 8Institute for Medical Biometry, Informatics, and Epidemiology University Hospital Bonn Bonn Germany
  9. 9Munich Center for Machine Learning (MCML) Munich Germany
Open at sourceDOICite

Abstract

Most original articles published in the medical literature report the results of multiple statistical tests. In a few simple cases, there is agreement on whether to adjust for the number of performed tests. For many cases encountered in practice, however, this is less clear, and the recommendations in the literature are contradictory, along different dimensions, or otherwise confusing. This lack of clear guidance may impair the conduct and interpretation of analyses, and encourage questionable research practices, ultimately jeopardizing the credibility of medical research. In this article, we refine, illustrate, and discuss a unifying guiding principle to assist both statisticians and applied researchers in deciding whether to adjust for multiple testing and, if so, over which set of tests. The principle is that multiple testing should be adjusted for if and only if authors, when reporting and interpreting their findings, put more emphasis on results of one or several of the tests because of their small p-value(s). We relate this principle to previously proposed rules and show how it can guide and clarify the choice of adjustment strategies in three complex multiple testing settings.

Related

  1. Quantum f-divergences from retrodictive tests: a unifying classical variational principle
    Francesco Buscemi
    arXivPreprintarXiv licenceFull textMathematical physicsQuantum physics & quantum informationarXiv
  2. Efficient Optimization on the Euclidean Sphere: Riemannian Gradient Alignment as a Unifying Principle
    Puqian Wang, Nikos Zarifis, Jelena Diakonikolas
    arXivPreprintarXiv licenceFull textOptimisation & controlarXiv
  3. A Model of Multiple Hypothesis Testing
    Davide Viviano, Kaspar Wüthrich, Paul Niehaus
    The Review of Economic StudiesPeer-reviewedLicence not statedFull text32 citedStatisticsOpenAlex
  4. Sample-Efficient Multiple Testing with Adaptive Data Collection
    Zhanran Lin, Wanteng Ma, Zhimei Ren +1
    arXivPreprintarXiv licenceFull textStatisticsarXiv
  5. How Many Significant LISA?: A multiple testing correction
    Giuseppe Arbia, Francesca Cardile, Vincenzo Nardelli +1
    arXivPreprintCC BY-NC-ND 4.0Full textStatisticsarXiv

Cite

BibTeXRIS
BibTeX
@article{hoffmann2026when,
  title = {{When to Adjust for Multiple Testing: A Unifying Guiding Principle}},
  author = {Sabine Hoffmann and Simon Lemster and Gary Collins and Alexander Hapfelmeier and Georg Heinze and Andreas J. Mayr and Matthias C. Schmid and Juliane C. Wilcke and Anne‐Laure Boulesteix},
  journal = {Biometrical Journal},
  year = {2026},
  month = jul,
  doi = {10.1002/bimj.70148},
  url = {https://doi.org/10.1002/bimj.70148},
  note = {CC BY},
}
RIS
TY  - JOUR
TI  - When to Adjust for Multiple Testing: A Unifying Guiding Principle
AU  - Sabine Hoffmann
AU  - Simon Lemster
AU  - Gary Collins
AU  - Alexander Hapfelmeier
AU  - Georg Heinze
AU  - Andreas J. Mayr
AU  - Matthias C. Schmid
AU  - Juliane C. Wilcke
AU  - Anne‐Laure Boulesteix
PY  - 2026
DA  - 2026/07/02
JO  - Biometrical Journal
DO  - 10.1002/bimj.70148
UR  - https://doi.org/10.1002/bimj.70148
AB  - Most original articles published in the medical literature report the results of multiple statistical tests. In a few simple cases, there is agreement on whether to adjust for the number of performed tests. For many cases encountered in practice, however, this is less clear, and the recommendations in the literature are contradictory, along different dimensions, or otherwise confusing. This lack of clear guidance may impair the conduct and interpretation of analyses, and encourage questionable research practices, ultimately jeopardizing the credibility of medical research. In this article, we refine, illustrate, and discuss a unifying guiding principle to assist both statisticians and applied researchers in deciding whether to adjust for multiple testing and, if so, over which set of tests. The principle is that multiple testing should be adjusted for if and only if authors, when reporting and interpreting their findings, put more emphasis on results of one or several of the tests because of their small p-value(s). We relate this principle to previously proposed rules and show how it can guide and clarify the choice of adjustment strategies in three complex multiple testing settings.
SN  - 0323-3847
SN  - 1521-4036
ER  -