Too-good-to-be-true: explaining large effects from common exposures in epidemiologic studies
Abstract
Background: Epidemiologic studies reporting large effects attract high attention in the scientific community compared to studies with smaller effects or null findings. Claims about strong effects of everyday exposures on common diseases also gain interest in non-scientific (social) media, even when they appear implausible (“too-good-go-be-true”) according to common sense. We assessed whether such effects are based on sound methodology.
Methods: We screened a major news magazine (Spiegel), two newspapers (Stuttgarter und Süddeutsche Zeitung) and the section “most read” of the leading German medical journal (Deutsches Ärzteblatt) for reports on epidemiologic studies with implausible effects published from July to December 2023. We examined the underlying publications regarding study population, methods and reporting to identify issues that potentially influencing the results.
Results: Five of 47 studies featured in the news media and four of 18 studies in the medical journal reported implausible effects, three studies were covered by two media. Reported findings included a 30% reduction in cardiovascular risk from low alcohol intake, up to 80% lower cancer risk and 67% lower heart failure risk from short intensive physical activity, up to 24 additional years of life associated with a healthy lifestyle, and a 50% increase in dementia risk from specific sugars. We identified underlying issues including inappropriate reference groups, arbitrary exposure cut-offs, selection and survivor bias, inadequate exposure measurement, extrapolation, selective reporting, and conflicts of interest.
Conclusions: Methodological flaws contribute to effects that seem “too-good-to-be-true” in epidemiologic studies. The majority could have been identified without advanced statistical expertise. Our findings underscore an urgent need for better scrutiny of “surprising” findings to prevent dissemination of misleading or even adverse public health information.
Related
- Three Case Studies of Two-stage MCMC for Fast Bayesian Inference of Large Spatio-temporal Data
- A Hilbert-Valued Functional Decomposition Framework for Explaining Time-Dependent Outputs
- Explaining Heterogeneous Treatment Effects with Causal Forests and the Hoeffding Functional Decomposition
- Incremental effects for continuous exposures
- Explaining the Saliency Map Sparsity of Adversarially-Trained Neural Networks
BibTeX
@article{stolpe2026toogoodtobetrue,
title = {{Too-good-to-be-true: explaining large effects from common exposures in epidemiologic studies}},
author = {S Stolpe and HC Vollmar and B Kowall},
journal = {The European Journal of Public Health},
year = {2026},
month = oct,
doi = {10.1093/eurpub/ckag169.535},
url = {https://doi.org/10.1093/eurpub/ckag169.535},
note = {CC BY-NC 4.0},
}
RIS
TY - JOUR
TI - Too-good-to-be-true: explaining large effects from common exposures in epidemiologic studies
AU - S Stolpe
AU - HC Vollmar
AU - B Kowall
PY - 2026
DA - 2026/10/01
JO - The European Journal of Public Health
DO - 10.1093/eurpub/ckag169.535
UR - https://doi.org/10.1093/eurpub/ckag169.535
AB - Background: Epidemiologic studies reporting large effects attract high attention in the scientific community compared to studies with smaller effects or null findings. Claims about strong effects of everyday exposures on common diseases also gain interest in non-scientific (social) media, even when they appear implausible (“too-good-go-be-true”) according to common sense. We assessed whether such effects are based on sound methodology. Methods: We screened a major news magazine (Spiegel), two newspapers (Stuttgarter und Süddeutsche Zeitung) and the section “most read” of the leading German medical journal (Deutsches Ärzteblatt) for reports on epidemiologic studies with implausible effects published from July to December 2023. We examined the underlying publications regarding study population, methods and reporting to identify issues that potentially influencing the results. Results: Five of 47 studies featured in the news media and four of 18 studies in the medical journal reported implausible effects, three studies were covered by two media. Reported findings included a 30% reduction in cardiovascular risk from low alcohol intake, up to 80% lower cancer risk and 67% lower heart failure risk from short intensive physical activity, up to 24 additional years of life associated with a healthy lifestyle, and a 50% increase in dementia risk from specific sugars. We identified underlying issues including inappropriate reference groups, arbitrary exposure cut-offs, selection and survivor bias, inadequate exposure measurement, extrapolation, selective reporting, and conflicts of interest. Conclusions: Methodological flaws contribute to effects that seem “too-good-to-be-true” in epidemiologic studies. The majority could have been identified without advanced statistical expertise. Our findings underscore an urgent need for better scrutiny of “surprising” findings to prevent dissemination of misleading or even adverse public health information.
SN - 1101-1262
SN - 1464-360X
ER -