Evaluating a candidate predictor beyond statistical significance: Atmospheric pressure and psychiatric day-care attendance as a worked example
- 1Department of Psychiatry, Sodegaura Satsukidai Hospital, Sodegaura, Chiba, Japan
Abstract
In observational research, statistical detectability, robustness to adjustment, and predictive utility are distinct properties of a candidate predictor, yet they are often conflated. A statistically significant association may weaken after accounting for co-varying factors and may add little to out-of-sample prediction. We illustrate this distinction using atmospheric pressure and daily attendance at a Japanese psychiatric day-care as a worked example. Using four consecutive fiscal years of facility-level daily attendance, we evaluated same-day atmospheric pressure as a pre-specified candidate predictor in three stages: statistical detectability, robustness to adjustment for co-varying meteorological factors and seasonality, and incremental out-of-sample predictive utility beyond institutional scheduling variables and recent attendance history. Same-day pressure was positively associated with attendance in all four years (incidence rate ratios 1.003–1.007 per hPa). However, effect-estimate stability varied across years, with marked attenuation in FY2022, moderate attenuation after precipitation adjustment in FY2023, and specification-sensitive precision in FY2024, whereas FY2025 was comparatively stable. In contrast, the structural baseline incorporating scheduling variables and recent attendance history provided strong out-of-sample prediction, explaining 55%–69% of the variance across years. Adding atmospheric pressure produced only small and unstable changes in the primary expanding-window cross-validated performance (mean ΔR2 ≈ +0.005), with increments varying in sign across years. Thus, a candidate predictor that was repeatedly statistically detectable showed limited stability of effect magnitude or precision under reasonable adjustment and little incremental predictive utility beyond a strong structural baseline. In this highly schedule-driven, single-facility setting, the weather signal examined contributed little incremental information for out-of-sample prediction of aggregate attendance. More broadly, this worked example illustrates why candidate predictors should be evaluated separately for detectability, robustness of effect estimates, and out-of-sample predictive utility before statistically significant associations are interpreted as meaningful or actionable.
Related
- Statistical Downscaling of Daily Precipitation from ERA5 Atmospheric Fields to IMERG over West Africa
- A comparison of data-driven and physics-informed statistical learning models for atmospheric remote sensing applications
- Evaluating LLMs for Translating Legacy Fortran Atmospheric Physics to GPU-Accelerated JAX
- Deglacial Laminated Diatom Mats (Ethmodiscus rex), in the Western Philippine Sea and their palaeoceanographic significance
- Explaining patterns learned by neural networks for detection of atmospheric surface fronts in gridded atmospheric data
BibTeX
@article{okuma2026evaluating,
title = {{Evaluating a candidate predictor beyond statistical significance: Atmospheric pressure and psychiatric day-care attendance as a worked example}},
author = {Takahiro Okuma and Minoru Ishige and Shuichi Kikuchi},
journal = {PLOS One},
year = {2026},
month = nov,
doi = {10.1371/journal.pone.0352385},
url = {https://doi.org/10.1371/journal.pone.0352385},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Evaluating a candidate predictor beyond statistical significance: Atmospheric pressure and psychiatric day-care attendance as a worked example
AU - Takahiro Okuma
AU - Minoru Ishige
AU - Shuichi Kikuchi
PY - 2026
DA - 2026/11/06
JO - PLOS One
DO - 10.1371/journal.pone.0352385
UR - https://doi.org/10.1371/journal.pone.0352385
AB - In observational research, statistical detectability, robustness to adjustment, and predictive utility are distinct properties of a candidate predictor, yet they are often conflated. A statistically significant association may weaken after accounting for co-varying factors and may add little to out-of-sample prediction. We illustrate this distinction using atmospheric pressure and daily attendance at a Japanese psychiatric day-care as a worked example. Using four consecutive fiscal years of facility-level daily attendance, we evaluated same-day atmospheric pressure as a pre-specified candidate predictor in three stages: statistical detectability, robustness to adjustment for co-varying meteorological factors and seasonality, and incremental out-of-sample predictive utility beyond institutional scheduling variables and recent attendance history. Same-day pressure was positively associated with attendance in all four years (incidence rate ratios 1.003–1.007 per hPa). However, effect-estimate stability varied across years, with marked attenuation in FY2022, moderate attenuation after precipitation adjustment in FY2023, and specification-sensitive precision in FY2024, whereas FY2025 was comparatively stable. In contrast, the structural baseline incorporating scheduling variables and recent attendance history provided strong out-of-sample prediction, explaining 55%–69% of the variance across years. Adding atmospheric pressure produced only small and unstable changes in the primary expanding-window cross-validated performance (mean ΔR2 ≈ +0.005), with increments varying in sign across years. Thus, a candidate predictor that was repeatedly statistically detectable showed limited stability of effect magnitude or precision under reasonable adjustment and little incremental predictive utility beyond a strong structural baseline. In this highly schedule-driven, single-facility setting, the weather signal examined contributed little incremental information for out-of-sample prediction of aggregate attendance. More broadly, this worked example illustrates why candidate predictors should be evaluated separately for detectability, robustness of effect estimates, and out-of-sample predictive utility before statistically significant associations are interpreted as meaningful or actionable.
SN - 1932-6203
ER -