Validation of Space Weather Forecasting at Leonid Space
Abstract
We validate Leonid Space's space weather forecasts through comprehensive cross-validation of retroforecasts scored against the entire F10.7 and Ap historical record. We built a unified forecast model that jointly forecasts F10.7 and Ap on all timescales from 3 hours up to multi-decade chains of future solar cycles. This was used to generate an archive of historical 15-year retroforecasts, after performing a leave-one-out fit of the model on held-out data and conditioning only on historical data available at the time. The accuracy of this archive was scored against measured space weather indices at all lookahead horizons from 1 day to 15 years, using the continuous ranked probability score (CRPS) as the primary quality metric. All other publicly available F10.7 and Ap forecast archives were scored using the same approach, including NASA's MSAFE and Schatten forecasts, SWPC's 27-day, 45-day, and Solar Cycle predictions, SWPC/USAF's RSGA, SIDC's Ursigram, CLS, Athens, KSO, BGS, SEPC's 3-day and 27-day forecasts, AFRL/NSO's SIFT, and BoM. Taken together, these 15 flux forecasts and 12 Ap forecasts compared on common lookahead metrics make up the most comprehensive cross-comparison of space weather forecasts we are aware of. Over a common forecast interval since 2000, Leonid had the lowest CRPS prediction error of any F10.7 or Ap prediction model: strictly lowest for Ap at all lookaheads and for F10.7 up to 5 years, and parity with MSAFE after 5 years. Much of this performance comes from our well-calibrated uncertainty bands, and we urge space weather forecasters to include calibrated uncertainties with their predictions and to issue retroforecast archives that can be used to rigorously evaluate their generating models.
Related
- Calculation of distant retrograde orbits and their use in space weather forecasting
- Multiviewpoint Tracking of a Coronal Mass Ejection Driven Shock Propagating to Mars: Implications for Space Weather Forecasting
- Preliminary Goniopolarimetric Simulations for Space Weather Radio (SWARA) Polarimeter
- PISCES: Physics-Informed Solar-wind Convolutional autoEncoder for Space-weather Anomaly Detection and Early Warning
- Major Space Weather Risks Identified via Coupled Physics‐Engineering‐Economic Modeling
BibTeX
@misc{shambaugh2026validation,
title = {{Validation of Space Weather Forecasting at Leonid Space}},
author = {Scott Shambaugh},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.15213},
eprint = {2610.15213},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.15213},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Validation of Space Weather Forecasting at Leonid Space
AU - Scott Shambaugh
PY - 2026
DA - 2026/10/10
PB - arXiv
DO - 10.48550/arxiv.2610.15213
UR - https://arxiv.org/abs/2610.15213
AB - We validate Leonid Space's space weather forecasts through comprehensive cross-validation of retroforecasts scored against the entire F10.7 and Ap historical record. We built a unified forecast model that jointly forecasts F10.7 and Ap on all timescales from 3 hours up to multi-decade chains of future solar cycles. This was used to generate an archive of historical 15-year retroforecasts, after performing a leave-one-out fit of the model on held-out data and conditioning only on historical data available at the time. The accuracy of this archive was scored against measured space weather indices at all lookahead horizons from 1 day to 15 years, using the continuous ranked probability score (CRPS) as the primary quality metric. All other publicly available F10.7 and Ap forecast archives were scored using the same approach, including NASA's MSAFE and Schatten forecasts, SWPC's 27-day, 45-day, and Solar Cycle predictions, SWPC/USAF's RSGA, SIDC's Ursigram, CLS, Athens, KSO, BGS, SEPC's 3-day and 27-day forecasts, AFRL/NSO's SIFT, and BoM. Taken together, these 15 flux forecasts and 12 Ap forecasts compared on common lookahead metrics make up the most comprehensive cross-comparison of space weather forecasts we are aware of. Over a common forecast interval since 2000, Leonid had the lowest CRPS prediction error of any F10.7 or Ap prediction model: strictly lowest for Ap at all lookaheads and for F10.7 up to 5 years, and parity with MSAFE after 5 years. Much of this performance comes from our well-calibrated uncertainty bands, and we urge space weather forecasters to include calibrated uncertainties with their predictions and to issue retroforecast archives that can be used to rigorously evaluate their generating models.
ER -