The hypervariational method: a general, automated technique based on innovations to tune the parameters of the Hybrid covariance matrix
Abstract
The Hybrid forecast covariance model is an operationally successful way to build a background error covariance matrix. It is constructed as a weighted sum, with scalar weights, of the climatological covariance matrix and a localized ensemble covariance matrix, with a localization matrix typically depending on a localization length. For best performance, the Hybrid needs to be optimized by tuning those weights and, possibly, other parameters, such as the localization length. Here, a systematic and automated method for this tuning operation is introduced. This technique is referred to as hypervariational method, and it is formulated in terms of a cost function, built using a time series of innovations as an essential ingredient. This cost function, which depends on the parameters, is minimized when they have optimal values. Equations for its minimization, based on Newton's method, are derived. Besides the aforementioned, commonly used Hybrid, the hypervariational method is applicable to any known Hybrid (and to any set of parameters on which it depends analytically), including recently developed Hybrids, not necessarily constructed as weighted sums. Here, the hypervariational method is applied to tune the ensemble covariance matrix weight in the seminal-formulation Hybrid (which does not feature localization). This is done in a synthetic experimental setting where a theoretically-derived expression for the optimal value for that weight is known. Moreover, in that experimental setting, a further method, based on a linear fit procedure, to determine that optimal value is possible. Both the hypervariational method and that linear fit procedure are shown to produce an optimal weight value matching (up to a tolerance of $0.1$) the optimal value known from the theory. Furthermore, Newton's minimization in the hypervariational method is shown to converge immediately (at the first recursive step).
Related
- Electrical resistivity technique for assessing groundwater tables in an unconfined aquifer system and the importance of Dar-Zarrouk parameters in aquifer investigation: a case study from Walajabad block, Tamil Nadu, India
- Résilience et réactivité des pêcheurs artisans sénégalais : la crise écologique comme moteur d’innovations
- From Wastewater to Bio-Hydrogen: Advancing Microbial Electrolysis Cells Through Challenges, Innovations, and Process Integration
- Ecological responsibility, wellbeing, and transformative reach: a three-dimensional framework for assessing ecovillages as social innovations toward sustainability
- Innovations in Agronomy and Their Impact on Greenhouse Vegetable Yields: Species-Specific Perspectives
BibTeX
@misc{sardelli2026hypervariational,
title = {{The hypervariational method: a general, automated technique based on innovations to tune the parameters of the Hybrid covariance matrix}},
author = {Francesco Sardelli and Craig H. Bishop},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.17561},
eprint = {2610.17561},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.17561},
}
RIS
TY - UNPB
TI - The hypervariational method: a general, automated technique based on innovations to tune the parameters of the Hybrid covariance matrix
AU - Francesco Sardelli
AU - Craig H. Bishop
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2610.17561
UR - https://arxiv.org/abs/2610.17561
AB - The Hybrid forecast covariance model is an operationally successful way to build a background error covariance matrix. It is constructed as a weighted sum, with scalar weights, of the climatological covariance matrix and a localized ensemble covariance matrix, with a localization matrix typically depending on a localization length. For best performance, the Hybrid needs to be optimized by tuning those weights and, possibly, other parameters, such as the localization length. Here, a systematic and automated method for this tuning operation is introduced. This technique is referred to as hypervariational method, and it is formulated in terms of a cost function, built using a time series of innovations as an essential ingredient. This cost function, which depends on the parameters, is minimized when they have optimal values. Equations for its minimization, based on Newton's method, are derived. Besides the aforementioned, commonly used Hybrid, the hypervariational method is applicable to any known Hybrid (and to any set of parameters on which it depends analytically), including recently developed Hybrids, not necessarily constructed as weighted sums. Here, the hypervariational method is applied to tune the ensemble covariance matrix weight in the seminal-formulation Hybrid (which does not feature localization). This is done in a synthetic experimental setting where a theoretically-derived expression for the optimal value for that weight is known. Moreover, in that experimental setting, a further method, based on a linear fit procedure, to determine that optimal value is possible. Both the hypervariational method and that linear fit procedure are shown to produce an optimal weight value matching (up to a tolerance of $0.1$) the optimal value known from the theory. Furthermore, Newton's minimization in the hypervariational method is shown to converge immediately (at the first recursive step).
ER -