A Data-Consistent Approach to Ensemble Filtering
Abstract
Ensemble data assimilation is challenging when the ensemble size is substantially smaller than the state and observation dimensions, leaving empirical covariance information rank deficient and sensitive to finite sampling. We introduce the Quantity-of-Interest Principal Component Analysis Ensemble Data-Consistent Filter (QPCA-EnDCF), a deterministic filtering method motivated by data-consistent inversion that regularizes the update in the whitened forecast--observation mismatch space. The method restricts the correction to a rank-κ empirical spectral subspace before mapping it to state space through an empirical EnDCF gain. We establish concentration and spectral-projector perturbation results and derive a bias--variance decomposition showing that truncation bias is governed by the discarded component of the population mean residual, while finite-ensemble variability depends on estimation of the retained correction. QPCA-EnDCF avoids the analysis-mean variance contribution from perturbed observations in stochastic EnKF updates, and Lorenz--96 experiments show improved calibration and lower mean-squared error relative to the tested stochastic and deterministic controls, primarily through lower across-realization variance despite increased bias. An idealized inlet shallow-water experiment further shows that QPCA-EnDCF can be implemented in a high-dimensional, severely rank-deficient nonlinear model and remains numerically viable through the prescribed shelf-crossing endpoints.
Related
- Negative-binomial ensemble filtering for discrete-time Hawkes network reconstruction
- Regularity-informed data assimilation: A hierarchical Bayesian approach to ensemble Kalman filtering for hyperbolic conservation laws
- AECSF: Adaptive Ensemble Conditional Score Filtering for High-Dimensional Nonlinear Data Assimilation
- Covariance-Informed Ensemble Score Filtering: Embedding Localized Covariance Geometry into Score-Based Transport
- Finite-Dimensional Recursions for Small-Noise Expansions in Nonlinear Filtering
BibTeX
@misc{spence2026dataconsistent,
title = {{A Data-Consistent Approach to Ensemble Filtering}},
author = {Rylan Spence and Troy Butler and Clint Dawson},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.11282},
eprint = {2605.11282},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.11282},
}
RIS
TY - UNPB
TI - A Data-Consistent Approach to Ensemble Filtering
AU - Rylan Spence
AU - Troy Butler
AU - Clint Dawson
PY - 2026
DA - 2026/10/16
PB - arXiv
DO - 10.48550/arxiv.2605.11282
UR - https://arxiv.org/abs/2605.11282
AB - Ensemble data assimilation is challenging when the ensemble size is substantially smaller than the state and observation dimensions, leaving empirical covariance information rank deficient and sensitive to finite sampling. We introduce the Quantity-of-Interest Principal Component Analysis Ensemble Data-Consistent Filter (QPCA-EnDCF), a deterministic filtering method motivated by data-consistent inversion that regularizes the update in the whitened forecast--observation mismatch space. The method restricts the correction to a rank-κ empirical spectral subspace before mapping it to state space through an empirical EnDCF gain. We establish concentration and spectral-projector perturbation results and derive a bias--variance decomposition showing that truncation bias is governed by the discarded component of the population mean residual, while finite-ensemble variability depends on estimation of the retained correction. QPCA-EnDCF avoids the analysis-mean variance contribution from perturbed observations in stochastic EnKF updates, and Lorenz--96 experiments show improved calibration and lower mean-squared error relative to the tested stochastic and deterministic controls, primarily through lower across-realization variance despite increased bias. An idealized inlet shallow-water experiment further shows that QPCA-EnDCF can be implemented in a high-dimensional, severely rank-deficient nonlinear model and remains numerically viable through the prescribed shelf-crossing endpoints.
ER -