Observation-Specific Uncertainty Modeling for Latent-Variable Regression in High-Dimensional Analytical Data
Abstract
Across many fields, including medical diagnostics, process monitoring, and quality control, decisions often rely on point predictions for single samples. Observation-specific uncertainty is needed to assess the risk of decision error when predictions are compared with thresholds. Latent-variable regression is widely used for such predictions, but observation-specific uncertainty remains difficult to estimate. Traditional methods assume unbiased predictions, attributing prediction uncertainty solely to variance. However, dimensionality reduction can introduce observation-specific bias, violating this assumption. We introduce a framework using Gaussian processes trained on latent scores and prediction errors to estimate observation-specific bias and error variance, separating systematic error from epistemic and aleatoric variance. In Raman measurements of a fructose-sucrose model system, estimated prediction-error distributions agreed with empirical distributions; bias correction reduced mean squared error, and decision limits incorporating bias and variance improved analyte detection over variance-only limits. An aqueous-solubility application using molecular descriptors demonstrated the framework's generalisation beyond spectroscopy.
Related
- Online Bayesian Model Averaging with Joint Uncertainty Quantification for Models and Regression Coefficients in Binary Regression
- Correctness and Predictive Uncertainty in Bayesian TSK Fuzzy Regression: A Reproducible Evaluation on Tabular and Building-Meter Data
- Three-Dimensional Analytical Model of Axial Flux Magnetic Gear Based on Harmonic Modeling
- Evaluating heart rate coupling with Multi-dimensional Recurrence Quantification Analysis (MdRQA) and Cross-Correlation (CC): Analytical decisions and implications
- Central limit theorems for high-dimensional ridge regression under uniform subsampling
BibTeX
@misc{eskildsen2026observationspecific,
title = {{Observation-Specific Uncertainty Modeling for Latent-Variable Regression in High-Dimensional Analytical Data}},
author = {Carl Emil Eskildsen and Georgios Papadopoulos and Kevin A. Giraldo and Mauricio Barahona and Molly M. Stevens},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.41815},
eprint = {2610.41815},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.41815},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Observation-Specific Uncertainty Modeling for Latent-Variable Regression in High-Dimensional Analytical Data
AU - Carl Emil Eskildsen
AU - Georgios Papadopoulos
AU - Kevin A. Giraldo
AU - Mauricio Barahona
AU - Molly M. Stevens
PY - 2026
DA - 2026/10/30
PB - arXiv
DO - 10.48550/arxiv.2610.41815
UR - https://arxiv.org/abs/2610.41815
AB - Across many fields, including medical diagnostics, process monitoring, and quality control, decisions often rely on point predictions for single samples. Observation-specific uncertainty is needed to assess the risk of decision error when predictions are compared with thresholds. Latent-variable regression is widely used for such predictions, but observation-specific uncertainty remains difficult to estimate. Traditional methods assume unbiased predictions, attributing prediction uncertainty solely to variance. However, dimensionality reduction can introduce observation-specific bias, violating this assumption. We introduce a framework using Gaussian processes trained on latent scores and prediction errors to estimate observation-specific bias and error variance, separating systematic error from epistemic and aleatoric variance. In Raman measurements of a fructose-sucrose model system, estimated prediction-error distributions agreed with empirical distributions; bias correction reduced mean squared error, and decision limits incorporating bias and variance improved analyte detection over variance-only limits. An aqueous-solubility application using molecular descriptors demonstrated the framework's generalisation beyond spectroscopy.
ER -