Action-conditioned latent Kalman filtering for robust spatio-temporal state estimation of deformable object
Abstract
Robust spatio-temporal state estimation from noisy observations is a critical challenge in robotic manipulation. Current latent space representations derived from dynamic point clouds often suffer from severe perceptual noise and occlusions, leading to discontinuous trajectories and pronounced temporal jitter. To mitigate these artifacts, we propose an Action-Conditioned Latent Kalman Filter (AC-LKF) framework. Leveraging a hybrid architecture of PointNet++ and Gated Recurrent Units (GRU), our method models non-linear temporal dynamics within a low-dimensional latent manifold. Instead of claiming to fully model the physical causality of deformable-object dynamics, AC-LKF incorporates the robot end-effector displacement as a low-dimensional, observable, and task-relevant kinematic prior to condition the latent state transition. Furthermore, a heteroscedastic latent filtering module adaptively estimates process and measurement uncertainties, balancing the action-conditioned prior with degraded visual inputs under noise and occlusion. Evaluations using quantitative temporal metrics and reconstruction errors, supplemented by principal component analysis (PCA) and t-distributed stochastic neighbor embedding (t-SNE) visualizations, show that AC-LKF improves latent trajectory smoothness, information retention, and dynamic robustness by reducing abrupt observation-induced state transitions. These results indicate that kinematic action conditioning can improve the temporal consistency and robustness of learned geometric state representations for interactive deformable-object manipulation.
Related
- Differentiable Forward and Back Projectors for Gradient-Based Deformable Motion Estimation
- Contact-Reliability-Aware Neural Residual Kalman Filtering for Legged Robot State Estimation
- Attentive temporal patch filtering for explainable streaming intrusion detection in medical IoT traffic
- Quantitative assessment of deformable image registration accuracy for cumulative dose estimation in multi-site stereotactic re-irradiation
- FIA-LLM: feedback conditioned learning state estimation with large language models
BibTeX
@article{gao2026actionconditioned,
title = {{Action-conditioned latent Kalman filtering for robust spatio-temporal state estimation of deformable object}},
author = {Duan Gao and Majed Melibary and Junlei Hu},
journal = {Scientific Reports},
year = {2026},
month = oct,
doi = {10.1038/s41598-026-77444-2},
url = {https://www.nature.com/articles/s41598-026-77444-2},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Action-conditioned latent Kalman filtering for robust spatio-temporal state estimation of deformable object
AU - Duan Gao
AU - Majed Melibary
AU - Junlei Hu
PY - 2026
DA - 2026/10/29
JO - Scientific Reports
DO - 10.1038/s41598-026-77444-2
UR - https://www.nature.com/articles/s41598-026-77444-2
AB - Robust spatio-temporal state estimation from noisy observations is a critical challenge in robotic manipulation. Current latent space representations derived from dynamic point clouds often suffer from severe perceptual noise and occlusions, leading to discontinuous trajectories and pronounced temporal jitter. To mitigate these artifacts, we propose an Action-Conditioned Latent Kalman Filter (AC-LKF) framework. Leveraging a hybrid architecture of PointNet++ and Gated Recurrent Units (GRU), our method models non-linear temporal dynamics within a low-dimensional latent manifold. Instead of claiming to fully model the physical causality of deformable-object dynamics, AC-LKF incorporates the robot end-effector displacement as a low-dimensional, observable, and task-relevant kinematic prior to condition the latent state transition. Furthermore, a heteroscedastic latent filtering module adaptively estimates process and measurement uncertainties, balancing the action-conditioned prior with degraded visual inputs under noise and occlusion. Evaluations using quantitative temporal metrics and reconstruction errors, supplemented by principal component analysis (PCA) and t-distributed stochastic neighbor embedding (t-SNE) visualizations, show that AC-LKF improves latent trajectory smoothness, information retention, and dynamic robustness by reducing abrupt observation-induced state transitions. These results indicate that kinematic action conditioning can improve the temporal consistency and robustness of learned geometric state representations for interactive deformable-object manipulation.
SN - 2045-2322
ER -