An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories
Abstract
Deep generative models provide flexible frameworks for modeling complex, structured data such as images, videos, 3D objects, and texts. However, when applied to sequences of human skeletons, standard variational autoencoders (VAEs) often allocate substantial capacity to nuisance factors-such as camera orientation, subject scale, viewpoint, and execution speed-rather than the intrinsic geometry of shapes and their motion. We propose the Elastic Shape - Variational Autoencoder (ES-VAE), a geometry-aware generative model for skeletal trajectories that leverages the transported square-root velocity field (TSRVF) representation on Kendall's shape manifold. This representation inherently removes rigid translations, rotations, and global scaling of shapes, and temporal rate variability of sequences, isolating the underlying shape dynamics. The ES-VAE encoder maps skeletal sequences to a low-dimensional latent space incorporating the Riemannian logarithm map, while the decoder reconstructs sequences using the corresponding exponential map. We demonstrate the effectiveness of ES-VAE on two datasets. First, we analyze skeletal gait cycles to predict clinical mobility scores and classify subjects into healthy and post-stroke groups. Second, we evaluate action recognition on the NTU RGB+D dataset. Across both settings, ES-VAE consistently outperforms standard VAEs and a range of sequence modeling baselines, including temporal convolutional networks, transformers, and graph convolutional networks. More broadly, ES-VAE provides a principled framework for learning generative models of longitudinal data on pose shape manifolds, offering improved latent representation and downstream performance compared to existing deep learning approaches.
Related
- xVGAE: A Hierarchical Variational Graph Autoencoder for Exchangeable Graphs
- Theoretical Study on the Evidential Learning-based Variational Autoencoder
- A novel Weighted Exponential-Lindley distribution with Bayesian estimation under progressive Type-II censoring and a Bayesian-Calibrated Variational Autoencoder for survival prediction.
- Elastic knots: compactness, contact, and variational structure
- The skeleton-blocks decomposition of Bienaymé trees, and applications to their local convergence
BibTeX
@misc{rahman2026elastic,
title = {{An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories}},
author = {Arafat Rahman and Shashwat Kumar and Laura E. Barnes and Anuj Srivastava},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.09231},
eprint = {2605.09231},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.09231},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories
AU - Arafat Rahman
AU - Shashwat Kumar
AU - Laura E. Barnes
AU - Anuj Srivastava
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2605.09231
UR - https://arxiv.org/abs/2605.09231
AB - Deep generative models provide flexible frameworks for modeling complex, structured data such as images, videos, 3D objects, and texts. However, when applied to sequences of human skeletons, standard variational autoencoders (VAEs) often allocate substantial capacity to nuisance factors-such as camera orientation, subject scale, viewpoint, and execution speed-rather than the intrinsic geometry of shapes and their motion. We propose the Elastic Shape - Variational Autoencoder (ES-VAE), a geometry-aware generative model for skeletal trajectories that leverages the transported square-root velocity field (TSRVF) representation on Kendall's shape manifold. This representation inherently removes rigid translations, rotations, and global scaling of shapes, and temporal rate variability of sequences, isolating the underlying shape dynamics. The ES-VAE encoder maps skeletal sequences to a low-dimensional latent space incorporating the Riemannian logarithm map, while the decoder reconstructs sequences using the corresponding exponential map. We demonstrate the effectiveness of ES-VAE on two datasets. First, we analyze skeletal gait cycles to predict clinical mobility scores and classify subjects into healthy and post-stroke groups. Second, we evaluate action recognition on the NTU RGB+D dataset. Across both settings, ES-VAE consistently outperforms standard VAEs and a range of sequence modeling baselines, including temporal convolutional networks, transformers, and graph convolutional networks. More broadly, ES-VAE provides a principled framework for learning generative models of longitudinal data on pose shape manifolds, offering improved latent representation and downstream performance compared to existing deep learning approaches.
ER -