BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting

Abstract

Multi-view video recordings are increasingly used to capture the 3D movements of animals in experimental settings, yet extracting rich 3D representations from these recordings remains challenging. Supervised pose estimation requires extensive manual annotation, while general-purpose 3D reconstruction models trained on generic scene datasets fail on the specialized imagery and sparse-view setting of laboratory experiments. We address these limitations with BEAST3D, a self-supervised pretraining framework that learns 3D visual representations from unlabeled, calibrated multi-view video. BEAST3D uses a vision transformer to predict 3D Gaussian splats that reconstruct held-out views through differentiable rendering, while simultaneously segmenting the animal from the background. BEAST3D reconstructs 3D structure with as few as four views by conditioning directly on known camera parameters--unlike general-purpose models, which must estimate camera geometry from dense overlapping viewpoints that are seldom available in lab settings. Through comprehensive evaluation across four species, we demonstrate that BEAST3D produces rich, viewpoint-invariant features that transfer effectively to three downstream tasks: novel view synthesis, which validates the quality of the learned 3D representations; multi-view pose estimation, which provides the sparse keypoint trajectories widely used in behavioral analysis; and neural encoding, which relates 3D behavioral features to simultaneously recorded neural activity. BEAST3D thus establishes a versatile framework for behavioral analysis that leverages 3D structure in modern multi-view laboratory recordings.

Luxar: Gaussian splatting for microscopy and scalable interactive web visualisation of multidimensional scientific data
L. A. Royer
bioRxiv22 Sep 2026PreprintCC BYFull textBioinformatics & computational biologybioRxiv
MedGSSR: Generalizable Medical Image Super-Resolution 3D Reconstruction via Hierarchical Feed-forward Gaussian Splatting
Chengkai Wang, Luoyu Hong, Yiting Zhao +7
arXiv6 Sep 2026PreprintCC BY 4.0Full textBioinformatics & computational biologyarXiv
From pose to behavior: SABER integrates identity-resolved multi-animal pose tracking with language-model-based behavioral factor discovery
J. Zheng, C. Peng, S.-Y. Zhang +8
bioRxiv2 Sep 2026PreprintCC BY-NCFull textZoology & animal behaviourbioRxiv
Animal Remains from Tell Asmar
Max Hilzheimer
16 Sep 2026Peer-reviewedLicence not stated14 citedEvolutionary biologyOpenAlex
A review on applying immunological methods in animal research
Animal nutrition (Zhongguo xu mu shou yi xue hui)8 Sep 2026Peer-reviewedCC BY-NC-NDFull textGeneral biologyEurope PMC

Cite

BibTeX RIS

BibTeX

@misc{wang2026beast3d,
  title = {{BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting}},
  author = {Yanchen Wang and Lenny Aharon and Wangshu Zhu and Kyle Daruwalla and Linghua Zhang and Jiaru Zou and Selmaan Chettih and Helen Hou and Liam Paninski and Matthew R Whiteway},
  howpublished = {arXiv},
  year = {2026},
  month = nov,
  doi = {10.48550/arxiv.2606.02937},
  eprint = {2606.02937},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.02937},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting
AU  - Yanchen Wang
AU  - Lenny Aharon
AU  - Wangshu Zhu
AU  - Kyle Daruwalla
AU  - Linghua Zhang
AU  - Jiaru Zou
AU  - Selmaan Chettih
AU  - Helen Hou
AU  - Liam Paninski
AU  - Matthew R Whiteway
PY  - 2026
DA  - 2026/11/05
PB  - arXiv
DO  - 10.48550/arxiv.2606.02937
UR  - https://arxiv.org/abs/2606.02937
AB  - Multi-view video recordings are increasingly used to capture the 3D movements of animals in experimental settings, yet extracting rich 3D representations from these recordings remains challenging. Supervised pose estimation requires extensive manual annotation, while general-purpose 3D reconstruction models trained on generic scene datasets fail on the specialized imagery and sparse-view setting of laboratory experiments. We address these limitations with BEAST3D, a self-supervised pretraining framework that learns 3D visual representations from unlabeled, calibrated multi-view video. BEAST3D uses a vision transformer to predict 3D Gaussian splats that reconstruct held-out views through differentiable rendering, while simultaneously segmenting the animal from the background. BEAST3D reconstructs 3D structure with as few as four views by conditioning directly on known camera parameters--unlike general-purpose models, which must estimate camera geometry from dense overlapping viewpoints that are seldom available in lab settings. Through comprehensive evaluation across four species, we demonstrate that BEAST3D produces rich, viewpoint-invariant features that transfer effectively to three downstream tasks: novel view synthesis, which validates the quality of the learned 3D representations; multi-view pose estimation, which provides the sparse keypoint trajectories widely used in behavioral analysis; and neural encoding, which relates 3D behavioral features to simultaneously recorded neural activity. BEAST3D thus establishes a versatile framework for behavioral analysis that leverages 3D structure in modern multi-view laboratory recordings.
ER  -

Abstract

Related

Cite