A promptable foundation model enables automated multi-task dataset construction and real-time pose estimation in mice
- 1Centre National Recherche Scientifique
Abstract
Behavioural experiments often hinge on moments when animals crowd together. In these situations, existing pose-estimation methods usually fail because detected crops include neighbouring animals, and their body parts are easily confused between individuals. We solved it with DeepLabSAM, a pipeline that segments and tracks every animal from a single text prompt using the SAM 3 foundation model, followed by a mask-gated pretrained pose estimation network from DeepLabCut. Across recordings of multiple interacting mice, the proposed method substantially reduces misassigned keypoints under heavy occlusion. Furthermore, we used DeepLabSAM to automatically construct the Multi-Task Mouse Behaviour (MTMB) dataset, comprising 14,300 frames spanning nine behavioural assays, with each instance annotated for bounding box, mask, identity, and pose. SuperAnimal pretrained pose models transferred unpredictably across this dataset. Conversely, MouseLite, our RF-DETR pose-estimation model trained on the MTMB dataset, matched their in-distribution accuracy, generalized better to unseen assays, and achieved real-time (faster-than-acquisition) inference. Combined, DeepLabSAM, MTMB dataset, and MouseLite provide a scalable framework for robust, cross-assay behavioural pose tracking of freely interacting mice.
Related
- Automated identification of individual birds by song enables multi-year recapture from passive acoustic monitoring data
- DAPE-PD: Diffusion adaptation of genomic foundation model enables tailored-generation of cis-regulatory DNA
- A single-cell multimodal foundation model enables cross modality analysis between transcriptome and chromatin
- Time-series foundation modeling enables accurate lake ecosystem forecasting
- Simultaneous hyperkinetic movement disorders phenotyping: a cross-cohort pediatric transfer study using routine videos, markerless pose estimation and a tabular foundation model
BibTeX
@misc{cobos2026promptable,
title = {{A promptable foundation model enables automated multi-task dataset construction and real-time pose estimation in mice}},
author = {J. Cobos and S. Thirard and M. Belkaid and J. Naude},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.02.756254},
url = {https://www.biorxiv.org/content/10.64898/2026.10.02.756254v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - A promptable foundation model enables automated multi-task dataset construction and real-time pose estimation in mice
AU - J. Cobos
AU - S. Thirard
AU - M. Belkaid
AU - J. Naude
PY - 2026
DA - 2026/10/02
PB - bioRxiv
DO - 10.64898/2026.10.02.756254
UR - https://www.biorxiv.org/content/10.64898/2026.10.02.756254v1
AB - Behavioural experiments often hinge on moments when animals crowd together. In these situations, existing pose-estimation methods usually fail because detected crops include neighbouring animals, and their body parts are easily confused between individuals. We solved it with DeepLabSAM, a pipeline that segments and tracks every animal from a single text prompt using the SAM 3 foundation model, followed by a mask-gated pretrained pose estimation network from DeepLabCut. Across recordings of multiple interacting mice, the proposed method substantially reduces misassigned keypoints under heavy occlusion. Furthermore, we used DeepLabSAM to automatically construct the Multi-Task Mouse Behaviour (MTMB) dataset, comprising 14,300 frames spanning nine behavioural assays, with each instance annotated for bounding box, mask, identity, and pose. SuperAnimal pretrained pose models transferred unpredictably across this dataset. Conversely, MouseLite, our RF-DETR pose-estimation model trained on the MTMB dataset, matched their in-distribution accuracy, generalized better to unseen assays, and achieved real-time (faster-than-acquisition) inference. Combined, DeepLabSAM, MTMB dataset, and MouseLite provide a scalable framework for robust, cross-assay behavioural pose tracking of freely interacting mice.
ER -