Learning fly pose from vision-language annotations
- 1Joseph Henry Laboratories of Physics and Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ 08544, USA
Abstract
Animal pose estimation typically requires manual annotation of body parts to train neural networks for each new recording system or environmental condition. Here we use a general-purpose vision-language model (VLM) to generate annotations in multi-animal recordings of fruit flies (Drosophila melanogaster). Initial annotations follow written anatomical descriptions and are reviewed by an independent VLM session using neighboring video frames. On 120 held-out fly images, AI annotations had a median distance of 2.28 pixels from human labels, with 84.0% agreement within eight pixels, averaged across landmark types. The comparison included 10 landmarks, with head and eyes analyzed separately because of systematic differences in placement. Review improved agreement primarily at the feet, with the largest remaining discrepancies at positions with limited visual evidence. Visual inspection revealed cases in which the AI annotations were more anatomically accurate than those of an experienced human annotator, particularly at the feet. A convolutional network trained using SLEAP on 300 AI-annotated fly images showed similar agreement with human labels and closely reproduced the AI annotations on held-out images, with a median distance of 1.56 pixels. We also applied the procedure to recordings from another laboratory, using new AI annotations to train a separate pose network. Although VLM annotation required minutes per fly, the trained networks enabled pose estimation throughout more than 700,000 frames. These results demonstrate that VLM annotations can support fast animal pose estimation without manually labeled training images.
Related
- Region-Aware Degradation-Conditioned Vision-Language Learning for Blind Image-Quality Assessment of Power Field-Operation Imagery
- From pose to behavior: SABER integrates identity-resolved multi-animal pose tracking with language-model-based behavioral factor discovery
- MIEN: a deep-learning model for pocket-conditioned glycan pose and conformation discrimination
- A Vision-Language Model as a Teacher for Bird Vocalization Detection
- MEMOIR-VLM—a multimodal vision-language model for Alzheimer's disease classification and question answering
BibTeX
@misc{shaevitz2026learning,
title = {{Learning fly pose from vision-language annotations}},
author = {J. W. Shaevitz and M. G. Sierra and S. K. Kim and S. D. Pletcher},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.30.755712},
url = {https://www.biorxiv.org/content/10.64898/2026.09.30.755712v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Learning fly pose from vision-language annotations
AU - J. W. Shaevitz
AU - M. G. Sierra
AU - S. K. Kim
AU - S. D. Pletcher
PY - 2026
DA - 2026/09/30
PB - bioRxiv
DO - 10.64898/2026.09.30.755712
UR - https://www.biorxiv.org/content/10.64898/2026.09.30.755712v1
AB - Animal pose estimation typically requires manual annotation of body parts to train neural networks for each new recording system or environmental condition. Here we use a general-purpose vision-language model (VLM) to generate annotations in multi-animal recordings of fruit flies (Drosophila melanogaster). Initial annotations follow written anatomical descriptions and are reviewed by an independent VLM session using neighboring video frames. On 120 held-out fly images, AI annotations had a median distance of 2.28 pixels from human labels, with 84.0% agreement within eight pixels, averaged across landmark types. The comparison included 10 landmarks, with head and eyes analyzed separately because of systematic differences in placement. Review improved agreement primarily at the feet, with the largest remaining discrepancies at positions with limited visual evidence. Visual inspection revealed cases in which the AI annotations were more anatomically accurate than those of an experienced human annotator, particularly at the feet. A convolutional network trained using SLEAP on 300 AI-annotated fly images showed similar agreement with human labels and closely reproduced the AI annotations on held-out images, with a median distance of 1.56 pixels. We also applied the procedure to recordings from another laboratory, using new AI annotations to train a separate pose network. Although VLM annotation required minutes per fly, the trained networks enabled pose estimation throughout more than 700,000 frames. These results demonstrate that VLM annotations can support fast animal pose estimation without manually labeled training images.
ER -