A fully immersive Virtual Reality setup using DeepLabCut reveals species-specific behavioural responses of freely moving coral reef fishes to conspecifics and predators
Abstract
A fully immersive Virtual Reality setup using DeepLabCut reveals species-specific behavioural responses of freely moving coral reef fishes to conspecifics and predators Abstract 1. Virtual reality (VR) coupled with video-based tracking offers unique opportunities to study freely moving animal behaviour under controlled, ecologically relevant conditions. However, existing VR technology for fish is either head-fixed, has limited swimming depth (shallow bowls), or have basic virtual simulations that limit realism and ecological validity of fish responses. We developed a fully immersive VR system for freely moving fishes that integrates in depth 3D tracking, DeepLabCutbased posture detection, real-time viewpoint updating (closed-loop rendering) and highly realistic virtual scenarios.
2. This proof-of-concept study quantified the behavioural responses of three wild coral reef fish species: convict tang surgeonfish (Acanthurus triostegus), humbug damselfish (Dascyllus enamo), and Polynesian anemonefish (Amphiprion maohiensis) to a baseline sand environment and to virtual conspecific and predator shoals across three phases: arrival, static display and departure.
3. Underwater calibration and triangulation, enabled precise positional tracking and real-time updating to the rendering viewpoint, delivering a fully immersive three-dimensional VR experience.
DeepLabCut-based posture tracking provided detailed behavioural metrics, including time swimming forward and gaze direction in the real/virtual reference frame, providing a comprehensive analysis of fish behaviour and perception. Behavioural responses were species-specific across multiple metrics (X, Y and Z positions, velocity, forward swimming, and gaze direction): surgeonfish consistently avoided both conspecifics and predators, damselfish strongly approached conspecifics and avoided predators, while anemonefish showed variable responses. These patterns likely reflect interspecific differences in social organisation, group size and mobility. Similar responses were observed in social species tested either singly or in groups, highlighting the applicability of the VR system to groupliving fishes. Updating the rendering viewpoint did not alter behavioural responses but occasionally enhancing them. Novel species-specific patterns of gaze and stimulus-motion direction revealed distinct scanning, tracking and vigilance strategies across species. Moreover, behavioural responses to a virtual conspecific were qualitatively similar to those elicited by a real conspecific in the setup.
4. Together, these findings validate our fully immersive VR system and demonstrate its potential for uncovering species-specific behaviour, decision-making, sensory-motor integration and social interactions under realistic yet experimentally controlled conditions. This platform opens new opportunities for studying animal cognition, collective behaviour and responses to environmental change.
Related
- Should I stay or should I go? Frog breeding responses to drought events and alien predators
- Under pressure: personality, behavioural and physiological responses to handling in captive Addax nasomaculatus antelopes
- Evolutionary decoupling of morphological and behavioural antipredator responses and their hormonal basis across anuran species
- Quantification of group and individual behavioural responses to a nutritional challenge in dairy goats
- Foraging in a changing world: behavioural and demographic responses to climate and land-use change in Barbary macaque (Macaca sylvanus)
BibTeX
@misc{mills2026fully,
title = {{A fully immersive Virtual Reality setup using DeepLabCut reveals species-specific behavioural responses of freely moving coral reef fishes to conspecifics and predators}},
author = {Suzanne C Mills and Ricardo Beldade and Julie Lafaye and David Lecchini and Manuel Vidal},
howpublished = {HAL},
year = {2026},
month = sep,
url = {https://hal.science/hal-05761670v1},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - A fully immersive Virtual Reality setup using DeepLabCut reveals species-specific behavioural responses of freely moving coral reef fishes to conspecifics and predators
AU - Suzanne C Mills
AU - Ricardo Beldade
AU - Julie Lafaye
AU - David Lecchini
AU - Manuel Vidal
PY - 2026
DA - 2026/09/24
PB - HAL
UR - https://hal.science/hal-05761670v1
AB - A fully immersive Virtual Reality setup using DeepLabCut reveals species-specific behavioural responses of freely moving coral reef fishes to conspecifics and predators Abstract 1. Virtual reality (VR) coupled with video-based tracking offers unique opportunities to study freely moving animal behaviour under controlled, ecologically relevant conditions. However, existing VR technology for fish is either head-fixed, has limited swimming depth (shallow bowls), or have basic virtual simulations that limit realism and ecological validity of fish responses. We developed a fully immersive VR system for freely moving fishes that integrates in depth 3D tracking, DeepLabCutbased posture detection, real-time viewpoint updating (closed-loop rendering) and highly realistic virtual scenarios. 2. This proof-of-concept study quantified the behavioural responses of three wild coral reef fish species: convict tang surgeonfish (Acanthurus triostegus), humbug damselfish (Dascyllus enamo), and Polynesian anemonefish (Amphiprion maohiensis) to a baseline sand environment and to virtual conspecific and predator shoals across three phases: arrival, static display and departure. 3. Underwater calibration and triangulation, enabled precise positional tracking and real-time updating to the rendering viewpoint, delivering a fully immersive three-dimensional VR experience. DeepLabCut-based posture tracking provided detailed behavioural metrics, including time swimming forward and gaze direction in the real/virtual reference frame, providing a comprehensive analysis of fish behaviour and perception. Behavioural responses were species-specific across multiple metrics (X, Y and Z positions, velocity, forward swimming, and gaze direction): surgeonfish consistently avoided both conspecifics and predators, damselfish strongly approached conspecifics and avoided predators, while anemonefish showed variable responses. These patterns likely reflect interspecific differences in social organisation, group size and mobility. Similar responses were observed in social species tested either singly or in groups, highlighting the applicability of the VR system to groupliving fishes. Updating the rendering viewpoint did not alter behavioural responses but occasionally enhancing them. Novel species-specific patterns of gaze and stimulus-motion direction revealed distinct scanning, tracking and vigilance strategies across species. Moreover, behavioural responses to a virtual conspecific were qualitatively similar to those elicited by a real conspecific in the setup. 4. Together, these findings validate our fully immersive VR system and demonstrate its potential for uncovering species-specific behaviour, decision-making, sensory-motor integration and social interactions under realistic yet experimentally controlled conditions. This platform opens new opportunities for studying animal cognition, collective behaviour and responses to environmental change.
ER -