JEPA-SORT: World-Model Embeddings Improve Deep-Sea Multi-Object Tracking on Visually Homogeneous Fauna
- 1OceanX
Abstract
Protecting deep-sea sessile communities as Marine Protected Areas (MPAs) requires accurate estimates of their spatial distribution and abundance. Existing computer-vision detection methods require careful calibration to avoid counting errors; multi-object tracking (MOT) theoretically offers a more robust alternative. In practice, however, poor lighting and turbidity limit the efficacy of purely kinematic MOT algorithms in deep-sea environments. At the same time, the intra-species visual homogeneity typical of deep-sea scenes means traditional per-frame appearance features are often too weak to maintain stable identities in deep-association MOT algorithms. In this paper we introduce JEPA-SORT, an MOT algorithm that injects V-JEPA 2.1 embeddings into detection-to-track association, to show that video world-model embeddings - which capture not only physical appearance but spatiotemporal trajectory across viewpoints - can help maintain stable identities on visually homogeneous specimens. We provide a benchmark of twelve ROV transects surveying deep-sea sessile communities and evaluate JEPA-SORT against standard MOT algorithms across a range of simulated optical environments. We also propose the Biodensity Utility Metric (BDU), which measures tracker performance in the context of downstream ecological modeling. JEPA-SORT outperforms all benchmarked trackers on IDF1, MOTA, and BDU and is competitive on specimen-count accuracy. Ablations indicate its advantage stems primarily from the discriminative power of world-model embeddings on visually homogeneous footage.
Related
- Open-world fungal ITS embeddings improve higher-rank placement and cross-view retrieval, but percent identity is stronger on a matched ITS2 novelty benchmark
- A Visually Interpretable Histopathology-Based Immune Model Predicts T-effector Biology and Response to Immune checkpoint inhibition in Clear Cell Renal Cell Carcinoma Clinical Trial and Contemporary Real-World Datasets
- VIJB: a companion of the JBROWSE genome browser for the visually impaired people
- Cows visually discriminate and cross-modally recognise familiar and unfamiliar human faces in videos
- A Semi-Dry, Visually Comfortable c-VEP BCI with Short Calibration, Conservative Idle-State Behavior, and a VR Proof of Concept
BibTeX
@misc{tillinghastraby2026jepasort,
title = {{JEPA-SORT: World-Model Embeddings Improve Deep-Sea Multi-Object Tracking on Visually Homogeneous Fauna}},
author = {A. Tillinghast-Raby and M. Kravetz and J. Kamau-Weng and A. Weinnig and Y. Rodrigues and F. Varini and V. Pieribone},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.07.757484},
url = {https://www.biorxiv.org/content/10.64898/2026.10.07.757484v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - JEPA-SORT: World-Model Embeddings Improve Deep-Sea Multi-Object Tracking on Visually Homogeneous Fauna
AU - A. Tillinghast-Raby
AU - M. Kravetz
AU - J. Kamau-Weng
AU - A. Weinnig
AU - Y. Rodrigues
AU - F. Varini
AU - V. Pieribone
PY - 2026
DA - 2026/10/07
PB - bioRxiv
DO - 10.64898/2026.10.07.757484
UR - https://www.biorxiv.org/content/10.64898/2026.10.07.757484v1
AB - Protecting deep-sea sessile communities as Marine Protected Areas (MPAs) requires accurate estimates of their spatial distribution and abundance. Existing computer-vision detection methods require careful calibration to avoid counting errors; multi-object tracking (MOT) theoretically offers a more robust alternative. In practice, however, poor lighting and turbidity limit the efficacy of purely kinematic MOT algorithms in deep-sea environments. At the same time, the intra-species visual homogeneity typical of deep-sea scenes means traditional per-frame appearance features are often too weak to maintain stable identities in deep-association MOT algorithms. In this paper we introduce JEPA-SORT, an MOT algorithm that injects V-JEPA 2.1 embeddings into detection-to-track association, to show that video world-model embeddings - which capture not only physical appearance but spatiotemporal trajectory across viewpoints - can help maintain stable identities on visually homogeneous specimens. We provide a benchmark of twelve ROV transects surveying deep-sea sessile communities and evaluate JEPA-SORT against standard MOT algorithms across a range of simulated optical environments. We also propose the Biodensity Utility Metric (BDU), which measures tracker performance in the context of downstream ecological modeling. JEPA-SORT outperforms all benchmarked trackers on IDF1, MOTA, and BDU and is competitive on specimen-count accuracy. Ablations indicate its advantage stems primarily from the discriminative power of world-model embeddings on visually homogeneous footage.
ER -