An end-to-end multi-modal pipeline for person search in unconstrained CCTV environments
4 affiliations
- 1Department of Computer Engineering, Pimpri Chinchwad College of Engineering
- 2Department of Computer Science, Applied College, Princess Nourah bint Abdulrahman University
- 3School of Computer Sciences, University of Petroleum and Energy Studies (UPES)
- 4Department of Computer Science, King Khalid University
Abstract
In real-world surveillance, situations arise in which two people wear nearly identical clothing or in which identifying features are obscured by heavy occlusions and shifting poses. In these environments, traditional uni-modal systems that rely on static appearance do not perform well and often produce false matches. This paper presents a multi-modal Re-ID framework that combines appearance, facial features, and gait to accurately perform this task. In the first stage, robust person tracklets are extracted from video using a YOLOv8 detector and a DeepSORT tracker. A multi-modal feature extractor is then applied to each person crop through three parallel streams. A custom GaitSet-inspired network produces a 256-dimensional gait embedding from RGB tracklet frames. An OSNet backbone is used to get a 1024-dimensional omni-scale appearance embedding. For facial biometrics, a Multi-Task Cascaded Convolutional Network (MTCNN) is used to explicitly detect the face and five-point landmarks followed by a facial landmark regression model. Each crop is aligned to a landmark and the aligned face is sent to FaceNet for computing the 128 dimensional facial embedding. In the last step, an Ensemble Concatenation module merges these three normalised embeddings into a single ℝ1408 identity descriptor ( 1024 + 256 + 128 ). The discriminability is significantly improved through the combination of facial features and temporal motion cues. Furthermore, the fusion strategy prevents any single modality from causing identity leakage, thereby providing a strong defense against appearance-based identity errors.
Related
- Participant perspectives on a person-centred care model for mobile individuals in HIV care from the SEARCH Dynamic Choice Treatment trial in rural Kenya and Uganda
- An integrated fuzzy multi-criteria decision making and mixed-integer linear programming optimization framework for project risk assessment and response strategy selection: A case study of a liquefied petroleum gas pipeline construction project
- Remote vs. In-Person Study Evaluation (RISE) Trial 1 of 3 to examine the impact of decentralized trial methods on accrual efficiency, retention, biospecimen capture, and treatment utilization: study protocol for a multi-center randomized controlled trial in the context of smoking cessation (RISE Above Smoking)
- Human-in-the-loop validation of a sequential multi-LLM medical education pipeline
- A Portable Multi-Stream Telemedicine System for First-Level Emergency Support in Crisis Environments
BibTeX
@article{shinde2026endtoend,
title = {{An end-to-end multi-modal pipeline for person search in unconstrained CCTV environments}},
author = {Swati Shinde and Sahil Hirve and Tushar Badlani and Shashank Dhudhane and Saurabh Dhakite and Lubna A. Gabralla and Tanupriya Choudhury and Minakshi Singh},
journal = {Scientific Reports},
year = {2026},
month = sep,
doi = {10.1038/s41598-026-69393-7},
url = {https://doi.org/10.1038/s41598-026-69393-7},
note = {CC BY},
}
RIS
TY - JOUR
TI - An end-to-end multi-modal pipeline for person search in unconstrained CCTV environments
AU - Swati Shinde
AU - Sahil Hirve
AU - Tushar Badlani
AU - Shashank Dhudhane
AU - Saurabh Dhakite
AU - Lubna A. Gabralla
AU - Tanupriya Choudhury
AU - Minakshi Singh
PY - 2026
DA - 2026/09/01
JO - Scientific Reports
DO - 10.1038/s41598-026-69393-7
UR - https://doi.org/10.1038/s41598-026-69393-7
AB - In real-world surveillance, situations arise in which two people wear nearly identical clothing or in which identifying features are obscured by heavy occlusions and shifting poses. In these environments, traditional uni-modal systems that rely on static appearance do not perform well and often produce false matches. This paper presents a multi-modal Re-ID framework that combines appearance, facial features, and gait to accurately perform this task. In the first stage, robust person tracklets are extracted from video using a YOLOv8 detector and a DeepSORT tracker. A multi-modal feature extractor is then applied to each person crop through three parallel streams. A custom GaitSet-inspired network produces a 256-dimensional gait embedding from RGB tracklet frames. An OSNet backbone is used to get a 1024-dimensional omni-scale appearance embedding. For facial biometrics, a Multi-Task Cascaded Convolutional Network (MTCNN) is used to explicitly detect the face and five-point landmarks followed by a facial landmark regression model. Each crop is aligned to a landmark and the aligned face is sent to FaceNet for computing the 128 dimensional facial embedding. In the last step, an Ensemble Concatenation module merges these three normalised embeddings into a single ℝ^1408 identity descriptor ( 1024 + 256 + 128 ). The discriminability is significantly improved through the combination of facial features and temporal motion cues. Furthermore, the fusion strategy prevents any single modality from causing identity leakage, thereby providing a strong defense against appearance-based identity errors.
SN - 2045-2322
ER -