TempoCross: instance-aware sparse representation for multimodal temporal fusion in 3D detection
- 1School of Geomatics, Liaoning Technical University
Abstract
With rising performance demands for autonomous driving perception systems, bird’s-eye-view (BEV) perception that combines heterogeneous sensor data and temporal information has become a key research focus. Recent studies have advanced either multimodal temporal detection through BEV-level feature aggregation or sparse object-centric temporal modeling. However, integrating these two directions while preserving modality-specific temporal states remains less explored for LiDAR-camera detection. In this work, we present TempoCross, a 3D detection method based on instance-aware sparse representations for multimodal temporal fusion. TempoCross encodes features from different timestamps and sensor modalities in a unified instance space, enabling adaptive extraction of critical target information from modality- and time-specific states. Initially, both branches perform a preliminary cross-modal fusion to generate queries for the current frame. In the motion compensation module, a hybrid motion modeling strategy reduces alignment discrepancies between historical and current instances caused by complex object motion. This strategy combines explicit rigid-body transformations with implicit learnable deformation residuals, improving both accuracy and robustness in cross-frame instance association. Next, temporal-aware enhancement integrates the initial queries and current-frame features with motion-compensated historical instances. Finally, a lightweight cross-attention fuses current and historical instances from both branches. This formulation reduces the reliance on repeatedly propagating full-scene fused BEV features and concentrates temporal interaction on target-related instance states. On the nuScenes test set, TempoCross achieves 74.1% mAP and 75.7% NDS, outperforming mainstream baseline detectors. The results support the effectiveness of combining LiDAR-camera fusion with instance-aware sparse temporal modeling.
Related
- Multimodal Multiple Instance Learning for Fibroepithelial Tumor Diagnosis in Breast Ultrasound
- An interpretable cell-centric representation of pancreatic ductal adenocarcinoma matches multiple-instance learning for clinical outcomes and leads to novel pathologist-driven histological biomarkers
- GRASS-MIL: graph-based representation and discovery of phenotype-associated spatial structures with multiple instance learning
- Weakly-Supervised Detection of Facial Motor Tics in Video Data Using Score-Driven Top-k Multiple Instance Learning
- FragMIL: A Fragment-Token Multiple-Instance Learning Framework for Cancer Detection from Cell-Free DNA Fragmentomics
BibTeX
@article{wang2026tempocross,
title = {{TempoCross: instance-aware sparse representation for multimodal temporal fusion in 3D detection}},
author = {Changqiang Wang and Yuanqi Liang and Bolin Zhang and Xin Sui and Jiaxin Gao and Song Gao},
journal = {Scientific Reports},
year = {2026},
month = jul,
doi = {10.1038/s41598-026-62822-7},
url = {https://doi.org/10.1038/s41598-026-62822-7},
note = {CC BY},
}
RIS
TY - JOUR
TI - TempoCross: instance-aware sparse representation for multimodal temporal fusion in 3D detection
AU - Changqiang Wang
AU - Yuanqi Liang
AU - Bolin Zhang
AU - Xin Sui
AU - Jiaxin Gao
AU - Song Gao
PY - 2026
DA - 2026/07/01
JO - Scientific Reports
DO - 10.1038/s41598-026-62822-7
UR - https://doi.org/10.1038/s41598-026-62822-7
AB - With rising performance demands for autonomous driving perception systems, bird’s-eye-view (BEV) perception that combines heterogeneous sensor data and temporal information has become a key research focus. Recent studies have advanced either multimodal temporal detection through BEV-level feature aggregation or sparse object-centric temporal modeling. However, integrating these two directions while preserving modality-specific temporal states remains less explored for LiDAR-camera detection. In this work, we present TempoCross, a 3D detection method based on instance-aware sparse representations for multimodal temporal fusion. TempoCross encodes features from different timestamps and sensor modalities in a unified instance space, enabling adaptive extraction of critical target information from modality- and time-specific states. Initially, both branches perform a preliminary cross-modal fusion to generate queries for the current frame. In the motion compensation module, a hybrid motion modeling strategy reduces alignment discrepancies between historical and current instances caused by complex object motion. This strategy combines explicit rigid-body transformations with implicit learnable deformation residuals, improving both accuracy and robustness in cross-frame instance association. Next, temporal-aware enhancement integrates the initial queries and current-frame features with motion-compensated historical instances. Finally, a lightweight cross-attention fuses current and historical instances from both branches. This formulation reduces the reliance on repeatedly propagating full-scene fused BEV features and concentrates temporal interaction on target-related instance states. On the nuScenes test set, TempoCross achieves 74.1% mAP and 75.7% NDS, outperforming mainstream baseline detectors. The results support the effectiveness of combining LiDAR-camera fusion with instance-aware sparse temporal modeling.
SN - 2045-2322
ER -