Hybrid Federated Learning for Multimodal Fusion under Missing Modalities and Site Heterogeneity
Abstract
Multimodal fusion is studied under hybrid federation, where data are partitioned across samples and feature blocks. The proposed two-dimensional federated smooth multiple-kernel large-margin learning method (2D-FedSMKL) integrates availability-corrected modality scores, group-kernel regularization, and smooth worst-site risk aggregation. For fixed maps and propensities, the objective is convex and smooth. Curvature bounds and exact synchronous block-gradient equivalence are established for binary and multiclass scores, giving a conditional accelerated rate. An oracle-propensity binary risk bound and a propensity-perturbation result are also derived. Simulations and two public activity-recognition benchmarks assess site heterogeneity and modality absence. Worst-site aggregation improves worst-site performance under heterogeneity, including reversed and size-balanced designs. On the benchmarks, validation selects the correction; the selected model is not significantly worse than the strongest reference at the worst site, and group-regularization benefits vary by dataset.
Related
- FedRSPO+: A Heterogeneity-aware Algorithm for Decision-focused Federated Learning
- On the Gradient Heterogeneity Dynamics of Adversarially Robust Federated Regression
- D3MI: an efficient and powerful federated imputation method for bias reduction in the analysis of distributed incomplete data by accounting for within-site correlation and between-site heterogeneity.
- Uncertainty-Aware Missing-Data Multimodal Latent for Fetal-Growth Analysis
- Progressive-Resolution Secure Aggregation for Federated Learning
BibTeX
@misc{wu2026hybrid,
title = {{Hybrid Federated Learning for Multimodal Fusion under Missing Modalities and Site Heterogeneity}},
author = {Xiaofei Wu and Rongmei Liang and Jian Qing Shi},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.03032},
eprint = {2611.03032},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.03032},
}
RIS
TY - UNPB
TI - Hybrid Federated Learning for Multimodal Fusion under Missing Modalities and Site Heterogeneity
AU - Xiaofei Wu
AU - Rongmei Liang
AU - Jian Qing Shi
PY - 2026
DA - 2026/11/02
PB - arXiv
DO - 10.48550/arxiv.2611.03032
UR - https://arxiv.org/abs/2611.03032
AB - Multimodal fusion is studied under hybrid federation, where data are partitioned across samples and feature blocks. The proposed two-dimensional federated smooth multiple-kernel large-margin learning method (2D-FedSMKL) integrates availability-corrected modality scores, group-kernel regularization, and smooth worst-site risk aggregation. For fixed maps and propensities, the objective is convex and smooth. Curvature bounds and exact synchronous block-gradient equivalence are established for binary and multiclass scores, giving a conditional accelerated rate. An oracle-propensity binary risk bound and a propensity-perturbation result are also derived. Simulations and two public activity-recognition benchmarks assess site heterogeneity and modality absence. Worst-site aggregation improves worst-site performance under heterogeneity, including reversed and size-balanced designs. On the benchmarks, validation selects the correction; the selected model is not significantly worse than the strongest reference at the worst site, and group-regularization benefits vary by dataset.
ER -