Cross-modal alignment with language tokens for unified multilingual image-to-text generation.
- 1School of Information Engineering, Fujian Key Lab of Agriculture Internet-of-Things (IOT) Application, Sanming University, Sanming, China.
- 2School of Physics and Electronic Engineering, Hanshan Normal University, Chaozhou, China.
- 3Faculty of Computing, Bayero University, Kano, Nigeria.
Abstract
Current multimodal translation systems typically employ a cascaded pipeline where image captioning and machine translation are performed sequentially, leading to error propagation and semantic drift. This paper introduces a unified neural architecture that integrates visual encoding, English caption generation and multilingual translation into a single end-to-end model. Our first key contribution is this unified framework which jointly optimizes both tasks through shared latent representations enabling richer cross-modal learning and more semantically consistent multilingual outputs than conventional systems. The architecture incorporates two core technical innovations. First, we introduce a cross-modal representation alignment mechanism that directly fuses hierarchical visual features from a CNN encoder with multilingual linguistic embeddings in the Transformer decoder. This ensures visual semantics (e.g., gender, count, and object attributes) are preserved during translation effectively grounding multilingual outputs in the visual context and rectifying a common failure mode of text-only NMT. Second, we design a language token driven conditioning strategy prepending learnable target language tokens to guide a single multilingual model. This allows dynamic adaptation to diverse syntax and morphology without requiring separate decoders for each language. Empirical evaluations on MS COCO and OPUS multilingual datasets demonstrate notable improvements over existing multimodal and cascaded baselines achieving a BLEU-4 score of 44.8 for English captioning and an average BLEU of 41.1 across five target languages (Chinese, French, Arabic, Spanish, and Hindi), representing gains of 2.2 and 2.0 points, respectively, over cascaded baselines. COMET scores averaging 0.732 further confirm the semantic fidelity of our visually grounded translations. The proposed framework establishes a scalable, visually grounded and linguistically adaptive solution for multilingual caption generation contributing a meaningful advancement toward integrated vision-language translation research and real-world accessibility applications.
Related
- English-Persian cross-language plagiarism detection: Multilingual deep learning approach
- What Shapes Brain Alignment of Large Language Models during Dialogue? Insights from Language Production and Comprehension
- CrossScale-GLIO: Topology-Preserving Vision-Language Alignment of MRI and Whole-Slide Histopathology for Diffuse Glioma
- A Low-Rank Space Captures Linear Alignment Between LLM and Human Neural Activity During Naturalistic Language Listening
- Generative Language Modeling for Antibody CDR Grafting and Alignment-driven De Novo Design
BibTeX
@article{gezawa2026crossmodal,
title = {{Cross-modal alignment with language tokens for unified multilingual image-to-text generation.}},
author = {Abubakar Sulaiman Gezawa and Naveed Ur Rehman Junejo and Saud Adam Abdulkadir and Chibiao Liu and Guanjun Lin},
journal = {Frontiers in computational neuroscience},
year = {2026},
month = oct,
doi = {10.3389/fncom.2026.1826455},
url = {https://europepmc.org/article/MED/42952814},
note = {CC BY},
}
RIS
TY - JOUR
TI - Cross-modal alignment with language tokens for unified multilingual image-to-text generation.
AU - Abubakar Sulaiman Gezawa
AU - Naveed Ur Rehman Junejo
AU - Saud Adam Abdulkadir
AU - Chibiao Liu
AU - Guanjun Lin
PY - 2026
DA - 2026/10/14
JO - Frontiers in computational neuroscience
DO - 10.3389/fncom.2026.1826455
UR - https://europepmc.org/article/MED/42952814
AB - Current multimodal translation systems typically employ a cascaded pipeline where image captioning and machine translation are performed sequentially, leading to error propagation and semantic drift. This paper introduces a unified neural architecture that integrates visual encoding, English caption generation and multilingual translation into a single end-to-end model. Our first key contribution is this unified framework which jointly optimizes both tasks through shared latent representations enabling richer cross-modal learning and more semantically consistent multilingual outputs than conventional systems. The architecture incorporates two core technical innovations. First, we introduce a cross-modal representation alignment mechanism that directly fuses hierarchical visual features from a CNN encoder with multilingual linguistic embeddings in the Transformer decoder. This ensures visual semantics (e.g., gender, count, and object attributes) are preserved during translation effectively grounding multilingual outputs in the visual context and rectifying a common failure mode of text-only NMT. Second, we design a language token driven conditioning strategy prepending learnable target language tokens to guide a single multilingual model. This allows dynamic adaptation to diverse syntax and morphology without requiring separate decoders for each language. Empirical evaluations on MS COCO and OPUS multilingual datasets demonstrate notable improvements over existing multimodal and cascaded baselines achieving a BLEU-4 score of 44.8 for English captioning and an average BLEU of 41.1 across five target languages (Chinese, French, Arabic, Spanish, and Hindi), representing gains of 2.2 and 2.0 points, respectively, over cascaded baselines. COMET scores averaging 0.732 further confirm the semantic fidelity of our visually grounded translations. The proposed framework establishes a scalable, visually grounded and linguistically adaptive solution for multilingual caption generation contributing a meaningful advancement toward integrated vision-language translation research and real-world accessibility applications.
SN - 1662-5188
ER -