From Images and Words to Modality-Independent Semantic Representations: When and Where They Emerge
- 1University of Geneva
Abstract
Conceptual knowledge can be accessed from different modalities, such as seeing an image or reading a written word. While prior neuroimaging studies implicate anterior temporal cortex in modality-independent semantic representations, when and how representations evoked by images and words converge onto a modality-independent code remains unclear. Moreover, evidence for modality-independent category representations comes largely from tasks that explicitly required semantic processing, leaving open whether such representations emerge when semantic category and stimulus modality are task-irrelevant. We magnetoencephalography (MEG) to determine when and where modality-independent semantic category representations emerge across images and words when participants (n = 32) encoded six-item sequences containing an exemplar of six semantic categories counterbalanced across positions and reported the ordinal position of a probed item. Decoding performance was above chance for all categories in each modality from approximately 70 ms relative to item presentation. Generalization peaked at similar latencies for the two modalities (~520-530 ms), where image and word classifiers had the most similar coefficients. In both transfer directions, activity elicited by images generalized mostly to later activity elicited by words, and a given period of the image response shared its category code with a longer period of the word response. Source-space decoding around the peak of generalization identified the same regions for both transfer directions: left and right anterior temporal lobe, perisylvian cortex (more extensive in the left hemisphere), right visual cortex, and bilateral orbitofrontal cortex, indicating a modality-independent category code distributed across the cortex.
Related
- Multidimensional semantic representations emerge from multi-frequency representational similarity learning
- Modality-Independent Core Brain Network for Language as Proved by Sign Language
- Shared neural geometries for bilingual semantic representations in human hippocampal neurons
- ACMNet: An attention-convolution mixed neural network for semantic segmentation of high-resolution remote sensing images
- When meaning becomes decodable: Linking neural markers of semantic processing and semantic representations
BibTeX
@misc{piron2026from,
title = {{From Images and Words to Modality-Independent Semantic Representations: When and Where They Emerge}},
author = {N. Piron and T. Desbordes and I. M. Olasagasti and S. Schwartz and N. Kazanina},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.30.763035},
url = {https://www.biorxiv.org/content/10.64898/2026.10.30.763035v1},
note = {CC BY-ND},
}
RIS
TY - UNPB
TI - From Images and Words to Modality-Independent Semantic Representations: When and Where They Emerge
AU - N. Piron
AU - T. Desbordes
AU - I. M. Olasagasti
AU - S. Schwartz
AU - N. Kazanina
PY - 2026
DA - 2026/10/30
PB - bioRxiv
DO - 10.64898/2026.10.30.763035
UR - https://www.biorxiv.org/content/10.64898/2026.10.30.763035v1
AB - Conceptual knowledge can be accessed from different modalities, such as seeing an image or reading a written word. While prior neuroimaging studies implicate anterior temporal cortex in modality-independent semantic representations, when and how representations evoked by images and words converge onto a modality-independent code remains unclear. Moreover, evidence for modality-independent category representations comes largely from tasks that explicitly required semantic processing, leaving open whether such representations emerge when semantic category and stimulus modality are task-irrelevant. We magnetoencephalography (MEG) to determine when and where modality-independent semantic category representations emerge across images and words when participants (n = 32) encoded six-item sequences containing an exemplar of six semantic categories counterbalanced across positions and reported the ordinal position of a probed item. Decoding performance was above chance for all categories in each modality from approximately 70 ms relative to item presentation. Generalization peaked at similar latencies for the two modalities (~520-530 ms), where image and word classifiers had the most similar coefficients. In both transfer directions, activity elicited by images generalized mostly to later activity elicited by words, and a given period of the image response shared its category code with a longer period of the word response. Source-space decoding around the peak of generalization identified the same regions for both transfer directions: left and right anterior temporal lobe, perisylvian cortex (more extensive in the left hemisphere), right visual cortex, and bilateral orbitofrontal cortex, indicating a modality-independent category code distributed across the cortex.
ER -