Head-level lesion-symptom mapping of picture naming in vision-language models
- 1University of South Carolina
Abstract
Researchers in artificial intelligence increasingly intervene on language models to study how their functions are organized, silencing weights and attention components in ways reminiscent of the brain lesions long used to map language in post-stroke aphasia. Under this program of mechanistic interpretability, a common move is to ablate an attention head and read the resulting drop in a behavior as evidence that the head implements it. How much that move can establish about where a behavior is computed remains unclear, because a head whose removal disrupts a behavior is necessary for it but need not be the site where it is computed. We examined this question for the picture naming task, taking the loss of naming (anomia) that defines aphasia as the behavior of interest. Across six vision-language models, spanning three language backbones and a range of parameter scales, we ablated each attention head in turn during single-word picture naming and measured accuracy before and after. The degree of localization varied widely across models. In LLaVA-1.6-Vicuna-13B, a single early head (layer 0, head 20) was necessary: removing it alone reduced naming accuracy from 0.99 to 0.006. The same head was not sufficient, because retaining it while ablating the other 1{,}599 heads also produced 0\% accuracy. Two Mistral-backbone models (LLaVA-Mistral-7B and Idefics2-8B) had no critical head. An early-layer dependence was present in every model but varied in strength, whereas dependence on any single head ranged from dominant to absent. Within Qwen2.5-VL, a dominant head was present in the 7B model but not the 3B model, indicating that this concentration emerged with scale rather than being fixed across a model family, and it was not explained by attention type. These results show that ablating the head whose removal disrupts naming does not establish that the head computes the behavior, that the result generalizes across models, or that the behavior localizes to a head at all.
Related
- The Role of the Left Inferior Frontal Gyrus in Resolving Immediate and Carryover Semantic Interference During Picture Naming
- Robust lesion network mapping reveals symptom-specific circuits
- Microstructure of white matter association tracts in the temporal lobe predicts the timing of electrophysiological activity and behaviour in picture naming
- Color naming needs a non-uniform perceptual prior: Individual differences in hue scaling and the vision–language interface
- Beyond one-to-one mappings: Modelling distributed lesion-symptom relationships with multilayer networks
BibTeX
@misc{nemati2026headlevel,
title = {{Head-level lesion-symptom mapping of picture naming in vision-language models}},
author = {S. Nemati and R. D. Newman-Norlund and S. Ahmadi and X. Guan and K. Warren and Y. Yang and S. Nelakuditi and C. Rorden and L. Bonilha and J. Fridriksson},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.15.751760},
url = {https://www.biorxiv.org/content/10.64898/2026.09.15.751760v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Head-level lesion-symptom mapping of picture naming in vision-language models
AU - S. Nemati
AU - R. D. Newman-Norlund
AU - S. Ahmadi
AU - X. Guan
AU - K. Warren
AU - Y. Yang
AU - S. Nelakuditi
AU - C. Rorden
AU - L. Bonilha
AU - J. Fridriksson
PY - 2026
DA - 2026/09/15
PB - bioRxiv
DO - 10.64898/2026.09.15.751760
UR - https://www.biorxiv.org/content/10.64898/2026.09.15.751760v1
AB - Researchers in artificial intelligence increasingly intervene on language models to study how their functions are organized, silencing weights and attention components in ways reminiscent of the brain lesions long used to map language in post-stroke aphasia. Under this program of mechanistic interpretability, a common move is to ablate an attention head and read the resulting drop in a behavior as evidence that the head implements it. How much that move can establish about where a behavior is computed remains unclear, because a head whose removal disrupts a behavior is necessary for it but need not be the site where it is computed. We examined this question for the picture naming task, taking the loss of naming (anomia) that defines aphasia as the behavior of interest. Across six vision-language models, spanning three language backbones and a range of parameter scales, we ablated each attention head in turn during single-word picture naming and measured accuracy before and after. The degree of localization varied widely across models. In LLaVA-1.6-Vicuna-13B, a single early head (layer 0, head 20) was necessary: removing it alone reduced naming accuracy from 0.99 to 0.006. The same head was not sufficient, because retaining it while ablating the other 1{,}599 heads also produced 0\% accuracy. Two Mistral-backbone models (LLaVA-Mistral-7B and Idefics2-8B) had no critical head. An early-layer dependence was present in every model but varied in strength, whereas dependence on any single head ranged from dominant to absent. Within Qwen2.5-VL, a dominant head was present in the 7B model but not the 3B model, indicating that this concentration emerged with scale rather than being fixed across a model family, and it was not explained by attention type. These results show that ablating the head whose removal disrupts naming does not establish that the head computes the behavior, that the result generalizes across models, or that the behavior localizes to a head at all.
ER -