What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models
Abstract
Vision--language models (VLMs) promise to make computational pathology queryable and able to draft reports, but how much language adds beyond the vision backbone is unclear, because almost every system is evaluated by its developers on test sets of their own choosing. We grade published evidence by how it was produced and test four claims with controlled probes that reuse released checkpoints and public data, varying one input at a time. Report text adds prognostic signal: adding a frozen embedding of the stage-redacted report to a slide embedding raises the pooled C-index by 0.019 for progression-free interval and by 0.016 for overall survival across 1{,}633 TCGA patients, and restoring the stage statement adds nothing. Prompts add no measurable accuracy at inference: with features and pooling fixed, zero-shot classification is statistically indistinguishable from a supervised linear probe for lung and renal cancer subtyping. Slide-level question answering is barely grounded: SlideChat answers 74.82\% of SlideBench-VQA(TCGA) items correctly with the slide and 74.41\% without it, whereas its language backbone alone reaches 43.52\%. Paraphrasing leaves accuracy unchanged but flips 8--23\% of answers. Self-reported gains from agentic navigation and few-shot language priors cannot yet be attributed to language. Language thus adds real but narrow value, and we propose minimum controls that would expose these failure modes in future evaluations.
Related
- What’s taking so long for conscious vision?
- A Vision-Language Model as a Teacher for Bird Vocalization Detection
- MEMOIR-VLM—a multimodal vision-language model for Alzheimer's disease classification and question answering
- What we can learn from nature to restore human vision
- On Color and Emotion: What Can Vision Science Learn from the Rorschach?
BibTeX
@misc{zhang2026what,
title = {{What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models}},
author = {Wenhao Zhang and Zhongliang Zhou and Yi Ping and Zirui Wang and Sheng Li},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.26805},
eprint = {2610.26805},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.26805},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models
AU - Wenhao Zhang
AU - Zhongliang Zhou
AU - Yi Ping
AU - Zirui Wang
AU - Sheng Li
PY - 2026
DA - 2026/10/19
PB - arXiv
DO - 10.48550/arxiv.2610.26805
UR - https://arxiv.org/abs/2610.26805
AB - Vision--language models (VLMs) promise to make computational pathology queryable and able to draft reports, but how much language adds beyond the vision backbone is unclear, because almost every system is evaluated by its developers on test sets of their own choosing. We grade published evidence by how it was produced and test four claims with controlled probes that reuse released checkpoints and public data, varying one input at a time. Report text adds prognostic signal: adding a frozen embedding of the stage-redacted report to a slide embedding raises the pooled C-index by 0.019 for progression-free interval and by 0.016 for overall survival across 1{,}633 TCGA patients, and restoring the stage statement adds nothing. Prompts add no measurable accuracy at inference: with features and pooling fixed, zero-shot classification is statistically indistinguishable from a supervised linear probe for lung and renal cancer subtyping. Slide-level question answering is barely grounded: SlideChat answers 74.82\% of SlideBench-VQA(TCGA) items correctly with the slide and 74.41\% without it, whereas its language backbone alone reaches 43.52\%. Paraphrasing leaves accuracy unchanged but flips 8--23\% of answers. Self-reported gains from agentic navigation and few-shot language priors cannot yet be attributed to language. Language thus adds real but narrow value, and we propose minimum controls that would expose these failure modes in future evaluations.
ER -