Representation geometry and task performance in vision-language modeling for computed tomography enterography
4 affiliations
- 1University of Michigan, Gilbert S. Omenn Department of Computational Medicine and Bioinformatics, Ann Arbor, Michigan, United States
- 2University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan, United States
- 3University of Michigan, Department of Electrical Engineering and Computer Science, Ann Arbor, Michigan, United States
- 4University of Michigan, Department of Gastroenterology, Ann Arbor, Michigan, United States
Abstract
Purpose: Computed tomography (CT) enterography is central to inflammatory bowel disease evaluation, but optimal representation choices for vision–language learning on this volumetric modality are unclear. We establish baseline performance and quantify how aggregation, slice encoding, and retrieval context affect classification, retrieval, and report generation.
Approach: We fine-tuned vision–language models on abdominal CT enterography with parameter-efficient adapters and compared (i) mean versus attention pooling of slice embeddings, (ii) multiwindow red–green–blue Hounsfield-unit encoding versus multiplanar sampling for added spatial coverage, and (iii) report generation with versus without retrieval-augmented generation (RAG). A three-teacher pseudolabel framework enabled evaluation without expert annotations. Classification and retrieval comparisons used three training seeds.
Results: Mean and attention pooling achieved the same mean three-class disease classification accuracy (55.7%; standard deviations of 2.0 and 2.4 percentage points) at the largest low-rank adaptation capacity. Neither consistently outperformed the other at matched capacity with central sampling. Sixteen slices at 20% to 80% had higher classification accuracy than 8 slices, 32 slices, and 10% to 90% sampling in all three seeds for both pooling methods. The tested multiplanar configurations had lower classification accuracy. For report generation, fine-tuning without retrieval performed at prevalence-matched chance for within-1 report-label agreement (70.4% versus 71.0% random). RAG improved within-1 report-label agreement by 7 to 14 points over chance and reduced ordinal mean absolute error from 0.98 to 0.80–0.89.
Conclusions: Slice sampling and tissue-contrast–focused encoding affect performance in CT enterography vision–language systems, and RAG improved report-label agreement over the fine-tuned baseline. These results provide initial baselines and practical guidance for multimodal learning in volumetric CT.
Related
- Clinical Usefulness of Computed Tomography Enterography as a Diagnostic Modality for Isolated Terminal Ileitis
- Deep Learning Optimizes the Diagnostic Performance of Coronary Computed Tomography Angiography in Heavily Calcified Coronary Plaques: A Chinese Multicenter Study
- Recent Advances in Knowledge Representation and Management in Healthcare: the Emerging Role of Large Language Models
- Distinguishing Post-LASIK Ablation Patterns on Corneal Tomography: Comparing Clinician Versus Large Language Model
- Semi-supervised clustering with knowledge-guided representation learning in cryo-electron tomography
BibTeX
@article{minoccheri2026representation,
title = {{Representation geometry and task performance in vision-language modeling for computed tomography enterography}},
author = {Cristian Minoccheri and Emily Wittrup and Kayvan Najarian and Ryan Stidham},
journal = {Journal of Medical Imaging},
year = {2026},
month = nov,
doi = {10.1117/1.jmi.13.6.064501},
url = {https://doi.org/10.1117/1.jmi.13.6.064501},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Representation geometry and task performance in vision-language modeling for computed tomography enterography
AU - Cristian Minoccheri
AU - Emily Wittrup
AU - Kayvan Najarian
AU - Ryan Stidham
PY - 2026
DA - 2026/11/10
JO - Journal of Medical Imaging
DO - 10.1117/1.jmi.13.6.064501
UR - https://doi.org/10.1117/1.jmi.13.6.064501
AB - Purpose: Computed tomography (CT) enterography is central to inflammatory bowel disease evaluation, but optimal representation choices for vision–language learning on this volumetric modality are unclear. We establish baseline performance and quantify how aggregation, slice encoding, and retrieval context affect classification, retrieval, and report generation. Approach: We fine-tuned vision–language models on abdominal CT enterography with parameter-efficient adapters and compared (i) mean versus attention pooling of slice embeddings, (ii) multiwindow red–green–blue Hounsfield-unit encoding versus multiplanar sampling for added spatial coverage, and (iii) report generation with versus without retrieval-augmented generation (RAG). A three-teacher pseudolabel framework enabled evaluation without expert annotations. Classification and retrieval comparisons used three training seeds. Results: Mean and attention pooling achieved the same mean three-class disease classification accuracy (55.7%; standard deviations of 2.0 and 2.4 percentage points) at the largest low-rank adaptation capacity. Neither consistently outperformed the other at matched capacity with central sampling. Sixteen slices at 20% to 80% had higher classification accuracy than 8 slices, 32 slices, and 10% to 90% sampling in all three seeds for both pooling methods. The tested multiplanar configurations had lower classification accuracy. For report generation, fine-tuning without retrieval performed at prevalence-matched chance for within-1 report-label agreement (70.4% versus 71.0% random). RAG improved within-1 report-label agreement by 7 to 14 points over chance and reduced ordinal mean absolute error from 0.98 to 0.80–0.89. Conclusions: Slice sampling and tissue-contrast–focused encoding affect performance in CT enterography vision–language systems, and RAG improved report-label agreement over the fine-tuned baseline. These results provide initial baselines and practical guidance for multimodal learning in volumetric CT.
SN - 2329-4302
SN - 2329-4310
ER -