Understanding Performance Plateaus and Representation Collapse in 3D MRI Transformers Trained to Classify Brain Disease
- 1University of Southern California
Abstract
In medical imaging AI, we commonly assume that diagnostic classifiers will increase in accuracy if they are trained on more data, but in reality, their performance often plateaus. To understand this behavior, we ran a spectral analysis of 3D vision transformers, a common supervised deep learning model that can be trained to classify disease based on 3D brain MRI. For simpler classification methods such as linear discriminant analysis, the classifier performance AUC(N), when trained on a sample of size N, is known to depend on the encoder's eigenspectrum. Each class-discriminative direction in the latent space contributes only after the sample size N reaches a threshold, so the spectrum influences how performance scales with training data. Building on this theory, we show that supervised transformers compress almost all of their discriminative signal into one principal component (EVR@1 = 0.90--0.96): after this representation collapse happens, no further directions cross the threshold as training data increases, and accuracy plateaus. In principle, this plateau in performance could be due to either the encoder's learned feature space or the difficulty of the task. To distinguish these cases, we trained three types of supervised 3D transformers (ViT3D, MINiT, NIT) and, as a non-collapsing alternative, a frozen self-supervised encoder (DINO), to classify people with dementia or mild cognitive impairment (MCI) versus cognitively normal (CN) individuals from 3D brain MRI, using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). We froze each encoder and applied it to a new task, so that only the task difficulty varied. An easy task encoder (Dementia vs CN), transferred to the harder task (MCI vs CN), did not reproduce the plateau (MINiT threshold 9.2), but full retraining did (2758.5). The plateau is therefore due to the learned encoder, rather than task difficulty. Ranking the feature directions by the proportion of the disease signal they contain, we were able to identify a sign of representation collapse, i.e., a single dominant spike in the latent feature eigenspectrum. This spike was 11--26 times larger than predicted from the rest of the spectrum for the disease-trained (supervised) encoders, but was absent for DINO (0.66). To predict how transformer accuracy scales with the number of training scans, a large spike in the encoder spectrum suggests that the representation has collapsed and additional scans will not help; a small spike means that the signal is spread across many encoder latent space directions, and more training data should boost the classifier performance.
Related
- Community Assembly Mechanisms Facilitate Understanding of Microbial Responses to Habitat Transition along a Spatial Environmental Continuum in Alpine Plateaus
- PaECTER: Patent-level representation learning using citation-informed transformers
- Associations between stand structure and the predictability of multiple forest carbon pools in the transition zone between the Qinghai–Tibet and Loess Plateaus
- Combined active-passive remote sensing inversion of grassland aboveground biomass on the Qinghai-Tibet plateaus and exploration of its driving mechanisms with optical-radar-meteorological systems
- Vision Transformers Enable Advanced Plant Phenotyping in Controlled Environments
BibTeX
@misc{chattopadhyay2026understanding,
title = {{Understanding Performance Plateaus and Representation Collapse in 3D MRI Transformers Trained to Classify Brain Disease}},
author = {T. Chattopadhyay and K. Shelar and S. I. Thomopoulos and P. M. Thompson},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.15.759510},
url = {https://www.biorxiv.org/content/10.64898/2026.10.15.759510v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Understanding Performance Plateaus and Representation Collapse in 3D MRI Transformers Trained to Classify Brain Disease
AU - T. Chattopadhyay
AU - K. Shelar
AU - S. I. Thomopoulos
AU - P. M. Thompson
PY - 2026
DA - 2026/10/15
PB - bioRxiv
DO - 10.64898/2026.10.15.759510
UR - https://www.biorxiv.org/content/10.64898/2026.10.15.759510v1
AB - In medical imaging AI, we commonly assume that diagnostic classifiers will increase in accuracy if they are trained on more data, but in reality, their performance often plateaus. To understand this behavior, we ran a spectral analysis of 3D vision transformers, a common supervised deep learning model that can be trained to classify disease based on 3D brain MRI. For simpler classification methods such as linear discriminant analysis, the classifier performance AUC(N), when trained on a sample of size N, is known to depend on the encoder's eigenspectrum. Each class-discriminative direction in the latent space contributes only after the sample size N reaches a threshold, so the spectrum influences how performance scales with training data. Building on this theory, we show that supervised transformers compress almost all of their discriminative signal into one principal component (EVR@1 = 0.90--0.96): after this representation collapse happens, no further directions cross the threshold as training data increases, and accuracy plateaus. In principle, this plateau in performance could be due to either the encoder's learned feature space or the difficulty of the task. To distinguish these cases, we trained three types of supervised 3D transformers (ViT3D, MINiT, NIT) and, as a non-collapsing alternative, a frozen self-supervised encoder (DINO), to classify people with dementia or mild cognitive impairment (MCI) versus cognitively normal (CN) individuals from 3D brain MRI, using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). We froze each encoder and applied it to a new task, so that only the task difficulty varied. An easy task encoder (Dementia vs CN), transferred to the harder task (MCI vs CN), did not reproduce the plateau (MINiT threshold 9.2), but full retraining did (2758.5). The plateau is therefore due to the learned encoder, rather than task difficulty. Ranking the feature directions by the proportion of the disease signal they contain, we were able to identify a sign of representation collapse, i.e., a single dominant spike in the latent feature eigenspectrum. This spike was 11--26 times larger than predicted from the rest of the spectrum for the disease-trained (supervised) encoders, but was absent for DINO (0.66). To predict how transformer accuracy scales with the number of training scans, a large spike in the encoder spectrum suggests that the representation has collapsed and additional scans will not help; a small spike means that the signal is spread across many encoder latent space directions, and more training data should boost the classifier performance.
ER -