Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram
Abstract
We introduce scale-selective Proper Orthogonal Decomposition (POD) for transformer attention fields, inspired by the use of POD for extracting energetically dominant modes from turbulent flow ensembles. The Morlet continuous wavelet transform identifies dominant temporal scales in the attention lag structure across a document ensemble; POD then extracts the energetically dominant modes at each scale from the ensemble of attention fields. The resulting modes reveal layer-dependent scale organisation, with early layers emphasising fine scales and later layers shifting toward coarser scales. We define a spectral concentration index from the POD eigenvalue decay rate and show empirically that it differentiates layers by their attention field complexity. By the classical POD optimality theorem, the extracted modes minimise the average L2 reconstruction error over the ensemble (Theorem 1), giving a data-driven effective rank for each layer. The method requires no architectural modification and no linguistic annotations: dominant attention patterns emerge from ensemble statistics alone. The turbulence analogy is structural rather than physical: we borrow ensemble covariance and modal analysis, not fluid dynamics itself.
Related
- A generative pre-trained transformer with Kerr-soliton attention
- Quantum Reservoir Computing for Attention-like Selective Reproduction
- Label-Free Finite-Volume-Residual Training of Attention Graph Neural Networks for Coupled Thermo-Fluid Fields
- SCANO: Restoring Boundary-Condition Completeness in Sampled Meshes with a Sliced-Cross-Attention Neural Operator for 3D Thermal Fields
- Continuous vision transformer-based neural operator for scalable beam propagation modeling in optical fibers
BibTeX
@misc{zeris2026multiscale,
title = {{Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram}},
author = {Athanasios Zeris},
howpublished = {arXiv},
year = {2026},
month = jun,
doi = {10.48550/arxiv.2606.06573},
eprint = {2606.06573},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.06573},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram
AU - Athanasios Zeris
PY - 2026
DA - 2026/06/04
PB - arXiv
DO - 10.48550/arxiv.2606.06573
UR - https://arxiv.org/abs/2606.06573
AB - We introduce scale-selective Proper Orthogonal Decomposition (POD) for transformer attention fields, inspired by the use of POD for extracting energetically dominant modes from turbulent flow ensembles. The Morlet continuous wavelet transform identifies dominant temporal scales in the attention lag structure across a document ensemble; POD then extracts the energetically dominant modes at each scale from the ensemble of attention fields. The resulting modes reveal layer-dependent scale organisation, with early layers emphasising fine scales and later layers shifting toward coarser scales. We define a spectral concentration index from the POD eigenvalue decay rate and show empirically that it differentiates layers by their attention field complexity. By the classical POD optimality theorem, the extracted modes minimise the average L2 reconstruction error over the ensemble (Theorem 1), giving a data-driven effective rank for each layer. The method requires no architectural modification and no linguistic annotations: dominant attention patterns emerge from ensemble statistics alone. The turbulence analogy is structural rather than physical: we borrow ensemble covariance and modal analysis, not fluid dynamics itself.
ER -