Flow-based Spectral Kernel Learning for Nonstationary Attention
Abstract
Kernelized attention provides a useful perspective for understanding self-attention as a similarity-based aggregation mechanism, but existing approaches often rely on fixed or stationary kernels, limiting their ability to adapt the similarity structure to complex token interactions. In this paper, we propose flow-based spectral kernel attention (FSKA), a framework for learning expressive nonstationary attention kernels. FSKA formulates kernelized attention through spectral density learning and parameterizes the resulting bivariate spectral density with normalizing flows. This design enables flexible and regularizable kernel learning while remaining compatible with scalable linear-attention computation. We further show that expressive kernel learning enables a simplified attention design with removed query--key projections and fixed orthogonal value projections, resulting in a more parameter-efficient architecture. Controlled experiments on classification, ablation, and scalability profiling show that FSKA achieves competitive or improved performance over existing attention mechanisms, while reducing projection-related trainable parameters by approximately 40%--47% compared with most baselines and maintaining favorable linear-attention memory scaling. Code Available: https://github.com/ziyang18/FSKA.
Related
- Asymptotic spectral properties and extreme eigenvalues of sample correlation matrices from nonstationary high-dimensional VAR processes
- Quasi Linear Kernel Attention with Infinite Capacity
- Gradient estimates for the Green kernel under spectral Ricci bounds, and the stable Bernstein theorem in ℝ⁴
- Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms
- Uniform many-particle spectral gap inequality and heat kernel bounds for strong attractive interactions
BibTeX
@misc{yang2026flowbased,
title = {{Flow-based Spectral Kernel Learning for Nonstationary Attention}},
author = {Zi Yang and Ying Li and Alejandro Lancho and Darío C. Larese and Pablo M. Olmos and Michael Minyi Zhang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.30465},
eprint = {2610.30465},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.30465},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Flow-based Spectral Kernel Learning for Nonstationary Attention
AU - Zi Yang
AU - Ying Li
AU - Alejandro Lancho
AU - Darío C. Larese
AU - Pablo M. Olmos
AU - Michael Minyi Zhang
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.30465
UR - https://arxiv.org/abs/2610.30465
AB - Kernelized attention provides a useful perspective for understanding self-attention as a similarity-based aggregation mechanism, but existing approaches often rely on fixed or stationary kernels, limiting their ability to adapt the similarity structure to complex token interactions. In this paper, we propose flow-based spectral kernel attention (FSKA), a framework for learning expressive nonstationary attention kernels. FSKA formulates kernelized attention through spectral density learning and parameterizes the resulting bivariate spectral density with normalizing flows. This design enables flexible and regularizable kernel learning while remaining compatible with scalable linear-attention computation. We further show that expressive kernel learning enables a simplified attention design with removed query--key projections and fixed orthogonal value projections, resulting in a more parameter-efficient architecture. Controlled experiments on classification, ablation, and scalability profiling show that FSKA achieves competitive or improved performance over existing attention mechanisms, while reducing projection-related trainable parameters by approximately 40%--47% compared with most baselines and maintaining favorable linear-attention memory scaling. Code Available: https://github.com/ziyang18/FSKA.
ER -