Adaptive Signed Transformations for Budgeted Transformer Memory: Learned Activation Routing, CReLU Sign Preservation, and Exponential Retention
Abstract
Adaptive Signed Transformations for Budgeted Transformer Memory proposes a Transformer memory architecture that combines learned routing over multiple mathematical transformations, sign-preserving CReLU representations, and learned exponential retention within a bounded working-memory system. The proposed pipeline is
Wx+b→adaptive transformation→CReLU→α-based retention→BWM.Wx+b \rightarrow \text{adaptive transformation} \rightarrow \text{CReLU} \rightarrow \alpha\text{-based retention} \rightarrow \text{BWM}.
The central hypothesis is that preserving both positive and negative activation evidence before learning a decay coefficient may provide a richer signal for deciding what information should be retained, compressed, or forgotten under a fixed memory budget.
The work defines parameter- and compute-matched ablations for evaluating routing behavior, signed delayed recall, retention versus delay, compression ratio, OOD performance, and memory-efficiency tradeoffs. It deliberately separates the proposed architecture from existing experimental evidence: the complete adaptive/CReLU/learned-retention pipeline has not yet been experimentally validated, while previously executed BWM experiments are reported only as supporting background.
Related
- Vector Field Invariance and Unitary Boundary Transformations in Spacetime Horizons: Resolving the Hawking-Einstein Information Preservation Debate
- Asymptotic preservation of an exponential Euler method under Hamiltonian fast advection with multiple critical points
- Complex-Plane-Inspired Magnitude Rectifier: An Exploratory Transformer Activation Study
- Mitofusin-2 overexpression in adipose tissue-derived mesenchymal stem cells protects the testis against testicular torsion-induced ischemia-reperfusion injury through preservation of mitochondrial homeostasis associated with AMPK activation.
- An explainable swarm-optimized hybrid GRU–Transformer framework with wavelet-based spatial–temporal feature modeling and activation function analysis for rainfall classification across Pakistan’s climatic regimes
BibTeX
@misc{kanth2026adaptive,
title = {{Adaptive Signed Transformations for Budgeted Transformer Memory: Learned Activation Routing, CReLU Sign Preservation, and Exponential Retention}},
author = {Chaman Kanth},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23621092},
url = {https://zenodo.org/records/23621092},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Adaptive Signed Transformations for Budgeted Transformer Memory: Learned Activation Routing, CReLU Sign Preservation, and Exponential Retention
AU - Chaman Kanth
PY - 2026
DA - 2026/10/24
PB - Zenodo
DO - 10.5281/zenodo.23621092
UR - https://zenodo.org/records/23621092
AB - Adaptive Signed Transformations for Budgeted Transformer Memory proposes a Transformer memory architecture that combines learned routing over multiple mathematical transformations, sign-preserving CReLU representations, and learned exponential retention within a bounded working-memory system. The proposed pipeline is Wx+b→adaptive transformation→CReLU→α-based retention→BWM.Wx+b \rightarrow \text{adaptive transformation} \rightarrow \text{CReLU} \rightarrow \alpha\text{-based retention} \rightarrow \text{BWM}. The central hypothesis is that preserving both positive and negative activation evidence before learning a decay coefficient may provide a richer signal for deciding what information should be retained, compressed, or forgotten under a fixed memory budget. The work defines parameter- and compute-matched ablations for evaluating routing behavior, signed delayed recall, retention versus delay, compression ratio, OOD performance, and memory-efficiency tradeoffs. It deliberately separates the proposed architecture from existing experimental evidence: the complete adaptive/CReLU/learned-retention pipeline has not yet been experimentally validated, while previously executed BWM experiments are reported only as supporting background.
ER -