Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
Abstract
Normalization layers in modern large language models (LLMs) consist of a deterministic normalization operation and a learnable scale vector. While the normalization operation has been extensively studied, the scale vector remains poorly understood despite its ubiquitous use. In this work, we present a systematic study of scale vectors in LLMs from the perspectives of expressivity, optimization, and architectural structure. First, we show empirically that although scale vectors constitute only a negligible fraction of model parameters, removing them substantially degrades LLM pre-training. Our theory further shows that, in Pre-Norm architectures, scale vectors do not increase expressivity; instead, they improve optimization through a self-amplifying preconditioning effect on subsequent linear mappings. Second, we investigate the role of weight decay for scale vectors. By distinguishing Input-Norm and Output-Norm layers, we theoretically show that weight decay is beneficial for the former but harmful for the latter, due to their distinct roles in optimization and expressivity. Third, motivated by this understanding, we propose three lightweight and complementary improvements to scale vectors: branch-specific heterogeneity, improved placement around linear mappings, and magnitude-direction reparameterization. Both theory and experiments show that each improvement yields consistent gains. Finally, we combine these improvements into a unified scale-vector strategy and evaluate it through extensive LLM pre-training experiments on dense and mixture-of-experts models ranging from 0.12B to 2B parameters, across multiple optimizers and learning rate schedules, under industrial-scale token budgets. The unified strategy consistently achieves lower terminal loss than well-tuned baselines and exhibits more favorable scaling behavior, while adding negligible parameter and computational overhead.
Related
- Large-Scale Comparison of Metaheuristic Algorithms via Behavioral Verification of Large Language Model-Generated Codes
- Large signed sums of unit vectors: the first linearly dependent case
- Large smooth twins from short lattice vectors
- Limiting moments of sample covariance matrices from large fold tensor vectors
- Do I Even Need a Multilevel Model? Negligible Effect Significance Testing for Intraclass Correlation Coefficients
BibTeX
@misc{wang2026negligible,
title = {{Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models}},
author = {Mingze Wang and Shuchen Zhu and Yuxin Fang and Binghui Li and Kai Shen and Shu Zhong},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2605.26895},
eprint = {2605.26895},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.26895},
}
RIS
TY - UNPB
TI - Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
AU - Mingze Wang
AU - Shuchen Zhu
AU - Yuxin Fang
AU - Binghui Li
AU - Kai Shen
AU - Shu Zhong
PY - 2026
DA - 2026/08/28
PB - arXiv
DO - 10.48550/arxiv.2605.26895
UR - https://arxiv.org/abs/2605.26895
AB - Normalization layers in modern large language models (LLMs) consist of a deterministic normalization operation and a learnable scale vector. While the normalization operation has been extensively studied, the scale vector remains poorly understood despite its ubiquitous use. In this work, we present a systematic study of scale vectors in LLMs from the perspectives of expressivity, optimization, and architectural structure. First, we show empirically that although scale vectors constitute only a negligible fraction of model parameters, removing them substantially degrades LLM pre-training. Our theory further shows that, in Pre-Norm architectures, scale vectors do not increase expressivity; instead, they improve optimization through a self-amplifying preconditioning effect on subsequent linear mappings. Second, we investigate the role of weight decay for scale vectors. By distinguishing Input-Norm and Output-Norm layers, we theoretically show that weight decay is beneficial for the former but harmful for the latter, due to their distinct roles in optimization and expressivity. Third, motivated by this understanding, we propose three lightweight and complementary improvements to scale vectors: branch-specific heterogeneity, improved placement around linear mappings, and magnitude-direction reparameterization. Both theory and experiments show that each improvement yields consistent gains. Finally, we combine these improvements into a unified scale-vector strategy and evaluate it through extensive LLM pre-training experiments on dense and mixture-of-experts models ranging from 0.12B to 2B parameters, across multiple optimizers and learning rate schedules, under industrial-scale token budgets. The unified strategy consistently achieves lower terminal loss than well-tuned baselines and exhibits more favorable scaling behavior, while adding negligible parameter and computational overhead.
ER -