Activation Flow: Manufacturing Activations for Steering
Abstract
Difference-in-means steering requires activations recorded while a model shows the desired behavior, which a sandbagging model withholds by deliberately underperforming. We introduce Activation Flow (ActFlow), which manufactures these activations from k correct labels without fine-tuning. ActFlow sets target logits that rank each labeled item's correct answer first, and moves the logits toward them by adding one vector x to all k residual streams at one layer. ActFlow is a family of ordinary differential equations for x, one for each rule that maps the required logit change to the velocity of x. The smallest-norm rule lands exactly on the targets, while the others keep only the top singular directions of the Jacobian. We test ActFlow on three instruction-tuned models, each locked by a sandbagging prompt and by a password-locked LoRA. At k=40, ActFlow keeping five singular directions raises the mean held-out ARC-Easy accuracy over the six locked models from $0.05$ to $0.85$, against $0.88$ for rank-$16$ LoRA fine-tuning, which trains about 3,000 times as many parameters, and $0.92$ for the honest models. Furthermore, it scores higher than the smallest-norm rule in 16 of the 18 combinations of locked model and k, and its steering direction is nearly orthogonal to the honest difference-in-means direction. It also unlocks two LoRA locks where the honest direction fails. Surprisingly, with one labeled item, the single-step smallest-norm rule raises the Qwen2.5-7B prompt lock from $0.04$ to $0.59$.
Related
- Radon Measure Representations for Infinite-Width Neural Networks with Singular Activations
- What Do Sparse Autoencoder Feature Discovery Scores Identify? From Activations to Interventions
- Fully Deterministic Shallow Neural Networks with Localized Activations. Part II: Condition Number Estimates
- More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
- Fully Deterministic Shallow Neural Networks with Localized Activations. Part I: Construction and Optimal Sobolev Approximation
BibTeX
@misc{tan2026activation,
title = {{Activation Flow: Manufacturing Activations for Steering}},
author = {Hong Kiat Tan and Linh Le and David Williams-King},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.32530},
eprint = {2609.32530},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.32530},
}
RIS
TY - UNPB
TI - Activation Flow: Manufacturing Activations for Steering
AU - Hong Kiat Tan
AU - Linh Le
AU - David Williams-King
PY - 2026
DA - 2026/10/02
PB - arXiv
DO - 10.48550/arxiv.2609.32530
UR - https://arxiv.org/abs/2609.32530
AB - Difference-in-means steering requires activations recorded while a model shows the desired behavior, which a sandbagging model withholds by deliberately underperforming. We introduce Activation Flow (ActFlow), which manufactures these activations from k correct labels without fine-tuning. ActFlow sets target logits that rank each labeled item's correct answer first, and moves the logits toward them by adding one vector x to all k residual streams at one layer. ActFlow is a family of ordinary differential equations for x, one for each rule that maps the required logit change to the velocity of x. The smallest-norm rule lands exactly on the targets, while the others keep only the top singular directions of the Jacobian. We test ActFlow on three instruction-tuned models, each locked by a sandbagging prompt and by a password-locked LoRA. At k=40, ActFlow keeping five singular directions raises the mean held-out ARC-Easy accuracy over the six locked models from $0.05$ to $0.85$, against $0.88$ for rank-$16$ LoRA fine-tuning, which trains about 3,000 times as many parameters, and $0.92$ for the honest models. Furthermore, it scores higher than the smallest-norm rule in 16 of the 18 combinations of locked model and k, and its steering direction is nearly orthogonal to the honest difference-in-means direction. It also unlocks two LoRA locks where the honest direction fails. Surprisingly, with one labeled item, the single-step smallest-norm rule raises the Qwen2.5-7B prompt lock from $0.04$ to $0.59$.
ER -