Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too

Abstract

Post-training pruning enables efficient deployment, but conventional methods do not explicitly preserve internal-gradient alignment between sparse and dense models. We study novel connections between downstream performance of compressed models and the ability to retain gradient information via influence-function preservation. Central to this connection is a curvature-aware objective that accounts for both layer activations and backpropagated gradients. Despite its promise, this objective remains underexplored in pruning due to a lack of efficient solvers. We introduce CACTUS, a novel ADMM-based optimizer supporting both pruning and sparse-plus-low-rank decompositions of pre-trained weights. Compared to LLM-Surgeon which targets the same objective, CACTUS can achieve over an order-of-magnitude speedup. At 50% pruning of Qwen3-4B, CACTUS reduces the C4 perplexity gap to the dense model by 45%, improves compressed-to-dense per-sample gradient cosine similarity on WikiText-2 from $0.670$ to $0.755$, and runs over 28× faster than LLM-Surgeon on substantially weaker hardware. By preserving influence functions, CACTUS supports effective data selection. On Llama-3.2-3B, fine-tuning a fresh dense model on the 5% of data selected by CACTUS achieves 49.44% accuracy on BIG-Bench Hard, compared with 47.22% using ALPS and 49.54% using dense-model selection. Finally, integrating CACTUS with LoGRA, a recent gradient-projection strategy, on Qwen3-4B with MLP layers pruned to 2:4 sparsity and a rank-64 correction retains a $0.711$ cosine alignment with dense LoGRA-projected C4 gradients.

Convexity of a ratio function involving polygamma functions and monotonicity of the sectional curvature of the Fisher--Rao manifold
Feng Qi
arXiv28 Oct 2026PreprintCC BY 4.0Full textGeneral mathematicsarXiv
Remarks on prescribed scalar curvature functions
Mingxiang Li, Quoc Anh Ngo, Xingwang Xu
arXiv14 Oct 2026PreprintarXiv licenceFull textAnalysis & PDEs Geometry & topologyarXiv
Idleness Functions for Ollivier-Ricci Curvature on Hypergraphs
Qing Xia
arXiv29 Sep 2026PreprintarXiv licenceFull textCombinatorics & discrete mathematics ProbabilityarXiv
On the Fourier Entropy-Influence Conjecture for Boolean Plateaued Functions
Vladimir N. Potapov
arXiv18 Sep 2026PreprintarXiv licenceFull textCombinatorics & discrete mathematicsarXiv
A Dual Representation of Influence Functions for Linearizable Models
Zhenhuan Sun, Shahrokh Valaee
arXiv25 Sep 2026PreprintCC BY 4.0Full textStatisticsarXiv

Cite

BibTeX RIS

BibTeX

@misc{makni2026price,
  title = {{Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too}},
  author = {Mehdi Makni and Natalia Ponomareva and Rahul Mazumder},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.18516},
  eprint = {2610.18516},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.18516},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too
AU  - Mehdi Makni
AU  - Natalia Ponomareva
AU  - Rahul Mazumder
PY  - 2026
DA  - 2026/10/13
PB  - arXiv
DO  - 10.48550/arxiv.2610.18516
UR  - https://arxiv.org/abs/2610.18516
AB  - Post-training pruning enables efficient deployment, but conventional methods do not explicitly preserve internal-gradient alignment between sparse and dense models. We study novel connections between downstream performance of compressed models and the ability to retain gradient information via influence-function preservation. Central to this connection is a curvature-aware objective that accounts for both layer activations and backpropagated gradients. Despite its promise, this objective remains underexplored in pruning due to a lack of efficient solvers. We introduce CACTUS, a novel ADMM-based optimizer supporting both pruning and sparse-plus-low-rank decompositions of pre-trained weights. Compared to LLM-Surgeon which targets the same objective, CACTUS can achieve over an order-of-magnitude speedup. At 50% pruning of Qwen3-4B, CACTUS reduces the C4 perplexity gap to the dense model by 45%, improves compressed-to-dense per-sample gradient cosine similarity on WikiText-2 from $0.670$ to $0.755$, and runs over 28× faster than LLM-Surgeon on substantially weaker hardware. By preserving influence functions, CACTUS supports effective data selection. On Llama-3.2-3B, fine-tuning a fresh dense model on the 5% of data selected by CACTUS achieves 49.44% accuracy on BIG-Bench Hard, compared with 47.22% using ALPS and 49.54% using dense-model selection. Finally, integrating CACTUS with LoGRA, a recent gradient-projection strategy, on Qwen3-4B with MLP layers pruned to 2:4 sparsity and a rank-64 correction retains a $0.711$ cosine alignment with dense LoGRA-projected C4 gradients.
ER  -

Abstract

Related

Cite