PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compression
Abstract
Mixture-of-Experts (MoE) models have emerged as an effective paradigm for scaling large language models, yet their massive parameter footprint poses severe memory challenges for deployment. Existing compression methods typically treat expert pruning and merging as disjoint operations and rely on greedy heuristics, often leading to irreversible information loss or representational pollution. In this paper, we propose PRIME, a unified optimization framework for MoE compression via adaptive PRunIng and MErging. PRIME casts the discrete prune-merge assignment step as a global Integer Linear Programming over a frequency-weighted surrogate objective, jointly selecting pruning and merging decisions while accounting for expert usage. We motivate the frequency-aware objective through an error-bound analysis under fixed representative assignments, and show that the formulation favors high-frequency experts as centroids, promoting routing stability. Extensive experiments on Qwen3-30B-A3B and Qwen3.5-122B-A10B demonstrate that PRIME achieves strong accuracy retention across knowledge, coding, math, and long-context tasks, while serving benchmarks of merged models quantify the memory and throughput benefits of expert compression in vLLM.
Related
- Dependency, Compression, and Synergy: A Unified Information-Theoretic View of Multimodal Learning
- IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs
- PMR-Aided Shifted-Pruning SCL-Flip Decoding for CRC-Aided Polar Codes
- EntroPrefill: Renyi-Guided Context Pruning with Conditional Stability Guarantees for Retrieval-Augmented Generation
- Geometry-induced formation and merging of multiple dead zones in diffusion-reaction problems
BibTeX
@misc{huang2026prime,
title = {{PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compression}},
author = {Shanfeng Huang and Rongqian Zhang and Zhou Fang and Song Xiao and Hai Du},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.13982},
eprint = {2610.13982},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.13982},
}
RIS
TY - UNPB
TI - PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compression
AU - Shanfeng Huang
AU - Rongqian Zhang
AU - Zhou Fang
AU - Song Xiao
AU - Hai Du
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.13982
UR - https://arxiv.org/abs/2610.13982
AB - Mixture-of-Experts (MoE) models have emerged as an effective paradigm for scaling large language models, yet their massive parameter footprint poses severe memory challenges for deployment. Existing compression methods typically treat expert pruning and merging as disjoint operations and rely on greedy heuristics, often leading to irreversible information loss or representational pollution. In this paper, we propose PRIME, a unified optimization framework for MoE compression via adaptive PRunIng and MErging. PRIME casts the discrete prune-merge assignment step as a global Integer Linear Programming over a frequency-weighted surrogate objective, jointly selecting pruning and merging decisions while accounting for expert usage. We motivate the frequency-aware objective through an error-bound analysis under fixed representative assignments, and show that the formulation favors high-frequency experts as centroids, promoting routing stability. Extensive experiments on Qwen3-30B-A3B and Qwen3.5-122B-A10B demonstrate that PRIME achieves strong accuracy retention across knowledge, coding, math, and long-context tasks, while serving benchmarks of merged models quantify the memory and throughput benefits of expert compression in vLLM.
ER -