Publications

Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs

The 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), Main Track, Acceptance rate 15.4%, Budapest, Hungary, 2026.

Seungwoo Jung, Dohyeok Kwon, Seungmin Cha, Junseok Lee, Yeonho Yoo, Chuck Yoo, Gyeongsik Yang

Download PDF

Abstract

Mixture-of-experts (MoE) architectures scale large language models efficiently, but they demand massive GPU memory. To cope with such demand, models are commonly compressed to reduce their memory footprint. Residual sparsification is a representative compression technique that decomposes each projection matrix of an expert into a shared base matrix and per-expert residual matrix, and then compresses the residuals. Existing sparsification methods compress each residual matrix independently by minimizing its compression error, thereby minimizing the error of each projection matrix. However, our analysis shows that this objective is misaligned with preserving model accuracy after compression. In an expert, the final output is produced through computations coupled across multiple projections and hidden representations. Therefore, even small errors in individual matrices can propagate through hidden representations and projection interactions, leading to large expert output errors and accuracy degradation. To address this misalignment, we propose PARSER, a new residual sparsification method that shifts the compression objective from minimizing isolated matrix errors to preserving the expert output error. PARSER achieves this by introducing output importance, which measures the actual contribution to the expert output error. Our experiments show that, compared with existing methods, PARSER narrows the accuracy gap to the uncompressed model by 1.41× on Qwen and 1.44× on DeepSeek, while achieving the same peak memory reduction.

Paper summary

Preserve each expert’s output

Compressing an expert model around the outputs it produces can save memory while retaining more of its accuracy.

Mixture-of-experts models activate only a few experts per token, yet their stored parameters still consume substantial GPU memory. Residual sparsification reduces that footprint by sharing a base matrix and compressing each expert’s remaining differences. The paper shows why minimizing errors in those matrices separately can still distort the expert’s final output: the projections interact through nonlinear hidden representations.

Score what a removed dimension changes

PARSER estimates output importance: how much an expert’s output changes when a hidden dimension loses its expert-specific residual contribution. It uses a calibration dataset to estimate this effect, accounting for the coupled up, gate and down projections.

A dimension’s contribution to the output guides its retention.

Scores are pooled across experts within each MoE layer. PARSER removes the lowest-scoring dimensions up to the compression budget, allowing more important experts to retain more dimensions. It removes the corresponding residual rows and columns together while keeping the shared base.

PARSER uses output-aware importance to select dimensions retained in compressed expert residuals.View full-size figure
PARSER uses output-aware importance to select dimensions retained in compressed expert residuals.

Accuracy at a comparable memory footprint

The main evaluation runs Qwen1.5-MoE-A2.7B and DeepSeek-V2-Lite on one NVIDIA B200 GPU. It averages zero-shot accuracy across seven tasks, using bfloat16 and batch size 64. The reported setting compresses 90% of residual dimensions within the last two-thirds of layers.

Against D2MoE, the strongest accuracy baseline in this comparison, PARSER narrows the gap to the uncompressed model by 1.41× on Qwen and 1.44× on DeepSeek. Its average per-task peak GPU memory is 24.26 GB and 28.97 GB respectively, comparable to the compressed baselines.

Average accuracy across seven tasks (%)
ModelUncompressedD2MoEPARSER
Qwen1.5-MoE-A2.7B59.1245.2249.24
DeepSeek-V2-Lite59.6441.6647.16

Key takeaways

  1. Compression error matters through the expert’s output. Matrix-by-matrix objectives miss interactions between projections that output importance captures.
  2. Experts need not lose equal proportions of their dimensions. Pooling importance scores within a layer lets the compression budget follow their relative contribution.
  3. The comparison preserves more accuracy at similar memory use. PARSER outperforms D2MoE on the reported seven-task averages for both evaluated models.