arxiv: 2604.04356 · v1 · submitted 2026-04-06 · 💻 cs.AI · cs.CL· cs.LG· cs.PF

Recognition: 2 theorem links

· Lean Theorem

REAM: Merging Improves Pruning of Experts in LLMs

Saurav Jha , Maryam Hashemzadeh , Ali Saheb Pasand , Ali Parviz , Min-Joong Lee , Boris Knyazev

Authors on Pith no claims yet

Pith reviewed 2026-05-10 19:51 UTC · model grok-4.3

classification 💻 cs.AI cs.CLcs.LGcs.PF

keywords Mixture-of-Expertsexpert mergingmodel compressionLLM pruningparameter reductioncalibration dataMoE optimization

0 comments

The pith

Merging groups of experts preserves more performance than pruning in MoE LLMs

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large Mixture-of-Experts models deliver strong results but demand too much memory for many deployment settings. Standard pruning drops low-activation experts to shrink size, yet it often degrades accuracy on both multiple-choice and open-ended tasks. REAM instead clusters experts by their router-weighted activation statistics on a calibration set and merges the weights within each cluster. Experiments across several MoE LLMs show that this merging approach frequently matches the original model more closely than pruning baselines, although the best balance between multiple-choice and generative scores shifts with the mix of general, math, and coding calibration data.

Core claim

By grouping experts according to router-weighted activation patterns and merging their weights rather than discarding them, REAM reduces expert count while retaining higher performance on both multiple-choice question answering and generative benchmarks than REAP or other baselines, and in many cases reaches accuracy levels comparable to the uncompressed original model when the calibration data mix is controlled.

What carries the argument

Router-weighted Expert Activation Merging (REAM), which groups experts by activation statistics on a calibration set and combines their parameters instead of removing experts.

If this is right

REAM delivers higher retained accuracy than expert pruning on the same set of MoE architectures and benchmarks.
The relative strength on multiple-choice versus generative tasks can be shifted by changing the proportion of general, math, and coding examples in the calibration set.
Several REAM settings produce models whose scores are statistically close to those of the full uncompressed model.
The method works across multiple different MoE LLMs and consistently beats the REAP baseline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the calibration step can be made task-agnostic, REAM could enable routine compression of MoE models before they reach edge hardware.
Merging might be extended to dynamic, online grouping that updates as new data arrives rather than relying on a static calibration pass.
The same grouping-plus-merge logic could be tested on other modular networks such as sparse transformers or mixture-of-experts vision models.

Load-bearing premise

That merging weights of experts grouped by activation on a chosen calibration mix will preserve behavior across new tasks without creating fresh failure modes.

What would settle it

Measure whether an REAM-compressed model falls below the accuracy of its REAP-pruned counterpart on a completely unseen benchmark category that was absent from the calibration data.

Figures

Figures reproduced from arXiv: 2604.04356 by Ali Parviz, Ali Saheb Pasand, Boris Knyazev, Maryam Hashemzadeh, Min-Joong Lee, Saurav Jha.

**Figure 1.** Figure 1: Illustration of REAM components: a) Comparison of expert compression strategies reducing N=9 experts to N′=4. HC-SMoE merging (Chen et al., 2025) clusters all experts by output similarity regardless of saliency (e.g., E1 and E7 grouped together). Pruning retains the top-4 salient experts unchanged and discards the rest. Our REAM’s pseudo-pruning selects the top-4 experts as protected centroids and absorbs… view at source ↗

**Figure 2.** Figure 2: Discriminative (MC) vs. Generative (GEN) trade-off depending on the calibration data mixture: benchmark scores with 64 (left) and 96 (right) experts for REAP, HC-SMoE, and REAM across ten mixing ratios of the calibration data with Qwen3-30B-A3BInstruct-2507. The marker sizes are proportional to the The-Stack-Smol share of the mixture. We report the mean score within each suite. Since generative tasks are… view at source ↗

**Figure 3.** Figure 3: Additional analyses for 96 experts: a) Pearson correlation r between calibration datasets (C4, Math, Code) and MC/GEN scores, and between MC and GEN scores themselves, for each merging method. b) Pareto frontiers where each point is one of 10 calibration mixtures. Filled markers denote Pareto-optimal configurations not simultaneously dominated on MC and GEN by any other mixture of the same method, and ho… view at source ↗

**Figure 4.** Figure 4: Ablation of REAM components with 96 experts: (a) MC and GEN scores for each ablation variant; (b) Per-task score drop (∆) relative to the full REAM performance. Our second-largest drop stems from removing gate softmax scaling (σ(x) in Eq. (8)) before computing pairwise output similarity (∆AVG = −5.9, ∆GEN = −11.5) during grouping. This reaffirms that ignoring the router’s confidence in grouping similarity … view at source ↗

**Figure 5.** Figure 5: Correlation between avg. prelogit ranks and AVG benchmark scores across 10 calibration ratios for 96 experts. Rank analyses. To study whether expert merging strategies that better preserve the representational capacity of the compressed model translate into higher benchmark scores, we compute the average numerical rank of the pre-logit embeddings for each method across all ten calibration mixtures an… view at source ↗

**Figure 6.** Figure 6: Effect of calibration data mixture on MC–GEN trade-off. Each panel shows discriminative (MC) vs. generative (GEN) benchmark scores for Freq, REAP, HC-SMoE, and REAM across ten mixing ratios of C4, Math, and Code datasets, with marker size proportional to each dataset’s share of the mixture. Results are shown at two expert-count targets: 64 (50% reduction) and 96 (25% reduction). The star denotes the perfor… view at source ↗

**Figure 7.** Figure 7: Pareto frontiers of expert-merging methods at 64 retained experts. Each point is one of 10 calibration mixtures; filled markers denote Pareto-optimal configurations (not simultaneously dominated on both MC and GEN by any other mixture of the same method) and hollow markers denote dominated ones. The hypervolume (HV) measures the area of the MC×GEN plane dominated by each method’s frontier relative to a sha… view at source ↗

read the original abstract

Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

REAM replaces expert pruning with router-weighted merging and shows it can match full-model performance on tuned calibration mixes, but the evidence is mostly comparative benchmarks without strong controls for generalization.

read the letter

The core idea is simple: instead of dropping experts in MoE models, group them by how the router activates them on a calibration set and merge their weights. REAM does this and reports better retention of performance than REAP or other pruning baselines across several MoE LLMs, often staying close to the uncompressed model on MC and generative tasks. The useful part is the explicit calibration analysis. By varying the mix of general, math, and coding data they trace a clear MC-versus-GEN trade-off and identify operating points on the Pareto front. That practical angle is the main contribution worth noting. The experiments cover multiple models and benchmark types, which gives the comparison some scope. The soft spots are proportionate to the claims. The write-up gives limited implementation detail on the exact grouping and averaging steps, and the abstract-level summary shows no error bars, seed variance, or statistical tests. The central assumption—that merged experts keep the router’s selection behavior intact—rests on the calibration set matching future inputs. The paper flags the MC/GEN sensitivity but does not appear to add direct checks for activation statistics on out-of-mix data, so the stress-test concern about distorted routing on new tasks is reasonable and unaddressed. This work is aimed at people who deploy or compress large MoE models and need concrete compression options. A reader focused on inference efficiency would pick up the calibration trade-off and the merging-versus-pruning comparison. It is coherent enough on its own terms to deserve a serious referee, even though the current evidence would need tighter experimental protocol and robustness tests to hold up under review.

Referee Report

2 major / 1 minor

Summary. The paper proposes Router-weighted Expert Activation Merging (REAM) for Mixture-of-Experts LLMs: experts are grouped by router-weighted activation statistics computed on a calibration mix of general/math/coding data, their weights are averaged, and the merged experts replace the original set. This is positioned as an improvement over Router-weighted Expert Activation Pruning (REAP) and other baselines. Experiments on multiple MoE models compare REAM to pruning methods on multiple-choice QA and generative benchmarks, reporting a controllable MC/GEN performance trade-off and claiming that REAM frequently outperforms the baselines while remaining comparable to the uncompressed model.

Significance. If the empirical claims hold under rigorous verification, the work would be a useful incremental contribution to MoE compression: merging rather than discarding experts can retain more functional capacity than pure pruning while still reducing memory footprint. The explicit control of the calibration-data mix and the resulting Pareto analysis of the MC/GEN trade-off is a concrete, falsifiable observation that could guide practical deployment decisions.

major comments (2)

[Abstract and experimental evaluation] The abstract and experimental sections report comparative benchmark results but supply no implementation details, error bars, statistical significance tests, or full experimental protocol (data splits, number of runs, exact calibration-set sizes, merging formula). This absence makes it impossible to assess whether the reported outperformance of REAM over REAP and baselines is reproducible or statistically reliable.
[Method description and results discussion] The central claim that merged experts 'better preserve original performance' and are 'comparable to the original uncompressed models' rests on the untested assumption that router selection dynamics and per-token expert contributions remain essentially unchanged after weight averaging. No analysis of post-merge routing histograms, activation statistics on held-out inputs, or capacity measurements is provided, even though the paper itself notes that performance depends on the calibration mix.

minor comments (1)

[Method] Notation for the router-weighted activation score and the precise merging operation (simple average, weighted average, etc.) should be formalized with an equation in the method section.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below and commit to revisions that improve reproducibility and strengthen the analysis without misrepresenting our current results.

read point-by-point responses

Referee: [Abstract and experimental evaluation] The abstract and experimental sections report comparative benchmark results but supply no implementation details, error bars, statistical significance tests, or full experimental protocol (data splits, number of runs, exact calibration-set sizes, merging formula). This absence makes it impossible to assess whether the reported outperformance of REAM over REAP and baselines is reproducible or statistically reliable.

Authors: We agree that the manuscript currently lacks these details, which limits independent verification. In the revised version we will add: the exact merging formula with mathematical notation, pseudocode for expert grouping and weight averaging, full calibration-set composition and sizes, data splits for all benchmarks, number of evaluation runs, error bars on all metrics, and statistical significance tests (e.g., bootstrap confidence intervals or paired tests) comparing REAM against REAP and other baselines. revision: yes
Referee: [Method description and results discussion] The central claim that merged experts 'better preserve original performance' and are 'comparable to the original uncompressed models' rests on the untested assumption that router selection dynamics and per-token expert contributions remain essentially unchanged after weight averaging. No analysis of post-merge routing histograms, activation statistics on held-out inputs, or capacity measurements is provided, even though the paper itself notes that performance depends on the calibration mix.

Authors: We acknowledge that direct post-merge routing analysis is absent. Our claims rest on consistent outperformance or parity with the uncompressed model across held-out MC and GEN benchmarks, which indirectly indicates that merged experts retain functional capacity under the original router. The paper already highlights the dependence on calibration mix and demonstrates explicit control of the MC/GEN Pareto frontier. In revision we will add a dedicated paragraph discussing the routing assumption, include any available indirect evidence from evaluation-time activation counts, and list the lack of histogram analysis as a limitation. We will not perform new large-scale routing experiments but will clarify that benchmark results serve as the primary validation. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical method proposal and benchmark comparison

full rationale

The paper introduces REAM as a practical merging procedure for MoE experts, motivated by prior pruning work, and supports its claims exclusively through experimental comparisons on MC and GEN benchmarks using controlled calibration mixes. No mathematical derivation, uniqueness theorem, or predictive equation is presented whose output is forced by construction from the same data or self-citations used to evaluate it. The observed MC/GEN trade-off is reported as an empirical finding rather than a fitted or self-defined quantity. Self-reference to REAP is limited to motivation and does not bear the load of the performance claims.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The central claim rests entirely on empirical benchmarking; no free parameters, mathematical axioms, or new postulated entities are stated or required in the provided abstract.

pith-pipeline@v0.9.0 · 5501 in / 990 out tokens · 51587 ms · 2026-05-10T19:51:13.717163+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

REAM groups experts by router-weighted activation and merges weights... pseudo-pruning... combined cost matrix C⟨ci,j⟩=Cact+Cwt
IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

trade-off between MC and GEN performance that depends on the mix of calibration data

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts
cs.LG 2026-05 unverdicted novelty 8.0

HodgeCover isolates the harmonic kernel of a simplicial Laplacian on an expert 2-complex to identify irreducible merge cycles and selects experts for aggressive compression, matching or exceeding baselines on open-wei...

Reference graph

Works this paper leans on

7 extracted references · 6 canonical work pages · cited by 1 Pith paper · 3 internal anchors

[1]

Task-specific expert pruning for sparse mixture-of-experts.arXiv preprint arXiv:2206.00277, 2022

5, 15 Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. Task-specific expert pruning for sparse mixture-of-experts.ArXiv, abs/2206.00277, 2022. URLhttps://api.semanticscholar.org/CorpusID:249240535. 2 Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xi...

work page arXiv 2022
[2]

Towards efficient mixture of experts: A holistic study of compression techniques.arXiv preprint arXiv:2406.02500, 2024

2 Shwai He, Daize Dong, Liang Ding, and Ang Li. Demystifying the compression of mixture- of-experts through a unified framework.arXiv preprint arXiv:2406.02500, 2, 2024. 1, 2, 3 Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conferen...

work page arXiv 2024
[3]

Mixtral of Experts

1, 2, 3 Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024. 1 Yinsicheng Jiang, Yao Fu, Yeqi Huang, Ping Nie, Zhan Lu, Leyang Xue, Congjie He, Man-Kit Sit, Jilong Xue...

work page internal anchor Pith review Pith/arXiv arXiv 2024
[4]

DeepSeek-V3 Technical Report

2 Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. InFirst Workshop on Interpolation Regularizers and Beyond at NeurIPS 2022, 2022. URLhttps://openreview.net/forum?id=SQgVgE2Sq4. 2 Pingzhi Li, Zhenyu Zhang, Prateek Yad...

work page internal anchor Pith review Pith/arXiv arXiv 2022
[5]

Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts,

URLhttps://openreview.net/forum?id=jfZF7nJnqx. 2 12 Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pp. 2381–2391, 2018. 15 Mohammed Muqeeth, Haokun Liu, and Coli...

work page arXiv 2018
[6]

Instruction-Following Evaluation for Large Language Models

Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/ v1/2025.findings-acl.4. URLhttps://aclanthology.org/2025.findings-acl.4/. 2 Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URLhttps://arxiv.org/abs/2...

work page internal anchor Pith review Pith/arXiv arXiv 2025
[7]

GPQA-Diamond is evaluated without chain-of-thought (CoT) reasoning using 5 shots

with a HuggingFace or vLLM backend (Kwon et al., 2023) and default task set- tings. GPQA-Diamond is evaluated without chain-of-thought (CoT) reasoning using 5 shots. For LiveCodeBench-v6 we use their official evaluation code. But to evalua- tion GLM-4.5-Air on HumanEval and LiveCodeBench we use the evaluation tool from https://github.com/zai-org/glm-simpl...

2023