Pith. sign in

REVIEW 4 major objections 5 minor 19 references

A single ternary matrix, rotated per expert, cuts MoE memory from O(N·d^2) to O(d^2 + N·d log d), giving 150x compression at 256 experts with negligible accuracy loss.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:27 UTC pith:ADUU6TAJ

load-bearing objection Novel orbital MoE parameterization with correct memory arithmetic, but the accuracy claim is untested and internal numbers conflict. the 4 major comments →

arxiv 2601.13563 v5 pith:ADUU6TAJ submitted 2026-01-20 cs.LG cs.AI

ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbits

Aryan Karmore This is my paper
classification cs.LG cs.AI
keywords mixture-of-expertsmodel compressionternary quantizationbutterfly matricesstructured matricessub-linear memoryexpert diversityedge deployment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that MoE experts need not be stored independently. It parameterizes each expert as W_i = B(phi_i) * W_base * B(theta_i)^T, where W_base is a shared ternary matrix and B(phi_i), B(theta_i) are butterfly rotations with O(d log d) parameters. This changes per-expert memory from O(d^2) to O(d log d), so total memory becomes O(d^2 + N d log d), and the compression ratio grows with expert count, reaching 150x at 256 experts. The paper also argues that learned per-expert rotations suppress activation outliers, which stabilizes extreme low-bit training that static methods fail at. If correct, MoE memory stops being a bottleneck for edge deployment, allowing hundreds or thousands of experts in a few megabytes.

Core claim

The central claim is that the diversity needed across experts can be generated geometrically: a single ternary 'substrate' matrix stores the shared computation, and each expert is a different orientation of that substrate obtained by multiplying with learned butterfly rotations on the left and right. Because butterfly matrices have O(d log d) parameters, expert identity becomes cheap, and total memory no longer grows linearly with the number of experts. The paper reports 150x compression at 256 experts with negligible accuracy loss, a better memory-accuracy tradeoff than an equal-memory dense baseline, a 97% reduction in quantization error from trained rotations, and stable training without

What carries the argument

The central object is the group-orbit parameterization W_i = B(phi_i) * W_base * B(theta_i)^T, where W_base is ternary-quantized to {-1,0,+1} and B(·) is a butterfly matrix built from block-diagonal Givens rotations and fixed permutations, using O(d log d) learnable angles per expert. This parameterization does three jobs: it makes per-expert storage O(d log d) instead of O(d^2); it gives each expert a different view of shared capacity so diversity does not require redundant weights; and the learned input rotation redistributes activation energy across dimensions to suppress outliers before the ternary multiply, making 1.58-bit quantization trainable.

Load-bearing premise

The load-bearing premise is that one shared ternary matrix contains enough total knowledge that rotating it yields experts as accurate as independently trained ones.

What would settle it

A concrete falsifier: compute the singular values of the matrix whose rows are the flattened effective expert matrices B(phi_i) * W_base * B(theta_i)^T for a trained model. If only a few singular values are large, the generated experts lie in a low-dimensional subspace and cannot match the diversity of independent experts, directly disproving the expressivity premise.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • MoE memory becomes sub-linear in expert count; adding an expert costs only O(d log d) additional parameters, so models with hundreds of experts fit in a few megabytes.
  • Compression ratio improves as N grows: at 64 experts the model uses about 1.9 MB versus 256 MB for standard MoE, and at 256 experts about 4.7 MB versus 1024 MB.
  • Extreme low-bit training becomes stable through learned rotations, reducing quantization error by about 97% relative to an untrained substrate, with no clipping or recovery stages.
  • Because the substrate is ternary, the per-expert multiply becomes additions only, lowering energy per operation and enabling on-the-fly expert synthesis without materializing weight matrices.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The shared-substrate design suggests a broader principle: expert identity can live in transformation space rather than in parameters, so the total knowledge capacity of a model need not scale with the number of experts.
  • Because the rotation angles are continuous, routing could interpolate between expert angles to create a continuum of experts, enabling smooth specialization beyond discrete top-k selection, a direction the paper does not explore.
  • If the shared substrate is task-agnostic, fine-tuning on a new task might require updating only the rotation angles while keeping W_base frozen, which would make multi-task edge deployment dramatically cheaper, though this remains untested.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. ButterflyMoE proposes replacing N independently stored MoE expert matrices with expert-specific butterfly rotations applied to one shared ternary weight matrix, W_i = B(phi_i) W_base B(theta_i)^T. The claimed memory is O(d^2 + N d log d), with per-expert cost reduced to O(d log d). The paper derives closed-form memory formulas, reports a quantization-error reduction from learned rotations, measures expert output diversity via cosine similarity, and estimates energy savings. The central claim is that this yields up to 150x compression at 256 experts with 'negligible accuracy loss' and that ButterflyMoE 'outperforms an equal memory dense baseline'.

Significance. If the accuracy claim held, this would be a significant architectural idea: making MoE expert memory grow only logarithmically in expert count would directly attack the edge-deployment bottleneck. The memory formula in Eq. (8) is simple arithmetic and is internally coherent under the stated parameterization. The quantization-stability experiment (Figure 4) is a useful, falsifiable measurement. However, the paper reports no downstream task accuracy or perplexity for either ButterflyMoE or any baseline, so the load-bearing part of the claim is untested. The manuscript also contains a concrete numerical inconsistency between its own Eq. (8) and Table 1 for the 256-expert case, which further undermines confidence.

major comments (4)
  1. [Section 4.1, Table 1 vs. Eq. (8)] There is a direct inconsistency. For d_model=512, d_ff=2048, N_E=256, Eq. (8) gives M = 1.58/8*512*2048 + 256*(256*9 + 1024*11)*2 bytes = 207,094 + 256*27,136 = 7,151,? bytes, i.e. about 7.15 MB. Table 1 reports 4.70 MB for 256 experts, and the text says 150x compression. The standard-MoE memory is 1024 MB, so 7.15 MB gives ~143x, not 150x. The asymptotic limit in Eq. (9) is ~154.5x, which is also not what Table 1 reports for 64 experts. At least one of Eq. (8), Eq. (9), or Table 1 must be wrong. Because the compression ratio is the paper's headline quantitative claim, this must be resolved.
  2. [Abstract, Sections 1 and 4] The central claim of 'negligible accuracy loss' and 'outperforms an equal memory dense baseline' is not supported by any downstream evaluation. Section 4.1 states that ButterflyMoE is evaluated on Wiki-Text and compared against dense and standard-MoE baselines, but no perplexity, accuracy, or loss numbers appear anywhere in the paper. The only quantitative results are memory, quantization MSE, expert cosine similarity, and energy estimates. Without a task metric, the compression ratio measures bytes, not preserved task performance. This is the load-bearing gap in the paper.
  3. [Eqs. (1)-(3), Section 3.5] The expressivity justification is incomplete. Each B(theta) and B(phi) is a product of Givens rotations and permutations and is therefore orthogonal. Left- and right-multiplication by orthogonal matrices preserves the singular-value spectrum of W_base exactly. Consequently every expert W_i has identical singular values; only their singular vectors can vary. Independently trained MoE experts can have heterogeneous spectra, and no argument or experiment is provided to show that this heterogeneity is irrelevant to expert specialization and routing quality. The cited Monarch-matrix result [6] addresses approximation of general linear transforms, not task accuracy after imposing a shared isospectral substrate. This concern would remain even after adding accuracy numbers, because a positive result would be needed to establish that spectral diversity is not necessary.
  4. [Table 2 and Section 5] The runtime claims are also unsupported. Table 2 is captioned as an ablation of butterfly layers but reports only throughput and speedup; there is no inference speed comparison to a standard dense baseline. The conclusion states that inference is up to 6.6x slower on T4 GPUs without kernel support and that a custom Triton kernel 'matches' dense baselines, but no kernel implementation, latency measurements, or accuracy comparison is provided. This is presented as a resolved limitation rather than as an experimental result.
minor comments (5)
  1. [Abstract and Section 4.1] The compression-ratio narrative is inconsistent across the paper: the abstract says 80x at 8 experts and 150x at 256 experts, the introduction says 150x at 256 experts, and Table 1 labels a column 'Scaling Ratio (64)' with 150x. Please clarify which expert count yields which compression ratio.
  2. [Section 4.1] The text says 'Our model uses d_model=512, d_ff=2048, and N_E=8 experts' and then discusses 64 and 256 experts. The experimental setup should state which configuration produced each reported number.
  3. [Section 3.5] The sentence 'Monarch matrices [6] shows that O(log d) butterfly layers approximate a broad class of orthogonal transformations' is too vague. Specify the approximation measure and conditions, or give a citation to the exact theorem.
  4. [Throughout] There are numerous typos and grammatical errors, e.g., 'acheives', 'competetive', 'maintaing', 'limitaion', 'meaninfully', and 'efficency'. The notation in Eq. (10) mixes d with d_model/d_ff; please define all symbols consistently.
  5. [Section 3.9] The phrase 'memory-reducing per-expert cost from O(d^2) to O(d log d)' in the abstract is confusing: the total memory still contains an O(d^2) substrate term. Clarify that the per-expert marginal cost is O(d log d).

Circularity Check

0 steps flagged

No circular derivation: compression claims are definitional byte-counting; no fitted quantity is relabeled as a prediction, and no load-bearing self-citation appears.

full rationale

The derivation chain starts from Eq. (1), which defines each expert as W_i = B(phi_i) W_base B(theta_i)^T. Eq. (8) then counts the bytes required to store the shared ternary substrate plus the butterfly angles, and Eq. (9) divides the standard O(N d^2) MoE byte count by that figure to obtain the asymptotic 150x compression. This is definitional arithmetic, not an empirical prediction: the compression ratio is entailed by the chosen parameterization and the baseline byte model, and no fitted parameter is later presented as a predicted outcome. The quantization-error reduction (51.3% to 1.43%) is a measurement of the training objective (minimizing quantization error via learned rotations), not a held-out forecast, so it does not fit the 'fitted input called prediction' pattern. The paper has no self-citations by the author; the cited expressivity support (Butterfly [7], Monarch [6]) is external. The abstract's accuracy claims ('negligible accuracy loss', 'outperforms an equal memory dense baseline') are unsupported because Section 4 reports only memory, quantization MSE, cosine similarity, and energy, with no perplexity or accuracy; and the stated 4.70 MB at 256 experts is inconsistent with Eq. (8)'s ~7.0 MB. These are evidence/correctness concerns, not circularity. The conclusion's own limitations (small-scale experiments, inference slowdown without custom kernels) are acknowledged and do not create a circular dependency. Therefore no specific circular step can be quoted, and the circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

The memory claim rests on standard matrix-complexity arithmetic plus a domain assumption about butterfly expressivity. The accuracy claim rests on the same expressivity assumption plus untested ternary-training stability. No fitted parameter is used to derive the compression ratio.

free parameters (3)
  • lambda_balance = 0.01
    Hand-chosen load-balancing coefficient (Switch Transformer standard), affects routing/training but not the memory formula.
  • butterfly angle initialization std = 0.01
    Random init N(0,0.01^2) chosen to prevent expert collapse; accuracy depends on it.
  • number of butterfly layers = 2 (default)
    Selected from Table 2 ablation based on throughput/speedup; affects accuracy and compute.
axioms (3)
  • domain assumption Butterfly matrices with O(d log d) parameters can approximate the orthogonal transforms needed to diversify experts.
    Section 3.5 cites Monarch [6] for general matrix approximation; no MoE-specific guarantee.
  • domain assumption Linear mode connectivity of whole networks transfers to experts as reorientations of one substrate.
    Section 3.3 uses LMC results [1,10] as motivation; not proven for expert matrices.
  • domain assumption Ternary quantization with STE preserves accuracy at 1.58 bits for this architecture.
    Section 3.6 relies on prior 1-bit LLM work [16]; the paper gives no accuracy numbers to confirm.
invented entities (1)
  • shared ternary substrate W_base no independent evidence
    purpose: Single prototype from which all N experts are synthesized via rotations
    Architectural invention at the core of the method; no external falsifiable handle outside the paper's own experiments.

pith-pipeline@v1.3.0-alltime-deepseek · 8790 in / 13530 out tokens · 117100 ms · 2026-08-03T09:27:52.905702+00:00 · methodology

0 comments
read the original abstract

In current Mixture of Experts (MoE) architectures, linear memory scaling is present, the memory grows as the number of experts increases. $N$ independent expert weight matrices require $\mathcal{O}(N \cdot d^2)$ memory which exceeds the memory budget of edge devices. Current compression methods like quantization, pruning, and low-rank factorization reduce constant factors, but the scaling bottleneck is still unresolved. We introduce ButterflyMoE, a method which treats experts not as independent matrices but as geometric reorientations of a shared quantized substrate. Diversity amongst the experts arises from viewing different angles of the shared capacity and not from redundant storage. Learned rotations are applied to a shared ternary prototype. With this, each expert yields $\mathcal{O}(d^2 + N \cdot d \log d)$ memory-reducing per-expert cost from $\mathcal{O}(d^2)$ to $\mathcal{O}(d \log d)$. The key insight is that training these rotations with quantization reduces activation outliers and stabilizes extreme low-bit training where other static methods collapse. Across language modeling benchmarks, ButterflyMoE achieves 80$\times$ memory reduction at 8 experts with a highly favorable memory-accuracy tradeoff.At this 80x compression ButterflyMoE outperforms an equal memory dense baseline, showing that orbital parameterization extracts fundamentally more utility per byte. When scaled up to 256 experts, ButterflyMoE asymptotically compresses the memory by 150 $\times$. ButterflyMoE reduces the constant factor of linear scaling with compression ratio growing with the expert count.

Figures

Figures reproduced from arXiv: 2601.13563 by Aryan Karmore.

Figure 1
Figure 1. Figure 1: The proposed Butterfly MoE layer replaces the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Top-k gating instantiates experts via lightweight rotations of a shared ternary base matrix Wbase achieving O(d logd) parameters per expert and weighted sum aggre￾gation. Stage 1: Prototype Quantization Training Wbase ∈ R dff×dmodel and quantizing to ternary values {−1,0,+1} using Straight-Through Estimator (STE) gradi￾ents. Stage 2: Orbital Expert Generation Each expert i is parameterized by rotation vari… view at source ↗
Figure 3
Figure 3. Figure 3: Figure 3: Memory consumption as a function of [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: shows that per-expert rotations B(θi) reduce the impact of activation outliers during training. quantization er￾ror decreases from 51.3% (untrained) to 1.43% (trained), a 97.2% reduction. The resulting weight distribution (top right) is tightly con￾centrated around the ternary grid {−1,0,+1}, whereas the untrained model spans [−4,+4]. 4.3 Expert Specialization Experts maintain distinct behaviour despite op… view at source ↗
Figure 5
Figure 5. Figure 5: Expert output similarity. Pairwise cosine similarity [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 10 linked inside Pith

  1. [1]

    Git re-basin: Merging mod- els modulo permutation symmetries.arXiv preprint arXiv:2209.04836, 2022

    Samuel K Ainsworth, Jonathan Hayase, and Sid- dhartha Srinivasa. Git re-basin: Merging mod- els modulo permutation symmetries.arXiv preprint arXiv:2209.04836, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    Quarot: Outlier-free 4-bit inference in rotated llms

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximil- ian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213–100240, 2024

    2024

  3. [3]

    Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432, 2013

    Yoshua Bengio, Nicholas L ´eonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432, 2013

    Pith/arXiv arXiv 2013

  4. [4]

    Mobe: Mixture-of-basis- experts for compressing moe-based llms.arXiv preprint arXiv:2508.05257, 2025

    Xiaodong Chen, Mingming Ha, Zhenzhong Lan, Jing Zhang, and Jianguo Li. Mobe: Mixture-of-basis- experts for compressing moe-based llms.arXiv preprint arXiv:2508.05257, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    On the represen- tation collapse of sparse mixture of experts.Advances in Neural Information Processing Systems, 35:34600– 34613, 2022

    Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Ba- jaj, Xia Song, Xian-Ling Mao, et al. On the represen- tation collapse of sparse mixture of experts.Advances in Neural Information Processing Systems, 35:34600– 34613, 2022

    2022

  6. [6]

    Monarch: Expres- sive structured matrices for efficient and accurate train- ing

    Tri Dao, Beidi Chen, Nimit S Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher R´e. Monarch: Expres- sive structured matrices for efficient and accurate train- ing. InInternational Conference on Machine Learning, pages 4690–4721. PMLR, 2022

    2022

  7. [7]

    Learning fast algorithms for lin- ear transforms using butterfly factorizations

    Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, and Christopher R ´e. Learning fast algorithms for lin- ear transforms using butterfly factorizations. InInter- national conference on machine learning, pages 1517–

  8. [8]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learn- ing Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learn- ing Research, 23(120):1–39, 2022

    2022

  9. [9]

    Qmoe: Practical sub- 1-bit compression of trillion-parameter models.arXiv preprint arXiv:2310.16795, 2023

    Elias Frantar and Dan Alistarh. Qmoe: Practical sub- 1-bit compression of trillion-parameter models.arXiv preprint arXiv:2310.16795, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Loss surfaces, mode connectivity, and fast ensembling of dnns.Ad- vances in neural information processing systems, 31, 2018

    Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns.Ad- vances in neural information processing systems, 31, 2018

    2018

  11. [11]

    Towards efficient mixture of experts: A holistic study of compression techniques.arXiv preprint arXiv:2406.02500, 2024

    Shwai He, Daize Dong, Liang Ding, and Ang Li. Towards efficient mixture of experts: A holistic study of compression techniques.arXiv preprint arXiv:2406.02500, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    1.1 computing’s energy problem (and what we can do about it)

    Mark Horowitz. 1.1 computing’s energy problem (and what we can do about it). In2014 IEEE International Solid-State Circuits Conference Digest of Technical Pa- pers (ISSCC), pages 10–14, 2014

    2014

  13. [13]

    Lora: Low-rank adaptation of large lan- guage models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large lan- guage models.ICLR, 1(2):3, 2022

    2022

  14. [14]

    Mixture compressor for mixture-of-experts llms gains more.arXiv preprint arXiv:2410.06270, 2024

    Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and Xi- aojuan Qi. Mixture compressor for mixture-of-experts llms gains more.arXiv preprint arXiv:2410.06270, 2024. 8

    Pith/arXiv arXiv 2024

  15. [15]

    Mixture of quantized experts (moqe): Com- plementary effect of low-bit quantization and robust- ness.arXiv preprint arXiv:2310.02410, 2023

    Young Jin Kim, Raffy Fahim, and Hany Hassan Awadalla. Mixture of quantized experts (moqe): Com- plementary effect of low-bit quantization and robust- ness.arXiv preprint arXiv:2310.02410, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    The era of 1-bit llms: All large language models are in 1.58 bits.arXiv preprint arXiv:2402.17764, 1(4), 2024

    Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits.arXiv preprint arXiv:2402.17764, 1(4), 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Smoothquant: Ac- curate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Ac- curate and efficient post-training quantization for large language models. InInternational conference on ma- chine learning, pages 38087–38099. PMLR, 2023

    2023

  18. [18]

    MoE-I 2: Compressing mix- ture of experts models through inter-expert pruning and intra-expert low-rank decomposition.arXiv preprint arXiv:2411.01016, 2024

    Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. MoE-I 2: Compressing mix- ture of experts models through inter-expert pruning and intra-expert low-rank decomposition.arXiv preprint arXiv:2411.01016, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Puzzle- moe: Efficient compression of large mixture-of-experts models via sparse expert merging and bit-packed infer- ence.arXiv preprint arXiv:2511.04805, 2025

    Yushu Zhao, Zheng Wang, and Minjia Zhang. Puzzle- moe: Efficient compression of large mixture-of-experts models via sparse expert merging and bit-packed infer- ence.arXiv preprint arXiv:2511.04805, 2025. 9

    Pith/arXiv arXiv 2025