Pith. sign in

REVIEW 4 major objections 4 minor 7 cited by

Grove MoE's variable-size experts, activated by token complexity, let a 33B-parameter model run with 3.14–3.28B active parameters while matching larger open models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Grove MoE uses unequal-size adjugate experts with complexity-based activation to run 33B-parameter models at roughly 3.1 to 3.3B active parameters while matching larger open models in benchmarks.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Heterogeneous-size experts with complexity-based activation is a plausible efficiency idea, but the paper as it stands can't be verified — the full text is corrupted and the routing mechanism is undisclosed. the 4 major comments →

arxiv 2508.07785 v1 pith:QK5DIFKI submitted 2025-08-11 cs.CL

Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts

Haoyuan Wu , Haoxing Chen , Xiaodong Chen , Zhanchao Zhou , Tieyuan Chen , Yihong Zhuang , Guoshan Lu , Zenan Huang
show 5 more authors
Junbo Zhao Lin Liu Zhenzhong Lan Bei Yu Jianguo Li
This is my paper
classification cs.CL
keywords mixture of expertsheterogeneous expertsdynamic activationtoken complexityadjugate expertsefficient LLM inferenceupcyclingsparse activation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the standard Mixture-of-Experts recipe, in which every expert is the same size and every token activates a fixed number of experts, wastes computation on easy tokens. Grove MoE replaces uniform experts with adjugate experts of varying sizes, and adds a dynamic activation mechanism that chooses how much capacity a token receives based on its complexity. The authors build two 33B-parameter models by upcycling the Qwen3-30B-A3B-Base checkpoint, and report that only 3.14–3.28B parameters are active per token while performance stays comparable to open-source models of similar or larger size. If this holds, MoE inference can buy large-model quality with a much smaller compute budget.

Core claim

Grove MoE's central proposal is to make sparsity complexity-adaptive rather than count-adaptive. In a conventional MoE, activation is sparse but homogeneous: each token uses the same number of same-sized experts, so a trivial token pays as much as a difficult one. Grove MoE instead maintains experts in different sizes and lets a router allocate more parameters to tokens that need them. The resulting models, GroveMoE-Base and GroveMoE-Inst, have 33B total parameters yet activate only 3.14–3.28B per token, and the paper reports that they match state-of-the-art open-source models of comparable or larger size. The design is presented as an upcycling of an existing strong MoE checkpoint, so the a

What carries the argument

The load-bearing mechanism is the adjugate expert: a variable-size expert unit used inside the MoE layers, paired with a dynamic activation policy that reads per-token complexity and decides which expert size to invoke. This turns sparse activation from a fixed-count selection into a continuous capacity budget—the source of the 3.14–3.28B activated-parameter range. The big.LITTLE-inspired design is what allows the model to carry 33B parameters while spending only a tenth of them on an average token.

Load-bearing premise

The whole efficiency claim rests on the router being able to tell hard tokens from easy ones and pick the right expert size reliably, while the routing and load-balancing overhead stays small enough to be worth it.

What would settle it

Take the trained GroveMoE-Base and run it with dynamic activation disabled, first forcing all tokens to the smallest experts and then forcing all to the largest, at matched total FLOPs. If either fixed policy achieves the same benchmark scores as the dynamic policy, then token-complexity routing is not the source of the reported efficiency; the claim collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • MoE quality no longer needs to be tied to a fixed activated-parameter count; capacity can be spread across expert sizes and spent selectively.
  • Inference costs become input-dependent, so serving systems can schedule harder tokens on larger experts and easy tokens on small ones, lowering average FLOPs per request.
  • Existing strong MoE checkpoints can be upcycled into heterogeneous-expert models, preserving quality while adding dormant capacity.
  • The reported 3.14–3.28B range implies fine-grained control over the compute-quality frontier, making per-token compute budgeting a usable design axis.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • I would expect the dynamic activation policy to create a measurable quality-per-FLOP frontier: if a fixed-size expert matches the dynamic model at the same average activated parameters, then the complexity signal is not buying anything; if it does not, complexity-routing is doing real work.
  • The 3.14–3.28B spread is narrow enough that a distribution analysis over tokens would clarify which tokens actually get the large experts; a concentrated distribution would mean most of the benefit comes from a single size, not from complexity adaptation.
  • The same adjugate idea could be applied to the number of layers or attention heads a token passes through, not just expert width, since the big.LITTLE principle is about allocating capacity where it matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes Grove MoE, an MoE architecture with heterogeneous ``adjugate'' experts of varying sizes, inspired by the big.LITTLE CPU design, and a dynamic activation mechanism that selects experts according to per-token complexity. The authors report GroveMoE-Base and GroveMoE-Inst, 33B-parameter models upcycled from Qwen3-30B-A3B-Base, claiming 3.14-3.28B dynamically activated parameters and performance comparable to SOTA open-source models of similar or larger size. The supplied full text is a mojibake rendering and no equations, tables, ablations, training details, or experimental protocols can be audited; the abstract is the only legible substantive component.

Significance. If the central efficiency claim were fully supported, Grove MoE would be a valuable step toward input-dependent MoE computation, and the big.LITTLE analogy is well motivated. The active-parameter range (3.14-3.28B) is specific and falsifiable in principle. However, because the full text cannot be read, none of the architectural or empirical support can be verified, and no code, proofs, or reproducibility artifacts are visible. The current manuscript therefore cannot yet contribute beyond its abstract; a complete, correctly submitted manuscript is required before its significance can be judged.

major comments (4)
  1. [Full text (passim)] The supplied full text is largely unreadable mojibake; no equation, table, or experimental description can be inspected. The abstract's claims about the architecture, dynamic activation, upcycling, and benchmark performance cannot be checked. This is the central load-bearing issue for the paper. Please resubmit a clean, correctly rendered PDF and source files.
  2. [Abstract and full text] The dynamic activation mechanism is not specified. The abstract states that 3.14-3.28B parameters are activated "based on token complexity" but does not define the complexity signal, the routing criterion, or the threshold policy. Without this, the reported active-parameter range is uninterpretable and the efficiency claim is not reproducible. A revision must state the routing rule and provide evidence that the router remains stable under load-balancing and entropy-regularization losses.
  3. [Abstract (upcycling comparison)] The comparison to Qwen3-30B-A3B-Base is a potential confound. GroveMoE has 33B total parameters versus the approximately 30B of the source model, so any performance improvement over the backbone could be due to the extra parameters rather than to the adjugate-expert/dynamic-activation design. Please provide controlled ablations with equal total parameter counts, equal active parameter counts, and a homogeneous-expert MoE baseline of the same scale.
  4. [Full-text header] The supplied full text contains the line "arXiv:2508.07795v3 [cs.CV] 14 Mar 2026", which does not match the claimed arXiv identifier (2508.07785) or field (cs.CL). This mismatch makes it impossible to confirm that the reviewed full text is the paper described in the abstract. Please submit the correct manuscript and verify the arXiv metadata.
minor comments (4)
  1. [Abstract] Define "adjugate experts" at first use; the term is not standard and the abstract does not explain what makes an expert "adjugate" beyond having a different size.
  2. [Abstract] Clarify whether the 3.14-3.28B active-parameter range is an empirical measurement across a benchmark or an analytic bound, and report the distribution of token routing across expert sizes.
  3. [Abstract] The phrase "similar or even larger size" is vague. Name the concrete SOTA baselines and report the win/loss/tie summary on each benchmark.
  4. [Full text (tables)] The readable fragments of the tables appear to contain blank cells and unexplained column headers. If these are model outputs, please report the full numerical values, standard errors, and the precise metric names.

Circularity Check

0 steps flagged

No circularity detected: benchmarks are external and the active-parameter claim is empirical, not definitional; the garbled full text prevents identifying any reduction.

full rationale

The central claims in the abstract are comparisons to external open-source models ("performance comparable to SOTA open-source models of similar or even larger size") and a reported active-parameter range ("dynamically activate 3.14-3.28B parameters"). These are empirical measurements against public benchmarks and fixed model configurations, not quantities defined in terms of one another. Nothing in the readable portion of the manuscript defines the reported active-parameter count as the output of the same fitted quantity later called a prediction, and no fitted parameter is renamed as a prediction. The upcycling from Qwen3-30B-A3B-Base introduces a potential confound: some of the gains could come from the added total parameters rather than the Grove architecture. That is a validity concern, not circularity. The full text is largely mojibake, so equations and ablations cannot be inspected; there is also an inserted arXiv identifier (2508.07795v3 [cs.CV]) inconsistent with the paper ID, which makes verification harder. However, under the required evidentiary standard, no specific equation or construction shows that the derivation reduces to its own inputs. Therefore the appropriate honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

Everything above is inferred from the abstract because the supplied full text is encoding-corrupted and unreadable. The ledger flags the routing policy, expert-size schedule, and data recipe as likely design decisions that are invisible here, plus the upcycling and complexity-signal assumptions that carry the architecture.

free parameters (3)
  • Token-complexity routing criteria and activation thresholds = undisclosed in abstract
    The 3.14-3.28B active-parameter range must come from some routing or threshold policy; if tuned on the evaluation benchmarks, the reported efficiency would be partly fitted.
  • Expert size configuration (count and width of small versus large experts) = undisclosed in abstract
    The heterogeneous expert schedule is a design choice typically set by experiment; not visible at abstract level.
  • Mid-training and post-training data mixing ratios = undisclosed in abstract
    Upcycled model quality depends on data recipe choices; absent from the abstract.
axioms (3)
  • domain assumption Weights upcycled from Qwen3-30B-A3B-Base preserve and extend capability during mid-training and post-training
    The abstract stakes the models entirely on upcycling; if the backbone does not transfer, the central claim fails.
  • ad hoc to paper Per-token complexity is a learnable signal that predicts how much expert capacity a token needs
    Core premise of the dynamic activation mechanism; the abstract supplies no evidence that this signal is reliable at scale.
  • standard math Standard MoE routing and training math from prior literature holds for heterogeneous expert sizes
    The paper extends existing MoE machinery; correctness of that underlying machinery is assumed.
invented entities (1)
  • Adjugate experts (heterogeneous-size experts with dynamic activation) independent evidence
    purpose: Allow per-token variable compute by mixing small and large experts inside one MoE layer
    The design is falsifiable through the reported benchmarks and the 3.14-3.28B active-parameter claim, but that evidence is not shown in the abstract.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts." pith.science (2026). https://pith.science/paper/QK5DIFKI

@misc{pith2026250807785,
  author       = {Pith},
  title        = {Pith review of: Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QK5DIFKI}},
  note         = {Machine review of arXiv:2508.07785}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The Mixture of Experts (MoE) architecture is a cornerstone of modern state-of-the-art (SOTA) large language models (LLMs). MoE models facilitate scalability by enabling sparse parameter activation. However, traditional MoE architecture uses homogeneous experts of a uniform size, activating a fixed number of parameters irrespective of input complexity and thus limiting computational efficiency. To overcome this limitation, we introduce Grove MoE, a novel architecture incorporating experts of varying sizes, inspired by the heterogeneous big.LITTLE CPU architecture. This architecture features novel adjugate experts with a dynamic activation mechanism, enabling model capacity expansion while maintaining manageable computational overhead. Building on this architecture, we present GroveMoE-Base and GroveMoE-Inst, 33B-parameter LLMs developed by applying an upcycling strategy to the Qwen3-30B-A3B-Base model during mid-training and post-training. GroveMoE models dynamically activate 3.14-3.28B parameters based on token complexity and achieve performance comparable to SOTA open-source models of similar or even larger size.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

    cs.LG 2026-04 unverdicted novelty 7.0

    Expert upcycling duplicates experts in an existing MoE checkpoint and continues pre-training to match fixed-size baseline performance with 32% less compute.

  2. Post-Trained MoE Can Skip Half Experts via Self-Distillation

    cs.LG 2026-05 unverdicted novelty 6.0

    ZEDA injects zero-output experts and uses two-stage self-distillation to adapt post-trained MoE models into dynamic ones that skip over half the experts, yielding 1.2x inference speedup with small accuracy drops.

  3. Post-Trained MoE Can Skip Half Experts via Self-Distillation

    cs.LG 2026-05 unverdicted novelty 6.0

    ZEDA turns post-trained static MoE models into dynamic ones via zero-output expert injection and two-stage self-distillation, cutting over 50% expert FLOPs on Qwen3-30B-A3B and GLM-4.7-Flash with small accuracy drops ...

  4. SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs

    cs.CV 2026-04 unverdicted novelty 6.0

    SMoES improves MoE-VLM performance and efficiency via soft modality-guided expert routing and inter-bin mutual information regularization, yielding 0.9-4.2% task gains and 56% communication reduction.

  5. Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

    cs.LG 2026-04 unverdicted novelty 6.0

    Expert upcycling expands MoE models by duplicating experts and continuing pre-training, matching baseline performance while saving 32% GPU hours in 7B-13B experiments.

  6. Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts

    cs.LG 2025-10 unverdicted novelty 5.0

    Orthogonal growth recycles pre-trained MoE checkpoints via layer copying and noisy expert duplication, delivering 10.6% higher accuracy than training from scratch with equivalent extra compute.

  7. Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling

    cs.LG 2026-06 unverdicted novelty 4.0

    A 120B sparse MoE model with 460 experts was trained on one 8-GPU node to loss 1.78 using reversible recurrence and state-preserving scaling from a 1.78B dense seed, with 5.93B active parameters.

Reference graph

Works this paper leans on

51 extracted references · 18 canonical work pages · cited by 5 Pith papers · 1 internal anchor

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    AIME Problems and Solutions , 2025

    AIME . AIME Problems and Solutions , 2025. URL https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions

    work page 2025

  3. [3]

    Alternating Updates for Efficient Transformers

    Cenk Baykal, Dylan Cutler, Nishanth Dikkala, Nikhil Ghosh, Rina Panigrahy, and Xin Wang. Alternating Updates for Efficient Transformers . In Annual Conference on Neural Information Processing Systems (NIPS), 2023

    work page 2023

  4. [4]

    MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation . IEEE Transactions on Software Engineering, 49 0 (7): 0 3675--3691, 2023

    work page 2023

  5. [5]

    Parallel Scaling Law for Language Models

    Mouxiang Chen, Binyuan Hui, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Jianling Sun, Junyang Lin, and Zhongxin Liu. Parallel Scaling Law for Language Models . arXiv preprint arXiv:2505.10475, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training Verifiers to Solve Math Word Problems . arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  7. [7]

    SuperGPQA: Scaling llm evaluation across 285 graduate disciplines

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. SuperGPQA: Scaling llm evaluation across 285 graduate disciplines . arXiv preprint arXiv:2502.14739, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning over Paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning over Paragraphs . arXiv preprint arXiv:1903.00161, 2019

    Pith/arXiv arXiv 1903

  9. [9]

    Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models . arXiv preprint arXiv:2410.07985, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Gemma 3 Technical Teport

    Gemma, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram \'e , Morgane Rivi \`e re, et al. Gemma 3 Technical Teport . arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Gemini2.5 Pro

    Google DeepMind . Gemini2.5 Pro . https://deepmind.google/technologies/gemini/pro/, 2025

    work page 2025

  12. [12]

    The Llama 3 Herd of Models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 Herd of Models . arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    big.LITTLE Processing with ARL Cortex-A15 & Cortex-A7

    Peter Greenhalgh. big.LITTLE Processing with ARL Cortex-A15 & Cortex-A7 . ARM White paper, 17, 2011

    work page 2011

  14. [14]

    CruxEval: A Benchmark for Code Reasoning, Understanding and Execution

    Alex Gu, Baptiste Rozi \`e re, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. CruxEval: A Benchmark for Code Reasoning, Understanding and Execution . arXiv preprint arXiv:2401.03065, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models

    Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin. Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models . arXiv preprint arXiv:2405.14297, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems . arXiv preprint arXiv:2402.14008, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding . arXiv preprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  18. [18]

    Measuring Mathematical Problem Solving with the Math Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving with the Math Dataset . arXiv preprint arXiv:2103.03874, 2021

    Pith/arXiv arXiv 2021

  19. [19]

    Harder Tasks Need More Experts: Dynamic Routing in MoE Models

    Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder Tasks Need More Experts: Dynamic Routing in MoE Models . arXiv preprint arXiv:2403.07652, 2024

    arXiv 2024

  20. [20]

    C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models . Advances in Neural Information Processing Systems, 36: 0 62991--63010, 2023

    work page 2023

  21. [21]

    dots.llm1 Technical Report

    Bi Huo, Bin Tu, Cheng Qin, Da Zheng, Debing Zhang, Dongjie Zhang, En Li, Fu Guo, Jian Yao, Jie Lou, et al. dots.llm1 Technical Report . arXiv preprint arXiv:2506.05767, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    GPT-4o System Card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o System Card . arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code . arXiv preprint arXiv:2403.07974, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Mixtral of Experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of Experts . arXiv preprint arXiv:2401.04088, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts

    Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts . arXiv preprint arXiv:2410.07348, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Sparse Upcycling: Training mixture-of-experts from dense checkpoints

    Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse Upcycling: Training mixture-of-experts from dense checkpoints . In International Conference on Learning Representations (ICLR), 2023

    work page 2023

  27. [27]

    CMMLU: Measuring Massive Multitask Language Understanding in Chinese

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: Measuring Massive Multitask Language Understanding in Chinese . arXiv preprint arXiv:2306.09212, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and Benchbuilder Pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and Benchbuilder Pipeline . arXiv preprint arXiv:2406.11939, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Let's Verify Step by Step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's Verify Step by Step . In The Twelfth International Conference on Learning Representations, 2023

    work page 2023

  30. [30]

    DeepSeek-V3 Technical Report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 Technical Report . arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation . Advances in Neural Information Processing Systems, 36: 0 21558--21572, 2023

    work page 2023

  32. [32]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization . arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  33. [33]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation , 2025

    Meta-AI . The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation , 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence/

    work page 2025

  34. [34]

    Mistral-Small-3.1

    Mistral AI . Mistral-Small-3.1 . https://mistral.ai/news/mistral-small-3-1, 2025

    work page 2025

  35. [35]

    Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-Initialization

    Taishi Nakamura, Takuya Akiba, Kazuki Fujii, Yusuke Oda, Rio Yokota, and Jun Suzuki. Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-Initialization . arXiv preprint arXiv:2502.19261, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    GPQA: A Graduate-Level Google-Proof Q&Q Benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A Graduate-Level Google-Proof Q&Q Benchmark . In First Conference on Language Modeling, 2024

    work page 2024

  37. [37]

    sgl-project . SGLang . https://github.com/sgl-project/sglang, 2025

    work page 2025

  38. [38]

    MoE Travels 3 , 2025

    Jianlin Su. MoE Travels 3 , 2025. URL https://kexue.fm/archives/10757

    work page 2025

  39. [39]

    Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent

    Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, et al. Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent . arXiv preprint arXiv:2411.02265, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Challenging Big-Bench Tasks and Whether Chain-of-Thought Can Solve Them

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging Big-Bench Tasks and Whether Chain-of-Thought Can Solve Them . arXiv preprint arXiv:2210.09261, 2022

    Pith/arXiv arXiv 2022

  41. [41]

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark . In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024 a

    work page 2024

  42. [42]

    ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

    Ziteng Wang, Jun Zhu, and Jianfei Chen. ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing . arXiv preprint arXiv:2412.14711, 2024 b

    Pith/arXiv arXiv 2024

  43. [43]

    Magicoder: Empowering Code Generation with OSS-Instruct

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering Code Generation with OSS-Instruct . arXiv preprint arXiv:2312.02120, 2023

    Pith/arXiv arXiv 2023

  44. [44]

    Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

    Haoyuan Wu, Haisheng Zheng, Zhuolun He, and Bei Yu. Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks . In Empirical Methods in Natural Language Processing (EMNLP), 2024

    work page 2024

  45. [45]

    Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing . arXiv preprint arXiv:2406.08464, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Patil, Ion Stoica, and Joseph E

    Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley Function Calling Leaderboard . https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html, 2024

    work page 2024

  47. [47]

    Qwen2.5 Technical Report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 Technical Report . arXiv preprint arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 Technical Report . arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models

    Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models . arXiv preprint arXiv:2406.13233, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  50. [50]

    AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models . arXiv preprint arXiv:2304.06364, 2023

    Pith/arXiv arXiv 2023

  51. [51]

    Instruction-Following Evaluation for Large Language Models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-Following Evaluation for Large Language Models . arXiv preprint arXiv:2311.07911, 2023

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.