Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

A redesigned memory layer matches 8-expert MoE at equal compute and parameters—and wins on long-context recall.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A redesigned memory-layer architecture with five engineering improvements reaches performance parity with 8-expert MoE at similar compute, with lower memory access and stronger long-context memorization.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The architecture is a genuine step forward for memory layers and the small-scale matched comparisons support parity, but the flagship proprietary claim compares 60B total params against 30B, so the headline result is not yet proven at scale. the 4 major comments →

arxiv 2508.18756 v1 pith:3GYUN2UR submitted 2025-08-26 cs.LG

UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning

Zihao Huang , Yu Bao , Qiyang Min , Siyan Chen , Ran Guo , Hongzhi Huang , Defa Zhu , Yutao Zeng
show 3 more authors
Banggu Wu Xun Zhou Siyuan Qiao
This is my paper
classification cs.LG
keywords memory layersmixture of expertssparse modelslong-context learningproduct key memoryinference efficiencyactivation densityMoE parity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a memory-layer architecture—where the model retrieves stored values from a large table instead of routing tokens through expert networks—can now match the performance of state-of-the-art 8-expert mixture-of-experts models under equal computation and parameter counts. Previous memory-layer designs, including the authors' own UltraMem, only matched 2-expert MoE. UltraMemV2 closes that gap with five changes, and goes further on memory-heavy jobs: long-context memorization, multi-round recall, and in-context learning improve over an MoE counterpart by 1.6, 6.2, and 7.9 points respectively. The paper also argues, from a scale-up to 120B total parameters, that how many values a token activates matters more than how many sparse parameters exist. If right, this makes memory layers a credible low-memory-access alternative for large sparse language models.

Core claim

UltraMemV2 is a memory-layer architecture that places a memory layer in every transformer block, retrieves values via Tucker-decomposed query-key scoring (TDQKR), and processes the activated values with a single shared projector and an FFN-style value computation taken from PEER. With principled parameter initialization and a rebalanced memory-to-FFN compute split (about 17% to memory), the authors report that 2.5B-activated-parameter models over 60B-120B total parameters match SeedMoE baselines with 8 activated experts at equal compute and parameters, after continued training. On retention-heavy benchmarks the memory layer wins: +1.6 on long-context memorizing, +6.2 on multi-round memorizin

What carries the argument

The load-bearing object is the UltraMemV2 memory layer: row and column key scores pass through a small Tucker core to form a top-m grid over a huge factorized value table, and only the top-m values are read—which is what keeps memory access far below expert routing. The five changes—memory layer in every block, a single shared value projector, PEER-style FFN values, initialization variance matched to the FFN's, and a roughly 17% memory compute share—are what make the retrieved values competitive with routed experts.

Load-bearing premise

The parity claim assumes the proprietary SeedMoE baselines were trained with comparable data quality, compute budget, and hyperparameter tuning; the paper reports no FLOP-matched training curves, latency numbers, or baseline tuning details, so a weaker baseline would make the parity result an artifact of the comparison.

What would settle it

Train an 8-expert MoE and an UltraMemV2 model on identical data with matched FLOPs and equal hyperparameter-tuning effort, then compare loss curves and benchmark scores at every checkpoint: if the MoE leads at equal compute throughout training, the parity claim fails. A second checkable observation: the paper claims significantly low memory access but reports no measured per-token bytes read or serving latency, so directly measuring memory traffic in a deployed setting would test that part of the claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Memory-layer architectures reach performance parity with 8-expert MoE under equal compute and parameters while reading far less memory at inference, making them a practical alternative for efficient sparse models.
  • On memory-intensive workloads—long-context memorization, multi-round dialogue recall, and in-context learning—memory layers beat expert routing by 1.6, 6.2, and 7.9 points respectively.
  • Activation density beats parameter count: with matched activated parameters, a 60B table with top-768 outperforms a 120B table with top-256 on the same benchmarks.
  • Training simplifies: the Tucker core penalty loss, balance loss, and decaying value learning-rate schedule—all used in prior memory-layer work—can be removed without hurting final performance.
  • The design scales to 2.5B activated / 120B total parameters, with the rule Dk ≈ h/2 and key count growing only as the square root of hidden size keeping the memory compute share stable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's own table shows a large MoE advantage on multi-hop reasoning (17.7 vs 34.8) even where memory layers win overall—suggesting memory layers store and retrieve associations well but chain them poorly; testing a hybrid that keeps a few routing experts in late blocks is a natural follow-up the paper leaves implicit.
  • Parity is demonstrated after continued training on high-quality data; at the 1.6T-token pretraining checkpoint MoE leads on math, code, and reasoning. A FLOP-matched training-curve comparison would reveal when, and whether, the crossover happens—something the paper does not report.
  • If activation density is the dominant lever, the next bottleneck is retrieval cost growing with sequence length; measuring the crossover where top-m memory access exceeds expert-routing cost would decide how far this architecture class can scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. UltraMemV2 proposes a redesigned memory-layer architecture intended to close the performance gap between memory-layer models and state-of-the-art 8-expert MoE models. The five main changes are: inserting memory layers into every transformer block, simplifying implicit value expansion to a single shared linear projection, replacing value embeddings with an FFN-based value processing derived from PEER (with the activation removed), a new initialization scheme, and a rebalanced memory-to-FFN computation ratio. The paper reports proprietary SeedMoE comparisons, open-source OLMoE/Memory+/UltraMem comparisons at 227M/1.2B and 1B/7B, extensive component ablations, and a scaling study up to a 2.5B-activated/120B-total model. The headline claims are performance parity with 8-expert MoE under the same computation and parameters, superior long-context memorization, multi-round memorization and in-context learning, and a design principle that activation density matters more than total sparse parameter count.

Significance. If the parity claim survives a properly controlled comparison, this is a significant result for sparse model design: memory layers with very low memory access are an appealing alternative to MoE, and the paper gives concrete architectural guidance (memory layer per block, shared single projector, PEER-style values, initialization, and computation split). The paper also contributes careful open-source comparisons at two matched parameter scales, detailed ablations of value processing, head count, memory placement, sharing topologies, and learning-rate schedules, and it explicitly acknowledges training-stage limitations. These are valuable even if the proprietary flagship comparison needs revision. The strongest weakness is that the flagship proprietary comparison is not iso-parameter, which directly affects the abstract's central claim.

major comments (4)
  1. [Section 4.1, Tables 1-2] The flagship parity claim and the long-context advantages (+1.6, +6.2, +7.9) compare UltraMemV2-2.5B/60B-top768 trained on 3.9T PT + 500B CT to SeedMoE-2.5B/30B at the same schedule. SeedMoE-2.5B/60B is only reported at 1.6T PT + 250B CT. Since the notation is activated/total parameters, the comparison is not iso-parameter (60B vs 30B total), and the abstract's 'same computation and parameters' is not satisfied. The 2x total sparse parameter gap is a first-order confound: the results are also consistent with extra memorization capacity. Please either add a SeedMoE-2.5B/60B run at 3.9T+500B, or explicitly restrict the parity claim to iso-parameter open-source comparisons and state the proprietary comparison as not iso-parameter.
  2. [Section 4.1, Table 2] Multi-hop reasoning drops from 34.8 (SeedMoE-2.5B/30B) to 17.7 (UltraMemV2-2.5B/60B-top768). The paper says performance variations are 'attributable to architectural differences rather than parameter count disparities' with no supporting analysis. Given the comparison is already confounded by total parameter count, this 17-point regression is a load-bearing caveat that must be reconciled or at least analyzed (e.g., by task-level ablations or a SeedMoE-2.5B/60B baseline) before the 'superior long-context learning' title claim is supported.
  3. [Section 4.1, bullet 3; Table 1] The claim 'activation density has greater impact on performance than total sparse parameter count' is supported by comparing UltraMemV2-2.5B/60B-top768 to UltraMemV2-2.5B/120B-top256, but both TopM and total parameter count change between the two runs. This is not a controlled ablation for activation density. The observed differences are small and mixed across sub-benchmarks (e.g., OpenBench knowledge 80.7 vs 80.3, but OpenBench All 68.3 vs 69.1). Please provide a controlled sweep (vary TopM under fixed total parameters and fixed compute) or rephrase the conclusion as a config comparison.
  4. [Section 4.1, training protocol] The parity claim rests on 'same computation and parameters', but the paper does not report FLOP counts, training wall-clock, or inference latency/memory-access measurements, nor baseline hyperparameter budgets for SeedMoE. Without these, the efficiency parity cannot be verified; lower memory access is asserted but not measured in this work.
minor comments (6)
  1. [Table 3 and Table 10] The open-source model is called 'UltraMemV2-1M/7B' in Table 3 and Table 10, but the text and context indicate '1B/7B'. Please correct the typo.
  2. [Section 3.6 vs Section 4.2.7] Section 3.6 says the two auxiliary losses are 'NOT used in UltraMemV2', but Section 4.2.7 reports ablation experiments with and without them. Clarify that they are not used in the final model, while being studied in ablations.
  3. [Section 3.4] The paper says it 'adopt[s] PEER' but then removes the activation function, and states 'this change will be uniformly abbreviated as PEER'. Calling a modified variant 'PEER' can confuse readers; consider a distinct name or explicit notation.
  4. [Appendix A] The initialization derivation relies on calibrating the top-k score mean/variance using randomly generated data whose distribution is assumed to match training-time distributions. This is a heuristic calibration step rather than a closed-form derivation; state this clearly in the main text if it is part of the claimed 'principled initialization'.
  5. [Section 4.2.4] The choice of MCP=17% and the scaling rule Dk=h/2 are based on a single small-scale ablation (500M/6B) with no error bars or multiple seeds. This may be acceptable, but the lack of uncertainty makes the rule look more deterministic than the evidence supports.
  6. [Throughout] There are several typos and formatting issues: 'Prelimilary' (Section 3.1), 'Resent MOEs' (Section 2), 'dots.llm1[19]' reference formatting, and inconsistent use of 'top-k'/'TopM'. These do not affect the technical content.

Circularity Check

0 steps flagged

No load-bearing circularity: empirical architecture comparisons with minor self-citations but no reduction of predictions to inputs.

full rationale

The paper's main claims are empirical: UltraMemV2 is compared against proprietary SeedMoE models and open-source OLMoE/Memory+/UltraMem baselines. No fitted constant is later renamed as a prediction, and no result is derived from an equation that presupposes the conclusion. The initialization in Section 3.5 and Appendix A sets the memory-layer output variance equal to the FFN output variance; this is an explicit design target for stability, not a circular input/output relation. The derivation of sigma_V is a calibration rule, and the top-k score statistics are estimated from random data in Appendix A.2, again a calibration choice rather than a claimed prediction. The adoption of PEER's FFN-based value processing is attributed to an external prior work [12], and the paper's modification (removing the activation) is motivated by an empirical argument, not by a self-citation chain. The 'activation density has greater impact than total sparse parameter count' statement is a post-hoc interpretation of two large runs that differ in two variables simultaneously; this is a confound and a weakness in inference, but it is not a circular derivation from the model equations. Self-citations to UltraMem [18] and Over-tokenized Transformer [16] are present, but they are used for background, baseline comparisons, or ablation context, and they are not invoked as a uniqueness theorem or as the sole justification for the central parity claim. The parity claim is anchored by direct open-source comparisons in Table 3 where total parameters are matched. The proprietary comparison in Table 1/2 compares UltraMemV2-2.5B/60B with SeedMoE-2.5B/30B, i.e., a 2x total-parameter difference; this is a legitimate fairness/correctness concern but not circularity, since the result is not forced by construction. Overall, no step in the paper reduces to its own inputs; the low score reflects only the presence of minor, non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical or latent entities; it recombines existing architectural components. The free parameters listed are tuned on small-scale ablations and then baked into the scaled-up models, which is standard practice but means the final configuration is partly fitted to the validation objectives.

free parameters (5)
  • Memory compute proportion (MCP) = 17% of compute
    Chosen by ablation across 12.0% to 27.5% in Table 4/Figure 5; the paper later uses Dk=h/2 at scale, which changes MCP.
  • Key dimension Dk = h/2 at scale
    Adopted from ablation in Section 4.2.4 as 'a reasonable configuration'.
  • Value/pre-value dimension ratio Dp:Dv = 1:3
    Chosen from ablation in Section 4.2.5 (Table 5, Figure 6 right).
  • TopM (activated values per layer) = 32 to 768 depending on config
    Design choice balancing compute, memory access, and performance; e.g., top768/60B vs top256/120B.
  • Memory layer count = memory layer in every block (e.g., 20 or 24)
    Ablation in Section 4.2.6 shows downstream gains from more layers despite saturated validation loss.
axioms (4)
  • standard math Central Limit Theorem applies to sums in the initialization variance derivation
    Used in Appendix A to derive output variance of memory layer and FFN.
  • domain assumption Swish-activated gate distribution can be approximated as a truncated normal
    Appendix A.3 uses this approximation to compute FFN output variance.
  • ad hoc to paper Top-k score distribution can be calibrated to mean 1 via query/key norm initialization, and the variance measured on random data represents training
    Appendix A.2 uses random-data simulation rather than a closed-form derivation.
  • domain assumption Performance on the proprietary OpenBench/HardBench/long-context benchmarks transfers to general capability
    All headline comparisons in Section 4.1 rely on in-house benchmarks not released.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning." pith.science (2026). https://pith.science/paper/3GYUN2UR

@misc{pith2026250818756,
  author       = {Pith},
  title        = {Pith review of: UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3GYUN2UR}},
  note         = {Machine review of arXiv:2508.18756}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

While Mixture of Experts (MoE) models achieve remarkable efficiency by activating only subsets of parameters, they suffer from high memory access costs during inference. Memory-layer architectures offer an appealing alternative with very few memory access, but previous attempts like UltraMem have only matched the performance of 2-expert MoE models, falling significantly short of state-of-the-art 8-expert configurations. We present UltraMemV2, a redesigned memory-layer architecture that closes this performance gap. Our approach introduces five key improvements: integrating memory layers into every transformer block, simplifying value expansion with single linear projections, adopting FFN-based value processing from PEER, implementing principled parameter initialization, and rebalancing memory-to-FFN computation ratios. Through extensive evaluation, we demonstrate that UltraMemV2 achieves performance parity with 8-expert MoE models under same computation and parameters but significantly low memory access. Notably, UltraMemV2 shows superior performance on memory-intensive tasks, with improvements of +1.6 points on long-context memorization, +6.2 points on multi-round memorization, and +7.9 points on in-context learning. We validate our approach at scale with models up to 2.5B activated parameters from 120B total parameters, and establish that activation density has greater impact on performance than total sparse parameter count. Our work brings memory-layer architectures to performance parity with state-of-the-art MoE models, presenting a compelling alternative for efficient sparse computation.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MIDUS: Memory-Infused Depth Up-Scaling

    cs.LG 2025-12 unverdicted novelty 7.0

    MIDUS replaces duplicated FFN branches in depth up-scaling with head-wise memory layers using product-key retrieval and HIVE to deliver lightweight, head-conditioned residual capacity.

  2. SinkRec: Mitigating Semantic State Sink in Long Sequence Recommendation with Memory-Conditioned Gated Delta Networks

    cs.LG 2026-06 unverdicted novelty 5.0

    SinkRec proposes a memory-conditioned architecture with TDGD to mitigate semantic state sink in linear attention for long-sequence recommendation.

Reference graph

Works this paper leans on

43 extracted references · 7 canonical work pages · cited by 2 Pith papers

  1. [1]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  2. [2]

    Memory layers at scale.arXiv preprint arXiv:2412.09764, 2024

    Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, and Gargi Ghosh. Memory layers at scale.arXiv preprint arXiv:2412.09764, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. InThirty-FourthAAAI Conference on Artificial Intelligence, 2020

    work page 2020

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  5. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1, 2018

    Pith/arXiv arXiv 2018

  6. [6]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  7. [7]

    Approximating two-layer feedforward networks for efficient transformers

    Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. Approximating two-layer feedforward networks for efficient transformers. arXiv preprint arXiv:2310.10837, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. InProc. of NAACL, 2019

    work page 2019

  10. [10]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  11. [11]

    Fastmoe: A fast mixture-of-expert training system

    Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. Fastmoe: A fast mixture-of-expert training system. arXiv preprint arXiv:2103.13262, 2021

    Pith/arXiv arXiv 2021

  12. [12]

    Mixture of a million experts.arXiv preprint arXiv:2407.04153, 2024

    Xu Owen He. Mixture of a million experts.arXiv preprint arXiv:2407.04153, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Aligning ai with shared human values.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  14. [14]

    Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  15. [15]

    Measuring mathematical problem solving with the math dataset.NeurIPS, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.NeurIPS, 2021

    2021

  16. [16]

    Over-tokenized transformer: Vocabulary is generally worth scaling.arXiv preprint arXiv:2501.16975, 2025

    Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, and Xun Zhou. Over-tokenized transformer: Vocabulary is generally worth scaling.arXiv preprint arXiv:2501.16975, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. InAdvances in Neural Information Processing Systems, 2023. 18

    work page 2023

  18. [18]

    Ultra-sparse memory network

    Zihao Huang, Qiyang Min, Hongzhi Huang, Defa Zhu, Yutao Zeng, Ran Guo, and Xun Zhou. Ultra-sparse memory network. arXiv preprint arXiv:2411.12364, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Bi Huo, Bin Tu, Cheng Qin, Da Zheng, Debing Zhang, Dongjie Zhang, En Li, Fu Guo, Jian Yao, Jie Lou, et al. dots. llm1 technical report.arXiv preprint arXiv:2506.05767, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Product quantization for nearest neighbor search.IEEE transactions on pattern analysis and machine intelligence, 33(1):117–128, 2010

    Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search.IEEE transactions on pattern analysis and machine intelligence, 33(1):117–128, 2010

    2010

  21. [21]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Matt Gardner Johannes Welbl, Nelson F. Liu. Crowdsourcing multiple choice science questions. 2017

    2017

  23. [23]

    triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.arXiv e-prints, art

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.arXiv e-prints, art. arXiv:1705.03551, 2017

    Pith/arXiv arXiv 2017

  24. [24]

    Large product key memory for pretrained language models.arXiv preprint arXiv:2010.03881, 2020

    Gyuwan Kim and Tae-Hwan Jung. Large product key memory for pretrained language models.arXiv preprint arXiv:2010.03881, 2020

    Pith/arXiv arXiv 2010

  25. [25]

    Scaling laws for fine-grained mixture of experts

    Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Large memory layers with product keys.Advances in Neural Information Processing Systems, 32, 2019

    Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large memory layers with product keys.Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  27. [27]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InEMNLP, 2018

    2018

  29. [29]

    Olmoe: Open mixture-of-experts language models.arXiv preprint arXiv:2409.02060, 2024

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models.arXiv preprint arXiv:2409.02060, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Transformers without tears: Improving the normalization of self-attention

    Toan Q Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv:1910.05895, 2019

    Pith/arXiv arXiv 1910

  31. [31]

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. InInternational conference on machine learning, pages 18332–18346. PMLR, 2022

    2022

  32. [32]

    Winogrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, 2019

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, 2019

    Pith/arXiv arXiv 1907

  33. [33]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Pith/arXiv arXiv 2002

  34. [34]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprintarXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprintarXiv:1701.06538, 2017

    Pith/arXiv arXiv 2017

  35. [35]

    Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    Pith/arXiv arXiv 2022

  36. [36]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Volume1 (Long and Short Papers), pages 4149–4158, Minneapolis, M...

    doi:10.18653/v1/n19-1421 2019

  37. [37]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 19

    work page 2017

  38. [38]

    Moec: Mixture of expert clusters

    Yuan Xie, Shaohan Huang, Tianyu Chen, and Furu Wei. Moec: Mixture of expert clusters. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13807–13815, 2023

    work page 2023

  39. [39]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Expert race: A flexible routing strategy for scaling diffusion transformer with mixture of experts.arXiv preprint arXiv:2503.16057, 2025

    Yike Yuan, Ziyu Wang, Zihao Huang, Defa Zhu, Xun Zhou, Jingyi Yu, and Qiyang Min. Expert race: A flexible routing strategy for scaling diffusion transformer with mixture of experts.arXiv preprint arXiv:2503.16057, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  42. [42]

    Ape210k: A large-scale and template-rich dataset of math word problems.arXiv preprint arXiv:2009.11506, 2020

    Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. Ape210k: A large-scale and template-rich dataset of math word problems.arXiv preprint arXiv:2009.11506, 2020

    Pith/arXiv arXiv 2009

  43. [43]

    Pre-values

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023. 20 Appendix A Optimized Initialization First, we list the important theorems required for the derivation, starting with theCentral Limit Theorem(CLT): If x1, x2, ..., x...

    work page 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.