Pith. sign in

REVIEW 5 major objections 6 minor 68 references

Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Token importance decides expert count, beating fixed top-k

desk verdict A well-described recombination of known MoE/attention ideas whose central empirical claim is untested: no long-sequence benchmark anywhere, impossible compute claims, and internally inconsistent GLUE tables. read the letter →

arxiv 2509.10530 v1 pith:HV364GEF submitted 2025-09-05 cs.LG cs.AI

Cheng Li , Jiexiong Liu , Yixuan Chen , Jie ji This is my paper · ORCID
classification cs.LGcs.AI
keywords mixtureofexpertsdynamicroutingtokenimportancegroupedmulti-headattentionlong-sequencemodelingdual-scalesharedadaptivesparseactivation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes DASG-MoE, a mixture-of-experts transformer in which the number and depth of experts assigned to a token depend on an importance score extracted from the attention distribution rather than a fixed top-k. The authors claim this changes the efficiency-accuracy tradeoff: low-information tokens use one lightweight expert while significant tokens can use several, and splitting the sequence into sixteen windowed groups cuts the attention cost. The supporting experiments report that the model beats the baseline architecture on six of eight tasks in an established language-understanding benchmark, with a maximum gain of 4.64 points, and that it also posts the highest average in the paper's broad comparison of open models. If correct, this offers a practical way to scale long-context mixture-of-experts models without paying full attention cost or wasting expert capacity on trivial tokens. The main caveat, visible in the setup section, is that the reported pre-training scale is hard to reconcile with the described hardware.

What carries the argument

The argument turns on three coupled modules. Grouped Multi-Head Attention (GMHA) splits the input sequence $X \in \mathbb{R}^{N\times d}$ into $G=16$ subgroups, applies sliding-window multi-head attention inside each group with a mask $M_{\text{window}}$, passes each group's output through its own MLP, and concatenates the results; this is claimed to cut attention cost from $O(N^2)$ to $O(N^2/G)$ with a local window of size $w$. Dual-Scale Shared Experts (DSSE) provide two tiers: eight shallow one-layer-MLP experts for low-dimensional features and eight deep three-layer-MLP experts for complex semantics, with the deep experts' first layer copied from pre-trained shallow weights and frozen. Adaptive Dynamic Routing (ADR) connects the two: a lightweight two-layer evaluator maps each token's attention weights to an importance score $I_i$ and a complexity score $C_i$, a global router selects the shallow or deep module using a softmax decision trained with time and accuracy rewards, and a local router then activates the top two experts inside the chosen module. A threshold rule with $\theta_s=0.3$ and $\theta_d=0.7$ decides when a token is simple enough for shallow-only handling or complex enough for the deep module, and the number of experts allocated grows with $I_i$ through $K_i = \lceil L_i K \rceil$.

What would settle it

Compute the wall-clock time implied by 5 trillion tokens on eight accelerators; if a single 8-billion-parameter run requires many months, the training curves in the paper cannot come from the described setup. A smaller reproduction at the stated 0.6-billion-parameter scale, with the adaptive router enabled versus disabled, should either restore or erase the reported benchmark deltas.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that routing by token importance is enough to improve both accuracy and efficiency over a fixed-top-k mixture-of-experts baseline. In the pre-training comparison, DASG-MoE outperforms the baseline on six of the eight benchmark tasks, with the largest improvement of 4.64 points in accuracy or F1 at the largest tested size, and the average scores move from 66.72 to 68.12. In the fine-tuning comparison, the model again wins six of eight tasks, losing only on the grammatical-acceptability and question-inference tasks. The paper also presents a broad comparison in which the instruction-tuned model averages 81.6 across eight evaluation suites, above the listed comparison models, and the ablation studies attribute the gains to the adaptive routing module and to the sixteen-group attention setting.

Load-bearing premise

The reported pre-training of an 8-billion-parameter model on 5 trillion tokens, and a 3-billion-parameter model on 2.2 trillion tokens, is attributed to a setup of eight high-end accelerators, a scale that does not obviously fit that hardware, and all empirical comparisons rest on that premise.

Editorial extensions

If this is right

  • Fixed top-k routing is not required: a token-importance-dependent expert count can raise average benchmark accuracy while cutting computation on low-information tokens, if the reported comparisons hold.
  • Sixteen sequence groups is the reported sweet spot for grouped windowed attention; using 32 or 64 groups degrades accuracy, with paraphrase detection suffering the most.
  • The failure on the paraphrase and sentiment tasks is attributed to over-routing simple tokens into deep experts, so routing thresholds must be task-aware rather than global.
  • Scaling the architecture from 0.6B to 8B parameters both accelerates training convergence and increases the margin over the baseline, supporting the claim that adaptive allocation scales well.
  • Because the framework is described as generic, it can be dropped into other mixture-of-experts transformer variants, not only the baseline it was tested in.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same importance evaluator could plausibly be reused to prune tokens or skip attention heads, since the paper only uses it to choose expert count and depth.
  • The described compute budget suggests either a much larger cluster than stated or a different experimental lineage; reproducing the smallest reported run at the stated token count would resolve which.
  • A task-conditional routing prior, such as forcing paraphrase tasks to use mostly shallow experts, would likely close the observed paraphrase gap; the paper's threshold analysis points there but stops short of testing that variant.
  • The reported $O(N^2/G)$ complexity claim should be checked against the actual sliding-window implementation, because the mask still limits each token to a window of size $w$ and the group count mainly changes parallelism and aggregation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes DASG-MoE, a Mixture-of-Experts Transformer variant that combines grouped multi-head attention (GMHA) with sliding windows, a dual-scale shared expert structure (DSSE) with shallow and deep expert modules, and an adaptive dynamic routing (ADR) mechanism that allocates a variable number of experts per token based on a learned importance score. The authors claim that this architecture outperforms state-of-the-art models on multiple long-sequence benchmarks. The experiments reported in Sections 5.1-5.6 cover GLUE tasks, MMLU, CMMLU, CEval, BBH, GSM8K, MATH500, MBPP, and HumanEval, together with ablations on the routing mechanism and the number of attention groups.

Significance. Variable expert allocation is a plausible and potentially useful extension of static top-k MoE routing, and the paper identifies a real limitation of fixed expert budgets. However, the significance cannot be assessed from the present manuscript because the central empirical claims are unsupported: no long-sequence benchmark is evaluated, the reported pretraining scale is incompatible with the stated hardware, and the GLUE numbers are internally inconsistent. On the positive side, the paper provides ablations for the number of attention groups and for routing versus random forwarding, and it reports confidence intervals for HumanEval and MBPP; these are useful elements, but they do not compensate for the missing evidence on the core claims. The manuscript also lacks code, data, or sufficient training details to be reproducible.

major comments (5)
  1. [Abstract, Sections 5.2-5.3, Conclusion] The central claim of state-of-the-art performance on 'multiple long-sequence benchmark datasets' is not tested anywhere in the paper. Section 5.2 evaluates GLUE tasks (SST-2, MRPC, MNLI, etc.), whose instances are short sentences, and Table 1 lists MMLU, CMMLU, CEval, BBH, GSM8K, MATH500, MBPP, and HumanEval, none of which is a long-sequence benchmark. No LongBench, Long Range Arena, SCROLLS, or similar long-context evaluation appears in the manuscript. The conclusion itself admits that 'conducting extensive evaluations on larger-scale long-sequence datasets' is future work, which confirms that the claimed long-sequence evaluation was not performed.
  2. [Sections 5.1 and 5.2] The reported training setup is not credible. Section 5.1 states that the infrastructure is 8 NVIDIA A100 GPUs, while Section 5.2 reports pre-training an 8B-parameter model on 5T tokens and a 3B model on 2.2T tokens. Even under optimistic assumptions about A100 throughput and MoE sparsity, training an 8B model on 5T tokens requires on the order of 10^23 FLOPs, which would take many months or years on a few A100s. The paper provides no training time, FLOP count, parallelism strategy, or data-mixture details. Without a credible explanation of how this computation was performed, the empirical results cannot be considered reliable.
  3. [Tables 2 and 3, Section 5.2] The GLUE results are internally inconsistent. Table 2 reports an 8B model with SST-2 at 80.13, MRPC at 73.01, and MNLI at 74.79, while Table 3, described as fine-tuning after C4 pre-training, reports 95.89, 88.42, and 90.02 on the same tasks. The text in Section 5.2 says the 8B configuration 'outperforms the ST baseline on 6 out of 8 GLUE tasks, with the exception of the MRPC and SST-2 tasks,' which matches Table 2 but not Table 3, where CoLA and QNLI are the exceptions. These tables appear to describe different models or protocols, and the paper does not reconcile them.
  4. [Section 3, Eq. (19), Table 5] The adaptive-allocation result is largely built into the definitions. Equations (2)-(3) define K_i as a non-decreasing function of L_i, so 'important tokens receive more experts' holds by construction; the empirical question is whether the learned importance L_i from the lightweight evaluator in Eq. (19) tracks semantic importance. The paper does not validate the evaluator against any ground-truth importance measure, and the Table 5 ablation replaces adaptive routing with random forwarding rather than with a static equal-allocation baseline. Thus the experiments do not establish that the importance-based allocation rule, rather than the extra capacity or the routing signal generally, is responsible for the reported gains.
  5. [Section 3, Eqs. (9)-(10)] The complexity claim is internally inconsistent. The text says GMHA reduces attention complexity from O(N^2) to O(Nw), but the last paragraph of Section 3 says it reduces complexity to O(N^2/g). If tokens are split into g groups and attention is computed within each group, the cost is O(N^2/g), which is still quadratic in N; only the sliding-window mask of Eq. (10) yields O(Nw). The paper should state which mechanism is responsible for linear scaling and provide actual FLOP counts, because the long-sequence efficiency argument depends on this distinction.
minor comments (6)
  1. [Table 2 caption] The caption says 'B and E represent millions, billions, and experts,' but B is used both for billion and for baseline; the caption and column headers should be clarified.
  2. [Section 5.1] The text refers to 'MiniCPM4' as the model whose benchmarks are being selected, although the paper's model is DASG-MoE; this appears to be a copy-paste error.
  3. [Section 5.5] Section 5.5 says the threshold-refinement results are shown in Table 6, but Table 6 is the grouping ablation; the threshold experiments appear in Table 4, and the cross-references are incorrect.
  4. [References] Several references are mismatched: [59] and [69] both cite DenseNet for unrelated claims, and [66] cites a linear-attention theory paper for the source of Hugging Face ST model weights.
  5. [Section 3, Eq. (8)] Equation (8) uses g(C_i) on the left but Eq. (5) defines g(C_i,T_j), and the right-hand side mixes scalar and vector quantities; the notation should be made consistent.
  6. [Sections 5.2 and 5.6] A paragraph about dynamic routing and group ablation is duplicated nearly word-for-word between Sections 5.2 and 5.6; the repeated text should be removed.

Circularity Check

3 steps flagged · score 7.0 of 10

Adaptive routing is a definitional monotone map (Eqs. 2-3), the evaluator is 'verified' by restating it (Eq. 19), and routing thresholds are tuned on the same benchmarks later reported as predictions.

  1. self definitional [Section 3, Problem Statement, Equations (2)-(3)]
    "K_i = f(I_i) (2) ... f(I_i) = ⌈L_i × K⌉ (3) ... In this scheme, tokens with larger I_i receive more experts, and tokens with smaller I_i receive fewer, improving resource efficiency."

    The claimed behavior—important tokens get more experts—is not derived or tested; it is inserted as the definition of f. Eq. (2) requires f to be non-decreasing and Eq. (3) makes K_i a non-decreasing function of L_i, so the statement 'larger I_i -> more experts, smaller I_i -> fewer experts' is exactly what the formula enforces. Any downstream claim of dynamic adaptability inherits this tautology rather than being confirmed by experiment.

  2. self definitional [Section 4.4, Lightweight Evaluator, Equation (19)]
    "[I_i, C_i]^T = W_2·ReLU(W_1·AttnWeights(x_i)+b_1)+b_2 (19) ... Taking the sentiment analysis in Figure 1 as an example, the I_i=0.82 of 'pretty' triggers 3 experts, while the I_i=0.11 of 'the' only activates 1 expert, which verifies the effectiveness of our evaluator."

    The evaluator's output I_i is the direct input to the monotone allocation of Eq. (3), so showing that a high I_i triggers more experts is a restatement of the routing rule, not evidence that the evaluator is effective. The example 'verifies' the mechanism by replaying the same formula: K_i = ceil(L_i*K) forces high-I tokens to receive more experts. There is no independent criterion or held-out task in this verification.

1 more flagged steps
  1. fitted input called prediction [Section 5.5, Threshold Refinement for Code Tasks (HumanEval/MBPP)]
    "To refine the complexity threshold C_i and analyze its impact on code generation tasks, we conducted experiments on HumanEval (n=164 samples) and MBPP (n=500 samples). Key variables include: deep expert activation rate (proportion of tokens routed to deep experts), C_i threshold (tuned from 0.3 to 0.7), and prediction accuracy."

    The routing threshold is a free parameter of the adaptive mechanism, and Section 5.5 tunes it on HumanEval and MBPP and reports the resulting accuracy relationships. These same two tasks are then presented in Table 1 as evidence that DASG-MoE-Instruct outperforms baselines (HumanEval 88.03, MBPP 76.59). The reported scores are therefore partly produced by fitting the controller to the evaluation set, not by an independent prediction. Similarly, Eq. (8)'s thresholds theta_s=0.3 and theta_d=0.7 are 'empirically tuned' with MRPC-style semantic-equivalence behavior in mind, making the later MRPC discussion post hoc.

full rationale

The central adaptivity claim reduces to its own definitions. Equation (2) defines K_i as a non-decreasing function f(I_i), and Eq. (3) instantiates it as ceil(L_i * K), so 'important tokens receive more experts' is true by construction and cannot be falsified by the experiments. The evaluator verification in Section 4.4 is the same tautology replayed with an example. The router's thresholds are not derived from principles: theta_s and theta_d are empirically tuned, and Section 5.5 tunes C_i directly on HumanEval and MBPP, the very tasks whose Table 1 numbers are later offered as evidence of superiority. That makes part of the empirical claim a fitted input rather than a prediction. Separately, the abstract's headline that experiments on 'multiple long-sequence benchmark datasets' show SOTA performance is unsupported because every listed benchmark (GLUE, MMLU, CMMLU, CEval, BBH, GSM8K, MATH500, MBPP, HumanEval) is short-sequence; this is a coverage gap rather than a circularity, but it compounds the confidence problem. The GMHA grouping and dual-scale expert construction are independent architectural contributions, so this is not a pure renaming, but the dynamic-allocation mechanism that motivates the paper is definitional and the headline numbers are partly fit to the evaluation tasks. Score 7 reflects partial circularity plus an untested central empirical claim; not 8-10 because the architecture still has independent components that could in principle be validated externally.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim rests on hand-tuned routing thresholds theta_s and theta_d, the choice of 16 attention groups, top-k=2 expert selection, and a task vector example; none of these are derived. The model also assumes attention-derived importance scores are learnable and useful for routing, and that grouped local attention with a final aggregation retains long-range information. These are domain assumptions, not established facts. The evaluator that produces the scores is trained jointly with the model, so it is not an independent entity.

free parameters (5)
  • Routing thresholds theta_s and theta_d = theta_s=0.3, theta_d=0.7
    Set in Eq. 8 and described as 'empirically tuned'; used to decide shallow vs deep expert allocation.
  • Number of attention groups G = 16
    Chosen after sweeping 1 to 64 groups in Table 6; the 16-group result is then used as the main model.
  • Top-k for local expert selection = 2
    Fixed in Algorithm 1 and Section 4.1 without derivation; number of active experts per module.
  • Task vector T_j for MRPC = [0.2, 0.8]
    Given as an example in the context of Eq. 7; hand-specified to emphasize deep experts.
  • Sliding window size w = not stated
    Used in Eq. 9 and Eq. 10 as a mask parameter but its value is never reported.
assumptions (4)
  • domain assumption Attention-derived importance scores can be read from attention weights by a two-layer MLP evaluator.
    Eq. 19 defines the evaluator without training details or validation that the scores are meaningful.
  • domain assumption Grouped attention with local sliding windows and a single global aggregation preserves sufficient long-range information.
    Section 4.2 asserts this; no theoretical or empirical proof beyond the final task accuracy.
  • domain assumption Routing more experts to high-importance tokens improves both accuracy and efficiency.
    Eq. 2 and Eq. 3 assert K_i = f(I_i) as non-decreasing; the benefit is the premise of the paper, not established independently.
  • standard math Standard softmax attention and MLP computations are correct.
    Used throughout the architecture; not at issue.
invented entities (1)
  • Lightweight Evaluator (two-layer MLP over attention weights)
    purpose: Produces token importance I_i and complexity C_i that drive expert count and tier selection.
    The evaluator is trained with the model and no external validation is given; its output is the only basis for routing, so routing behavior cannot confirm it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts." pith.science (2026). https://pith.science/paper/HV364GEF

@misc{pith2026250910530,
  author       = {Pith},
  title        = {Pith review of: Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HV364GEF}},
  note         = {Machine review of arXiv:2509.10530}
}
read the original abstract

Transformer models based on the Mixture of Experts (MoE) architecture have made significant progress in long-sequence modeling, but existing models still have shortcomings in computational efficiency and the ability to capture long-range dependencies, especially in terms of the dynamic adaptability of expert resource allocation. In this paper, we propose a Dynamic Adaptive Shared Expert and Grouped Multi-Head Attention Hybrid Model (DASG-MoE) to enhance long-sequence modeling capabilities by integrating three modules. First, we employ the Grouped Multi-Head Attention (GMHA) mechanism to effectively reduce the computational complexity of long sequences. By parallel processing through sequence grouping, local sliding window attention, and feature aggregation, we address long-range dependency issues and the model's lack of generalization for local information. Second, we design a Dual-Scale Shared Expert Structure (DSSE), where shallow experts use lightweight computations to quickly respond to low-dimensional features, while deep experts process high-dimensional complex semantics through pre-training transfer and post-training optimization, achieving a dynamic balance between efficiency and accuracy. Third, we propose a hierarchical Adaptive Dynamic Routing (ADR) mechanism that dynamically selects expert levels based on feature complexity and task requirements, and optimizes resource allocation through a local expert activation strategy. Experiments on multiple long-sequence benchmark datasets demonstrate that our DASG-MoE model outperforms state-of-the-art models.

Figures

Figures reproduced from arXiv: 2509.10530 by the authors.

Figure 1
Figure 1. An example of a sentiment analysis task. input tokens[14] regardless of their relative significance, creating an ongoing research challenge: developing sophisticated MoE architectures that incorporate token importance measures[15] to advance both predictive accuracy and computational efficiency in large-scale Transformer models. Within Transformer architectures, attention layers compute attention scores by evaluatin… view at source ↗
Figure 2
Figure 2. Our adaptive routing mechanism and MoE system diagram of the dual-scale shared expert module allow the model to assign corresponding hierarchical experts based on token importance. • Adaptive Dynamic Routing: First, the feature vectors of the input sequence are extracted through a grouped multi-head attention mechanism, and a lightweight evaluator is used to calculate the feature complexity score and task urgency in… view at source ↗
Figure 3
Figure 3. Comparison of DASG-MoE and ST baseline model training logs. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: “The service is pretty good.” Example of attention weight diagram for 12 heads in grouped multi-head attention [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Token importance score diagram in emotional analysis and interpretation task examples. knowledge transfer between shallow modules for rapid feature identification and deep modules for comprehensive semantic understanding demonstrates the adaptive intelligence of our mu…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 16 canonical work pages

  1. [1]

    The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020

    arXiv 2020

  2. [2]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  3. [3]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    arXiv 2001

  4. [4]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  5. [5]

    Outra- geously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outra- geously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    arXiv 2017

  6. [6]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  7. [7]

    Adaptive mixtures of local experts

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991

    1991

  8. [8]

    What does bert look at? an analysis of bert’s attention.arXiv preprint arXiv:1906.04341, 2019

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention.arXiv preprint arXiv:1906.04341, 2019

    arXiv 1906

Show all 68 references
  1. [9]

    Analyzing multi-head self-attention: Spe- cialized heads do the heavy lifting, the rest can be pruned.arXiv preprint arXiv:1905.09418, 2019

    Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Spe- cialized heads do the heavy lifting, the rest can be pruned.arXiv preprint arXiv:1905.09418, 2019

    1905 arXiv

  2. [10]

    Bert: Pre-training of deep bidirectional transform- ers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transform- ers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human lang...

    2019

  3. [11]

    Chatgpt: Optimizing language models for dialogue, Nov 2022

    OpenAI. Chatgpt: Optimizing language models for dialogue, Nov 2022

    2022

  4. [12]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  5. [13]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. InInternational conference on machine learning, pages 5547–5569. PMLR, 2022

    2022

  6. [14]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. InInternational conference on machine learning, pages 5547–5569. PMLR...

    2022

  7. [15]

    St-moe: Designing stable and transferable sparse expert models.arXiv preprint arXiv:2202.08906, 2022

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models.arXiv preprint arXiv:2202.08906, 2022

    2022 arXiv

  8. [16]

    Dai, Zhifeng Chen, Quoc V

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M. Dai, Zhifeng Chen, Quoc V. Le, and James Laudon. Mixture-of-experts with expert choice routing.arXiv preprint arXiv:2202.09368, feb 2022

    2022 arXiv

  9. [17]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023

    2023

  10. [18]

    The efficiency misnomer.arXiv preprint arXiv:2110.12894, 2021

    Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency misnomer.arXiv preprint arXiv:2110.12894, 2021

    2021 arXiv

  11. [19]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635, 2018

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635, 2018

    2018 arXiv

  12. [20]

    Big bird: Transformers for longer sequences.Advances in neural information processing systems, 33:17283–17297, 2020

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences.Advances in neural information processing systems, 33:17283–17297, 2020

    2020

  13. [21]

    Long range arena: A benchmark for efficient transformers.arXiv preprint arXiv:2011.04006, 2020

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers.arXiv preprint arXiv:2011.04006, 2020

    2011 arXiv

  14. [22]

    Base layers: Simplifying training of large, sparse models

    Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. InInternational Conference on Machine Learning, pages 6265–6274. PMLR, 2021

    2021

  15. [23]

    Sparse is enough in scaling transformers.Advances in Neural Information Processing Systems, 34:9895–9907, 2021

    Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojciech Gajewski, Henryk Michalewski, and Jonni Kanerva. Sparse is enough in scaling transformers.Advances in Neural Information Processing Systems, 34:9895–9907, 2021

    2021

  16. [24]

    Hash layers for large sparse models.advances in neural information processing systems, 34:17555–17566, 2021

    Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models.advances in neural information processing systems, 34:17555–17566, 2021

    2021

  17. [25]

    Synthesizer: Rethinking self-attention for transformer models

    Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. InInternational conference on machine learning, pages 10183–10192. PMLR, 2021

    2021

  18. [26]

    Random feature attention.arXiv preprint arXiv:2103.02143, 2021

    Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention.arXiv preprint arXiv:2103.02143, 2021

    2021 arXiv

  19. [27]

    Skyformer: Remodel self-attention with gaussian kernel and nystr\" om method.Advances in Neural Information Processing Systems, 34:2122–2135, 2021

    Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. Skyformer: Remodel self-attention with gaussian kernel and nystr\" om method.Advances in Neural Information Processing Systems, 34:2122–2135, 2021

    2021

  20. [28]

    Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models

    Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models. InProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, p...

    2022

  21. [29]

    Megablocks: Efficient sparse training with mixture- of-experts.arXiv preprint arXiv:2211.15841, nov 2022

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture- of-experts.arXiv preprint arXiv:2211.15841, nov 2022

    2022 arXiv

  22. [30]

    Flex- moe: Scaling large-scale sparse pre-trained model training via dynamic device placement.Proceedings of the ACM on Management of Data, 1(1):1–19, 2023

    Xiaonan Nie, Xupeng Miao, Zilong Wang, Zichao Yang, Jilong Xue, Lingxiao Ma, Gang Cao, and Bin Cui. Flex- moe: Scaling large-scale sparse pre-trained model training via dynamic device placement.Proceedings of the ACM on Management of Data, 1(1):1–19, 2023

    2023

  23. [31]

    Go wider instead of deeper

    Fuzhao Xue, Ziji Shi, Futao Wei, Yuxuan Lou, Yong Liu, and Yang You. Go wider instead of deeper. InProceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8779–8787, 2022

    2022

  24. [32]

    Albert: A lite bert for self-supervised learning of language representations.arXiv preprint arXiv:1909.11942, 2019

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations.arXiv preprint arXiv:1909.11942, 2019

    1909 arXiv

  25. [33]

    Efficientnet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. InInternational conference on machine learning, pages 6105–6114. PMLR, 2019

    2019

  26. [34]

    Mobilenets: Efficient convolutional neural networks for mobile vision applications.arXiv preprint arXiv:1704.04861, 2017

    Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications.arXiv preprint arXiv:1704.04861, 2017

    2017 arXiv

  27. [35]

    Dolan and Chris Brockett

    William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. InProceedings of the Third International Workshop on Paraphrasing, 2005

    2005

  28. [36]

    Efficient large scale language modeling with mixtures of experts.arXiv preprint arXiv:2112.10684, 2021

    Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts.arXiv preprint arXiv:2112.10684, 2021

    2021 arXiv

  29. [37]

    Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020. 14

    2006 arXiv

  30. [38]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  31. [39]

    Reformer: The efficient transformer.arXiv preprint arXiv:2001.04451, 2020

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.arXiv preprint arXiv:2001.04451, 2020

    2001 arXiv

  32. [40]

    Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    2009 arXiv

  33. [42]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  34. [43]

    Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  35. [44]

    Glue: A multi-task benchmark and analysis platform for natural language understanding.arXiv preprint arXiv:1804.07461, 2018

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding.arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  36. [45]

    Superglue: A stickier benchmark for general-purpose language understanding systems.Advances in neural information processing systems, 32, 2019

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems.Advances in neural information processing systems, 32, 2019

    2019

  37. [46]

    Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations, 2021

    2021

  38. [47]

    Cmmlu: Measuring massive multitask language understanding in chinese.arXiv preprint arXiv:2306.09212, 2023

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Liu, Yeyun Guo, Derry Tanti Wijaya, et al. Cmmlu: Measuring massive multitask language understanding in chinese.arXiv preprint arXiv:2306.09212, 2023

    2023 arXiv

  39. [48]

    C-eval: A comprehensive chinese evaluation suite for foundation models.arXiv preprint arXiv:2305.08322, 2023

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. C-eval: A comprehensive chinese evaluation suite for foundation models.arXiv preprint arXiv:2305.08322, 2023

    2023 arXiv

  40. [49]

    Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowd- hery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    arXiv

  41. [50]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  42. [51]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Stein- hardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  43. [52]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  44. [53]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  45. [54]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025

    2025

  46. [55]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie H...

    2024

  47. [56]

    Gemma 3 technical report, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, ..., Armand Joulin, and Léonard Hussenot. Gemma 3 technical report, 2025

    2025

  48. [57]

    The Llama 3 Herd of Models, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, and Goyal et al.”]. The Llama 3 Herd of Models, 2024. 15

    2024

  49. [58]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Rus- sell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, ...

    2024

  50. [59]

    Multi-scale dense networks for resource efficient image classification.arXiv preprint arXiv:1703.09844, 2017

    Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification.arXiv preprint arXiv:1703.09844, 2017

    2017 arXiv

  51. [60]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 Conference on Empirical Methods in Natural Language Process...

    2013

  52. [61]

    Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. InTransactions of the Association for Computational Linguistics, volume 7, pages 625–641, 2019

    2019

  53. [62]

    Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understand- ing through inference. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi...

    2018

  54. [63]

    Squad: 100,000+ questions for machine com- prehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine com- prehension of text. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016

    2016

  55. [64]

    The pascal recognising textual entailment challenge

    Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. InMachine Learning Challenges Workshop, pages 177–190, 2005

    2005

  56. [65]

    Palm 2 technical report.arXiv preprint arXiv:2305.10403, 2023

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report.arXiv preprint arXiv:2305.10403, 2023

    2023 arXiv

  57. [66]

    Linear attention is (maybe) all you need (to understand transformer optimization).arXiv preprint arXiv:2310.01082, 2023

    Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jadbabaie, and Suvrit Sra. Linear attention is (maybe) all you need (to understand transformer optimization).arXiv preprint arXiv:2310.01082, 2023

    2023 arXiv

  58. [67]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  59. [68]

    Fnet: Mixing tokens with fourier transforms

    James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.03824, 2021

    2021 arXiv

  60. [69]

    Densenet: densely connected convolutional networks.arXiv preprint arXiv:1608.06993, pages 362–371, 2018

    M Steen, S Downe, N Bamford, and L Edozien. Densenet: densely connected convolutional networks.arXiv preprint arXiv:1608.06993, pages 362–371, 2018. 16

    2018 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.