Pith. sign in

REVIEW 5 major objections 5 minor 42 references

cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A normalized mixture of linear experts is exactly an input-dependent 1×1 convolution; cMoLLM uses this to scale LLM capacity by adding streams at near-dense compute.

desk verdict The headline theorem is a tautology and the 'matched compute' claim is contradicted by the paper's own FLOPs formula — the empirical gains are ~4.5x compute at N=8, not matched. read the letter →

arxiv 2607.22577 v1 pith:VBOZMZY5 submitted 2026-06-06 cs.AI

Xin Yang , Yemin Wang , Mingda Liu , Letian Li , Shuaishuai Cao , Zhengxiao He , Ryan Dong This is my paper
classification cs.AI
keywords mixture-of-expertsdynamicconvolutionsoftroutingpipeline-levelscalinghorizontallawconditionalcomputationlanguagemodel1x1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish a precise identity: a mixture-of-experts layer with linear experts and normalized routing is not merely analogous to a dynamic 1×1 convolution—it is one, with the effective kernel equal to the input-weighted average of the expert kernels. On top of that identity it builds cMoLLM, which treats entire LLM streams as experts, mixes them with soft differentiable gating, and applies the mixed kernel through grouped pointwise convolution. If the identity extends to the full pipeline, model capacity can be scaled by adding streams while per-token computation stays near the dense baseline, roughly (N+1) times the FFN cost for N streams. In decoder-only models trained on a large public web corpus, the paper reports consistent perplexity and downstream accuracy improvements over a dense baseline at three model sizes, with stable stream utilization and no Top-K routing, virtual tokens, or auxiliary heads. A sympathetic reader would care because this points to a scaling axis—stream count—that is much cheaper than scaling width or depth.

What carries the argument

The load-bearing object is the MoE–dynamic-convolution equivalence: a 1×1 convolution applies the same matrix at every token position, and the theorem shows that mixing expert outputs with normalized weights is exactly convolution by the input-dependent averaged kernel. The architecture built on it is cMoLLM, where N parallel streams each own a kernel K_k, a lightweight gating network produces soft weights g_k(x), and the effective kernel K~(x)=Σ_k g_k(x)K_k is applied through grouped pointwise convolution. This identity does the work of replacing discrete expert routing with a differentiable, hardware-friendly kernel-averaging operation, and it motivates the load-balancing auxiliary loss th

What would settle it

Train cMoLLM with N=8 streams and a dense baseline whose per-token FLOPs are matched to cMoLLM's actual cost (about 4.5 times the dense FFN cost at N=8) on the same data; if the dense model matches or beats cMoLLM's perplexity, the horizontal scaling law is an artifact of an unequal compute comparison rather than a genuine capacity gain.

Watch

Extended reading notes

Core claim

At the center is Theorem 4.1: for linear experts E_k(x)=xW_k^T and routing weights g_k(x) summing to one, MoE(x)=Σ_k g_k(x)E_k(x)=Conv1×1(x; K~(x)) with K~(x)=Σ_k g_k(x)W_k. Routing is kernel averaging. cMoLLM instantiates this by giving each of N end-to-end streams its own 1×1 kernel, computing soft mixture weights from the input, and applying the averaged kernel via grouped pointwise convolution; a load-balancing loss keeps the streams from collapsing. The paper's horizontal scaling law follows from the FLOP count: N stream convolutions plus one down-projection cost about (N+1)dd_ff per token, versus about 2dd_ff for a dense FFN, so for bounded N the effective capacity grows roughly linear

Load-bearing premise

The load-bearing premise is that the per-layer identity—a weighted sum of linear experts equals one input-dependent 1×1 convolution—still holds for the full multi-stream cMoLLM pipeline as implemented, and that comparing N=8 streams to a dense model is a fair 'matched compute' comparison.

Editorial extensions

If this is right

  • Soft MoE routing and dynamic convolution become the same operation, so any soft mixture layer can be implemented as a single input-conditioned convolution without Top-K truncation.
  • For bounded N, adding streams grows effective capacity roughly linearly in N while per-token FLOPs stay within a constant factor of the dense baseline—a horizontal scaling law alongside the usual vertical one.
  • The design eliminates virtual tokens and auxiliary prediction branches, which the paper argues removes the compute overhead and training instability those mechanisms introduce.
  • At every model size tested (85M, 350M, 760M parameters), the multi-stream model improves loss, perplexity, and downstream accuracy over the dense baseline, suggesting the benefit does not fade as models grow.
  • The paper itself flags that its experiments stop at 760M parameters and that validation at 7B+ is still needed, so the scaling claim is currently supported only at small scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not proven in the paper: the single-layer identity is exact for linear experts, and Corollary 4.2 extends it to two-layer nonlinear experts via a data-dependent mask, but the full multi-layer pipeline in Algorithm 1 and Eq. (12) routes over already-mixed stream outputs and inserts Transformer layers between convolutions; the theorem does not literally cover that architecture, so its validity at th
  • The 'matched compute' comparison is generous to cMoLLM: N=8 streams cost about 4.5 times the dense FFN FLOPs, so a dense model given the same compute budget is a stronger baseline than the one reported.
  • If the identity holds at scale, the most direct testable extension is to measure whether the per-stream perplexity gain persists when a dense baseline is given matching FLOPs and parameters; another is to see whether the advantage concentrates on token clusters that correlate with the learned routing patterns, as the toy cluster model suggests.
  • The equivalence also suggests a cross-fertilization: convolution-optimized hardware and kernel-fusion libraries could accelerate MoE-style LLM training directly, since the mixture reduces to one grouped pointwise convolution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes cMoLLM, a pipeline-level mixture-of-LLMs in which each of N 'streams' is associated with a 1x1 convolution kernel and a soft router mixes the kernels. The authors claim a formal equivalence between MoE layers and dynamic convolutions (Theorem 4.1), derive a 'horizontal scaling law' (Section 4.7, Eq. (22)), and report GPT-2-style experiments on FineWeb where cMoLLM improves perplexity, GLUE, and SQuAD 'under matched compute' relative to a dense baseline. The architecture is described in Section 4.2 and Algorithm 1, with four gating variants and scaling experiments at 85M, 350M, and 760M parameters.

Significance. If the matched-compute claim held, cMoLLM would be a useful contribution to conditional-computation scaling. The paper is also transparent in reporting a FLOP model, hyperparameters, and 3-seed mean/std statistics. However, the central claim is not supported: the paper's own Eq. (21) shows that for N=8 the proposed model costs about 4.5x the FLOPs of a dense FFN, so Tables 2-3 are not matched-compute comparisons. Moreover, Eq. (22) is mathematically impossible as stated, and Theorem 4.1 is a definitional identity that does not license the multi-layer architecture. The advertised comparisons to ParaScale and AltUp are not reported numerically. On the evidence in the manuscript, the headline efficiency/scaling claims fail.

major comments (5)
  1. [§4.7, Eq. (21); Tables 2-3] The central 'matched compute' claim is contradicted by the paper's own FLOP model. Eq. (21) gives FLOPs_cMoLLM ≈ (N+1) d d_ff + gate versus FLOPs_dense ≈ 2 d d_ff. For N=8, the configuration with the best PPL in Table 2 and called 'representative' in Appendix F, the ratio is 9/2 = 4.5 before gating. No iso-FLOP comparison (e.g., reduced width, depth, or d_ff to equalize compute) is reported, so the gains in Tables 2-3 could simply reflect 4.5x more compute and parameters. This directly undermines the abstract and §5.3 headline claim of improvement 'under matched compute'.
  2. [§4.7, Eq. (22)] The horizontal scaling-law bound is mathematically impossible as stated. Eq. (22) asserts existence of a constant C such that FLOPs_cMoLLM ≤ C·FLOPs_dense whenever N ≤ C^{-1}. But Eq. (21) implies the LHS/RHS ratio is about (N+1)/2. For N=8 this ratio is ≈4.5, requiring C ≥ 4.5; the condition N ≤ C^{-1} then requires C ≤ 1/8. No such C exists for N≥2. The 'within a constant factor' claim is therefore false for the experimental configurations, and the paper's 'horizontal scaling law' is not established.
  3. [§4.1, Theorem 4.1; §4.2 Eq. (12); Algorithm 1] Theorem 4.1 is a definitional identity: by defining K~(x) := Σ_k g_k(x) W_k, Eq. (4) holds by linearity, so it is not a substantive equivalence that licenses the full cMoLLM pipeline. Algorithm 1 computes U_i = H_i K~_i^T, then a shared nonlinearity and down-projection, which is a single linear-expert MoE layer. In contrast, §4.2 Eq. (12) describes applying Transformer blocks to concatenated per-branch outputs X_input = [X_1,...,X_N]. The theorem does not cover the Eq. (12) structure, and no approximation or equivalence is proved for the actual multi-layer architecture used in the experiments.
  4. [Corollary 4.2; Appendix A] Corollary 4.2 is stated for general Lσ-Lipschitz activations satisfying Assumption 3.2, but Appendix A proves the 'data-dependent mask' representation only for ReLU. The key step uses the identity ReLU(z) = I(z>0)·z, which has no analogue for GELU or other smooth Lipschitz activations. The corollary is therefore unproved in the stated generality; the statement should be restricted to ReLU or a proof for the general case must be supplied.
  5. [§5.2, §5.3, Appendix F] The abstract and §5.3 claim that cMoLLM is better than 'ParaScale- and AltUp-style pipeline mixtures,' but Tables 2-3 contain only dense and cMoLLM results. Appendix F says ParaScale and AltUp were reimplemented and compared, yet no numerical results for those baselines appear anywhere in the manuscript. The comparative claims are unsupported by the presented evidence.
minor comments (5)
  1. [Corollary 4.2] Corollary 4.2 says 'Under Theorem 3.2'; the reference should be to Assumption 3.2, since no Theorem 3.2 is stated.
  2. [Proposition 4.4 proof] The proof refers to 'as discussed in Theorem 4.3' for the binary mask; the intended reference appears to be Remark 4.3.
  3. [Appendix B, Theorem B.2] Theorem B.2 says 'as in Theorem B.1,' but the limitation of a single linear classifier is Proposition B.1, not Theorem B.1.
  4. [Section 6] The sentence 'Extended results on downstream tasks and scaling are provided in ??' contains a broken cross-reference; the target section is missing.
  5. [Figure 3] The caption says the right panel shows 'validation loss and perplexity vs. stream count n or training steps,' but no training-step variation is displayed or discussed; the caption should match the actual content.

Circularity Check

2 steps flagged · score 8.0 of 10

The central theoretical 'equivalence' is true by definition (Eqs. 4-5) and the 'horizontal scaling law' is a tautological conditional (Eq. 22); the matched-compute claim is separately contradicted by the paper's own FLOPs equation, but that is a correctness issue, not circularity.

  1. self definitional [Section 4.1, Theorem 4.1, Eqs. (4)-(5)]
    "where the effective kernel is given by ˜K(x) := P N k=1 g k(x)W k. (5) ... MoE(x) = Conv1×1 (x; ˜K(x)) = x ˜K(x)⊤ (4)"

    The 'effective kernel' is defined as the gated sum of the expert weight matrices, so the claimed equality MoE(x)=Conv1x1(x;K~(x)) is just the linearity of matrix multiplication. No independent constraint is derived: any normalized linear-expert MoE satisfies Eq. (4) because K~(x) was chosen to make it so. The 'formal equivalence' is a definitional rewrite of the MoE sum as a dynamic convolution, not a result that could fail or be empirically tested.

  2. other [Section 4.7, Eq. (22)]
    "Then there exists a constant C (absorbing gating overhead) such that FLOPscMoLLM ≤ C · FLOPsdense whenever N ≤ C −1, while the number of distinct kernels (and thus the effective capacity of the mixture) grows linearly with N."

    Given Eq. (21), FLOPscMoLLM ≈ (N+1)dd_ff + gate and FLOPsdense ≈ 2dd_ff, the inequality holds whenever C ≥ (N+1)/2; the condition 'N ≤ C−1' is just a restatement of choosing C large enough. Thus the 'horizontal scaling law' is a tautological conditional with a free constant: it asserts only that the compute ratio is finite for any finite N, and 'capacity grows linearly with N' is true by construction because each stream contributes one kernel. It provides no falsifiable scaling prediction.

full rationale

The paper contains no load-bearing self-citation chain: the authors' prior works are cited only as context, and the one omitted proof (Prop. B.1) defers to Chen et al. (2022), not to the authors. The circularity lies in the two central theoretical constructions. Theorem 4.1 defines K~(x) as the gated sum of expert weight matrices, then 'proves' the MoE output equals the dynamic convolution with that kernel; this is a definitional rewrite via linearity, so the equivalence cannot fail and carries no independent content. Corollary 4.2/Appendix A repeats the same move: the 'data-dependent effective kernel' just absorbs the ReLU indicator into the weight product. Eq. (22) is a tautological conditional: since Eq. (21) gives FLOPs_cMoLLM ≈ (N+1)dd_ff + gate vs FLOPs_dense ≈ 2dd_ff, setting C≥(N+1)/2 makes the inequality true for any finite N, and the 'capacity grows linearly' clause is true by construction because each stream adds one kernel. Separately, but not as circularity, the abstract's 'matched compute' claim is contradicted by the paper's own FLOPs count: for the representative N=8 configuration (Appendix F), the ratio is 9/2 = 4.5, so Tables 2-3 compare unequal compute. There is also a theory-to-implementation gap: Theorem 4.1 covers a single layer with pre-mixing routing, while Algorithm 1 and Eq. (12) route over already-mixed streams and apply Transformers to concatenated branch outputs; that is a support gap, not a definitional reduction. Overall the central theoretical derivation is true by definition, so the score is 8.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central theorem is obtained by rewriting the MoE definition; it introduces no new entities but relies on an unproven transfer of the single-layer identity to the full cMoLLM pipeline, and Eq (22) is vacuous. The empirical comparisons select alpha, N, and gating variant post hoc, giving the paper two hand-set parameters that drive the headline gains.

free parameters (3)
  • Load-balancing coefficient alpha = 0.01
    In Eq (20), alpha is chosen by grid search over {0.001, 0.01, 0.1} and reported as the 'best value' in Table 6; all main results use this value.
  • Number of streams N = 8
    N is selected as the best validation-loss configuration in Table 2 (multi-head) and used for the scaling Table 3; compute scales linearly with N, so this choice conflates capacity with compute.
  • Gating variant = multi-head
    The scaling comparison selects multi-head gating after seeing Table 2 results; not a fitted number but a post-hoc model selection that drives the headline gains.
assumptions (5)
  • standard math Linear algebra identity: Sum_k g_k(x) x W_k^T = x (Sum_k g_k(x) W_k)^T
    Used in Theorem 4.1; valid, but it makes the 'MoE-dynamic convolution equivalence' a definitional restatement rather than a substantive theorem.
  • domain assumption Assumptions 3.1 and 3.2: bounded inputs and Lipschitz activation
    Stated in §3.1 and invoked for complexity/stability claims, but not actually needed for Theorem 4.1; they do not rescue Corollary 4.2 for general activations.
  • ad hoc to paper Activation function sigma can be represented by a binary diagonal mask M_k(x)
    Used in Corollary 4.2 / Appendix A Eqs (26)-(29). This representation holds for ReLU but not for GELU, so the corollary as stated for general Lipschitz activations is unsupported.
  • ad hoc to paper Single-layer MoE-convolution identity transfers to a multi-layer pipeline with token-dependent kernels
    The cMoLLM design assumes the per-token dynamic convolution from Theorem 4.1 remains the right abstraction when kernels are applied to intermediate hidden states of a multi-layer transformer; this transfer is never proved.
  • ad hoc to paper There exists a constant C such that FLOPs_cMoLLM <= C * FLOPs_dense whenever N <= C^{-1}
    Eq (22) in §4.7 defines the 'horizontal scaling law'. The existential constant makes the inequality vacuously true and untestable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs." pith.science (2026). https://pith.science/paper/VBOZMZY5

@misc{pith2026260722577,
  author       = {Pith},
  title        = {Pith review of: cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VBOZMZY5}},
  note         = {Machine review of arXiv:2607.22577}
}
abstract

Scaling large language models (LLMs) has driven their success, yet dense Transformers couple capacity and computation: every parameter is activated for every token, making training and inference costs grow linearly with model size-a critical bottleneck as models approach trillion-parameter regimes. We aim to scale capacity through MoE-style mixture throughout the LLM pipeline rather than only the FFN. Prior pipeline-level approaches include ParaScale, which introduces virtual tokens and parallel streams but incurs substantial overhead and suffers from homogenized routing and gradient collapse, and AltUp, which uses an auxiliary prediction branch but offers limited adaptivity and slow convergence. We establish that MoE-style mixture layers can be reformulated as variable-kernel dynamic convolutions, where each expert corresponds to a $1{\times}1$ convolutional kernel and routing implements input-conditioned kernel aggregation. Building on this equivalence, we introduce cMoLLM: a convolutionally gated mixture-of-LLMs that routes over end-to-end streams through fully differentiable dynamic convolution. In GPT-2-style models trained on FineWeb, cMoLLM improves language modeling perplexity and downstream GLUE and SQuAD accuracy under matched compute, with better stream utilization, more stable optimization, and favorable scaling compared to ParaScale- and AltUp-style baselines.

Figures

Figures reproduced from arXiv: 2607.22577 by the authors.

Figure 1
Figure 1. Teaser of cMoLLM. cMoLLM scales capacity at the pipeline level via convolutionally-gated mixture over end-to-end streams, yielding better perplexity and downstream accuracy than dense baselines under matched compute, without virtual tokens or auxiliary heads [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Experimental results ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 2 canonical work pages

  1. [1]

    Brown and Benjamin Chess and Rewon Child and Scott Gray and Alec Radford and Jeffrey Wu and Dario Amodei , title =

    Jared Kaplan and Sam McCandlish and Tom Henighan and Tom B. Brown and Benjamin Chess and Rewon Child and Scott Gray and Alec Radford and Jeffrey Wu and Dario Amodei , title =. CoRR , volume =. 2020 , url =. 2001.08361 , timestamp =

    arXiv 2020

  2. [2]

    Rae and Oriol Vinyals and Laurent Sifre , title =

    Jordan Hoffmann and Sebastian Borgeaud and Arthur Mensch and Elena Buchatskaya and Trevor Cai and Eliza Rutherford and Diego de Las Casas and Lisa Anne Hendricks and Johannes Welbl and Aidan Clark and Tom Hennigan and Eric Noland and Katie Millican and George van den Driessche and Bogdan Damoc and Aurelia Guy and Simon Osindero and Karen Simonyan and Eric...

    arXiv doi:10.48550/arxiv.2203.15556 2022

  3. [3]

    Jacobs and Michael I

    Robert A. Jacobs and Michael I. Jordan and Steven J. Nowlan and Geoffrey E. Hinton , title =. Neural Comput. , volume =. 1991 , url =. doi:10.1162/NECO.1991.3.1.79 , timestamp =

    doi:10.1162/neco.1991.3.1.79 1991

  4. [4]

    Jordan and Robert A

    Michael I. Jordan and Robert A. Jacobs , title =. Neural Comput. , volume =. 1994 , url =. doi:10.1162/NECO.1994.6.2.181 , timestamp =

    doi:10.1162/neco.1994.6.2.181 1994

  5. [5]

    Le and Geoffrey E

    Noam Shazeer and Azalia Mirhoseini and Krzysztof Maziarz and Andy Davis and Quoc V. Le and Geoffrey E. Hinton and Jeff Dean , title =. 5th International Conference on Learning Representations,. 2017 , url =

    2017

  6. [6]

    William Fedus and Barret Zoph and Noam Shazeer , title =. J. Mach. Learn. Res. , volume =. 2022 , url =

    2022

  7. [7]

    9th International Conference on Learning Representations,

    Dmitry Lepikhin and HyoukJoong Lee and Yuanzhong Xu and Dehao Chen and Orhan Firat and Yanping Huang and Maxim Krikun and Noam Shazeer and Zhifeng Chen , title =. 9th International Conference on Learning Representations,. 2021 , url =

    2021

  8. [8]

    ST-MoE: Designing Stable and Transferable Sparse Expert Models , author =

Show all 42 references
  1. [9]

    Zhao and Andrew M

    Yanqi Zhou and Tao Lei and Hanxiao Liu and Nan Du and Yanping Huang and Vincent Y. Zhao and Andrew M. Dai and Zhifeng Chen and Quoc V. Le and James Laudon , editor =. Mixture-of-Experts with Expert Choice Routing , booktitle =. 2022 , url =

    2022

  2. [10]

    CoRR , volume =

    Mouxiang Chen and Binyuan Hui and Zeyu Cui and Jiaxi Yang and Dayiheng Liu and Jianling Sun and Junyang Lin and Zhongxin Liu , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.10475 , eprinttype =. 2505.10475 , timestamp =

    2025 doi arXiv

  3. [11]

    Cutler and Nishanth Dikkala and Nikhil Ghosh and Rina Panigrahy and Xin Wang , editor =

    Cenk Baykal and Dylan J. Cutler and Nishanth Dikkala and Nikhil Ghosh and Rina Panigrahy and Xin Wang , editor =. Alternating Updates for Efficient Transformers , booktitle =. 2023 , timestamp =

    2023

  4. [12]

    Towards Understanding the Mixture-of-Experts Layer in Deep Learning , booktitle =

    Zixiang Chen and Yihe Deng and Yue Wu and Quanquan Gu and Yuanzhi Li , editor =. Towards Understanding the Mixture-of-Experts Layer in Deep Learning , booktitle =. 2022 , timestamp =

    2022

  5. [13]

    Proceedings of the 38th International Conference on Machine Learning,

    Mike Lewis and Shruti Bhosale and Tim Dettmers and Naman Goyal and Luke Zettlemoyer , editor =. Proceedings of the 38th International Conference on Machine Learning,. 2021 , url =

    2021

  6. [14]

    Dai and Simon Tong and Dmitry Lepikhin and Yuanzhong Xu and Maxim Krikun and Yanqi Zhou and Adams Wei Yu and Orhan Firat and Barret Zoph and Liam Fedus and Maarten P

    Nan Du and Yanping Huang and Andrew M. Dai and Simon Tong and Dmitry Lepikhin and Yuanzhong Xu and Maxim Krikun and Yanqi Zhou and Adams Wei Yu and Orhan Firat and Barret Zoph and Liam Fedus and Maarten P. Bosma and Zongwei Zhou and Tao Wang and Yu Emma Wang and Kellie Webster...

    2022

  7. [15]

    DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation

    Samyam Rajbhandari and Conglong Li and Zhewei Yao and Minjia Zhang and Reza Yazdani Aminabadi and Ammar Ahmad Awan and Jeff Rasley and Yuxiong He , editor =. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation. International Conference o...

    2022

  8. [16]

    Le and Jiquan Ngiam , editor =

    Brandon Yang and Gabriel Bender and Quoc V. Le and Jiquan Ngiam , editor =. CondConv: Conditionally Parameterized Convolutions for Efficient Inference , booktitle =. 2019 , timestamp =

    2019

  9. [17]

    Dauphin and Michael Auli , title =

    Felix Wu and Angela Fan and Alexei Baevski and Yann N. Dauphin and Michael Auli , title =. 7th International Conference on Learning Representations,. 2019 , url =

    2019

  10. [18]

    WeightNet: Revisiting the Design Space of Weight Networks , booktitle =

    Ningning Ma and Xiangyu Zhang and Jiawei Huang and Jian Sun , editor =. WeightNet: Revisiting the Design Space of Weight Networks , booktitle =. 2020 , url =. doi:10.1007/978-3-030-58555-6\_46 , timestamp =

    2020 doi

  11. [19]

    Dynamic Filter Networks , booktitle =

    Xu Jia and Bert De Brabandere and Tinne Tuytelaars and Luc Van Gool , editor =. Dynamic Filter Networks , booktitle =. 2016 , timestamp =

    2016

  12. [20]

    Yinpeng Chen and Xiyang Dai and Mengchen Liu and Dongdong Chen and Lu Yuan and Zicheng Liu , title =. 2020. 2020 , url =. doi:10.1109/CVPR42600.2020.01104 , timestamp =

    2020

  13. [21]

    Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation , journal =

    Yoshua Bengio and Nicholas L. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation , journal =. 2013 , url =. 1308.3432 , timestamp =

    2013 arXiv

  14. [22]

    6th International Conference on Learning Representations,

    Clemens Rosenbaum and Tim Klinger and Matthew Riemer , title =. 6th International Conference on Learning Representations,. 2018 , url =

    2018

  15. [23]

    Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen

    Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen. LoRA: Low-Rank Adaptation of Large Language Models , booktitle =. 2022 , url =

    2022

  16. [24]

    Parameter-Efficient Transfer Learning for

    Neil Houlsby and Andrei Giurgiu and Stanislaw Jastrzebski and Bruna Morrone and Quentin de Laroussilhe and Andrea Gesmundo and Mona Attariyan and Sylvain Gelly , editor =. Parameter-Efficient Transfer Learning for. Proceedings of the 36th International Conference on Machine Le...

    2019

  17. [25]

    Gomez and Lukasz Kaiser and Illia Polosukhin , editor =

    Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin , editor =. Attention is All you Need , booktitle =. 2017 , timestamp =

    2017

  18. [26]

    2019 , institution =

    Language Models are Unsupervised Multitask Learners , author =. 2019 , institution =

    2019

  19. [27]

    Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert. Language Models are Few-Shot Learners , booktitle =. 202...

    2020

  20. [28]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale , booktitle =

    Guilherme Penedo and Hynek Kydl. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale , booktitle =. 2024 , timestamp =

    2024

  21. [29]

    Bowman , title =

    Alex Wang and Amanpreet Singh and Julian Michael and Felix Hill and Omer Levy and Samuel R. Bowman , title =. 7th International Conference on Learning Representations,. 2019 , url =

    2019

  22. [30]

    Bowman , editor =

    Alex Wang and Yada Pruksachatkun and Nikita Nangia and Amanpreet Singh and Julian Michael and Felix Hill and Omer Levy and Samuel R. Bowman , editor =. SuperGLUE:. Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems ...

    2019

  23. [31]

    SQuAD: 100, 000+ Questions for Machine Comprehension of Text , booktitle =

    Pranav Rajpurkar and Jian Zhang and Konstantin Lopyrev and Percy Liang , editor =. SQuAD: 100, 000+ Questions for Machine Comprehension of Text , booktitle =. 2016 , url =. doi:10.18653/V1/D16-1264 , timestamp =

    2016 doi

  24. [32]

    Weld and Luke Zettlemoyer , editor =

    Mandar Joshi and Eunsol Choi and Daniel S. Weld and Luke Zettlemoyer , editor =. TriviaQA:. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics,. 2017 , url =. doi:10.18653/V1/P17-1147 , timestamp =

    2017 doi

  25. [33]

    Scaling Vision with Sparse Mixture of Experts , booktitle =

    Carlos Riquelme and Joan Puigcerver and Basil Mustafa and Maxim Neumann and Rodolphe Jenatton and Andr. Scaling Vision with Sparse Mixture of Experts , booktitle =. 2021 , timestamp =

    2021

  26. [34]

    2025 , url =

    Weilin Cai and Juyong Jiang and Fan Wang and Jing Tang and Sunghun Kim and Jiayi Huang , title =. 2025 , url =. doi:10.1109/TKDE.2025.3554028 , timestamp =

    2025

  27. [35]

    A Survey on Inference Optimization Techniques for Mixture of Experts Models , journal =

    Jiacheng Liu and Peng Tang and Wenfeng Wang and Yuhang Ren and Xiaofeng Hou and Pheng. A Survey on Inference Optimization Techniques for Mixture of Experts Models , journal =. 2026 , url =. doi:10.1145/3794845 , timestamp =

    2026 doi

  28. [36]

    CoRR , volume =

    Siyuan Mu and Sen Lin , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.07137 , eprinttype =. 2503.07137 , timestamp =

    2025 doi

  29. [37]

    Deciding How to Decide: Dynamic Routing in Artificial Neural Networks , booktitle =

    Mason McGill and Pietro Perona , editor =. Deciding How to Decide: Dynamic Routing in Artificial Neural Networks , booktitle =. 2017 , url =

    2017

  30. [38]

    CoRR , volume =

    Yikang Zhang and Jian Zhang and Qiang Wang and Zhao Zhong , title =. CoRR , volume =. 2020 , url =. 2004.10694 , timestamp =

    2020 arXiv

  31. [39]

    CoRR , volume =

    Xuxin Cheng and Ke Zeng and Zhiquan Cao and Linyi Dai and Wenxuan Gao and Fei Han and Ai Jian and Feng Hong and Wenxing Hu and Zihe Huang and Dejian Kong and Jia Leng and Zhuoyuan Liao and Pei Liu and Jiaye Lin and Xing Ma and Jingqing Ruan and Jiaxing Song and Xiaoyu Tan and ...

    2025 doi

  32. [40]

    CoRR , volume =

    Xin Yang and Letian Li and Abudukelimu Wuerkaixi and Xuxin Cheng and Cao Liu and Ke Zeng and Xunliang Cai and Wenyuan Jiang , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2603.03314 , eprinttype =. 2603.03314 , timestamp =

    2026 doi

  33. [41]

    2025 , howpublished =

    Introducing. 2025 , howpublished =

    2025

  34. [42]

    2025 , url =

    Xin Yang and Bintao Tang and Yuhao Wang and Zimo Ji and Wenyuan Jiang , title =. 2025 , url =. doi:10.1109/SMC58881.2025.11343280 , timestamp =

    2025

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.