Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture

Bingheng Wu; Jingze Shi

REVIEW 3 major objections 5 minor 24 references

Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cheems, a hybrid of state space duality, dynamic mask attention, and cross-domain mixture of experts, is claimed to outperform LLaMA3, Mamba2, and Jamba on most evaluated benchmarks at 320M and 1.3B scale.

desk verdict The architecture has interesting pieces, but the central 'Cheems beats the baselines' claim is undercut by the authors' own disclosure of a training bug, so the paper is conditional at best. read the letter →

arxiv 2412.11834 v3 pith:HXAFYFGC submitted 2024-12-16 cs.LG cs.AIcs.CL

Jingze Shi , Bingheng Wu This is my paper · ORCID

classification cs.LGcs.AIcs.CL

keywords statespacedualityrotarypositionembeddingdynamicmaskattentioncrossdomainmixtureofexpertshybridfoundationmodelarchitecturemulti-queryassociativerecalllanguagemodeling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a foundation model gets both more efficient and more effective when each of its two halves—sequence transformation and state transformation—is a combination rather than a single choice. It proposes three matrix-level components: rotary position embeddings proved to work inside state space duality, dynamic mask attention that lets causal attention selectively filter past states, and cross-domain mixture of experts that keeps retrieval fast when the expert count reaches thousands. Assembled into the Cheems architecture, these components are claimed to beat LLaMA3, Mamba2, and Jamba on most of seven downstream benchmarks at the 320M and 1.3B scales while training faster than LLaMA3 and Jamba. If the claim holds, a hybrid design can get state-space efficiency and attention effectiveness in one architecture rather than trading one for the other.

What carries the argument

The central mechanism is the semiseparable matrix identity $M = L \circ (CB^\top)$ that rewrites state space duality as causal linear attention, together with the proof that rotating $C$ and $B$ by position-dependent angles preserves relative position information, so $\langle f_C(x_m,m), f_B(x_n,n)\rangle = g(x_m,x_n,m-n)$. Dynamic mask attention adds a zero-order-hold mask, a mask whose value is held until the next non-continuous state, computed as $\exp(A\,\tau_\Delta([V_{\text{cache}},V]W_\Delta))$ on the attention score, letting the value state itself decide which past keys are attenuated or enhanced. Cross-domain mixture of experts projects each token against product keys, retrieves the top-$k$ expert rows from embedding tables, and adds a shared dense cross-domain path, so retrieval cost stays nearly flat as the expert count grows. The Cheems stack places one dynamic-mask attention layer after every seven SSD layers, with a CDMoE state transformation between modules.

What would settle it

Rerun the 320M and 1.3B training runs for Cheems, LLaMA3, Mamba2, and Jamba after the gradient-accumulation fix, using the same data, token counts, and hyperparameters, and compare the seven benchmark averages. If Cheems's margin over Mamba2 and Jamba shrinks or reverses, the central competitive claim is settled.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that sequence transformation and state transformation can be unified through compatible matrix algorithms: RoPE applies to the $C$ and $B$ matrices of SSD so the semiseparable product $L\circ(CB^\top)$ and the attention product $L\circ(QK^\top)$ carry the same relative position information; dynamic mask attention gives quadratic causal attention a trainable, value-derived mask with the selectivity of a gated state space model; and cross-domain mixture of experts combines dense cross-domain linear layers with sparse product-key embedding retrieval so expert count can scale to thousands without linear retrieval slowdown. The paper reports that Cheems-1.3B averages 52.44 on seven downstream tasks versus 50.36 for LLaMA3, 49.07 for Mamba2, and 51.07 for Jamba, and Cheems-320M averages 45.22 versus 43.99, 43.07, and 44.31 respectively, under matched parameter counts. It also reports that Cheems trains faster than LLaMA3 and Jamba while remaining close to Mamba2 in throughput.

Load-bearing premise

The load-bearing premise is that the four Table 4 models were trained under conditions fair enough that their downstream score differences reflect the architectures; the paper itself says the runs predate a gradient-accumulation error fix and that perplexity is not comparable, yet the downstream claims come from those same runs.

Editorial extensions

If this is right

At matched parameter counts, a hybrid SSD-attention-MoE stack can match or beat pure transformer, pure SSM, and transformer-SSM hybrid baselines on the evaluated downstream tasks, so hybridity is not just an efficiency compromise.
Rotary position embedding can serve as the single position-encoding scheme for both attention and SSD branches, which simplifies hybrid models and enables linear-time generation with relative positions.
Dynamic mask attention gives transformer-style attention a selectivity mechanism that preserves associative recall at sequence lengths where both quadratic causal attention and SSD degrade.
Cross-domain mixture of experts keeps expert retrieval fast beyond 1024 experts, so the number of stored experts can grow without the linear retrieval slowdown seen in routing MoE.
The same architecture that scores higher on average than the three baselines also trains and evaluates faster than LLaMA3 and Jamba, meaning efficiency and effectiveness are obtained together rather than traded off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the architecture is only demonstrated at 320M and 1.3B, so the paper leaves open whether CDMoE's flat retrieval cost keeps its advantage at larger widths and longer contexts, where the reported efficiency gap suggests it might.
Editorial extension: the Discussion's Doge variant—dynamic-mask attention with repeated CDMoE and no SSD—retains most benchmark scores, implying the SSD branch may be partly redundant; training Doge and Cheems under identical conditions would isolate SSD's contribution.
Editorial extension: DMAttn's 100% accuracy on multi-query associative recall is shown only up to 2048-token sequences and $d_{\text{model}}=32$; longer sequences or smaller heads would test whether the mechanism generalizes beyond the reported range.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Jingze and Bingheng's paper is a mixed bag. The headline claim—Cheems beats LLaMA3, Mamba2, and Jamba at matched parameter counts—does not survive contact with the paper's own Table 4 caption. That caption says all models were trained before a gradient-accumulation fix, that pre-training perplexity is not comparable, and that 'if our training results are reproduced in the future, the scores on these verification metrics may rise.' Gradient accumulation changes the effective batch size and optimizer noise; the authors offer no evidence the bug affected the three baselines and Cheems identically. With margins of 1-3 points, that is enough to reverse the ordering. The stress-test note is right: this is the load-bearing flaw.

That said, the paper is not empty. The RoPE-for-SSD proof in Appendix A is correct and cleanly written, even if it is a direct application of the standard RoPE two-dimensional rotation property to C and B—RetNet already did something similar. The DMAttn module shows genuinely strong performance on the multi-query associative recall task (Figure 6), and the CDMoE speed scaling in Figure 7 is a real empirical observation. The code is open-sourced, which is more than many submissions do.

The soft spots beyond the bug: all experiments are single runs with no error bars, so even the synthetic task and the module ablation tables (1-3) are statistically thin. The prose says DMAttn 'attenuates or enhances' states, but Listing 5 applies a hard threshold (masked_fill with -inf) rather than continuous scaling. That is a discrepancy worth flagging. The appendix code listings are partial and contain undefined variables, so reproduction is not turnkey.

Who should read this: anyone working on hybrid SSD-attention architectures or parameter-efficient MoE. The paper's ideas are plausible and the modular evaluations are suggestive, but the central comparative claim needs a retrain or a strong explanation of why the bug didn't matter. I think a serious editor should send this to review—the architecture is worth a look, and the authors' honesty about the bug is a point in their favor. But I would not cite the Cheems-vs-baselines numbers until they are reproduced on clean runs.

Referee Report

3 major / 5 minor

Summary. The paper proposes a hybrid foundation-model architecture called Cheems that combines State Space Duality (SSD) with quadratic causal self-attention, a new Dynamic Mask Attention (DMAttn), and a Cross Domain Mixture of Experts (CDMoE). It also proves that rotary position embedding can be applied to the SSD semiseparable matrix, reports perplexity and speed comparisons for the individual modules, and presents downstream benchmark results at the 320M and 1.3B scales claiming that Cheems outperforms LLaMA3, Mamba2, and Jamba on most evaluated tasks. The central architectural idea is to unify position encoding for hybrid sequence transformations, add a learnable mask to attention, and combine dense and embedding-based experts to improve parameter utilization and retrieval speed.

Significance. If the empirical claims were fully supported, the paper would make a useful contribution: a concrete hybrid architecture that combines the strengths of SSMs and attention, a clean proof that RoPE is compatible with SSD, and an expert-retrieval design that scales to thousands of experts at low activation cost. Credit is due for open-sourcing the code, for presenting a self-contained derivation in Appendix A, and for being transparent about a training bug that affects the reported results. However, the significance is currently limited by the fact that the main comparative claim rests on models trained before a known gradient-accumulation fix, with single runs and no error bars, so the headline ordering in Table 4 is not yet established.

major comments (3)

[Table 4 caption and Section 4.2] The central claim that Cheems outperforms LLaMA3, Mamba2, and Jamba is load-bearing but is based on runs that the authors themselves state were trained before a gradient-accumulation error fix in the Transformers library. The caption says that pretraining perplexity is therefore not comparable and that reproduced runs 'may rise' on the verification metrics, yet Table 4 and Figure 8 are produced from exactly those pre-fix runs. Gradient accumulation changes effective batch size and optimizer noise, and the paper provides no evidence that the bug affected all four architectures identically. Given that the reported margins are small (e.g., Cheems-1.3B average 52.44 vs Jamba 51.07; MMLU 39.08 vs 37.43), the ordering in Table 4 could plausibly reverse under fair training conditions. This issue must be resolved by retraining the models with the fixed library or by providing a convincing argument that the bug is architecture-symmetric; otherwise the headline comparison is unsupported.
[Section 3.2, Eqs. (5)-(7), and Appendix B.3, Listing 5] The mathematical description of DMAttn does not match the provided implementation. Equation (5) defines the dynamic mask as exp(τΔ(concat(V_cache, V) WΔ) A), and Eq. (7) applies it as a multiplicative, continuously attenuating or enhancing factor over the concatenated past and current value states. In Listing 5, however, dt is computed from the current V only (dt = self.dt_proj(rearrange(V, 'bhtp->btd'))), the dynamic mask is thresholded to a binary mask (dynamic_mask = ... < 1.0), and it is then added to the causal mask as -inf rather than multiplied with the attention scores. Because the reported MSE and MQAR results could correspond to either the equations or the code but not clearly to both, the method as presented is under-specified. The authors should align the description and the implementation, or explicitly state which version was evaluated.
[Section 4.1 and Figure 6] The abstract claims that DMAttn 'maintains 100% accuracy' and improves by 'more than 150%' on the multi-query associative recall task, but Figure 6 shows only point estimates without error bars or multiple seeds, and no numerical table is provided from which these percentages can be verified. The same single-run limitation applies to Tables 1, 2, and 3 and to Figures 7 and 8. Given that some of the reported differences are small, the absence of variance information makes it difficult to assess whether the observed improvements are significant. Please provide seed-averaged results with confidence intervals, or at least state the number of runs and the observed spread.

minor comments (5)

[Table 1] The column header 'a t' appears to be a typo; the caption refers to 'at' without explaining what this positional-encoding baseline is, and the table layout makes the comparison hard to parse.
[Table 7] The column header 'learing rate' is misspelled as 'learing rate'; it should be 'learning rate'.
[Appendix B.4, Listing 6] The variable name 'mamba_libray' in the SSD listing is a typo for 'mamba_library'.
[Section 5 and Figure 9] The discussion claims that the Doge architecture 'has no significant decrease in most verification metrics' without presenting any corresponding table or figure; either add the supporting results or soften this claim to a qualitative observation.
[Throughout] The model name is spelled inconsistently as 'LlaMa3' in tables and captions and 'Llama 3' in the references; please unify the spelling.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the RoPE-for-SSD proof is a direct algebraic derivation, and all claimed gains are empirical measurements on external benchmarks, not predictions forced by construction.

full rationale

No circular step is present. The RoPE-for-SSD argument in Appendix A is a self-contained algebraic proof that the inner product of rotated C and B vectors depends only on the relative position m-n; it assumes no target result and does not reduce to its own conclusion. DMAttn, CDMoE, and the combined Cheems architecture are evaluated on external benchmarks (multi-query associative recall, CEval, MMLU/TriviaQA/ARC/PIQA/HellaSwag/OBQA/Winogrande, and measured throughput), and no parameter fitted to those benchmarks is renamed as a prediction. The abstract's quantitative claims (over 4% perplexity reduction, 100% MQAR accuracy, 8-10x retrieval speed) are empirical readings of Table 1, Figure 6, and Figure 7, not consequences of definitions. There are no self-citations, no imported uniqueness theorem, and no ansatz smuggled through prior work by the same authors; the paper instead builds on external prior work (Mamba2, RoFormer, PEER, DeepSeek-MoE). The one serious caveat is a validity concern rather than circularity: the Table 4 caption explicitly states that all models were trained before a gradient-accumulation error fix in the Transformers library, that pretraining perplexity is therefore not comparable, and that reproduced runs may yield higher scores. That caveat reasonably undermines confidence in the reported benchmark ordering, but it does not make any derivation equivalent to its inputs, so the circularity score remains 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The paper's central claims rest on a few standard tools (SSD representation, RoPE) plus two newly proposed components (DMAttn, CDMoE). No new physical entities are introduced; the new components are model mechanisms whose behavior is defined in code. The main unexamined choices are the hard threshold in DMAttn and the hand-set 7:1 block ratio.

free parameters (4)

SSD-to-DMAttn block ratio = 7:1 per stack
Chosen by hand in Section 3.4; central architecture's performance likely depends on it, and no sensitivity analysis is provided.
Dynamic-mask hard threshold = 1.0 (mask if exp(A*softplus(dt)) < 1)
Chosen in Listing 5 and Section 3.2; converts a continuous gate into a hard mask, but the choice is arbitrary and untested.
CDMoE expert count = 3072 (320M) and 8192 (1.3B)
Set in Table 7 to match parameter counts; large expert count is central to the retrieval-speed claim and is not shown to be optimal.
Cross-domain dimension d_cd = not specified
Introduced in Eq. (8); its value is never reported, forcing reproduction guesses.

assumptions (5)

standard math SSD matrix equivalence M = L ◦ (C B^T) with scalar A
Used in Section 3.1 and Appendix A to justify applying RoPE to C and B; from Dao and Gu 2024.
standard math RoPE inner-product property for d=2 rotations
Appendix A derives the relative-position inner product; from Su et al. 2021.
standard math Extension of RoPE to even dimensions by block-diagonal rotations
Appendix A states 'for any even-dimensional RoPE... grouping the elements in pairs'.
ad hoc to paper Zero-order hold assumption for dynamic mask: states contain continuous correlation
Section 3.2: 'we will retain its value every time we receive a continuous correlated state'; this is a modeling choice without independent justification.
domain assumption Product-key top-k retrieval is a valid sparse expert layer
Section 3.3, Eqs. (9)-(10); relies on prior work by Lample et al. and He 2024.

invented entities (2)

Dynamic Mask Attention (DMAttn) independent evidence
purpose: Selectively mask key-value positions in quadratic attention based on a learned gate on V states.
Falsifiable via MQAR benchmark and downstream tasks; code in Appendix B.3 allows external testing.
Cross Domain Mixture of Experts (CDMoE) independent evidence
purpose: Combine a shared MLP with product-key top-k expert retrieval to reduce parameter redundancy while keeping retrieval fast.
Falsifiable via speed benchmark and CEval scores; code in Appendix B.4 allows external testing.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture." pith.science (2026). https://pith.science/paper/HXAFYFGC

@misc{pith2026241211834,
  author       = {Pith},
  title        = {Pith review of: Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HXAFYFGC}},
  note         = {Machine review of arXiv:2412.11834}
}

read the original abstract

In order to make the foundation model more efficient and effective, our idea is combining sequence transformation and state transformation. First, we prove the availability of rotary position embedding in the state space duality algorithm, which reduces the perplexity of the hybrid quadratic causal self-attention and state space duality by more than 4%, to ensure that the combining sequence transformation unifies position encoding. Second, we propose dynamic mask attention, which maintains 100% accuracy in the more challenging multi-query associative recall task, improving by more than 150% compared to quadratic causal self-attention and state space duality, to ensure that the combining sequence transformation selectively filters relevant information. Third, we design cross domain mixture of experts, which makes the computational speed of expert retrieval with more than 1024 experts 8 to 10 times faster than the mixture of experts, to ensure that the combining state transformation quickly retrieval mixture. Finally, we summarize these matrix algorithms that can form the foundation model: Wonderful Matrices, which can be a competitor to popular model architectures.

Figures

Figures reproduced from arXiv: 2412.11834 by the authors.

**Figure 1.** Wonderful Matrices Architecture. Shows the matrices used in the Wonderful Matrices Architecture, including the Rotary Position Embedding Matrix, State Space Duality Matrix, Dynamic Mask Attention Matrix, Cross Domain Mixture of Experts Matrix, and the process of using these matrices. To build a model that is both efficient and effective, the key is to balance the combination relationship between different sequence … view at source ↗

**Figure 2.** Rotary Position Embedding. Shows the algorithm of Rotary Position Embedding. In the case of input containing sequence dimension and hidden dimension, first add absolute position information𝑚 to the 𝑄 and𝐶 matrices, add absolute position information 𝑛 to the 𝐾 and 𝐵 matrices, then multiply the rotation matrix R 𝑑 Θ,𝑚 and R 𝑑 Θ,𝑛 with the 𝑄𝐾 or 𝐶𝐵 matrix to obtain the rotary position encoding matrix, and finally appl… view at source ↗

**Figure 3.** Dynamic Mask Attention. Shows the algorithm of Dynamic Mask Attention. The input is first projected through the projection function to obtain 𝑄𝐾𝑉 , then the attention score is calculated by the 𝑄 state and the concatenated past state 𝐾 state, the causal mask is applied to the attention score, and finally the score matrix is applied with the dynamic mask related to the concatenated past state 𝑉 state, and output to t… view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Cross Domain Mixture of Experts. Shows the algorithm of Cross Domain Mixture of Experts. The inputs first passes through the query projection, then calculates the dot product with the keys to obtain the affinity with the private experts, then activates the top K privat…

**Figure 5.** Figure 5: Wonderful Matrices in Language Modeling: Cheems. Shows the architecture of Wonderful Matrices applied in language modeling, including Word Embedding, RMSNorm, Residual, RoPE, SSD, DMAttn, CDMoE, LM Head modules. The black arrows indicate the calculation order of the mo…

**Figure 6.** Figure 6: Multi-Query Associative Recall. We introduced a more difficult version of the original multi-query associative recall task (Arora et al. 2024), including longer sequence lengths, smaller model dimensions, etc. For detailed parameters, see Appendix C.1. We compared the …

**Figure 7.** Figure 7: Multi-Expert Retrieval Mixture. We tested Shared Expert Isolation Mixture of Experts, Product Key Memory (Lample et al. 2019), Parameter Efficient Expert Retrieval and our method in the retrieval mixture speed at different activation ratios 𝑎𝑐𝑡𝑟𝑎𝑡𝑖𝑜 × √ 𝑛𝑒 . When the …

**Figure 8.** Figure 8: Efficient Benchmark. The LlaMa3 architecture that uses QCAttn as the sequence transformation, the Mamba2 architecture that uses SSD as the sequence transformation, the Jamba architecture that uses SSD and QCAttn as the sequence transformation, and the Cheems architectu…

**Figure 9.** Figure 9: Doge Architecture. Remove the SSD sequence transformation module in the Cheems architecture and modify the architecture to stack multiple CDMoE state transformation modules after a single DMAttn sequence transformation module. At the same time, Doge can also be underst…

**Figure 10.** Figure 10: Different Algorithms Parameters. At different dimensional scales, the number of parameters of DMAttn is not much different from QCAttn. SSD increases the number of parameters less when increasing the dimensional scale. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 10 canonical work pages

[1]

Zoology: Measuring and Improving Recall in Efficient Language Models

Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. “Zoology: Measuring and Improving Recall in Efficient Language Models”. In: The International Conference on Learning Representations (ICLR). 2024

work page 2024
[2]

Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra.SmolLM-Corpus. 2024. url: https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus

work page 2024
[3]

PIQA: Reasoning about Physical Commonsense in Natural Language

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. “PIQA: Reasoning about Physical Commonsense in Natural Language”. In: Proceedings of the AAAI conference on Artificial Intelligence . Vol. 34. 2020

2020
[4]

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. “Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge”. In:arXiv preprint arXiv:1803.05457 (2018)

arXiv 2018
[5]

DeepSeek- MoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. “DeepSeek- MoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models”. In: CoRR abs/2401.06066 (2024). url: https://arxiv.or...

arXiv 2024
[6]

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

Tri Dao and Albert Gu. “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality”. In: International Conference on Machine Learning (ICML) . 2024

work page 2024
[7]

A Framework for Few-shot Language Model Evaluation

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A Framework for Few-shot Language Model Evaluation . Version v0.0.1. Sept. 2021. doi: 10.5281/ zenodo.5371628. url: htt...

doi:10.5281/zenodo.5371628 2021
[8]

The Llama 3 Herd of Models

Aaron Grattafiori et al. The Llama 3 Herd of Models . 2024. arXiv: 2407.21783 [cs.AI]. url: https://arxiv.org/ abs/2407.21783

arXiv 2024

Show all 24 references

[9]

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Albert Gu and Tri Dao. “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”. In: arXiv preprint arXiv:2312.00752 (2023)

2023 arXiv
[10]

Mixture of A Million Experts

Xu Owen He. “Mixture of A Million Experts”. In: arXiv preprint arXiv:2407.04153 (2024)

2024 arXiv
[11]

Mea- suring Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. “Mea- suring Massive Multitask Language Understanding”. In: International Conference on Learning Representations. 2021

2021
[12]

C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. “C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models”. In: Advances in Neur...

2023
[13]

Weld, and Luke Zettlemoyer

Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension . 2017. arXiv: 1705.03551 [cs.CL]

2017 arXiv
[14]

Transformers are RNNs: Fast Au- toregressive Transformers with Linear Attention

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. “Transformers are RNNs: Fast Au- toregressive Transformers with Linear Attention”. In: International Conference on Machine Learning . PMLR. 2020, pp. 5156–5165

2020
[15]

Large Mem- ory Layers with Product Keys

Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. “Large Mem- ory Layers with Product Keys.” In: NeurIPS. 2019, pp. 8546–8557. url: http://papers.nips.cc/paper/9061- large-memory-layers-with-product-keys

2019
[16]

Jamba: A Hybrid Transformer-Mamba Language Model

Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. “Jamba: A Hybrid Transformer-Mamba Language Model”. In: arXiv preprint arXiv:2403.19887 (2024)

2024 arXiv
[17]

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. “Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering”. In: arXiv preprint arXiv:1809.02789 (2018)

2018 arXiv
[18]

PyTorch Container Image

Meta NVIDIA. PyTorch Container Image . https : / / catalog . ngc . nvidia . com / orgs / nvidia / containers / pytorch. 2022

2022
[19]

Winogrande: An Adversarial Winograd Schema Challenge at Scale

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. “Winogrande: An Adversarial Winograd Schema Challenge at Scale”. In: Communications of the ACM 64.9 (2021), pp. 99–106

2021
[20]

GLU Variants Improve Transformer

Noam Shazeer. “GLU Variants Improve Transformer”. In: arXiv preprint arXiv:2002.05202 (2020)

2020 arXiv
[21]

Roformer: Enhanced Transformer with Rotary Position Embedding

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. “Roformer: Enhanced Transformer with Rotary Position Embedding”. In: arXiv preprint arXiv:2104.09864 (2021)

2021 arXiv
[22]

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. “Attention Is All You Need”. In:Advances in Neural Information Processing Systems (NeurIPS). 2017

2017
[23]

Transformers: State-of-the-Art Natural Language Processing

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

2020
[24]

HellaSwag: Can a Machine Really Finish Your Sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. “HellaSwag: Can a Machine Really Finish Your Sentence?” In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . 2019. 13 A RoPE for SSD Proof of equation 2. by definit...

2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Zoology: Measuring and Improving Recall in Efficient Language Models

Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. “Zoology: Measuring and Improving Recall in Efficient Language Models”. In: The International Conference on Learning Representations (ICLR). 2024

work page 2024

[2] [2]

Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra.SmolLM-Corpus. 2024. url: https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus

work page 2024

[3] [3]

PIQA: Reasoning about Physical Commonsense in Natural Language

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. “PIQA: Reasoning about Physical Commonsense in Natural Language”. In: Proceedings of the AAAI conference on Artificial Intelligence . Vol. 34. 2020

2020

[4] [4]

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. “Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge”. In:arXiv preprint arXiv:1803.05457 (2018)

arXiv 2018

[5] [5]

DeepSeek- MoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. “DeepSeek- MoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models”. In: CoRR abs/2401.06066 (2024). url: https://arxiv.or...

arXiv 2024

[6] [6]

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

Tri Dao and Albert Gu. “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality”. In: International Conference on Machine Learning (ICML) . 2024

work page 2024

[7] [7]

A Framework for Few-shot Language Model Evaluation

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A Framework for Few-shot Language Model Evaluation . Version v0.0.1. Sept. 2021. doi: 10.5281/ zenodo.5371628. url: htt...

doi:10.5281/zenodo.5371628 2021

[8] [8]

The Llama 3 Herd of Models

Aaron Grattafiori et al. The Llama 3 Herd of Models . 2024. arXiv: 2407.21783 [cs.AI]. url: https://arxiv.org/ abs/2407.21783

arXiv 2024