Pith. sign in

REVIEW 5 major objections 8 minor 1 cited by

TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding

T0 review · 5 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TransXSSM claims that applying the same rotary position embedding to state-space update vectors as to attention queries and keys removes positional incoherence in hybrid model layers, yielding a language model that trains 42.3% faster…

desk verdict Simple, plausible hybrid recipe with controlled experiments, but the theoretical equivalence claim is undercut by a missing A-decay term and the empirical claims are slightly overstated. read the letter →

arxiv 2506.09507 v3 pith:G5G67JZ5 submitted 2025-06-11 cs.CL cs.AI

Bingheng Wu , Jingze Shi , Yifan Wu , Nan Tang , Yuyu Luo This is my paper
classification cs.CLcs.AI
keywords hybridlanguagemodelstatespace(SSM)rotarypositionembedding(RoPE)dualitypositionalencodinglong-contextmodelinglinear-timeinferencetransformer-SSM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TransXSSM sets out to solve a structural mismatch inside hybrid Transformer–state-space models: Transformers mark positions with explicit rotary embeddings, while SSMs carry position implicitly through their recurrence, and the transition between the two can scramble positional information. The paper proposes to give both component types the same rotary position embedding by rotating the SSM's C and B update vectors with the same phase schedule used for attention's Q and K. If that unification works, then a hybrid language model can get Transformer-level reasoning and SSM-level speed rather than compromising on one. The paper reports exactly that: over 4% higher accuracy than a matched Transformer on language-modeling benchmarks, training 42.3% faster and inference 29.5% faster at 4K sequence length, and larger gains when scaled to 1.3B parameters. The payoff would be a practical route to long-context models that are both accurate and cheap to run.

What carries the argument

The central object is Unified RoPE: four position-encoding functions $f_Q(q,m)=qe^{im\theta}$, $f_K(k,n)=ke^{in\theta}$, $f_C(c,m)=ce^{im\theta}$, $f_B(b,n)=be^{in\theta}$, implemented in real form as block-diagonal rotation matrices $R^d_{\Theta,m}$. It does the work of the argument by making both the attention score $\langle f_Q(q_m,m), f_K(k_n,n)\rangle$ and the SSD score $\langle f_C(c_m,m), f_B(b_n,n)\rangle$ equal to a rotated inner product that depends only on $m-n$, so positional phase flows continuously across heterogeneous layers. The proof that this holds proceeds by expanding the 2D complex product and extending to $d/2$ independent 2D blocks with frequencies $\theta_i = 10000^{-2i/d}$.

What would settle it

Pick random matrices $A, B, C$ and input $X$ with $d \ge 2$ and nonzero state decay $A$, and compare the exact SSD output $y_t = \sum_s C_t^\top A^{t:s} B_s x_s$ with the rotated-kernel output obtained from Eq. (7)'s $L \circ (C R_{m-n} B^\top) X$. If the two outputs differ materially for any such instance, the central equivalence that underlies Unified RoPE's linear-time claim is false.

Watch

Extended reading notes

Core claim

The paper's central claim is that RoPE is not a Transformer-only mechanism: the state-space duality view of masked attention already writes the SSD output as $L \circ (CB^\top)$ applied to the input, so the matrices $C$ and $B$ occupy the same role as $Q$ and $K$. Unified RoPE therefore rotates $C$ at position $m$ by $e^{im\theta}$ and $B$ at position $n$ by $e^{in\theta}$, making their inner product $c_m R^d_{\Theta,m-n} b_n$, a pure function of the relative position $m-n$. The same relative-position rotation then appears in attention scores and in SSD scores, giving every layer one shared positional phase. The paper argues that this removes the positional discontinuity between stacked SSM and attention layers, and because the rotation is applied to input vectors before the recurrence, it does not change the asymptotic cost: SSM layers keep their linear-time update and attention layers keep their cached linear-time generation. On this basis TransXSSM alternates seven SSM layers with one attention layer and reports higher accuracy, speed, and scaling than pure Transformer, pure SSM, and prior hybrid baselines.

Load-bearing premise

The load-bearing premise is that rotating C and B by the position phase faithfully reproduces the RoPE masked-attention kernel inside the SSD recurrence; the paper's Eq. (7) omits the state-decay factor A that appears in the Appendix's SSD definition, so the claim implicitly assumes $A^{j:i}$ commutes with or is absorbed by the rotation phase. If that interaction is not clean, the equivalence proof and the linear-time inference guarantee would need to be redone.

Editorial extensions

If this is right

  • A hybrid language model built on Unified RoPE can run long contexts at near-SSM cost: 42.3% faster training and 29.5% faster inference than a same-size Transformer at 4K sequence length.
  • Accuracy is not sacrificed for speed: under matched training settings TransXSSM beats a Transformer baseline by over 4% on average across seven language-modeling tasks.
  • The advantage compounds with scale: scaling from 320M to 1.3B parameters gives a 7.22-point average gain, larger than the roughly 6-point gains of the Transformer and SSM baselines.
  • Long-context retrieval also improves: TransXSSM-1.3B holds high needle-in-a-haystack accuracy across context lengths where a pure SSM's retrieval degrades.
  • Because all layers share the same positional phase, the 7:1 SSM-to-attention stacking ratio can be chosen for efficiency without creating positional incongruity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If spectral continuity is what Unified RoPE provides, then the 7:1 layer ratio is a tunable efficiency knob rather than a fixed architectural constraint; varying that ratio should trace a continuous accuracy–speed frontier for hybrid models.
  • The same 'rotate C and B like Q and K' recipe should transfer to any other linear-time sequence layer whose computation factors as a masked low-rank product, so the unification is not limited to Mamba-style SSMs.
  • The paper's own comment on RoPE's unbalanced positional distribution suggests a direct stress test: training TransXSSM on longer contexts than those in the training set should expose whether the high-bit generalization limits are inherited by the SSM layers.
  • A matched ablation that gives the same hybrid architecture separate positional encodings for its two component types would quantify how much of the reported gains come from unification itself rather than from the other architectural choices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper proposes Unified RoPE, which applies the same rotary position embedding to the C and B matrices of an SSM as to the Q and K matrices of self-attention, and introduces TransXSSM, a hybrid architecture with a 7:1 ratio of state-space to attention sublayers. The authors claim that this unified positional scheme resolves a 'positional spectrum discontinuity' in hybrid models, that it preserves linear-time SSM inference, and that TransXSSM outperforms Transformer, SSM, and hybrid baselines in both accuracy and speed. The main theoretical support is an appendix intended to show that rotating C and B reproduces the RoPE kernel inside the state-space-duality (SSD) formulation, and the main empirical support is a set of retrained 320M and 1.3B models evaluated on seven downstream tasks.

Significance. If the central claim were established, the paper would offer a simple and potentially useful design principle for hybrid Transformer-SSM models: unify positional encoding by placing the same rotation phase on C and B as on Q and K. The paper has real strengths: Table 2 provides a controlled comparison of position-encoding schemes, the baselines in Table 3 are retrained under identical conditions, and the throughput measurements in Figure 4 give concrete efficiency evidence. However, the theoretical derivation in Appendix A and Eq. (7) omits the state-decay factor A, the sign convention for the rotation matrix is internally inconsistent, and the headline accuracy claims in the abstract and Section 4.2 are not consistent with the numbers in Table 3. These are load-bearing issues, not presentation flaws.

major comments (5)
  1. [§2.2, Eq. (7); Appendix A] Equation (7) defines SSD(C,B,X)_i = sum_{j<=i} c_i R_{Theta,i-j} b_j^T x_j with no state-decay factor, but Appendix A defines the SSD matrix as M_ji = C_j^T A_{j:i} B_i and then represents it as a Toeplitz mask L composed with CB^T. For a general scalar A, the rotated recurrence gives y_i = sum_{j<=i} A^{i-j} c_i^T R_{i-j} b_j x_j, which reduces to Eq. (7) only when A=1. The paper neither states that the experiments use A=1 nor provides the position-dependent rescaling (e.g., B_j' = A^{-j} R_j^{-1} B_j, C_i' = A^i R_i^T C_i) that would absorb A into the rotated factors. Since the central claim that Unified RoPE reproduces the RoPE kernel while retaining the Mamba-2 linear-time algorithm depends on this equality, the omission is load-bearing and must be fixed by either specifying A=1 or deriving and verifying the modified recurrence against the chunked Mamba-2 kernel.
  2. [§2.2, Eq. (4); Appendix A] There is an internal sign inconsistency in the rotation matrix. Equation (4) states that the inner product of position-encoded vectors equals q_m R_{Theta,m-n} k_n^T, but a direct calculation gives <R_m q, R_n k> = q^T R_m^T R_n k = q^T R_{n-m} k. The long scalar expansion in Appendix A also evaluates to a rotation by n-m (equivalently -(m-n)), while the final displayed matrix in Appendix A is written as R_{m-n}. The authors should correct the convention and ensure Eq. (4), Eq. (7), and the appendix all use the same rotation direction.
  3. [Abstract; §4.2; Table 3] The headline accuracy claims are not supported by Table 3. The abstract says TransXSSM 'surpasses a Transformer baseline by over 4% on language modeling benchmarks,' and Section 4.2 says the 1.3B model 'surpasses all baselines by over 2 points on seven diverse tasks.' In Table 3, the average gain over LlaMa3 is 1.23 points at 320M and 2.08 points at 1.3B, and versus Jamba-1.3B only two of the seven tasks (TriviaQA and Winogrande) exceed 2 points; ARC and OBQA are slightly negative. The claims should be revised to match the reported per-task deltas, or additional evidence should be provided for the larger margins.
  4. [§4.2; Table 3] The empirical comparisons lack any measure of variability. Differences as small as 0.03 points (OBQA at 1.3B) and 0.13 points (ARC at 1.3B) are presented as evidence of 'consistently strong performance,' but without multiple seeds, confidence intervals, or significance tests, these differences are within likely noise. At minimum, the authors should report standard deviations or run additional seeds for the key comparisons.
  5. [§1; Table 1] The central conceptual notion of 'positional spectrum discontinuity' is never formally defined. The paper repeatedly invokes spectral continuity and spectral incompatibility, but it does not define a spectrum for positional encodings or state a formal condition under which two layers are spectrally compatible. Without such a definition, the claimed mechanism cannot be tested independently of the empirical results.
minor comments (8)
  1. [§2.2, Eq. (2)] The notation e_{i,m,θ} is nonstandard and should be written as e^{i m θ} or an equivalent explicit complex exponential.
  2. [§2.2, Eq. (5)] The set Θ is defined twice with inconsistent indexing: the first definition uses i in [0, ..., d/2-1] with θ_i = 10000^{-2i/d}, and the appendix uses i in [1, ..., d/2] with θ_i = 10000^{-2(i-1)/d}. These should be unified.
  3. [Table 2] The header 'Conv1d + Da t' is garbled and should read 'Conv1d + D' and 'a_t' (or similar); the current formatting makes the table difficult to interpret.
  4. [References] Reference [1] is cited for Llama3 but points to Ghahramani and Hinton (2000); the Llama3 citation should be [20] (Grattafiori et al.).
  5. [Appendix A] In the derivation, the line defining f_C uses f_C(x_m, n) where the second argument should be m; this is a typo, but it obscures an already notation-heavy proof.
  6. [Appendix A] The final paragraph introduces an extrapolation formula 'C×max(1, log_base n)' with no definition of C or the base, and it is not connected to any experiment in the paper. If it is meant as a contribution, define and evaluate it; otherwise remove it.
  7. [§3; Figure 3] The text says each module contains 7 SSM sub-layers followed by 1 attention sub-layer, but Figure 2 and Figure 3 show a single SS block and a single SA block per module; the relationship between the diagram and the 7:1 ratio should be clarified.
  8. [Table 4] The column header 'LEANINGRATE' is a typo for 'LEARNING_RATE'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Appendix's RoPE-for-SSD identity is a mathematical consequence of the rotation definitions, and the paper's performance claims rest on controlled empirical comparisons rather than on a fitted or self-cited premise.

full rationale

The paper's central claims—accuracy gains, speedups, and scaling trends—are supported by retrained baselines and benchmark evaluations under matched settings, so they are empirically testable rather than derived by construction. The theoretical part in Appendix A proves that applying the rotation functions f_C and f_B (defined in Eq. 2) makes the inner product depend on m-n; this is a valid trigonometric identity and not a fitted parameter renamed as a prediction. It is definitional in the sense that RoPE is being applied by construction, but the paper does not use the conclusion to justify the premise. The references to Mamba-2 and to hybrid-model ratios are external prior work, not self-citations by the present authors. One genuine concern is that Eq. (7) writes SSD(C,B,X)_i without the A^{i:j} state-decay factor that appears in the Appendix's own SSD definition (M_ji = A_{j:i} * C_j^T B_i); for general scalar A, the rotated recurrence should include A^{i:j} unless an explicit absorbing rescaling is stated. That is a correctness/derivation gap, not a circularity, because the paper does not assume what it sets out to prove. No step was found where an output is identical to an input by construction in a way that undercuts the paper's conclusions.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or model entities are posited. The load-bearing assumptions are the Mamba-2 duality, the RoPE geometric identity, the unverified compatibility of position-dependent rotations with non-trivial state decay A, and the unmeasured spectral discontinuity hypothesis. The 7:1 ratio is a hand-chosen hyperparameter, and no constants are fit to data.

free parameters (1)
  • SSM-to-attention stacking ratio (7:1) = 7:1
    Chosen based on prior hybrid studies and the authors' ablation; not derived from first principles. It directly affects the reported throughput and accuracy comparisons.
assumptions (4)
  • domain assumption Masked softmax attention is equivalent to an SSD with M = L elementwise product (C B^T) for suitable C, B (Mamba-2 duality).
    Invoked in Section 2.1 to justify transferring RoPE from Q,K to C,B; depends on the correctness of the identity (L elementwise QK^T)V = (L elementwise CB^T)X.
  • standard math Rotating both query-like and key-like vectors by angles m theta and n theta yields an inner product depending only on m-n.
    Used in Eq. (4) and Appendix A; correct for even-dimensional real vectors and is the standard RoPE property.
  • domain assumption Applying the same position-dependent rotation to C and B in the recurrent SSM update preserves the SSD equivalence and linear-time inference.
    Section 2.2 claims compatibility, but the paper does not explicitly derive the interaction with the state decay A in the recurrent, chunked Mamba-2 algorithm.
  • ad hoc to paper The positional spectrum discontinuity between Transformer and SSM layers is a real cause of hybrid underperformance and is fixed by unified RoPE.
    This causal hypothesis is asserted in Sections 1 and 5 but never measured directly; no diagnostic of positional spectrum continuity is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding." pith.science (2026). https://pith.science/paper/G5G67JZ5

@misc{pith2026250609507,
  author       = {Pith},
  title        = {Pith review of: TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G5G67JZ5}},
  note         = {Machine review of arXiv:2506.09507}
}
read the original abstract

Transformers exhibit proficiency in capturing long-range dependencies, whereas State Space Models (SSMs) facilitate linear-time sequence modeling. Notwithstanding their synergistic potential, the integration of these architectures presents a significant challenge, primarily attributable to a fundamental incongr inuity their respective positional encoding mechanisms: Transformers rely on explicit Rotary Position Embeddings (RoPE), while SSMs leverage implicit positional representations via convolutions. This divergence often precipitates discontinuities and suboptimal performance.To address this impediment, we propose a unified rotary position embedding (Unified RoPE) methodology, thereby establishing a consistent positional encoding framework for both self-attention and state-space components. Using this Unified RoPE, we introduce TransXSSM, a hybrid architecture that coherently integrates the Transformer and SSM layers under this unified positional encoding scheme. At a 4 sequenceK length, TransXSSM exhibits training and inference speeds that are 42.3% and 29.5% faster, respectively, relative to standard Transformer models. It also delivers higher accuracy: under comparable settings, it surpasses a Transformer baseline by over 4% on language modeling benchmarks.TransXSSM furthermore scales more effectively: TransXSSM-1.3B gains 7.22% in average accuracy over its 320M version (versus about 6% gains for equivalent Transformers or SSMs). Our results show that unified positional encoding resolves positional incompatibility in hybrid models, enabling efficient, high-performance long-context modeling.

Figures

Figures reproduced from arXiv: 2506.09507 by the authors.

Figure 1
Figure 1. Rotary Position Embedding Application. Application of Unified RoPE. Input vectors (Q, C, K, B) are enriched with absolute positions (m, n) via rotation matrices R d Θ,m or R d Θ,n (Eq. (3) and Eq.(5)), yielding position-encoded vectors. Their inner product captures relative position m − n (Eq. (4)). An optional masking step can follow. f{Q,C}(xm, m) =  cos(mθ) − sin(mθ) sin(mθ) cos(mθ)  x (1) m x (2) m ! , f{K,B}(… view at source ↗
Figure 2
Figure 2. TransXSSM Block. Structure of the TransXSSM block, integrating State-Space (SS), Multi-Layer Perceptron (MLP), and Transformer-style Self-Attention (SA) blocks. All blocks utilize the proposed Unified RoPE. Wor d Embedding Rotary Position Embedding Inputs RMSNor m Self-Attention Residual RMSNor m FeedForwar d Residual LM Head Outputs RMSNor m Rotary Position RMSNor Embeddingm State-Space Residual RMSNor m FeedForwar… view at source ↗
Figure 3
Figure 3. TransXSSM Language Modeling Architecture. Each input token is first converted to an embedding, then passes through N stacked hybrid modules. Each module contains 7 state-space (SS) layers and 1 self-attention (SA) layer, all employing the unified RoPE positional embedding. Every SS or SA layer is followed by an FFN (feed-forward network), and each pair (SS/SA + FFN) is wrapped with residual connections and RMSNorm n… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Throughput Evaluation. Training (left) and evaluation (right) throughput (iterations/sec￾ond) for LlaMa3, Mamba2, Jamba, and TransXSSM at the 1.3B parameter scale across varying sequence lengths. TransXSSM surpasses LlaMa3 and Jamba in efficiency, while being slightly …
Figure 5
Figure 5. Figure 5: Needle in a Haystack Performance (1.3B Models). Performance comparison of LlaMa3, Mamba2, Jamba, and TransXSSM (1.3B scale) on the “needle in a haystack” task. TransXSSM, with its unified position encoding, exhibits strong performance [PITH_FULL_IMAGE:figures/full_fig…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    cs.LG 2025-09 unverdicted novelty 7.0 of 10

    Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...

Reference graph

Works this paper leans on

31 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Variational learning for switching state-space models,

    Z. Ghahramani and G. E. Hinton, “Variational learning for switching state-space models,”Neural Compu- tation, vol. 12, no. 4, pp. 831–864, 2000

    work page 2000

  2. [2]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality,

    T. Dao and A. Gu, “Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality,” inInternational Conference on Machine Learning (ICML), 2024

    work page 2024

  3. [3]

    Jamba: A hybrid transformer-mamba language model,

    O. Lieber, B. Lenz, H. Bata, G. Cohen, J. Osin, I. Dalmedigos, E. Safahi, S. Meirom, Y . Belinkov, S. Shalev- Shwartzet al., “Jamba: A hybrid transformer-mamba language model,”arXiv preprint arXiv:2403.19887, 2024

    arXiv 2024

  4. [4]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  5. [5]

    An empirical study of mamba-based language models,

    R. Waleffe, W. Byeon, D. Riach, B. Norick, V . Korthikanti, T. Dao, A. Gu, A. Hatamizadeh, S. Singh, D. Narayananet al., “An empirical study of mamba-based language models,”arXiv preprint arXiv:2406.07887, 2024

    arXiv 2024

  6. [6]

    Roformer: Enhanced Transformer with rotary position embedding,

    J. Su, Y . Lu, S. Pan, A. Murtadha, B. Wen, and Y . Liu, “Roformer: Enhanced Transformer with rotary position embedding,”arXiv preprint arXiv:2104.09864, 2021

    arXiv 2021

  7. [7]

    Mamba-360: Survey of state space models as transformer alternative for long sequence modelling: Methods, applications, and challenges,

    B. N. Patro and V . S. Agneeswaran, “Mamba-360: Survey of state space models as transformer alternative for long sequence modelling: Methods, applications, and challenges,”arXiv preprint arXiv:2404.16112, 2024

    arXiv 2024

  8. [8]

    Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models,

    A. Blakeman, A. Basant, A. Khattar, A. Renduchintala, A. Bercovich, A. Ficek, A. Bjorlin, A. Taghibakhshi, A. S. Deshmukh, A. S. Mahabaleshwarkaret al., “Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models,”arXiv preprint arXiv:2504.03624, 2025

    arXiv 2025

Show all 31 references
  1. [9]

    Stuffed mamba: State collapse and state capacity of rnn-based long-context modeling,

    Y . Chen, X. Zhang, S. Hu, X. Han, Z. Liu, and M. Sun, “Stuffed mamba: State collapse and state capacity of rnn-based long-context modeling,”arXiv preprint arXiv:2410.07145, 2024

    2024

  2. [10]

    Efficient long sequence modeling via state space augmented Transformer,

    S. Zuo, X. Liu, J. Jiao, D. Charles, E. Manavoglu, T. Zhao, and J. Gao, “Efficient long sequence modeling via state space augmented Transformer,”arXiv preprint arXiv:2212.08136, 2022

    2022 arXiv

  3. [11]

    Block-state transformers,

    J. Pilault, M. Fathi, O. Firat, C. Pal, P.-L. Bacon, and R. Goroshin, “Block-state transformers,” inThirty- seventh Conference on Neural Information Processing Systems, 2023

    2023

  4. [12]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,”arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  5. [13]

    Hungry hungry hippos: Towards language modeling with state space models,

    D. Y . Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Re, “Hungry hungry hippos: Towards language modeling with state space models,” inThe Eleventh International Conference on Learning Representations, 2022

    2022

  6. [14]

    Hyena hierarchy: Towards larger convolutional language models,

    M. Poli, S. Massaroli, E. Nguyen, D. Y . Fu, T. Dao, S. Baccus, Y . Bengio, S. Ermon, and C. Ré, “Hyena hierarchy: Towards larger convolutional language models,” inThe International Conference on Machine Learning (ICML), 2023

    2023

  7. [15]

    StripedHyena: Mov- ing Beyond Transformers with Hybrid Signal Processing Models,

    M. Poli, J. Wang, S. Massaroli, J. Quesnelle, R. Carlow, E. Nguyen, and A. Thomas, “StripedHyena: Mov- ing Beyond Transformers with Hybrid Signal Processing Models,” https://github.com/togethercomputer/ stripedhyena, 2023

    2023

  8. [16]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnieret al., “Mistral 7b,”arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  9. [17]

    Multi-head state space model for speech recognition,

    Y . Fathullah, C. Wu, Y . Shangguan, J. Jia, W. Xiong, J. Mahadeokar, C. Liu, Y . Shi, O. Kalinli, M. Seltzer, and M. J. F. Gales, “Multi-head state space model for speech recognition,” inProceedings of INTER- SPEECH 2023, 2023, pp. 241–245

    2023

  10. [18]

    Diagonal state space augmented Transformers for speech recognition,

    G. Saon, A. Gupta, and X. Cui, “Diagonal state space augmented Transformers for speech recognition,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  11. [19]

    Can mamba learn how to learn? a comparative study on in-context learning tasks,

    J. Park, J. Park, Z. Xiong, N. Lee, J. Cho, S. Oymak, K. Lee, and D. Papailiopoulos, “Can mamba learn how to learn? a comparative study on in-context learning tasks,” inThe International Conference on Machine Learning (ICML), 2024. 16

    2024

  12. [20]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru,...

    2024 arXiv

  13. [21]

    Smollm-corpus,

    L. Ben Allal, A. Lozhkov, G. Penedo, T. Wolf, and L. von Werra, “Smollm-corpus,” 2024. [Online]. Available: https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus

    2024

  14. [22]

    Gpt-neox-20b: An open-source autoregressive language model,

    S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. Mc- Donell, J. Phanget al., “Gpt-neox-20b: An open-source autoregressive language model,”arXiv preprint arXiv:2204.06745, 2022

    2022 arXiv

  15. [23]

    Transformers: State-of-the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Transformers: State-of-the-art nat...

    2020

  16. [24]

    A framework for few-shot language model evaluation,

    L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “A framework for few-shot language model evaluation,” Sep. 2021. [Online]. Available: https://d...

    2021 doi

  17. [25]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” inInternational Conference on Learning Representations, 2021

    2021

  18. [26]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,

    M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer, “Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,” 2017

    2017

  19. [27]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try ARC, the AI2 reasoning challenge,”arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  20. [28]

    PIQA: Reasoning about physical commonsense in natural language,

    Y . Bisk, R. Zellers, J. Gao, Y . Choiet al., “PIQA: Reasoning about physical commonsense in natural language,” inProceedings of the AAAI conference on Artificial Intelligence, vol. 34, 2020

    2020

  21. [29]

    HellaSwag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “HellaSwag: Can a machine really finish your sentence?” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  22. [30]

    Can a suit of armor conduct electricity? a new dataset for open book question answering,

    T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,”arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  23. [31]

    Winogrande: An adversarial Winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial Winograd schema challenge at scale,”Communications of the ACM, vol. 64, no. 9, pp. 99–106, 2021. 18

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.