Pith. sign in

REVIEW 5 major objections 5 minor 30 references

Synergy: End-to-end Concept Model

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A learned router lets byte-level models beat tokenized Llama3 at the same scale.

desk verdict Promising architecture, honest limitations, but the central Llama3 advantage is not established by filtered single runs and unmatched compute. read the letter →

arxiv 2507.12769 v1 pith:HDXKGKBR submitted 2025-07-17 cs.CL cs.AI

Keli Zheng , Zerong Xie This is my paper
classification cs.CLcs.AI
keywords byte-levellanguagemodeltokenizer-freelearnedroutingconcepttokensMixtureofDepthsposition-independentconceptsbits-per-byteend-to-endabstraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Synergy is a byte-level decoder-only language model that does away with a fixed tokenizer. A learned router decides, for each byte, whether the byte also passes through an expensive middle transformer block; the router's weights double as a soft importance score, so the model learns its own segmentation of bytes into word-like concept tokens. The paper's central claim is that this end-to-end scheme matches or beats Llama3 with the same number of transformer layers and the same training data, measured in bits-per-byte, while using fewer concept tokens than a BBPE tokenizer would, and that the middle part performs best with no positional encoding at all. If true, it shows tokenizer-free byte-level transformers are a viable alternative to tokenized LLMs and can compress computation onto semantically meaningful positions. The paper also reports two costs: training is unstable because the top-k routing mask is non-differentiable, and Synergy uses about 1.5 times Llama3's FLOPs.

What carries the argument

The routing block that carries the argument is a top-k gated residual connection into the middle transformer: for each byte the encoder produces a scalar $w_i=\mathrm{Router}(x_i)$, a mask $m_i$ keeps only the $k$ largest weights, and the decoder input becomes $y_i = x_i + m_i\,\sigma_i\,\mathrm{Middle}(x_i)$ with gating factor $\sigma_i=\mathrm{sigmoid}(w_i)$. The mask is non-differentiable, so only the gating path supplies the training signal for which tokens get selected. Around this mechanism, the model uses local attention in the encoder and decoder and no positional encoding in the middle, which together push the middle part to operate on position-independent concepts at a coarser granularity.

What would settle it

Retrain the exact configuration on the same Wikipedia split with several random seeds and compare full bits-per-byte trajectories without filtering outliers. If Synergy's reported edge over Llama3 disappears once run-to-run variance is accounted for, or if lowering the concept-token count from 224 to 192 reliably degrades performance, the central claims would be contradicted. The paper states that reported results were filtered to remove poor outliers, so an unfiltered all-seeds comparison is the decisive check.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a single end-to-end byte-level transformer can spontaneously learn a byte-to-concept segmentation through a trainable router, and that this learned segmentation is more compact than BBPE: performance does not degrade until the number of concept tokens falls below 192, whereas the Llama3 tokenizer produces about 241 tokens for the same text. Trained on the same Wikipedia data with the same number of transformer layers, Synergy reaches a lower bits-per-byte than Llama3 once about 0.6e10 bytes of data have been seen, despite having 0.5B parameters versus Llama3's 0.8B because Synergy's vocabulary is just bytes plus special tokens. A further unexpected result is that removing positional encoding from the 24 middle layers improves bits-per-byte to 0.9906 versus 1.0164 with original positions, which the authors read as evidence that the middle part processes position-independent concepts whose position information is absorbed by the encoder. The authors present these findings as evidence of feasibility for tokenizer-free architectures, not as a production-ready system.

Load-bearing premise

The router can be trained well enough through its gating path even though the top-k selection itself has no gradient; if that learned routing signal is too weak or too unstable, the concept-token compression and the whole architecture collapse.

Editorial extensions

If this is right

  • Tokenizer-free byte-level transformers can match tokenized-model quality at the same layer count and data size, removing the need for a fixed vocabulary and its embedding cost.
  • Learned segmentation can be more compact than statistical tokenizers: concept-token counts down to 192 outperform BBPE's roughly 241 tokens for equal text, implying cheaper middle-block computation.
  • Removing positional encoding from the abstract middle layers improves loss, suggesting the architecture may extrapolate to longer sequences better than position-encoded models.
  • The byte-level interface makes the architecture applicable to raw multimodal input such as images and audio without adding a tokenizer per modality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the position-independent middle representations are real, they may transfer across contexts better than token embeddings, and the encoder could be retrained for new modalities without changing the middle.
  • The reported training instability suggests a concrete fix: replace the non-differentiable top-k with a differentiable soft top-k or a Gumbel-style relaxation, or add a REINFORCE-style baseline, and test whether glitches disappear.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript introduces Synergy, a byte-level decoder-only language model that uses a learned router to select a fixed number of byte positions to pass through a middle transformer stack, while encoder and decoder stacks process all bytes. The authors report three headline claims: (i) Synergy achieves lower Bits-Per-Byte (BPB) than a Llama3-style baseline after sufficient training data, under "same model scale and training dataset size"; (ii) it compresses bytes into fewer concept tokens than Byte-level Byte Pair Encoding (BBPE) tokenizers; and (iii) removing positional encoding from the middle stack improves performance, suggesting the emergence of position-independent concepts. The paper presents the architecture, BPB training curves, a positioning-mode ablation, router visualizations, and a discussion of limitations.

Significance. The proposed architecture is a plausible and interesting direction: learning where to allocate computation in a byte-level model is directly relevant to tokenizer-free language modeling, and the paper connects to Mixture-of-Depths, BLT, MegaByte, and Dynamic-Pooling Transformer. The router visualization and the positioning-mode ablation are thought-provoking. However, the empirical support for the headline claims is weak, and the paper's own limitation section concedes that the central comparison is based on filtered, single-run results. If the architecture were validated with matched baselines, multiple seeds, and released artifacts, it could become a useful contribution; in its current form, the evidence does not establish the claims.

major comments (5)
  1. [Sec. 3.2, Fig. 2] The claim of an advantage over Llama3 "under the same model scale" is contradicted by the paper's own reported numbers: Llama3 has 0.8B parameters while Synergy has 0.5B, a 60% difference, and only the layer count is matched. Matching layer count is not the same as matching model scale. In addition, Sec. 5.2 reports that Synergy consumes about 1.5 times more FLOPs than Llama3. Therefore the observed BPB crossover in Fig. 2 cannot be attributed to an architectural advantage; it may be an artifact of the unequal baseline.
  2. [Sec. 5.1, Fig. 2] The central BPB comparison is undermined by the acknowledged filtering of "poor outliers." The paper states that each training session may yield very different BPB values and that the presented results are partial results after filtering, with no seed count, no variance measure, and no stated selection protocol. Under acknowledged training instability, the crossover at roughly 0.6T bytes could be a selection artifact rather than a property of the architecture. The statement that "all results are confirmed to be reproducible" is unverifiable without code, checkpoints, or a precise protocol.
  3. [Fig. 2, Table 1, Sec. 3.1.3] There are serious unit inconsistencies. Fig. 2's x-axis is labeled "Trained Bytes 1e10" with values up to 1.4, implying at most 14 billion bytes, yet the text in Sec. 3.2 refers to a crossover at "0.6T bytes" (600 billion bytes). Table 1 reports BPB "at 5.5T token," while Sec. 3.1.3 states the training set is about 12G bytes. These inconsistencies make it impossible to interpret the training progress and the claimed crossover point; the authors must state units consistently and align the axis, the text, and the table.
  4. [Sec. 3.4, Fig. 5] The claim that Synergy produces "fewer concept tokens than BBPE tokenizers" is not fully supported. The number of concept tokens is a fixed hyperparameter k, not a quantity learned by the model; the router only selects which byte positions within that budget. The comparison with the BBPE average of 240.94 tokens per 1024 bytes relies on a single operating point (k=192) from Fig. 5, with no error bars and no report of run-to-run variability. Moreover, the claim that performance "does not decrease" until k falls below 192 needs a statistical basis before it can support the token-efficiency conclusion.
  5. [Sec. 3.3, Table 1] The conclusion that the middle part learns "position-independent concepts" is an interpretation, not a demonstrated property. The "None" positioning mode giving the best BPB could also be explained by the encoder absorbing positional information, by a change in optimization dynamics, or by reduced capacity in the middle stack. The paper acknowledges the first alternative, but no control experiment (e.g., probing position information in middle representations, or permuting positions at inference) is provided to distinguish these explanations. The claim should be softened or supported with additional analysis.
minor comments (5)
  1. [Abstract] The word "whiling" should be "while" in the abstract.
  2. [Eq. (4)] The notation m_i sigma_i is not fully explained; it would help to state explicitly which terms are differentiable and how gradients reach the router through the gating factor, especially given the non-differentiable top-k mask.
  3. [Sec. 3.1.2] The sentence "The 2/3 factor introduced by SwiGLU is not multiplied" is unclear; specify the actual intermediate dimension and activation logic.
  4. [Fig. 3] The six curves in Fig. 3 are difficult to distinguish in grayscale; use distinct line styles or markers.
  5. [Sec. 3.2] The baseline is called "Llama3," but it appears to be a Llama3-style model trained from scratch with a restricted configuration; this should be stated explicitly to avoid confusion with publicly released Llama3 checkpoints.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity: the central BPB-advantage claim is an external benchmark comparison with no self-citations; only the 'fewer concept tokens' count is fixed by the k hyperparameter rather than produced by the model.

  1. self definitional [Section 3.4 (Concept Tokens Number); Eq. 2; Section 3.1.2]
    "we conduct a series of experiments to test the influence of the concept token count (i.e., the number of tokens processed in the middle part) on the performance. As shown in fig. 5, the performance does not decrease until the number of concept tokens falls below 192, while the number of tokens from Llama3 tokenizer is about 1024/4.25 = 240.94. This result shows the potential of our model to tokenize bytes more efficiently, yielding fewer concept tokens than BBPE tokenizers."

    The paper defines the 'concept token count' as 'the number of tokens processed in the middle part' (Sec. 3.4), and Eq. 2 (mi = topkmask(wi, k)) fixes that number to the hyperparameter k (default 224, Sec. 3.1.2). The model therefore always routes exactly k tokens through the middle; it never 'produces' a variable count. The headline claim 'producing fewer concept tokens than BBPE tokenizers' thus reduces, in its count dimension, to choosing k = 192 < 240.94: the comparison count is the experimenter's hyperparameter, not a model output. The genuinely empirical, non-circular remainder is that BPB does not degrade at k = 192 and that the router's learned mask correlates with word boundaries (Fig. 4).

full rationale

Synergy's central claims are empirical measurements rather than derivations. The Llama3 comparison (Sec. 3.2, Fig. 2) is a measured BPB curve against an external baseline trained on the same clipped text segments, using the tokenizer-independent BPB metric adopted from external prior work (Xue et al. 2022; Yu et al. 2023; Wang et al. 2024; Pagnoni et al. 2024); nothing in the metric or the architecture is defined in terms of the target result. The router (Eqs. 1-4) is explicitly attributed to the external Mixture-of-Depths paper (Raposo et al. 2024), and the paper contains no self-citations at all, so the self-citation, uniqueness-import, and ansatz-smuggling patterns do not apply. The position-independence claim (Sec. 3.3) is an ablation ('none' mode best, Table 1) followed by a clearly labeled post hoc interpretation ('We assume that this is because...'), not a forced derivation. The one identified definitional element is the 'fewer concept tokens' headline (Sec. 3.4, Fig. 5): the concept-token count is defined as the number of tokens passed through the middle part, which Eq. 2 fixes to the hyperparameter k, so 'producing fewer concept tokens than BBPE' is, in its count dimension, the experimenter's chosen k rather than a model output; the empirical remainder (BPB holds at k = 192, and the learned mask tracks word boundaries) is independent. Correctness-relevant limitations are weighed as validity risk, not circularity: Sec. 5.1 concedes that 'the experimental results presented in this paper are partial results after being filtered out the poor outliers' with runs not repeated; Sec. 3.2 concedes a 0.8B vs 0.5B parameter gap with only layer count matched; Sec. 5.2 reports about 1.5x FLOPs. These undermine whether the claimed Llama3 advantage is established, but they are experimental-rigor concerns, not circular steps. Score: 1.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central experiments rest on standard LM machinery plus one design-specific assumption about router trainability; no new external entities are postulated.

free parameters (1)
  • middle token budget k (concept tokens per 1024 bytes) = 224 default; 192 viable in Fig.5 sweep
    Hand-chosen top-k count determines how many byte positions pass through the middle transformer; the claim of fewer concept tokens than BBPE comes from sweeping this number.
assumptions (3)
  • standard math Standard transformer building blocks (self-attention, SwiGLU MLP, RoPE) behave as in prior literature.
    Used throughout Section 2 and Section 3 without proof.
  • ad hoc to paper The router can be trained through the sigmoid gating path even though the top-k mask is non-differentiable.
    Eq.4 computes y_i = x_i + m_i sigma_i Middle(x_i); Section 5.1 admits the router training is imperfect and unstable.
  • domain assumption Bits-Per-Byte computed on held-out Wikipedia text is a sufficient metric for language modeling quality in this comparison.
    Section 3.2 uses BPB because tokenizers differ; no downstream-task evaluation is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Synergy: End-to-end Concept Model." pith.science (2026). https://pith.science/paper/HDXKGKBR

@misc{pith2026250712769,
  author       = {Pith},
  title        = {Pith review of: Synergy: End-to-end Concept Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HDXKGKBR}},
  note         = {Machine review of arXiv:2507.12769}
}
read the original abstract

In this paper, we present Synergy, a language model that bridges different levels of abstraction in an end-to-end fashion through a learned routing mechanism. Focusing on low-level linguistic abstraction, we trained our model as a byte-level language model. Our model spontaneously learns to tokenize bytes, producing fewer concept tokens than Byte-level Byte Pair Encoder (BBPE) tokenizers while keeping comparable performance. By comparing with Llama3, we observed an advantage of Synergy under the same model scale and training dataset size. Further studies show that the middle part (the higher abstraction part) of our model performs better when positional encodings are removed, suggesting the emergence of position-independent concepts. These findings demonstrate the feasibility of tokenizer-free architectures, paving the way for more robust and flexible pipelines.

Figures

Figures reproduced from arXiv: 2507.12769 by the authors.

Figure 1
Figure 1. The architecture of Synergy 3 Experiments 3.1 Experimental Setup 3.1.1 Dataset Configurations In all experiments, we use the English Wikipedia text corpus 1 , where 70% randomly sampled rows are used for training, remaining the rest for evalua￾tion and testing. We chose this dataset because we found it best highlights the characteristics of our model. To train our model Synergy as a byte-level lan￾guage model, we to… view at source ↗
Figure 3
Figure 3. BPB curves of different positioning modes [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Visualization of router output, where green color represents the weight factor (the darker green, the greater [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: The influence of concept tokens counts to [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 1 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Valentin Hofmann, Tomasz Limisiewicz, Yulia Tsvetkov, and Noah A. Smith. 2024. https://arxiv.org/abs/2407.08818 Magnet: Improving the multilingual fairness of language models with adaptive gradient-based tokenization . Preprint, arXiv:2407.08818

    arXiv 2024

  4. [4]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. https://arxiv.org/abs/2004.05150 Longformer: The long-document transformer . Preprint, arXiv:2004.05150

    arXiv 2020

  5. [5]

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. https://arxiv.org/abs/1904.10509 Generating long sequences with sparse transformers . Preprint, arXiv:1904.10509

    arXiv 2019

  6. [6]

    Tri Dao and Albert Gu. 2024. https://arxiv.org/abs/2405.21060 Transformers are ssms: Generalized models and efficient algorithms through structured state space duality . Preprint, arXiv:2405.21060

    arXiv 2024

  7. [7]

    Albert Gu and Tri Dao. 2024. https://arxiv.org/abs/2312.00752 Mamba: Linear-time sequence modeling with selective state spaces . Preprint, arXiv:2312.00752

    arXiv 2024

  8. [8]

    Namgyu Ho, Sangmin Bae, Taehyeon Kim, Hyunjik Jo, Yireun Kim, Tal Schuster, Adam Fisch, James Thorne, and Se-Young Yun. 2024. https://arxiv.org/abs/2406.02657 Block transformer: Global-to-local language modeling for fast inference . Preprint, arXiv:2406.02657

    work page Pith review arXiv 2024

Show all 30 references
  1. [9]

    Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. https://arxiv.org/abs/2407.02490 Minference 1.0: Accelerating pre-filling for long-context llms via dynami...

    2024 arXiv

  2. [10]

    Manning, Christopher Potts, and Róbert Csordás

    Julie Kallini, Shikhar Murty, Christopher D. Manning, Christopher Potts, and Róbert Csordás. 2025. https://arxiv.org/abs/2410.20771 Mrt5: Dynamic token merging for efficient byte-level language models . Preprint, arXiv:2410.20771

    2025 arXiv

  3. [11]

    Costa-jussà, David Dale, Hady Elsahar, Kevin Heffernan, João Maria Janeiro, Tuan Tran, Christophe Ropers, Eduardo Sánchez, Robin San Roman, Alexandre Mourachko, and 2 others

    LCM-team, Loïc Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R. Costa-jussà, David Dale, Hady Elsahar, Kevin Heffernan, João Maria Janeiro, Tuan Tran, Christophe Ropers, Eduardo Sán...

    2024 arXiv

  4. [12]

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang, Yuqing Yang, and Lili Qiu. 2024. https://arxiv.org/abs/2409.10516 Retrievalattention: Accelerating long-context llm inference via ...

    2024 arXiv

  5. [13]

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, and 6 others. 2025. https://arxiv.org/abs/2502.131...

    2025 arXiv

  6. [14]

    Piotr Nawrot, Jan Chorowski, Adrian Lancucki, and Edoardo Maria Ponti. 2023. https://doi.org/10.18653/v1/2023.acl-long.353 Efficient transformers with dynamic token pooling . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: ...

    2023 doi

  7. [15]

    Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski, Łukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. 2022. https://arxiv.org/abs/2110.13711 Hierarchical transformers are more efficient language models . Preprint, arXiv:2110.13711

    2022 arXiv

  8. [16]

    Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer. 2024. https://arxiv.org/abs/2412.09871 Byte latent transformer: Patch...

    2024 arXiv

  9. [17]

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, and ...

    2023 arXiv

  10. [18]

    Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Jiaju Lin, Niklas Mu...

    2024 arXiv

  11. [19]

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. 2024. https://arxiv.org/abs/2404.02258 Mixture-of-depths: Dynamically allocating compute in transformer-based language models . Preprint, arXiv:2404.02258

    2024 arXiv

  12. [20]

    Noam Shazeer. 2020. https://arxiv.org/abs/2002.05202 Glu variants improve transformer . Preprint, arXiv:2002.05202

    2020 arXiv

  13. [21]

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. https://arxiv.org/abs/2104.09864 Roformer: Enhanced transformer with rotary position embedding . Preprint, arXiv:2104.09864

    2023 arXiv

  14. [22]

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. 2023. https://arxiv.org/abs/2307.08621 Retentive network: A successor to transformer for large language models . Preprint, arXiv:2307.08621

    2023 arXiv

  15. [23]

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. https://arxiv.org/abs/2406.10774 Quest: Query-aware sparsity for efficient long-context llm inference . Preprint, arXiv:2406.10774

    2024 arXiv

  16. [24]

    Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M. Rush. 2024. https://arxiv.org/abs/2401.13660 Mambabyte: Token-free selective state space model . Preprint, arXiv:2401.13660

    2024 arXiv

  17. [25]

    Li, Madian Khabsa, Han Fang, and Hao Ma

    Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. https://arxiv.org/abs/2006.04768 Linformer: Self-attention with linear complexity . Preprint, arXiv:2006.04768

    2020 arXiv

  18. [26]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. https://arxiv.org/abs/2309.17453 Efficient streaming language models with attention sinks . Preprint, arXiv:2309.17453

    2024 arXiv

  19. [27]

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. https://doi.org/10.1162/tacl_a_00461 Byt5: Towards a token-free future with pre-trained byte-to-byte models . Transactions of the Association for Computatio...

    2022 doi

  20. [28]

    Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. https://arxiv.org/abs/2305.07185 Megabyte: Predicting million-byte sequences with multiscale transformers . Preprint, arXiv:2305.07185

    2023 arXiv

  21. [29]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025. https://arxiv.org/abs/2502.11089 Native sparse attention: Hardware-aligned ...

    2025 arXiv

  22. [30]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. https://arxiv.org/abs/2306.14048 H _2 o: Heavy-hitter oracle for efficient generative inference of lar...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.