Pith. sign in

REVIEW 3 major objections 5 minor 44 references

DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A learned router can send about one token in ten through full self-attention while still matching dense Transformer accuracy.

desk verdict A genuinely different token-routing design with solid 2K-context results, but the long-context FLOPs advantage is computed, not measured, and the paper should be revised to verify routing rates under YaRN. read the letter →

arxiv 2509.00925 v1 pith:25CCZVQK submitted 2025-08-31 cs.LG cs.CL

Aman Sharma , Saeed Najafi , Parsa Farinneya , Benyamin Jamialahmadi , Marzieh S. Tahaei , Yuhe Fan , Mehdi Rezagholizadeh , Boxing Chen
show 1 more author
Aref Jafari
This is my paper
classification cs.LGcs.CL
keywords dynamictokenroutingsparseattentiontransformerefficiencymixture-of-depthKV-cachememorylong-contextextrapolationtoken-choicelanguagemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DTRNet is an attempt to break the quadratic cost of Transformer attention without making tokens skip their updates. The paper's central claim is that a small learned router at each layer can send roughly 10% of tokens to full multi-head attention and route the remaining 90% through a cheap, token-local projection path, while keeping language-model accuracy at the level of a dense Transformer. The key to holding accuracy is that bypassed tokens still pass through shared value/output projections and then the MLP, so every token is updated at every layer. Against routing baselines that skip whole layers, DTRNet reports higher average accuracy at the same FLOP budget, lower KV-cache memory, and a growing FLOPs advantage as sequence length increases. The authors take this as evidence that attention and token updates can be decoupled, with most tokens not needing cross-token mixing most of the time.

What carries the argument

The DTRNet layer, interleaved with full Transformer layers. Its router is a two-layer SiLU MLP outputting softmax scores over two paths; a hard argmax picks the path at inference. The attention path is standard causal multi-head self-attention. The bypass path is the core mechanism: x_i W_V W_O, a token-local update using the same value and output projections as attention, followed by the shared MLP. Because the bypass uses the same W_V and W_O, skipped-token representations stay in the same space as attended-token representations. In matrix form, routing induces a dynamic sparse attention mask M = delta delta^T, which restricts attention to the routed tokens and turns the quadratic part of

What would settle it

Take a trained 360M DTRNet and retrain a variant where the router is forced to pick a random 10% of tokens per layer, keeping the same training budget and architecture. If the random-10% model matches DTRNet's 44.36 average accuracy instead of dropping, then the learned router is not what preserves performance; if it drops, the router's token selection is load-bearing.

Watch

Extended reading notes

Core claim

The paper's core discovery claim is that a Transformer can keep its accuracy while making self-attention a sparse, token-selective operation rather than a uniform one. In each DTRNet layer, a two-layer router scores each token for one of two paths: full multi-head attention, or a linear path x W_V W_O followed by the shared MLP. At inference the router makes a hard argmax choice; during training it uses the soft scores to weight both paths so gradients reach the router. With an L1 penalty on attention load designed to balance routing across layers, a trained 360M model routes only about 10% of tokens to attention per DTRNet block and averages 44.36 accuracy across eight benchmarks, versus 44

Load-bearing premise

The design assumes most tokens in a layer can get what they need from a token-local projection followed by the shared MLP, without cross-token mixing, and that the router learned under soft weighting at training length still picks correctly under hard routing at much longer sequences.

Editorial extensions

If this is right

  • At matched FLOPs (about 0.84-0.85 of a dense Transformer), DTRNet reaches or slightly beats the dense baseline's average accuracy on eight zero-shot tasks, so compute can be cut without a corresponding accuracy cut.
  • Because only about 10% of tokens are attended per DTRNet layer, the quadratic share of FLOPs shrinks as sequence length grows; the paper reports a ratio of 0.785 at 20K tokens.
  • KV-cache memory drops because unselected tokens never create key/value entries; the reported memory advantage grows with input length, unlike the masking-based eviction strategy used by D-LLM.
  • Uniform routing load across layers avoids D-LLM's layer starvation and MoD's train/inference router mismatch, making the efficiency stable at scale.
  • The design principle of retaining the MLP and giving bypassed tokens an explicit W_V W_O update is what preserves performance; ablations show removing the projections drops average accuracy from 44.36 to 41.99.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Implicit consequence: if most tokens genuinely need only a token-local projection plus MLP, the same idea could stack with linear-attention or sparse-window mechanisms on the attended path, pushing the quadratic share even lower.
  • The layerwise cosine-similarity redundancy the authors measure on a dense model suggests an adaptive budget policy: allocate more attention to boundary layers and less to the high-similarity middle layers, rather than a fixed interleaving pattern.
  • A testable extension is to make the routing threshold per-layer and per-input-difficulty at inference, since the trained router already outputs continuous scores that could be thresholded more aggressively under latency constraints.
  • If the 10% selection is truly driven by token content, DTRNet-style routers might transfer across domains or tasks after fine-tuning, though the paper does not test transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DTRNet, a Transformer variant in which each layer contains a learned two-layer router that sends each token either to the full self-attention path or to a linear projection path x W_V W_O followed by the shared MLP. Training uses soft weighting of both paths plus a load-balancing penalty; inference uses hard argmax routing. The architecture interleaves standard Transformer layers with DTRNet layers, with full attention in the first and last layers. Experiments at 360M and 1.3B scales on language modeling and downstream tasks report that DTRNet matches dense Transformer performance at about 0.84–0.85 FLOPs ratio, outperforms MoD and D-LLM under matched FLOPs, lowers KV-cache memory, and has a theoretical FLOPs ratio that decreases with sequence length (0.785 at 20K). The paper also includes ablations on routing strategy, architecture variants, and the necessity of the W_V W_O bypass path.

Significance. If the central claim holds, DTRNet offers a simple and practical way to reduce the quadratic cost of attention while retaining most of the model's quality: a token-local linear update plus the full MLP is enough for ~90% of tokens at each layer. The paper's strengths are concrete: code is released, the main comparisons are done under a fixed FLOPs budget, multiple ablations (A1–A5) probe design choices, and the long-context evaluation goes up to 20K tokens. The 10%-routing result is non-obvious and, if verified, would be a useful contribution to the growing literature on dynamic token routing. However, the efficiency and memory claims rest on an internal inconsistency in the attention definition and on an unverified assumption about routing behavior at extrapolated lengths, so the claims are not yet established in the submitted form.

major comments (3)
  1. [Methodology, Computation Paths, Eq. (4) and Eq. (6)] There is a direct contradiction between the two formal definitions of the attention path. Eq. (4) defines Attn(x_i) using K = X W^K and V = X W^V, where X is the full token sequence, so an attention-routed token attends to all tokens (subject to the causal mask). Eq. (6) defines the effective mask as M = \tilde{g} \tilde{g}^T, which restricts attention to interactions only among tokens routed to attention. These are different operations: the former requires computing and caching K,V for every token, while the latter only needs K,V for the ~10% routed tokens. The KV-cache savings in Figure 6 and the FLOPs ratios in Table 1 and Figure 4 presuppose the Eq. (6) semantics, since otherwise no KV-cache is saved. The paper must specify which semantics is actually implemented, align Eqs. (4) and (6), and re-derive the FLOPs/memory accounting accordingly. This is load-bearing for the paper's main
  2. [Results and Analysis, Figure 4] The statement 'At a 20K sequence length, DTRNet operates at a FLOPs ratio of 0.785' is based on a theoretical calculation that fixes k=0.10 for DTRNet throughout. The 10% figure is measured at the standard evaluation lengths (Figure 5), not at 8K–20K under YaRN(10.0). The router is a two-layer MLP trained on 2K sequences with soft path weights, so its hard-argmax decisions at extrapolated positions could shift. No measurement of the actual hard-routing fraction at 4K, 8K, 12K, or 20K is reported. If the fraction grows with length, 0.785 is a lower bound rather than an operating point, and the claimed long-context advantage over MoD/D-LLM may shrink. Please report measured routing fractions by layer at the extrapolated lengths, or substantially soften the claim.
  3. [Methodology, Training and Regularization; Results, Figure 5] The paper criticizes MoD for the inconsistency between its auxiliary router and the true routing decisions, but DTRNet itself uses soft scores to weight both paths during training (Eq. 3 and Eq. 5, with the soft score g) and hard argmax at inference (Eq. 2). This is a soft-to-hard mismatch of the same kind, and the paper does not evaluate it. The sensitivity analyses in Appendix A3 and A5 show that the composition of the two paths strongly affects accuracy (going from 44.36 to 42.50 without attention, and to 41.99 without W_V W_O), so the router's hard decisions are not a trivial detail. Please report at minimum the agreement rate between soft argmax and hard decisions during validation, or an ablation using straight-through/soft routing at inference. Without this, the claim that the trained model routes to 10% attention while preserving accuracy is not fully established for the deployed
minor comments (5)
  1. [Related Work] Mixture-of-Depths is attributed to '(Zhou et al. 2022)', but the reference list contains Zhou et al. as 'Mixture-of-Experts with Expert Choice Routing'. The MoD paper is Raposo et al. 2024. Please correct the citation.
  2. [Methodology, Eq. (7)] The loss in Eq. (7) is hard to parse: α_l is defined with δ (hard decisions) while the norm uses G (soft scores), and the text says 'penalize the product of the load and the aggregate attention scores'. Please rewrite the equation with explicit notation and explain the role of α_l. Also, f_att^l is a count but is later used in a normalized fraction; clarify the index ranges.
  3. [Experiments, DTRNet configurations] The notation for DTRNet-BiLayer and DTRNet-TriLayer is ambiguous. 'T-D-T-D-T-D' and 'T-D-D-T-D-D' do not make it clear how the 32-layer 360M and 24-layer 1.3B models are composed, particularly with first/last layers always being full Transformer layers. Please give the exact layer-by-layer pattern for the actual model sizes.
  4. [Results and Analysis, Figure 3 and Table 1] All results appear to come from a single training run with no error bars or multiple seeds. Given the small differences between DTRNet and the dense baseline (e.g., 44.36 vs 44.23 at 360M), a statement about run-to-run variance would help the reader judge the stability of the comparison.
  5. [Figure 4] The y-axis label '1e 1' is confusing; the plotted FLOPs ratios are around 0.8, not 10. Please use a normal 0–1 axis or clearly explain the scaling.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DTRNet's main claims rest on external benchmark comparisons and measured routing behavior; the long-context FLOPs curve is explicitly theoretical and arithmetically derived from a stated k, not a hidden fit.

full rationale

The paper's central assertions—accuracy versus SmolLM/MoD/D-LLM at matched FLOPs, lower perplexity on long-context benchmarks, and KV-cache memory savings—are empirical comparisons performed on external datasets (WikiText, LAMBADA, ARC, BoolQ, HellaSwag, PIQA, etc.) with all models trained from scratch under the same setup. No claim is obtained by fitting a parameter to a subset of data and then re-presenting a closely related quantity as a prediction. The ~10% attention routing rate is a measured outcome of the explicit penalty in Eq. (7), reported in Figure 5 as an observed average, not a fitted constant disguised as an emergent result. The self-citations (Balcony, SortedNet, SortedLLaMA) appear only in related-work discussion and are not load-bearing for any derivation. The only point that might resemble circularity is Figure 4, where the 20K FLOPs ratio of 0.785 is computed using k=0.10 for DTRNet. However, the figure is explicitly captioned 'Theoretical FLOPs comparison' and states the assumed k values (0.10, 0.70, 0.84), making the calculation a transparent arithmetic consequence of the stated routing fractions rather than a hidden reduction. Whether the hard-argmax router actually maintains 10% attention routing at 20K under YaRN extrapolation is a legitimate correctness/extrapolation risk, but it is not a circularity because the paper does not conceal the assumption as a measurement. The derivation chain is self-contained with respect to its empirical claims.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The ledger shows DTRNet's cost is concentrated in empirical hyperparameters: lambda controls the headline routing ratio, and the MoD/D-LLM baseline scores depend on the FLOPs-match values chosen by the authors. The most fragile premise is the ad hoc soft-to-hard routing transfer, which is neither formalized nor ablated. No invented entities: the router and linear path are built from standard Transformer components.

free parameters (4)
  • Routing penalty strength lambda = 8e-4 (360M), 6e-4 (1.3B)
    Controls the fraction of tokens routed to attention; the headline '~10% of tokens' is a consequence of this hand-chosen value, not a measured invariant of the architecture. No sensitivity analysis is reported.
  • MoD top-k routing ratio = 0.7
    Baseline hyperparameter chosen by the authors to match DTRNet's FLOPs. The baseline's score depends on it (k=0.125 gives 42.12 average vs 43.14 at k=0.7, Appendix A4), so the comparison rests on this choice.
  • D-LLM acceleration rate Omega = 0.85
    Baseline hyperparameter chosen to match FLOPs; the original paper's 0.55 configuration scores lower (42.31). Comparison fairness depends on the chosen value.
  • YaRN extrapolation factor = 10.0
    Applied to 1.3B models to reach 20K tokens for length-extrapolation benchmarks; matched across models, but the factor and evaluation lengths are chosen by hand.
assumptions (5)
  • domain assumption High adjacent-layer cosine similarity of token embeddings in a dense Transformer (Figure 1, S_{i,i+1} ~ 0.98) implies redundant token updates that a learned router can skip without loss.
    Section 'Motivation: Token Redundancy'. The entire efficiency hypothesis rests on this correlation being exploitable; the paper presents it as motivation, not as a causal guarantee.
  • ad hoc to paper Soft-weighted training of both paths (Eq. 3 and 5 with g as weights) transfers validly to hard argmax routing at inference (Eq. 2).
    Methodology, 'Computation Paths'. No Gumbel-Softmax or straight-through estimator is used, and the paper does not evaluate whether soft/hard mismatch harms routing quality, despite criticizing MoD for a similar mismatch.
  • domain assumption Zero-shot evaluation on ARC, BoolQ, HellaSwag, PIQA, Tiny MMLU, Winogrande, LAMBADA, and WikiText measures relative architecture quality at these model scales.
    Evaluation Benchmarks section. Standard practice in the subfield, but the comparative claims (Table 1) hinge on it.
  • domain assumption FineWeb-Edu pretraining at 15B/100B tokens is a sufficient and neutral testbed for all architectures compared.
    Training Setup section. Shared across models so it is neutral, but it bounds the generality of the conclusions.
  • standard math Standard transformer math: softmax attention, causal masking, FlashAttention varlen correctness, AdamW behavior.
    Equations 1 through 7 and the training protocol rely on these standard, unstated background facts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers." pith.science (2026). https://pith.science/paper/25CCZVQK

@misc{pith2026250900925,
  author       = {Pith},
  title        = {Pith review of: DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25CCZVQK}},
  note         = {Machine review of arXiv:2509.00925}
}
read the original abstract

Transformers achieve state-of-the-art results across many tasks, but their uniform application of quadratic self-attention to every token at every layer makes them computationally expensive. We introduce DTRNet (Dynamic Token Routing Network), an improved Transformer architecture that allows tokens to dynamically skip the quadratic cost of cross-token mixing while still receiving lightweight linear updates. By preserving the MLP module and reducing the attention cost for most tokens to linear, DTRNet ensures that every token is explicitly updated while significantly lowering overall computation. This design offers an efficient and effective alternative to standard dense attention. Once trained, DTRNet blocks routes only ~10% of tokens through attention at each layer while maintaining performance comparable to a full Transformer. It consistently outperforms routing-based layer skipping methods such as MoD and D-LLM in both accuracy and memory at matched FLOPs, while routing fewer tokens to full attention. Its efficiency gains, scales with sequence length, offering significant reduction in FLOPs for long-context inputs. By decoupling token updates from attention mixing, DTRNet substantially reduces the quadratic share of computation, providing a simple, efficient, and scalable alternative to Transformers.

Figures

Figures reproduced from arXiv: 2509.00925 by the authors.

Figure 1
Figure 1. Average Layerwise cosine similarity of token em [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Perplexity across increasing sequence lengths for LongLM benchmarks on 6 different tasks. DTRNet maintains lower [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Theoretical FLOPs comparison as sequence length [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Average Percentage of tokens going to attention per layer for D-LLM, MoD and DTRNet. We observe much less [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: KV cache memory for different sequence lengths [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 18 canonical work pages

  1. [1]

    Anthropic. 2023. Claude: A Next-Generation AI Assistant by Anthropic. https://www.anthropic.com/index/introducing-claude. Accessed: 2025-07-28

    work page 2023

  2. [2]

    Arora, S.; Eyuboglu, S.; Zhang, M.; Timalsina, A.; Alberti, S.; Zinsley, D.; Zou, J.; Rudra, A.; and R \'e , C. 2024. Simple linear attention language models balance the recall-throughput tradeoff. arXiv preprint arXiv:2402.18668

    arXiv 2024

  3. [3]

    E.; and Cohan, A

    Beltagy, I.; Peters, M. E.; and Cohan, A. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

    arXiv 2020

  4. [4]

    Bisk, Y.; Zellers, R.; Gao, J.; and Choi, Y. 2020. PIQA: Reasoning about physical commonsense in natural language. arXiv preprint arXiv:1911.11641

    arXiv 2020

  5. [5]

    Cai, R.; Muralidharan, S.; Heinrich, G.; Yin, H.; Wang, Z.; Kautz, J.; and Molchanov, P. 2024. Flextron: Many-in-one flexible large language model. arXiv preprint arXiv:2406.10260

    arXiv 2024

  6. [6]

    Cao, Z.; Yang, Y.; and Zhao, H. 2024. Head-wise shareable attention for large language models. arXiv preprint arXiv:2402.11819

    work page Pith review arXiv 2024

  7. [7]

    Child, R.; Gray, S.; Radford, A.; and Sutskever, I. 2019. Generating Long Sequences with Sparse Transformers. arXiv preprint arXiv:1904.10509

    arXiv 2019

  8. [8]

    Clark, C.; Lee, K.; Chang, M.-W.; Kwiatkowski, T.; Collins, M.; and Toutanova, K. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044

    arXiv 2019

Show all 44 references
  1. [9]

    Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  2. [10]

    Dao, T. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv preprint arXiv:2307.08691

    2023 arXiv

  3. [11]

    Dehghani, M.; Gouws, S.; Vinyals, O.; Uszkoreit, J.; and Kaiser, L. 2019. Universal Transformers. In International Conference on Learning Representations (ICLR)

    2019

  4. [12]

    Dong, Y.; Noci, L.; Khodak, M.; and Li, M. 2025. Attention Retrieves, MLP Memorizes: Disentangling Trainable Components in the Transformer. arXiv preprint arXiv:2506.01115

    2025 arXiv

  5. [13]

    Elhoushi, M.; Shrivastava, A.; Liskovich, D.; Hosmer, B.; Wasti, B.; Lai, L.; Mahmoud, A.; Acun, B.; Agarwal, S.; Roman, A.; et al. 2024. LayerSkip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710

    2024 arXiv

  6. [14]

    Fedus, W.; Zoph, B.; and Shazeer, N. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1): 5232--5270

    2022

  7. [15]

    Gemini Team, G. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv: 2403.05530

    2024 arXiv

  8. [16]

    Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  9. [17]

    E.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; and Chen, W

    Hu, J. E.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations

    2021

  10. [18]

    Jamialahmadi, B.; Kavehzadeh, P.; Rezagholizadeh, M.; Farinneya, P.; Rajabzadeh, H.; Jafari, A.; Chen, B.; and Tahaei, M. S. 2025. Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models. arXiv preprint arXiv:2503.05005

    2025 arXiv

  11. [19]

    Jiang, Y.; Wang, H.; Xie, L.; Zhao, H.; Zhang, C.; Qian, H.; and Lui, J. C. 2024. D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models. In Globerson, A.; Mackey, L.; Belgrave, D.; Fan, A.; Paquet, U.; Tomczak, J.; and Zhang, C., eds., Advanc...

    2024

  12. [20]

    Kavehzadeh, P.; Valipour, M.; Tahaei, M.; Ghodsi, A.; Chen, B.; and Rezagholizadeh, M. 2024. Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference. In Findings of the Association for Computational Linguistics: EACL 2024, 2129--2145

    2024

  13. [21]

    Kitaev, N.; Kaiser, .; and Levskaya, A. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451

    2020 arXiv

  14. [22]

    M.; Melis, G.; and Grefenstette, E

    Ko c isk \'y , T.; Schwarz, J.; Blunsom, P.; Dyer, C.; Hermann, K. M.; Melis, G.; and Grefenstette, E. 2018. The NarrativeQA Reading Comprehension Challenge. Transactions of the Association for Computational Linguistics, 6: 317--328

    2018

  15. [23]

    S.; McCann, B.; Xiong, C.; and Socher, R

    Kry \'s ci \'n ski, W.; Keskar, N. S.; McCann, B.; Xiong, C.; and Socher, R. 2021. BookSum: A Collection of Datasets for Long-form Narrative Summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL)

    2021

  16. [24]

    Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations (ICLR). ArXiv:1711.05101

    2019 arXiv

  17. [25]

    Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2016. Pointer Sentinel Mixture Models. arXiv preprint arXiv:1609.07843

    2016 arXiv

  18. [26]

    OpenAI. 2023. GPT-4 Technical Report. PREPRINT

    2023

  19. [27]

    N.; Bernardi, R.; Pezzelle, S.; Baroni, M.; Boleda, G.; and Fern \'a ndez, R

    Paperno, D.; Kruszewski, G.; Lazaridou, A.; Pham, Q. N.; Bernardi, R.; Pezzelle, S.; Baroni, M.; Boleda, G.; and Fern \'a ndez, R. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031

    2016 arXiv

  20. [28]

    Penedo, G.; Kydlíček, H.; Ben Allal, L.; Lozhkov, A.; Mitchell, M.; Raffel, C.; Von Werra, L.; and Wolf, T. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv preprint arXiv:2406.17557

    2024 arXiv

  21. [29]

    Peng, B.; Quesnelle, J.; Fan, H.; and Shippole, E. 2023. YaRN: Efficient Context Extension for Large Language Models. arXiv preprint arXiv:2309.00071

    2023 arXiv

  22. [30]

    W.; Potapenko, A.; Jayakumar, S

    Rae, J. W.; Potapenko, A.; Jayakumar, S. M.; and Lillicrap, T. P. 2020. Compressive Transformers for Long-Range Sequence Modelling. In International Conference on Learning Representations (ICLR)

    2020

  23. [31]

    C.; and Santoro, A

    Raposo, D.; Ritter, S.; Richards, B.; Lillicrap, T.; Humphreys, P. C.; and Santoro, A. 2024. Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258

    2024 arXiv

  24. [32]

    L.; Bhagavatula, C.; and Choi, Y

    Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2021. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641

    2021 arXiv

  25. [33]

    M.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q

    Shazeer, N. M.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q. V.; Hinton, G. E.; and Dean, J. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. International Conference on Learning Representations

    2017

  26. [34]

    Shin, S.; Oh, J.; and Oh, D. 2025. OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference. arXiv:2507.03865

    2025 arXiv

  27. [35]

    Touvron, H.; Martin, L.; Stone, K.; et al. 2023. LLaMA 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  28. [36]

    S.; Chen, B.; and Ghodsi, A

    Valipour, M.; Rezagholizadeh, M.; Rajabzadeh, H.; Tahaei, M. S.; Chen, B.; and Ghodsi, A. 2023. Sortednet, a place for every network and every network in its place: Towards a generalized solution for training many-in-one neural networks. CoRR

    2023

  29. [37]

    N.; Kaiser, L.; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention is All You Need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30

    2017

  30. [38]

    Wu, H.; and Tu, K. 2024. Layer-condensed kv cache for efficient inference of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 11175--11188

    2024

  31. [39]

    W.; Lin, Z.; et al

    Xu, C.; Ge, Y.; Zhang, S.; Yao, Z.; Yang, F. W.; Lin, Z.; et al. 2024. D-LLM: Dynamic Sparse Token Mixture of Experts for Efficient LLM Inference. arXiv preprint arXiv:2402.11961

    2024 arXiv

  32. [40]

    Yu, Z.; and Ananiadou, S. 2023. Neuron-level knowledge attribution in large language models. arXiv preprint arXiv:2312.12141

    2023 arXiv

  33. [41]

    Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  34. [42]

    M.; Le, Q

    Zhou, Y.; Lei, T.; Liu, H.; Du, N.; Huang, Y.; Zhao, V.; Dai, A. M.; Le, Q. V.; Laudon, J.; et al. 2022. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35: 7103--7114

    2022

  35. [43]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.