Pith. sign in

REVIEW 5 major objections 6 minor 27 references

You Do Not Fully Utilize Transformer's Representation Capacity

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Standard transformers collapse context into one hidden state, blurring distinctions at depth; LIMe routes attention through all earlier key-value buffers, cutting convergence cost 15.3% and lifting reasoning by 30 points.

desk verdict Useful cross-layer routing with a plausible but unproven collapse story; send to review with demands for seeds and causal tests. read the letter →

arxiv 2502.09245 v2 pith:65CNKRRN submitted 2025-02-13 cs.LG cs.CL

Gleb Gerasimov , Yaroslav Aksenov , Nikita Balagansky , Viacheslav Sinii , Daniil Gavrilov This is my paper · ORCID
classification cs.LGcs.CL
keywords representationcollapselayer-integratedmemoryinter-layerroutingkey-valuebufferstransformerarchitecturemulti-stepreasoningRényientropydepthscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard transformer decoders compress the whole context into the hidden state produced by the immediately previous layer. This paper argues that this single-stream design is a genuine flaw: in deep networks it induces representation collapse, with fine distinctions between tokens blurring until they become hard to separate, and the cost shows up as worse loss and failed multi-step reasoning. The proposed fix, Layer-Integrated Memory (LIMe), reuses the key-value buffers a transformer already keeps and learns per-head, per-layer routing weights so that each attention head can mix representations from all earlier layers, with essentially no added memory and around a percent of extra FLOPs. The paper reports that a 1B-parameter LIMe model converges 15.3% faster per FLOP, reaches 1.15% lower perplexity, gains 8.4 points on the ProsQA search task and 30 points on six-operand arithmetic, and makes a 64-layer model match a 128-layer baseline. The aim is to show that representation collapse, not depth or width alone, is the bottleneck in deep transformers, and that learned inter-layer memory removes much of it almost for free.

What carries the argument

The load-bearing object is the per-layer, per-head router tensor $R^{(\ell)} \in \mathbb{R}^{\ell \times H \times H}$, whose entry $R^{(\ell)}_{\ell',h',h}$ weights the contribution of head $h'$ at buffered layer $\ell'$ into head $h$ at layer $\ell$. It forms routed keys and values by the same weighted sum, $\tilde{K}_{\ell,h} = \sum_{\ell'=1}^{\ell}\sum_{h'=1}^{H} R^{(\ell)}_{\ell',h',h} B^{(K)}_{\ell',h'}$ and an analogous expression for values, then runs ordinary causal attention between the current layer's queries and that routed mixture. Because the buffers $B^{(K)}$ and $B^{(V)}$ are the ones vanilla transformers already maintain, the mechanism adds almost no memory, and because the router is a single matrix multiply, the added compute stays a small fraction. Initializing the diagonal slice to the identity and the remaining entries to Kaiming-uniform values lets the network start as a standard transformer and gradually open cross-layer pathways; the appendix ablation shows that replacing the learned router with a uniform average over past layers reproduces the baseline perplexity exactly, so the learned weighting is what carries the benefit.

What would settle it

Train a LLaMa baseline with an auxiliary objective that drives its value-vector Rényi entropy and token separability up to LIMe's measured levels; if its accuracy on ProsQA and six-operand arithmetic does not rise correspondingly, then the entropy/collapse channel is not the causal path behind LIMe's gains, and some other feature of the routing — extra gradient pathways, initialization, or the shared key-value weights — would be responsible.

Watch

Extended reading notes

Core claim

The central claim is that a standard decoder's habit of feeding each attention layer only the previous layer's hidden state is not a harmless design detail but the source of representation collapse: as layers deepen, the residual stream must carry every contextual distinction forward in a single vector, and under finite precision and capacity those distinctions merge, so similar tokens become indistinguishable exactly where the model needs them separated. LIMe's discovery is that the key-value buffers a transformer already allocates can serve as a persistent multi-layer memory: a small trainable router tensor per layer and head, initialized to the identity, learns to weight the buffered key-value states of all earlier layers and feeds the mixture into ordinary attention. The empirical case is that this changes measurable behavior — higher value-vector Rényi entropy, near-perfect linear separability of function words, and better separation of nearby numbers in arithmetic — and that this translates into faster convergence, lower perplexity per FLOP, large gains on ProsQA and the Arithmetic Expression Task, and depth scaling in which a 64-layer LIMe outperforms a 128-layer baseline. Analysis of the trained routers shows systematic reuse of embedding-layer features in early layers and retrieval of the first buffers in later layers, which the authors read as evidence that the inter-layer pathways, rather than extra parameters alone, are doing the work.

Load-bearing premise

The load-bearing premise is that the collapse measurements — the Rényi entropy of value vectors and the linear separability of four verb forms — capture the mechanism behind LIMe's task gains; the paper shows they move together with performance but never shows that raising entropy or separability is what lowers loss or raises accuracy.

Editorial extensions

If this is right

  • Depth scaling changes: a 64-layer LIMe model reaches lower training loss than a 128-layer LLaMa baseline, so the same compute budget buys more capability when deep layers can read earlier memory instead of carrying everything in the residual stream.
  • Multi-step reasoning improves on open-ended generation: +8.4 points on ProsQA (77.8% versus 69.4%) and about 30 points on six-operand arithmetic (71.6% versus 41.3%), where the baseline's intermediate values collapse.
  • The added cost is minimal: roughly 0.01% more parameters, 0.08% more FLOPs with grouped-query attention (1.22% without), essentially unchanged peak memory, and compatibility with efficient attention implementations such as FlashAttention.
  • The trained routers are used as designed: early layers lean on the embedding buffers, middle layers treat the immediate predecessor as an auxiliary memory bank, and final layers reach back to the first buffers, so the extra pathways are actually exercised rather than idle.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A test the paper leaves open is whether keys and values should share one router: Equation (2) couples them with a single tensor, and decoupling the two streams could reveal whether the value stream alone drives the gains or whether independent key routing helps further.
  • The routing heatmap hints that a sparse router — attending mainly to the first few buffers plus the immediate predecessor — might capture most of the benefit; this matters because the paper's limitations section concedes the vanilla router has $O(L^2)$ cost and adds GPU communication in pipeline-parallel training, both pressing against the depth scaling the method promises.
  • The entropy and separability diagnostics may be symptoms of routing rather than its causal channel; an intervention that raises a baseline's value-vector entropy without adding routing would decide whether 'mitigating collapse' explains the gains or merely accompanies them.
  • If the mechanism holds, LIMe's less-collapsed value states are a natural substrate for continuous latent-space reasoning, letting a model 'think' in earlier layers that have not yet squashed distinctions — a direction the paper flags for future work but does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Layer-Integrated Memory (LIMe), a modification of decoder-only Transformers in which each attention head at each layer learns a per-head, per-layer routing weight that mixes the key and value buffers of all previous layers (Eq. 2), instead of using only the current layer's projected K/V. The authors claim that standard Transformers suffer from representation collapse because they rely on the previous layer's hidden state, and that LIMe mitigates this collapse, leading to faster convergence per FLOP, lower perplexity, higher accuracy on ProsQA and arithmetic reasoning, and better value-space entropy/separability. Experiments include 1B-parameter language modeling, fine-tuning on ProsQA, training on arithmetic expressions, deep 32/64/128-layer training-loss comparisons, and a router ablation.

Significance. If validated, LIMe is a simple and plausible architectural fix: it reuses existing key-value buffers, adds negligible parameter overhead, and comes with clear pseudocode, a public code repository, and a broad set of experiments including GQA/full attention settings and comparisons with Hyper-Connections. The cross-layer routing idea is well motivated and the empirical gains on synthetic reasoning are potentially meaningful. However, the paper's central explanatory claim--that LIMe mitigates representation collapse--is not established by the evidence, and the reported efficiency numbers are internally inconsistent with the described O(L^2) router. The deep-network result (64-layer LIMe beats 128-layer LLaMA) rests only on training loss. These issues currently make the main claims uncertain, but they are addressable with additional analysis and clarifications.

major comments (5)
  1. [Section 5.2, Figures 2, 3, 7] The collapse-mitigation claim is not supported by the paper's own metrics. Figure 7(b) shows that LIMe hidden states are less linearly separable than LLaMA's, which by the definition of collapse cited in Section 2 (Voita et al., 2019: deeper layers lose fine-grained token distinctions) indicates more collapse, not less. The paper responds that hidden states need not carry all information because values do, but this is a re-interpretation rather than a measurement of the original phenomenon. Moreover, no experiment connects the favorable metrics (value entropy in Fig. 2(a), value separability in Fig. 2(b)) to next-token loss or synthetic-task accuracy. Because Eq. (2) uses the same learned router weights for keys and values, the reported value-space diversity could be a byproduct of linearly mixing many layers' buffers rather than the causal mechanism behind the task gains. To support the mechanistic claim, the authors should test the link directly, e.g., by intervening on the router to reduce value entropy/separability and measuring the effect on downstream accuracy, or by evaluating final-layer hidden-state separability for the specific function words used in the classifier.
  2. [Section 5.5, Figure 6] The deep-network claim that 'a 64-layer LIMe model outperforms a 128-layer LLaMA model' is load-bearing for the scaling conclusion, but it rests entirely on training-loss curves at 10B tokens, with no held-out validation loss or downstream evaluation. The section does not report the hidden dimension, KV-head configuration, training data, or total compute for these runs, and there are no multiple seeds or error bars. Please add evaluation on a held-out set, specify the exact architecture and compute budget, and report run-to-run variation; otherwise the claim that LIMe changes scaling behavior is not substantiated.
  3. [Table 8, Eq. (2), Appendix E] The reported FLOP overhead is inconsistent with the routing operation described in Eq. (2) and the pseudocode in Appendix E. In the pseudocode, the router at layer l performs a dense matmul between a [kv_h, (l+1)*kv_h] weight matrix and the full KV buffer, which costs O(l) per layer and O(L^2) overall. For the stated dimensions (L=16, d=2048, H=32, kv_h=8, dh=64) and a training sequence of 2048 tokens, this router matmul accounts for a much larger share of total FLOPs than the +0.08% (GQA) and +1.22% (full attention) reported in Table 8. This matters because the central iso-FLOP comparison (Fig. 1) and the 'negligible overhead' claim depend on how router FLOPs are counted. Please clarify whether the reported FLOPs are per-step or per-token, how the router matmul was included (or excluded), and provide wall-clock step times for the actual training configuration.
  4. [Table 7, Appendix D] Table 7 reports that the 'LIMe average' variant achieves validation perplexity 16.4611, exactly identical to the LLaMA baseline (16.4611). Since a uniform average of all buffered key-value representations necessarily changes the attention inputs relative to a standard transformer, an exactly equal perplexity is implausible and suggests a bug in the experiment or in the reporting. This makes the ablation's conclusion--that uniform pooling of past representations is insufficient--uninterpretable. Please correct the table or explain why the two rows coincide.
  5. [Tables 1, 4-6; Figures 1, 6; Section 5.3] All main empirical comparisons are based on single runs, with no confidence intervals, multiple seeds, or significance tests. The claimed margins are small (e.g., 1.15% perplexity difference in Section 5.1, 3.36% in Table 7, 8.4% on ProsQA) and the paper describes the results as 'consistently achieved'. Without variance information, the reader cannot judge whether these differences are within run-to-run noise. Please provide at least 3-5 seeds for the primary comparisons (language modeling loss, ProsQA, arithmetic, and the deep-network training curves), or state explicitly whether the training is deterministic and why.
minor comments (6)
  1. [Throughout] There are repeated typographical glitches in author names and math: 'V oita' appears in the abstract-related text and in the references, and 'R ´enyi' has a spurious accent. Please fix these.
  2. [Section 3, Eq. (2)] The notation using H for the number of heads is later reused as the second and third dimensions of the router tensor R^(l) in Eq. (2), but in the GQA experiments the router mixes only num_kv_heads (8) rather than num_heads (32), as shown in Appendix E. Define whether Eq. (2)'s sums over h' run over all heads or over KV heads, and make the pseudocode consistent with the equations.
  3. [Section 5.1] The precise definitions of '15.3% (8.9% with GQA) faster in FLOPs' and '1.15% (0.91% with GQA) lower perplexity' are not given. Please state the reference loss value and how the convergence speed is computed (e.g., FLOPs to reach a fixed loss).
  4. [Figure 3 caption] The sentence 'Unlike LLaMA, LIMe can make updates attending to the previous representations' is unclear; rephrase to say what the t-SNE panels show about hidden-state versus value-state separability.
  5. [Table 2, Appendix A] The router learning rate is set to 0.01 while the base LR is 0.001, and the identity initialization is described as important (Appendix A). No sensitivity analysis is reported for these hyperparameters; a brief ablation or a comment on stability would help.
  6. [Section 5.4, Figure 5] The analysis of the router weights uses absolute magnitudes because the sign is 'semantically ambiguous'. While reasonable, this discards the distinction between positive and negative routing, which may matter for interpretation. Please add at least one qualitative check on the signed weights or a small case study.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: LIMe's empirical gains are measured on held-out benchmarks and the collapse metrics are descriptive rather than fitted inputs.

full rationale

The paper's central empirical claims (faster convergence, lower perplexity, synthetic-task accuracy) are obtained by training LIMe with learned router weights and comparing against LLaMa/Hyper-Connections on held-out data, so no prediction is forced by a fitted input. The collapse analysis in Section 5.2 measures Rényi entropy and linear separability after training; these quantities are not used to define the loss or to select the router, and the router is trained only against next-token prediction / task loss. Eq. (2) defines routed keys and values as linear mixtures of prior buffers, which may contribute to larger value diversity, but the paper does not claim the entropy increase is derived from Eq. (2); it reports it as an empirical observation. The limitations section and appendices disclose the O(L^2) cost and pipeline-parallel communication issues, and no load-bearing appeal to a self-citation or uniqueness theorem appears: citations of prior collapse work (Voita et al., Barbero et al., Arefin et al.) are external evidence, not the authors' own unverified results. The weakest point is interpretive: the paper treats value-space separability and entropy as evidence that collapse is mitigated even though its hidden-state separability is lower, and it does not causally link the proxies to the task gains. That is a correctness or mechanism concern, not a circularity, because the measured quantities are not the inputs that produce the reported benchmark differences. Overall circularity score 1 reflects only the mild, non-load-bearing overlap between the method's design (mixing many layers) and the descriptive diversity metric.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The architecture is simple and mostly self-contained, but its motivation depends on unproven proxies for collapse and on several hand-tuned hyperparameters listed in Appendix A. No external theory or formal proof constrains the router design; the routing behavior is entirely learned from training data.

free parameters (3)
  • Router learning rate = 1e-2
    Appendix A reports that a higher router learning rate boosts performance and faster router convergence; this is a hand-tuned hyperparameter.
  • Identity initialization of current-layer router block = identity on the diagonal block
    Appendix A states that random initialization of all router weights resulted in worse performance, so identity initialization is a chosen design parameter.
  • Kaiming uniform initialization bound for router weights = sqrt(3 / ((layer_idx+1) * num_kv_heads))
    Appendix E sets this initialization heuristic to stabilize mixtures; it is an ad hoc choice rather than derived.
assumptions (4)
  • domain assumption Representation collapse, measured by Rényi entropy and linear classification accuracy of value vectors, is a valid proxy for Transformer representational capacity and is causally tied to task performance.
    Section 5.2 motivates LIMe through entropy and separability, but no proof is given that higher entropy causes lower loss or better reasoning; it could be a side effect of routing.
  • ad hoc to paper Using the same learned router weights to mix keys and values is sufficiently expressive and does not restrict useful routing patterns.
    Eq. 2 applies the same R^(l) to both K and V; no theoretical or ablation justification for this coupling is provided.
  • domain assumption Single training runs with the listed hyperparameters give a fair comparison to LLaMa and Hyper-Connections.
    Benchmark differences are often small, around one percent, so run-to-run variance could change conclusions; only one run is reported for the main models.
  • domain assumption Buffering all previous layer K/V states during training adds no extra memory because standard Transformers already keep those states.
    Asserted in Section 4 and Appendix F, but standard training does not typically maintain one global buffer of all layer K/V states; no training-time memory trace is shown.
invented entities (1)
  • LIMe router tensor R^(l), with per-layer, per-head routing weights
    purpose: Linearly combines buffered key and value states from all prior layers for each attention head at layer l (Eq. 2).
    The router is a trained model component rather than a predicted physical entity. It has no external falsifiable handle beyond in-paper benchmarks, and the optimal routing structure is neither derived nor validated on independent data outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of You Do Not Fully Utilize Transformer's Representation Capacity." pith.science (2026). https://pith.science/paper/65CNKRRN

@misc{pith2026250209245,
  author       = {Pith},
  title        = {Pith review of: You Do Not Fully Utilize Transformer's Representation Capacity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/65CNKRRN}},
  note         = {Machine review of arXiv:2502.09245}
}
read the original abstract

In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standard Transformers rely solely on the hidden state from the previous layer to represent the entire context. We show that this design choice induces representation collapse and degrades performance. To address this issue, we introduce Layer-Integrated Memory (LIMe), a lightweight extension that leverages existing key-value buffers and learns per-head, per-layer routing weights to integrate representations from all previous layers with negligible overhead. Through extensive experiments-including language modeling, synthetic reasoning benchmarks, and very deep architectures-LIMe consistently achieves faster convergence, lower perplexity per FLOP, and substantial accuracy improvements on synthetic tasks while preserving higher value-vector entropy and improved token separability. Finally, our analysis of the learned routing weights reveals systematic reuse of both local and long-distance features, demonstrating how LIMe mitigates collapse, unlocks richer representations without increasing hidden-state size, and points to promising directions for future research.

Figures

Figures reproduced from arXiv: 2502.09245 by the authors.

Figure 1
Figure 1. Training loss per FLOPs for LLaMa and LIMe. LIMe has a substantially lower loss with a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. (a) Matrix entropy of values on the FineWeb Edu subset by layer. LIMe has more diverse [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. (a) t-SNE of similar tokens’ hidden states among layers. Although hidden states are not [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: (a) LIMe exhibits consistently higher entropy of value vectors across layers, particularly in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Mean retrieval weight for each buffered representation across subsequent layers. Larger [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Training losses for deep architectures. The LIMe models consistently outperform their [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: (a) Matrix entropy of the hidden states across layers on the FineWeb Edu subset. We do [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: t-SNE of close numbers’ values representations of models trained on Arithmetic Expressions [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 10
Figure 10. Figure 10: Training loss per tokens trained on for LLaMa and LIMe with GQA. It shows that LIMe is more data efficient. See Section 5.1 for more details. 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 FLOPs 2.40 2.45 2.50 2.55 2.60 2.65 2.70 Loss 1e20 Training Loss LLaMa LIMe (a) 10 15 20 25 30…
Figure 11
Figure 11. Figure 11: Training loss for LLaMa and LIMe without GQA. (a) shows that LIMe has a substan￾tially lower loss with a similar amount of FLOPs. (b) shows that LIMe is more data efficient. See Section 5.1 for more details. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: LIMe routing scheme. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 7 canonical work pages

  1. [1]

    Transformer math 101

    Quentin Anthony, Stella Biderman, and Hailey Schoelkopf. Transformer math 101. blog.eleuther.ai/, 2023. URL https://blog.eleuther.ai/transformer-math/

    work page 2023

  2. [2]

    Seq-vcr: Preventing collapse in intermediate transformer representations for enhanced reasoning

    Md Rifat Arefin, Gopeshh Subbaraj, Nicolas Gontier, Yann LeCun, Irina Rish, Ravid Shwartz-Ziv, and Christopher Pal. Seq-vcr: Preventing collapse in intermediate transformer representations for enhanced reasoning. arXiv preprint arXiv: 2411.02344, 2024

    arXiv 2024

  3. [3]

    Training deeper neural machine translation models with transparent attention

    Ankur Bapna, Mia Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. Training deeper neural machine translation models with transparent attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3028--3033, 2018. doi:10.18653/v1/D18-1338. URL https://arxiv.org/abs/1808.07561. arXiv:1808.07561

    arXiv 2018

  4. [4]

    Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João G. M. Araújo, Alex Vitvitskyi, Razvan Pascanu, and Petar Veličković. Transformers need glasses! information over-squashing in language tasks. arXiv preprint arXiv: 2406.04267, 2024

    arXiv 2024

  5. [5]

    Flash A ttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

  6. [6]

    Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian L...

    arXiv 2024

  7. [7]

    Cross-layer retrospective retrieving via layer attention

    Yanwen Fang, Yuxi Cai, Jintai Chen, Jingyu Zhao, Guangjian Tian, and Guodong Li. Cross-layer retrospective retrieving via layer attention. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=pvgEL1yS3Ql

    work page 2023

  8. [8]

    Towards revealing the mystery behind chain of thought: A theoretical perspective, 2023

    Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: A theoretical perspective, 2023. URL https://arxiv.org/abs/2305.15408

    arXiv 2023

Show all 27 references
  1. [9]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  2. [10]

    Michael Hahn and Mark Rofin. Why are sensitive functions hard for transformers? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14973--15008, Bangk...

    2024 doi

  3. [11]

    Training large language models to reason in a continuous latent space

    Shibo Hao, Sainbayar Sukhbaatar, Dijia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  4. [12]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv: 1512.03385, 2015

    2015 arXiv

  5. [13]

    Weinberger

    Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, 2018. URL https://arxiv.org/abs/1608.06993

    2018 arXiv

  6. [14]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    arXiv

  7. [15]

    The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557

    2024 arXiv

  8. [16]

    Qwen2.5 technical report

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2024 arXiv

  9. [17]

    Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024. URL https://arxiv.org/abs/2404.02258

    2024 arXiv

  10. [18]

    Brown, Adam Santoro, Aditya Gupta, Adri \` a Garriga - Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adri \` a Garriga - Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocure...

    2023

  11. [19]

    Highway networks

    Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv: 1505.00387, 2015

    2015 arXiv

  12. [20]

    BERT rediscovers the classical NLP pipeline

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Llu \'i s M \`a rquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593--4601, Florence, Ita...

    2019 doi

  13. [21]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NEURIPS, 2017

    2017

  14. [22]

    The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives

    Elena Voita, Rico Sennrich, and Ivan Titov. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference o...

    2019

  15. [23]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Ws, 2018

    2018

  16. [24]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Neural Information Processing Systems, 2019

    2019

  17. [25]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 0 24824--24837, 2022

    2022

  18. [26]

    Hiformer: Sequence modeling networks with hierarchical attention mechanisms

    Xixin Wu, Hui Lu, Kun Li, Zhiyong Wu, Xunying Liu, and Helen Meng. Hiformer: Sequence modeling networks with hierarchical attention mechanisms. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31: 0 3993--4003, 2023. doi:10.1109/TASLP.2023.3313428. URL https://...

    2023

  19. [27]

    Hyper-connections

    Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, and Xun Zhou. Hyper-connections. arXiv preprint arXiv: 2409.19606, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.