Pith. sign in

REVIEW 4 major objections 5 minor 24 references

Inside a chess AI, correct moves often appear in intermediate layers, then get overridden in the final output.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

The paper demonstrates non-monotonic move-policy dynamics in a chess transformer, but its abstract claims a causal safety-prior override result that never appears in the body.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The body is a solid descriptive logit-lens study of Leela Chess Zero, but the front abstract claims a steering experiment and a 61.7% recovery rate that appear nowhere in the text. the 4 major comments →

arxiv 2508.21380 v3 pith:AUHIDRKV submitted 2025-08-29 cs.LG cs.AI

The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network

Elias Sandmann , Sebastian Lapuschkin , Wojciech Samek This is my paper
classification cs.LG cs.AI
keywords logit lensPost-LN transformerLeela Chess Zeroiterative inferencemechanistic interpretabilitypolicy networkchess puzzleslearned safety prior
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a neural network that has learned a correct algorithm will actually use it. The authors extend the logit lens to the 15-layer Post-LN transformer policy network of Leela Chess Zero, a strong chess engine, and find that on tactical puzzles the correct move—even an immediate checkmate—often becomes the top choice in a middle layer, then is discarded by the final policy. They argue the algorithm itself is not broken: the future moves of the correct continuation are still represented and causally relevant in later layers, matching prior evidence of learned look-ahead. Instead, late layers increasingly favor safe, material-preserving play, and steering the network against those preferences recovers a majority (61.7%) of the forgotten puzzles. The headline conclusion is that algorithmic structure does not guarantee algorithmic behavior—a model can internally solve a problem and still output the wrong answer.

Core claim

On the paper's own terms, the discovery is that the Leela Chess Zero policy network frequently computes the correct puzzle solution early in its forward pass and then throws it away. The claimed mechanism is not a failure of look-ahead: replicating prior analyses shows the winning continuation's future moves remain represented, causally important, and linearly decodable in later layers. Rather, the late layers shift the policy toward safe play, and this learned prior overrides the already-computed winning move. The paper presents 'forgotten puzzles' as evidence that a model can internally solve a problem and still output the wrong answer, since the final policy's argmax differs from the move

What carries the argument

The key technical instrument is a Post-LN extension of the logit lens: a zero-ablation forward pass that sets all attention and feed-forward sublayer outputs beyond layer k to zero, while preserving DeepNorm scaling, layer-normalization parameters, and recomputed normalization statistics, then projects the truncated representation through the policy head. This yields a layer-by-layer policy distribution, letting the authors ask what the network would output if it stopped at each depth. The lens defines the 'forgotten puzzles' phenomenon: a puzzle is forgotten when an intermediate layer's argmax is the correct move but the final output picks something else. The paper's causal evidence comes f

Load-bearing premise

The load-bearing premise is that the Post-LN zero-ablation logit lens faithfully reveals what the network computes at layer k; if this projection is not faithful, the 'forgotten puzzles' are measurement artifacts.

What would settle it

Take the puzzles classified as 'forgotten' and test whether the intermediate-layer argmax actually influences the final output through a causal intervention that does not rely on the lens, for example patching the layer-k representation into the final layers via activation replacement. If no such influence exists, the phenomenon is an artifact of the projection. Alternatively, a genuinely truncated network that never computes later layers should solve the same puzzles at the rate the lens predicts; if it does not, the lens is unfaithful.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The final policy can fail to solve puzzles that an intermediate layer had already solved correctly, so the network's final failure is not evidence that the computation never happened.
  • The learned look-ahead mechanism identified in prior work is still active and correct in the forgotten-puzzle positions; the override happens downstream of the look-ahead computation.
  • Move rankings remain poorly correlated with the final output until very late in the network, so intermediate layers are not simply converging on the final answer but are actively reevaluating options.
  • Steering the policy away from late-layer safe-play preferences recovers 61.7% of forgotten puzzles, indicating the override is caused by a learned safety prior rather than by a corrupted algorithm.
  • The overall monotonic increase in playing strength across layers coexists with non-monotonic policy trajectories, separating capability growth from how the model decides.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the logit lens is faithful, the same 'forgotten solution' pattern may appear in other game-playing and planning networks with Post-LN or DeepNorm architectures, and the safety-prior override might generalize beyond chess.
  • The 61.7% recovery rate, if confirmed, suggests a practical intervention: steering against a late-layer bias could improve a model's worst-case performance without retraining—but the remaining 38.3% implies other mechanisms also contribute.
  • The paper's stance implies that behavioral success (higher Elo, puzzle-solving) is an unreliable proxy for whether a network has computed the 'right' answer; interpretability metrics based on intermediate decoding may be a better predictor of latent capability than final outputs.
  • A testable extension is to apply the same Post-LN logit lens to other Post-LN transformer tasks, such as code or math, to see whether 'forgotten correct answers' are a general iterative-inference phenomenon or specific to adversarially tactical settings like chess puzzles.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies the 15-layer Post-LN transformer policy network of Leela Chess Zero. It extends the logit lens to this architecture by zero-ablating sublayer outputs beyond a given layer while preserving DeepNorm and LayerNorm scaling, and then uses the resulting layer-wise policies to measure puzzle-solving accuracy, tournament playing strength, Lichess ratings, and distributional metrics such as Jensen-Shannon divergence, entropy, and Kendall's tau. The body reports monotonic improvement in playing strength and puzzle-solving ability across layers, alongside non-monotonic policy trajectories: correct puzzle solutions sometimes appear in intermediate layers but are absent from the final policy ("forgotten puzzles"). The front-matter abstract and title go further, claiming that late layers prioritize safe play over aggression, that steering against this preference recovers 61.7% of forgotten puzzles, and that this provides causal evidence that learned safety priors override algorithmically computed look-ahead. The body, however, contains only descriptive observations and no steering experiment or operationalization of a safety prior.

Significance. If the descriptive findings are robust, the paper makes a useful contribution to mechanistic interpretability: it provides the first Post-LN logit-lens extension for a strong chess transformer, documents non-smooth policy trajectories that contrast with language-model behavior, and connects intermediate-layer policies to puzzle-solving and playing-strength metrics. The authors ship code and make extensive empirical efforts, including a 27,200-game internal tournament, real Lichess deployment, and detailed case studies. The paper also builds on and engages with the independent look-ahead findings of Jenner et al. (2024). However, the advertised causal claim about safety priors overriding look-ahead is not supported by the body, and the intermediate-layer observations are only as strong as the zero-ablation projection used to define them. The descriptive core is promising, but the manuscript currently overstates its own evidence.

major comments (4)
  1. [Title and front-matter Abstract] The title and abstract assert a causal result: late-layer safety priors override computed look-ahead, with a steering intervention recovering 61.7% of forgotten puzzles. I could not find this steering experiment, the 61.7% recovery rate, or any operationalization of a 'safety prior' anywhere in Sections 3, 4, or Appendices A-F. The body's own abstract is weaker and only claims that 'correct puzzle solutions that are discovered early but subsequently discarded.' This is load-bearing: the advertised central claim is not evidenced by the manuscript. Either the steering experiment must be added and the safety prior precisely defined, or the title and abstract must be revised to state the descriptive finding.
  2. [Appendix A.2/A.4 and §3.2] The existence of 'forgotten puzzles' is defined entirely by the Post-LN zero-ablation logit lens: sublayer outputs beyond layer k are zeroed, LayerNorm biases are ablated, and normalization statistics are recomputed. The manuscript validates intermediate policies by showing they play chess, but playing strength does not establish that the intermediate argmax corresponds to a computation the intact network actually performs at that layer. Appendix A.4 itself acknowledges that bias terms cannot be disentangled and that 'it is not clear how to disentangle these functions empirically.' This is a load-bearing premise. Please provide a faithfulness check (e.g., comparison with a tuned/affine lens, causal interventions, or robustness of the main metrics to the beta-ablation choice) or explicitly frame 'forgotten puzzles' as properties of the zero-ablation counterfactual model.
  3. [Abstract and §4] The claim that 'look-ahead operates normally' on forgotten puzzles is not demonstrated in the body. The abstract says future moves of the correct continuation are represented, causally important, and linearly decodable, but the main text does not report these analyses. Appendix F mentions Jenner et al. only in a single case study. A systematic replication on the forgotten-puzzle set is needed to support the claim that the algorithm itself is not failing.
  4. [§3.2 and Figure 3] The definitions of 'cumulative discoveries,' 'converged solutions,' and 'first discoveries' are not rigorous enough to support the quantitative claim that 'the final cumulative solve rate exceeds the final layer's performance.' For example, a puzzle is counted as 'discovered' at layer k if the argmax at that layer solves it, but no threshold is given for how long a solution must persist to be 'converged,' and no error bars or significance tests are reported. Please define these metrics precisely and report variability across puzzle subsets.
minor comments (5)
  1. [General] The full-text title ('Iterative Inference in a Chess-Playing Neural Network') differs from the arXiv title ('The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network'). This should be reconciled.
  2. [Figure 1 caption and §2.3] The Figure 1 caption says the correct move 'only emerges as a plausible candidate in the middle layers before becoming the decisive top choice in the final output,' but Appendix E shows it was already the top choice at layer 5 and then lost the top position. The caption should be adjusted to match the full trajectory.
  3. [Appendix A.3] The notation for elementwise multiplication is inconsistent: the text says 'J denotes element-wise multiplication,' but the equations use ⊙. Please unify.
  4. [Appendix C.2] The GPT-2 comparison lacks details on the number of prompts, sampling temperature, and whether the Pile subset is fixed across models. This makes the comparison hard to reproduce.
  5. [Appendix F.5] Typo: 'sacrifical' should be 'sacrificial.'

Circularity Check

0 steps flagged

No significant circularity; the body's observations are self-contained probes of a fixed public model, and the steering/safety-prior claims are absent from the body rather than circular.

full rationale

This paper reports layer-wise policy measurements obtained by extending the logit lens to a Post-LN transformer (Leela Chess Zero). The core observations—monotonic playing strength, non-monotonic policy trajectories, and 'forgotten puzzles'—are direct comparisons between a zero-ablated truncated network and the full network's policy head. No parameter is fitted to the target conclusion, no quantity is renamed as a prediction after being used as a fit, and no load-bearing argument reduces to a self-citation. The only prior-work anchor, Jenner et al. (2024), is external and independently published; the paper's reference to the L12H12 look-ahead head is used as an interpretive link, not as a derivational premise that forces the result. The main weakness is that the front abstract and title assert a steering intervention, a 61.7% recovery rate, and a causal 'safety prior' override, none of which appear in the body or appendices. That is a substantive evidentiary mismatch, but it is not circularity: an absent claim cannot be a derivation that reduces to its own inputs. The load-bearing methodological assumption is that the Post-LN zero-ablation logit lens faithfully reveals internal computation. The paper itself flags this in Appendix A.4 ('it's not clear how to disentangle these functions empirically given current interpretability techniques'). That is a measurement-validity limitation, not a circular derivation. The term 'forgotten puzzles' is operationally defined by the lens and then measured, but defining an observed phenomenon does not make the subsequent empirical claims equivalent to the definition. Since the derivation chain is self-contained against a fixed public model and the stated limitations are explicitly acknowledged, the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 1 invented entities

No free parameters are fitted to the central claim; the causal story in the abstract introduces an unmeasured 'safety prior.' The body's descriptive findings depend on assumptions about the Post-LN logit lens, puzzle ground truth, and the use of playing strength as a loss proxy.

axioms (4)
  • domain assumption Zero ablation of all sublayer outputs after layer k yields a faithful policy projection for Post-LN transformers.
    Invoked throughout Section 2.2 and Appendix A.2; if false, the layer-wise policies and the 'forgotten puzzles' phenomenon are artifacts.
  • ad hoc to paper LayerNorm biases can be set to zero for layers beyond k without changing the qualitative policy trajectory.
    Appendix A.2/A.4: the authors ablate beta biases for consistency and report only 'nuanced differences'; this choice is specific to the constructed lens and is not independently validated.
  • domain assumption The puzzle principal variations from Ruoss et al. (2024) are the correct ground truth for puzzle solution.
    Section 2.3 defines a solved puzzle as matching the PV or an immediate mate; if a PV has a stronger or equal alternative, solve rates are mismeasured.
  • domain assumption Intermediate-policy playing strength is a valid proxy for loss reduction during iterative inference.
    Section 1: 'we use playing strength as a proxy for loss reduction'; the transfer from Elo to internal loss dynamics is assumed.
invented entities (1)
  • Safety prior (safety-prior override) no independent evidence
    purpose: Postulated in the abstract to explain why correct puzzle solutions are overridden by safe play in the final output.
    The term appears only in the metadata abstract. The full text does not define, measure, or intervene on a safety prior, and no falsifiable prediction is attached to it.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network." pith.science (2026). https://pith.science/paper/AUHIDRKV

@misc{pith2026250821380,
  author       = {Pith},
  title        = {Pith review of: The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUHIDRKV}},
  note         = {Machine review of arXiv:2508.21380}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent mechanistic work has uncovered learned algorithms within neural networks, from modular arithmetic to search and planning in game-playing agents. But does algorithmic structure guarantee algorithmic behavior? We investigate this in Leela Chess Zero, the strongest neural chess engine, where prior work identified learned look-ahead. By extending the logit lens to its move-selecting policy network, we discover that correct puzzle solutions-including immediate checkmates-often appear in intermediate layers but are systematically overridden in the final output, a phenomenon we term "forgotten puzzles". Replicating prior analyses on these positions, we find that look-ahead operates normally-future moves of the correct continuation are represented, causally important, and linearly decodable-ruling out a failure of the algorithm itself. Instead, late layers increasingly shift toward prioritizing safe play over aggression. To test whether this shift drives the override, we steer the model against these preferences and recover 61.7% of forgotten puzzles, providing causal evidence that safety priors override algorithmically computed solutions. These findings demonstrate that algorithmic structure does not guarantee algorithmic behavior: a model can internally solve a problem and still output the wrong answer.

Figures

Figures reproduced from arXiv: 2508.21380 by Elias Sandmann, Sebastian Lapuschkin, Wojciech Samek.

Figure 1
Figure 1. Figure 1: Our extended logit lens reveals progressive policy refinement across transformer layers in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Puzzle-solving performance across transformer layers, stratified by Elo difficulty rating. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Layer-wise puzzle-solving performance across network depth showing raw solve rates, cu￾mulative discoveries, converged solutions, and first discoveries. Puzzle solution stability While overall capa￾bility improves smoothly, individual solution finding follows a more complex process. Fig￾ure 3 tracks four metrics: current layer solutions, cumulative solutions discovered up to each layer, solutions that rema… view at source ↗
Figure 4
Figure 4. Figure 4: Median Kendall’s τ ranking correlation across network depth, comparing the policy dy￾namics of Leela to GPT-2 language models. Distributional analysis Analysis of policy distributions reveals patterns distinct from the smooth convergence observed in language mod￾els (Lad et al., 2025; Belrose et al., 2023). Complete plots are provided in Appendix C. Kendall’s τ ranking correlation between inter￾mediate and… view at source ↗
Figure 5
Figure 5. Figure 5: Metrics of policy dynamics for Leela Chess Zero. The central line shows the median over [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Kendall’s τ move ranking correlation for Leela Chess Zero. The shaded regions represent the 5th-95th and 25th-75th percentile ranges. Language models Since we have not found analysis with this metric for language models, we conducted our own experiments for comparison using the GPT-2 series (small, large, XL) (Radford et al., 2019). To mitigate noise from low probability tokens, we only used tokens that ap… view at source ↗
Figure 7
Figure 7. Figure 7: Kendall’s τ token ranking correlation for GPT-2 models. The shaded regions represent the 5th-95th and 25th-75th percentile ranges. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Layer-wise policy evolution for the puzzle in Figure [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Layer-wise policy evolution for Puzzle 12864. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Layer-wise policy evolution for Puzzle 9745. [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Layer-wise policy evolution for Puzzle 483. [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Layer-wise policy evolution for Puzzle 215. [PITH_FULL_IMAGE:figures/full_fig_p028_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Layer-wise policy evolution for Puzzle 10363. [PITH_FULL_IMAGE:figures/full_fig_p031_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Layer-wise policy evolution for Puzzle 945. [PITH_FULL_IMAGE:figures/full_fig_p034_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 12 canonical work pages

  1. [1]

    Eliciting latent predictions from transformers with the tuned lens, 2023

    Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023. URL https://arxiv.org/abs/2303.08112

    Pith/arXiv arXiv 2023

  2. [2]

    Whole-history rating: A bayesian rating system for players of time-varying strength

    R \' e mi Coulom. Whole-history rating: A bayesian rating system for players of time-varying strength. In Computers and Games, 2008

    work page 2008

  3. [3]

    Discussion on layer normalization in transformer architecture

    dje\ dev. Discussion on layer normalization in transformer architecture. Forum post on Lc0 community chat, 2025. URL https://lc0.org/chat. Accessed: 2025-08-17

    work page 2025

  4. [4]

    The pile: An 800gb dataset of diverse text for language modeling, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/abs/2101.00027

    Pith/arXiv arXiv 2020

  5. [5]

    Residual connections encourage iterative inference, 2018

    Stanisław Jastrzębski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference, 2018. URL https://arxiv.org/abs/1710.04773

    Pith/arXiv arXiv 2018

  6. [6]

    Evidence of learned look-ahead in a chess-playing neural network, 2024

    Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell. Evidence of learned look-ahead in a chess-playing neural network, 2024. URL https://arxiv.org/abs/2406.00877

    Pith/arXiv arXiv 2024

  7. [7]

    The remarkable robustness of llms: Stages of inference?, 2025

    Vedang Lad, Jin Hwa Lee, Wes Gurnee, and Max Tegmark. The remarkable robustness of llms: Stages of inference?, 2025. URL https://arxiv.org/abs/2406.19384

    Pith/arXiv arXiv 2025

  8. [8]

    Leela Chess Zero

    Leela Chess Zero team . Leela Chess Zero . URL https://lczero.org/

    work page

  9. [9]

    A standard dataset, 2018

    Leela Chess Zero team . A standard dataset, 2018. URL https://lczero.org/blog/2018/09/a-standard-dataset/. Accessed: 2025-08-20

    work page 2018

  10. [10]

    How well do Lc0 networks compare to the greatest transformer network from DeepMind ?, 2024

    lepned. How well do Lc0 networks compare to the greatest transformer network from DeepMind ?, 2024. URL https://lczero.org/blog/2024/02/how-well-do-lc0-networks-compare-to-the-greatest-transformer-network-from-deepmind/

    work page 2024

  11. [11]

    Lichess bot: A bridge between lichess api and chess engines

    Lichess Bot Devs . Lichess bot: A bridge between lichess api and chess engines. https://github.com/lichess-bot-devs/lichess-bot, 2025. Accessed: 2025-08-18

    work page 2025

  12. [12]

    Lichess: Free online chess, 2025

    Lichess.org. Lichess: Free online chess, 2025. URL https://lichess.org. Accessed: 2025-08-18

    work page 2025

  13. [13]

    Encyclopaedia of Chess Openings

    Aleksandar Matanovi\' c . Encyclopaedia of Chess Openings. Batsford Limited, 1978

    work page 1978

  14. [14]

    Acquisition of chess knowledge in alphazero

    Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119 0 (47), November 2022. ISSN 1091-6490. doi:10.1073/pnas.2206625119. URL http://dx.doi.org/10.1073/pnas.2206625119

    doi:10.1073/pnas.2206625119 2022

  15. [15]

    How to dissect a muppet: The structure of transformer embedding spaces, 2022

    Timothee Mickus, Denis Paperno, and Mathieu Constant. How to dissect a muppet: The structure of transformer embedding spaces, 2022. URL https://arxiv.org/abs/2206.03529

    Pith/arXiv arXiv 2022

  16. [16]

    Mish: A self regularized non-monotonic activation function, 2020

    Diganta Misra. Mish: A self regularized non-monotonic activation function, 2020. URL https://arxiv.org/abs/1908.08681

    Pith/arXiv arXiv 2020

  17. [17]

    Transformer progress, February 2024

    Daniel Monroe. Transformer progress, February 2024. URL https://lczero.org/blog/2024/02/transformer-progress/. Accessed: 2025-08-17

    work page 2024

  18. [18]

    Interpreting gpt: the logit lens, 2020

    nostalgebraist. Interpreting gpt: the logit lens, 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    work page 2020

  19. [19]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. https://openai.com/research/language-unsupervised, 2019. OpenAI Blog

    work page 2019

  20. [20]

    Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2017. URL https://arxiv.org/abs/1710.05941

    Pith/arXiv arXiv 2017

  21. [21]

    Lewis, Joel Veness, and Tim Genewein

    Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt, John Reid, Cannada A. Lewis, Joel Veness, and Tim Genewein. Amortized planning with large-scale transformers: A case study on chess, 2024. URL https://arxiv.org/abs/2402.04494

    Pith/arXiv arXiv 2024

  22. [22]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362 0 (6419): 0 1140--1144, 2018

    2018

  23. [23]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, pages 5998--6008, 2017

    work page 2017

  24. [24]

    Deepnet: Scaling transformers to 1,000 layers, 2022

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers, 2022. URL https://arxiv.org/abs/2203.00555

    Pith/arXiv arXiv 2022

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.