Uncovering Uncertainty in Transformer Inference

Amber Tien; Greyson Brothers; John Winder; Willa Mannering

REVIEW 4 major objections 5 minor 15 references

Uncovering Uncertainty in Transformer Inference

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read How a transformer's layer-by-layer embedding trajectory predicts whether its next token is correct, with an AUC of 0.9239.

desk verdict The headline uncertainty detector is just final-layer -log p(^y); the residual-stream convergence story is never actually used, so the paper over-sells its central claim. read the letter →

arxiv 2412.05768 v1 pith:5ETU5X4Z submitted 2024-12-08 cs.CL cs.AI

Greyson Brothers , Willa Mannering , Amber Tien , John Winder This is my paper · ORCID

classification cs.CLcs.AI

keywords iterativeinferencehypothesisresidualstreamcross-entropyuncertaintydetectionlogitlensnext-tokenpredictionidiomcompletiontransformerinterpretability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a transformer's path toward its next-token prediction is visible inside its own residual stream and that the shape of that path reveals uncertainty. It reports that the embedding of the last input token moves through the model along a trajectory of decreasing loss, supporting the Iterative Inference Hypothesis, which treats each layer as an incremental correction toward the next token. It then defines a cross-entropy score between the model's layer-by-layer predictions and the single token it actually samples, and shows that this score separates correct from incorrect idiom completions: correct generations converge close to a one-hot distribution, incorrect ones stay diffuse, and the final score predicts correctness with an AUC of 0.9239. The motivation is practical as well as interpretive: a cheap, label-free measure of when a model is unsure could flag hallucinations before they are emitted.

What carries the argument

The central object is the residual stream, written as the recurrence $r_{i+1} = r_i + L_{i+1}(r_i)$, and specifically the trajectory of the $n$-th input token embedding as it is translated through token-embedding space by each layer. The central measurement is residual cross-entropy: after each layer, the intermediate embedding is passed through the model's output layer norm and unembedding head to form a residual prediction, and the negative log likelihood of a target token is computed. Two targets are used: the ground-truth next token $y$ (available in the idiom dataset, needed to test the IIH) and the deterministically sampled token $\hat{y}$ (available at inference time, used for the uncertainty detector). The choice of $\hat{y}$ implicitly penalizes high-entropy output distributions, which is what lets the final-layer score separate correct from incorrect generations.

What would settle it

Take a held-out set of prompts with known next tokens, compute residual predictions at every layer via the final layer norm and unembedding head, and check whether median cross-entropy to the true token falls at nearly every layer and whether final output cross-entropy separates correct and incorrect generations; if either fails on a broad dataset, the paper's central mechanism is refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that transformer inference can be watched as a convergence process in the residual stream: the representation of the last input token is updated by every layer, and reading each intermediate representation through the final layer norm and unembedding head yields a per-layer prediction whose cross-entropy with the ground-truth next token falls nearly monotonically. The rate and degree to which these residual predictions settle into a stable output representation tracks the correctness of the sampled token. When the model is right, the final output cross-entropy against the sampled token $\hat{y}$ is sharply concentrated near zero (mean 0.43); when it is wrong, the distribution is broad (mean 1.91), and the resulting separation yields an AUC of 0.9239 on the idiom dataset. The paper presents this as evidence that correctness and, plausibly, the amount of certainty for the next generated token can be measured from the convergence behavior of embeddings, not just from the final softmax.

Load-bearing premise

The argument rests on the assumption that feeding an intermediate residual embedding through the model's final layer norm and unembedding head yields a faithful distribution of what the model predicts at that layer; if that logit-lens reading is distorted, the per-layer loss trajectories and the convergence signal are not measuring what they appear to measure.

Editorial extensions

If this is right

Output cross-entropy against the sampled token can be computed during generation with no ground-truth label, so it is a candidate online flag for low-confidence or hallucinated tokens.
The per-layer trajectories give an interpretability handle: nearly every layer update reduces loss on average, with layers 20-38 doing the most work on correct idiom completions, so analysis of where convergence stalls can localize prediction failures.
If the result transfers beyond idioms, the same metric could be used as a threshold-based classifier; on this dataset, a final output cross-entropy above roughly 1.5 is very likely to be incorrect.
The IIH support means intermediate residual predictions carry real signal, so monitoring them layer by layer is a legitimate way to study and debug generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the anecdotal spike pattern on the 'Alan Turing' generation suggests the metric may flag factual errors in open-ended text, but the paper only shows one example; testing on a labeled hallucination benchmark would turn this into a claim.
Beyond the paper: because the target is the model's own argmax token rather than the true token, part of the AUC may reflect the metric penalizing high-entropy outputs by construction; separating 'uncertainty' from 'wrongness' would require comparing correct low-entropy and incorrect low-entropy cases.
Beyond the paper: the monotone decreasing loss may be partly an artifact of the logit-lens readout, since later layers are closer to the final output head; a control using random or shuffled layer norms would help isolate genuine iterative refinement.
Beyond the paper: the cross-entropy per token could be aggregated over a multi-token generation to produce a document-level confidence score, but the paper only measures single-token completions.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: the AUC of 0.9239 comes from final-layer cross-entropy with target ^y, which the paper's own appendix A.1 shows is just -log p(^y), the negative log likelihood of the model's top token. That number is available from the final softmax alone. The residual-stream dynamics—the 'rate and degree of convergence' promised in the abstract—are never used in the detector. The ROC uses one vertical slice; the trajectories in Figure 2 are shown but never quantified into a classifier or compared with final-layer confidence. So the central claim is not actually tested.

Credit where it's due: the paper is clear, honest, and small in scope. The IIH plots (Figure 2b) do show a roughly monotonic decrease in loss with respect to ground truth, which is consistent with the iterative inference picture. The authors are upfront about limitations, including confidently incorrect outputs and the difficulty of separating prompt uncertainty from model uncertainty. The idiom dataset is a reasonable testbed for a preliminary study.

The soft spots are fixable but real. The AUC is computed on the same 330 samples used to observe the effect, with no cross-validation or confidence interval. There is no baseline—softmax entropy, for example, would likely do as well. The logit lens assumption, that intermediate residuals can be faithfully mapped through the final layernorm and unembedding, is plausible for GPT-2 but unvalidated; it matters for the IIH evidence but not for the final-layer AUC. The paper also conflates a static confidence score with a dynamic convergence measurement, which is the load-bearing weakness.

Bottom line: this is a workshop-level exploratory paper with an honest write-up and a misleading headline. The IIH evidence is suggestive, not definitive, and the uncertainty detector reduces to a known quantity. I would not cite it as a new method, but it might be referenced as an early empirical look at IIH. A serious referee could push the authors to actually test the convergence-rate claim, add held-out evaluation and baselines, and temper the abstract. I'd send it to peer review, expecting major revision.

Referee Report

4 major / 5 minor

Summary. The paper studies the Iterative Inference Hypothesis (IIH) in GPT-2 XL by tracking the residual-stream embedding of the last input token across 48 layers on a dataset of 330 English idiom completions. Using a logit-lens-style mapping of intermediate residual embeddings to output distributions, it computes per-layer cross-entropy against two one-hot targets: the model's argmax token \hat{y} and the ground-truth token y. The authors report that cross-entropy decreases over layers in support of the IIH, that correct and incorrect generations are separated in these trajectories, and that the final-layer output cross-entropy with target \hat{y} yields an AUC of 0.9239 for discriminating correct from incorrect generations. They also give a qualitative example on an open-ended generation. The central claim is that the rate and degree of convergence of residual embeddings to stable output representations can measure generation uncertainty.

Significance. If the central claim were established, the proposed measure would be a cheap, inference-time uncertainty signal for autoregressive language models, and the paper would add empirical support for the IIH as a framework for interpreting transformer internals. The measurements are straightforward and the reported separation is visually clear. However, the paper does not actually use the rate or degree of convergence as a predictor; the uncertainty detector is the final-layer negative log-likelihood of the top token, which is a standard confidence score obtainable from the final softmax alone. No code or data are provided, and the evaluation is limited to one model, one dataset, and a single 330-sample set with no cross-validation or baseline comparison. The IIH evidence, which is the more mechanistically distinctive contribution, depends on an unvalidated logit-lens assumption. The paper is an honest preliminary study, but the evidential gap between the convergence narrative and the tested detector is load-bearing.

major comments (4)

[Sec. 3, Figs. 2 and 3] The central claim that 'the rate and degree to which embeddings converge to stable output representations' measures uncertainty is not tested. The ROC analysis in Fig. 3b uses only the final-layer output cross-entropy with target \hat{y}, which, as Appendix A.1 states, is just -log q(\hat{y}) and is available from the final softmax. No experiment quantifies the rate of convergence from Fig. 2, uses an intermediate-layer trajectory as a feature, or compares a convergence-based detector against the final-layer value. As written, the evidence cannot distinguish the IIH-based dynamic explanation from the trivial baseline that final-softmax confidence predicts correctness, which is expected when correctness is defined as \hat{y}=y. Please either add a detector that explicitly uses trajectory information (e.g., layer-wise slope, endpoint separation, or area under the per-layer CE curve) or revise the central claim to match what is actually measured.
[Sec. 3, Fig. 3b; Sec. 2.3] The reported AUC of 0.9239 is computed on the same 330 samples used to observe the effect, with no cross-validation, bootstrap confidence interval, or out-of-sample evaluation. There is also no comparison to simple baselines such as softmax entropy, maximum predicted probability, or the KL divergence to the output distribution, despite the paper's own observation in Sec. 2.1 and Appendix A.3 that the chosen CE target implicitly penalizes high-entropy outputs. Because correctness is defined as \hat{y}=y, the final CE is essentially a measure of confidence in the top token, so it is unclear what is gained relative to existing confidence scores. The statement that a Mann-Whitney U test 'corroborated' the AUC with the same value is also redundant: the AUC is a normalized Mann-Whitney U statistic, not an independent confirmation.
[Sec. 2.1 and Appendix A.3] The separation in Fig. 3a may be an artifact of the divergence-target choice rather than evidence about convergence dynamics. The paper acknowledges in Sec. 2.1 and Appendix A.3 that using a one-hot target for \hat{y} 'implicitly penalizes generations that have high output entropy' and that this bias is found 'useful for distinguishing correct and incorrect generations.' Since a high-entropy output distribution by construction has a large -log q(\hat{y}), the reported separation is expected for any detector based on softmax confidence. Please provide a direct comparison against softmax entropy or top-probability and report whether the convergence-based narrative adds predictive information beyond that baseline.
[Sec. 2.1 and Figs. 2, 5, 7] The evidence for the IIH relies on the logit-lens assumption that applying the output layer norm and unembedding head to an intermediate residual embedding yields a faithful distribution of the model's prediction at that layer. The paper does not validate this assumption for GPT-2 XL, and Appendix A.2 shows that intermediate argmax tokens can be semantically plausible but are not evaluated against any external criterion. If the logit-lens mapping is not faithful, the per-layer cross-entropy trajectories in Fig. 2b, the residual-prediction tables in Fig. 5, and the layer-wise loss table in Fig. 7 are not reliable evidence for the IIH. Please either provide a validation of the logit-lens reading (e.g., by probing with causal interventions or comparing against ablations) or present the IIH support as conditional on that assumption.

minor comments (5)

[Sec. 3, Fig. 3a] The text describes the correct-generation distribution as 'exponential' and the incorrect distribution as 'normal' without any goodness-of-fit test; please soften or substantiate these distributional claims.
[Sec. 3] There is a typo in the sentence 'Large cross-entropy values in this sequence are also observed on for tokens representing dates'—the 'on' appears to be extraneous.
[Appendix A.2] The phrase 'very common words that are could have many valid next tokens' contains a grammatical error; please revise.
[References] Reference [2] is malformed: 'LAYING DOWN and INTELLIGENCE ACT' should be replaced with the official title of the EU AI Act proposal.
[Sec. 2.2] The paper does not mention whether code or intermediate data will be released. Given the small dataset and single model, releasing prompts, predictions, and per-layer residuals would materially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline CE metric is -log q(hat-y), an external empirical confidence signal, not a fitted parameter; the untested 'rate of convergence' wording is an overgeneralization, not a circular derivation.

full rationale

The paper's uncertainty metric is H(one-hot(hat-y), q) = -log q(hat-y), derived explicitly in Appendix A.1. This is the negative log likelihood of the model's own top token (standard confidence), not a parameter fitted to correctness labels. Correctness is defined externally as hat-y = y, so the AUC of 0.9239 is an empirical correlation between model confidence and external labels; it is not forced by construction, since confidently incorrect and correctly low-confidence cases are logically possible. The per-layer traces in Figure 2 are descriptive; the conclusion that the 'rate and degree' of residual convergence measures certainty goes beyond the ROC experiment, which uses only the final-layer degree, but this is an unsupported generalization rather than a circular step. The logit lens assumption (Section 2) is load-bearing for the IIH evidence but is an interpretive assumption, not a self-citation or an equation that assumes the conclusion. No self-citations, imported uniqueness theorems, fitted inputs, or ansatz-by-citation chains appear. The authors' own limitation (Section 3, Figure 4: the metric cannot disambiguate prompt uncertainty from model uncertainty and misses confidently incorrect outputs) confirms that CE behaves as ordinary confidence, not as a separately derived convergence diagnostic. The lack of held-out evaluation is a statistical caution, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted free parameters or invented entities. Its central analysis depends on the logit lens assumption, the residual stream basis-preservation property, the idiom completion labeling, and the representativeness of GPT-2 XL. These are all domain assumptions rather than ad hoc additions, but they remain unvalidated in this work.

assumptions (4)

domain assumption Applying the final output layer norm and unembedding head to intermediate residual embeddings yields a valid distribution over tokens at that layer (logit lens).
The entire per-layer cross-entropy analysis depends on this mapping; the paper does not verify it for GPT-2 XL.
domain assumption The residual stream preserves the basis of the embedding space, so each layer update is a translation in token embedding space.
Used in Section 2 to interpret residual paths as paths through token embedding space; this is a known result cited from [3], but still an unproved assumption in this context.
domain assumption The final word of each idiom is the unique correct completion.
Used to define ground truth y and correctness; some idioms may have multiple valid completions, and the task depends on this labeling.
domain assumption GPT-2 XL is representative of transformer-based language models.
Used to draw general conclusions about transformer inference; the paper acknowledges this is preliminary.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Uncovering Uncertainty in Transformer Inference." pith.science (2026). https://pith.science/paper/5ETU5X4Z

@misc{pith2026241205768,
  author       = {Pith},
  title        = {Pith review of: Uncovering Uncertainty in Transformer Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ETU5X4Z}},
  note         = {Machine review of arXiv:2412.05768}
}

read the original abstract

We explore the Iterative Inference Hypothesis (IIH) within the context of transformer-based language models, aiming to understand how a model's latent representations are progressively refined and whether observable differences are present between correct and incorrect generations. Our findings provide empirical support for the IIH, showing that the nth token embedding in the residual stream follows a trajectory of decreasing loss. Additionally, we observe that the rate at which residual embeddings converge to a stable output representation reflects uncertainty in the token generation process. Finally, we introduce a method utilizing cross-entropy to detect this uncertainty and demonstrate its potential to distinguish between correct and incorrect token generations on a dataset of idioms.

Figures

Figures reproduced from arXiv: 2412.05768 by the authors.

**Figure 2.** Plots showing the cross-entropy between the residual prediction at each layer and a target [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** (Left) Distributions of correct and incorrect generations according to final layer cross [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Output cross-entropy per generated token given the open-ended prompt "Alan Turing". [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: A look into the residual stream for the idiom generations with highest and lowest output [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Computing KL divergence with respect to (a) the model output logits versus (b) the one-hot [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: The change in cross-entropy loss between the prediction in the residual stream and the [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: GPT-2 iteratively increases cosine between the embedding of the [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 9 canonical work pages

[1]

Analyzing transformers in embedding space

Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 16124–16170, Toronto, Canada, July 2023

work page 2023
[2]

LAYING DOWN and INTELLIGENCE ACT. Proposal for a regulation of the european parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts, 2021

work page 2021
[3]

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...
[4]

Srivastava, and Jürgen Schmidhuber

Klaus Greff, Rupesh K. Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. In International Conference on Learning Representations, 2017

work page 2017
[5]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770–778, 2016

2016
[6]

Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling, 2020

2020
[7]

Residual connections encourage iterative inference

Stanisław Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018

2018
[8]

The remarkable robustness of LLMs: Stages of inference? In ICML 2024 Workshop on Mechanistic Interpretability, 2024

Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of LLMs: Stages of inference? In ICML 2024 Workshop on Mechanistic Interpretability, 2024

2024

Show all 15 references

[9]

Interpreting gpt: The logit lens., 2020

Nostalgebraist. Interpreting gpt: The logit lens., 2020

2020
[10]

Park, Simon Goldstein, Aidan O’Gara, Michael Chen, and Dan Hendrycks

Peter S. Park, Simon Goldstein, Aidan O’Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions. Patterns, pages 1–16, 2024

2024
[11]

Language models are unsupervised multitask learners

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI, 2019

2019
[12]

Epie dataset: A corpus for possible idiomatic expressions

Prateek Saxena and Soma Paul. Epie dataset: A corpus for possible idiomatic expressions. In Text, Speech, and Dialogue: 23rd International Conference, TSD 2020, Brno, Czech Republic, September 8–11, 2020, Proceedings, page 87–94, Berlin, Heidelberg, 2020. Springer-Verlag

2020
[13]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, volume 30. Curran Associates, Inc., 2017. 6

2017
[14]

Transformers learn in-context by gradient descent

Johannes V on Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jon...

2023
[2021]

https://transformer-circuits.pub/2021/framework/index.html

2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Analyzing transformers in embedding space

Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 16124–16170, Toronto, Canada, July 2023

work page 2023

[2] [2]

LAYING DOWN and INTELLIGENCE ACT. Proposal for a regulation of the european parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts, 2021

work page 2021

[3] [3]

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...

[4] [4]

Srivastava, and Jürgen Schmidhuber

Klaus Greff, Rupesh K. Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. In International Conference on Learning Representations, 2017

work page 2017

[5] [5]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770–778, 2016

2016

[6] [6]

Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling, 2020

2020

[7] [7]

Residual connections encourage iterative inference

Stanisław Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018

2018

[8] [8]

The remarkable robustness of LLMs: Stages of inference? In ICML 2024 Workshop on Mechanistic Interpretability, 2024

Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of LLMs: Stages of inference? In ICML 2024 Workshop on Mechanistic Interpretability, 2024

2024