{"id":"f2777ccd-f504-4b5a-8474-9acef991c9aa","arxiv_id":"2412.05768","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On an idiom completion dataset, the negative log likelihood of GPT-2 XL's top token, measured across residual stream layers, predicts whether the completion is correct with an AUC of 0.92.","lead":"This paper tracks how GPT-2 XL's internal representations change layer by layer and finds that a cross-entropy measure of the model's top-token confidence separates correct from incorrect idiom completions. The result suggests a cheap inference-time signal for flagging uncertain or hallucinated text, but the evidence is limited to one model and one small dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline AUC comes from final-layer output cross-entropy, which is just -log p(^y); the residual-stream 'rate and degree of convergence' is never used in the uncertainty detector, so the central claim is not actually tested.","rationale":"The paper provides a transparent, honest empirical study, and the authors explicitly frame it as preliminary. The IIH evidence (decreasing cross-entropy with respect to ground truth across layers) is plausibly visible in the figures, though it rests on the unvalidated logit lens assumption flagged by the reader. My stronger concern is different: the paper's novelty claim is about residual-stream convergence dynamics, but the uncertainty detector—the part that produces the quantitative AUC—collapses to the negative log likelihood of the model's top token at the final layer. This is a well-known confidence score, and the paper gives no comparison showing that residual trajectories add predictive power beyond that final score. That gap is load-bearing for the strongest claim ('the rate and degree to which embeddings converge... reflect uncertainty'), but it is fixable: a proper feature comparison on held-out data would either establish the dynamic contribution or reveal that the contribution is just final-layer confidence. The paper's other limitations—small dataset, no baselines, no confidence intervals—are real but do not contradict the reported data. The appropriate verdict remains CONDITIONAL: the central claim needs additional analysis before it can be accepted, and this specific test is the key missing piece.","tokens_in":6345,"tokens_out":3819,"duration_ms":41589,"concrete_test":"On the same 330 idiom prompts (and ideally a held-out set), compute two predictors: (i) final output CE = -log p(^y); (ii) a logistic regression on the full per-layer CE trajectory, e.g., CE at layers 5, 10, 20, 30, 40 plus slopes over layers 20-38. Compare held-out AUC via cross-validation, with bootstrap confidence intervals, and also compare against max-softmax-probability and output-entropy baselines. If predictor (i) alone matches or beats (ii), the paper's dynamic convergence claim is not supported by its own data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result, AUC=0.9239, is computed from output cross-entropy at the final layer with target ^y (Figures 3a/3b). As Appendix A.1 states, this is -log q(^y), the negative log likelihood of the model's top token. That number is available from the final softmax alone and carries no information about residual-stream trajectories. Yet the central claim is that 'the rate and degree to which embeddings converge to stable output representations' measures uncertainty. No experiment in the paper uses the rate of convergence: the ROC analysis uses one vertical slice (the last layer), while Figure 2's trajectories are displayed but never quantified into a classifier or compared with final-layer confidence. Consequently, the evidence cannot distinguish the IIH-based dynamic explanation from the trivial baseline that final softmax confidence predicts correctness—which is expected when correctness is defined as ^y=y and correct cases are those where the model assigned top probability to y. The paper's own limitations (confidently incorrect outputs evade the metric; inability to separate prompt uncertainty from model uncertainty) reinforce that output CE is a standard confidence measure, not a convergence diagnostic. The logit lens assumption is a secondary issue: it affects the IIH evidence but not the final-layer AUC.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the Iterative Inference Hypothesis (IIH) in GPT-2 XL by tracking the residual-stream embedding of the last input token across 48 layers on a dataset of 330 English idiom completions. Using a logit-lens-style mapping of intermediate residual embeddings to output distributions, it computes per-layer cross-entropy against two one-hot targets: the model's argmax token \\hat{y} and the ground-truth token y. The authors report that cross-entropy decreases over layers in support of the IIH, that correct and incorrect generations are separated in these trajectories, and that the final-layer output cross-entropy with target \\hat{y} yields an AUC of 0.9239 for discriminating correct from incorrect generations. They also give a qualitative example on an open-ended generation. The central claim is that the rate and degree of convergence of residual embeddings to stable output representations can measure generation uncertainty.","tokens_in":6531,"tokens_out":3138,"duration_ms":33942,"significance":"If the central claim were established, the proposed measure would be a cheap, inference-time uncertainty signal for autoregressive language models, and the paper would add empirical support for the IIH as a framework for interpreting transformer internals. The measurements are straightforward and the reported separation is visually clear. However, the paper does not actually use the rate or degree of convergence as a predictor; the uncertainty detector is the final-layer negative log-likelihood of the top token, which is a standard confidence score obtainable from the final softmax alone. No code or data are provided, and the evaluation is limited to one model, one dataset, and a single 330-sample set with no cross-validation or baseline comparison. The IIH evidence, which is the more mechanistically distinctive contribution, depends on an unvalidated logit-lens assumption. The paper is an honest preliminary study, but the evidential gap between the convergence narrative and the tested detector is load-bearing.","major_comments":[{"comment":"The central claim that 'the rate and degree to which embeddings converge to stable output representations' measures uncertainty is not tested. The ROC analysis in Fig. 3b uses only the final-layer output cross-entropy with target \\hat{y}, which, as Appendix A.1 states, is just -log q(\\hat{y}) and is available from the final softmax. No experiment quantifies the rate of convergence from Fig. 2, uses an intermediate-layer trajectory as a feature, or compares a convergence-based detector against the final-layer value. As written, the evidence cannot distinguish the IIH-based dynamic explanation from the trivial baseline that final-softmax confidence predicts correctness, which is expected when correctness is defined as \\hat{y}=y. Please either add a detector that explicitly uses trajectory information (e.g., layer-wise slope, endpoint separation, or area under the per-layer CE curve) or revise the central claim to match what is actually measured.","section":"Sec. 3, Figs. 2 and 3"},{"comment":"The reported AUC of 0.9239 is computed on the same 330 samples used to observe the effect, with no cross-validation, bootstrap confidence interval, or out-of-sample evaluation. There is also no comparison to simple baselines such as softmax entropy, maximum predicted probability, or the KL divergence to the output distribution, despite the paper's own observation in Sec. 2.1 and Appendix A.3 that the chosen CE target implicitly penalizes high-entropy outputs. Because correctness is defined as \\hat{y}=y, the final CE is essentially a measure of confidence in the top token, so it is unclear what is gained relative to existing confidence scores. The statement that a Mann-Whitney U test 'corroborated' the AUC with the same value is also redundant: the AUC is a normalized Mann-Whitney U statistic, not an independent confirmation.","section":"Sec. 3, Fig. 3b; Sec. 2.3"},{"comment":"The separation in Fig. 3a may be an artifact of the divergence-target choice rather than evidence about convergence dynamics. The paper acknowledges in Sec. 2.1 and Appendix A.3 that using a one-hot target for \\hat{y} 'implicitly penalizes generations that have high output entropy' and that this bias is found 'useful for distinguishing correct and incorrect generations.' Since a high-entropy output distribution by construction has a large -log q(\\hat{y}), the reported separation is expected for any detector based on softmax confidence. Please provide a direct comparison against softmax entropy or top-probability and report whether the convergence-based narrative adds predictive information beyond that baseline.","section":"Sec. 2.1 and Appendix A.3"},{"comment":"The evidence for the IIH relies on the logit-lens assumption that applying the output layer norm and unembedding head to an intermediate residual embedding yields a faithful distribution of the model's prediction at that layer. The paper does not validate this assumption for GPT-2 XL, and Appendix A.2 shows that intermediate argmax tokens can be semantically plausible but are not evaluated against any external criterion. If the logit-lens mapping is not faithful, the per-layer cross-entropy trajectories in Fig. 2b, the residual-prediction tables in Fig. 5, and the layer-wise loss table in Fig. 7 are not reliable evidence for the IIH. Please either provide a validation of the logit-lens reading (e.g., by probing with causal interventions or comparing against ablations) or present the IIH support as conditional on that assumption.","section":"Sec. 2.1 and Figs. 2, 5, 7"}],"minor_comments":[{"comment":"The text describes the correct-generation distribution as 'exponential' and the incorrect distribution as 'normal' without any goodness-of-fit test; please soften or substantiate these distributional claims.","section":"Sec. 3, Fig. 3a"},{"comment":"There is a typo in the sentence 'Large cross-entropy values in this sequence are also observed on for tokens representing dates'—the 'on' appears to be extraneous.","section":"Sec. 3"},{"comment":"The phrase 'very common words that are could have many valid next tokens' contains a grammatical error; please revise.","section":"Appendix A.2"},{"comment":"Reference [2] is malformed: 'LAYING DOWN and INTELLIGENCE ACT' should be replaced with the official title of the EU AI Act proposal.","section":"References"},{"comment":"The paper does not mention whether code or intermediate data will be released. Given the small dataset and single model, releasing prompts, predictions, and per-layer residuals would materially improve reproducibility.","section":"Sec. 2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is an honest preliminary study, and the visual separation in the figures is real, but the headline claim as stated is not supported by the experiments: the ROC analysis uses final-layer cross-entropy, which is standard confidence, and the convergence-rate language is never operationalized. The logit-lens dependence of the IIH evidence is an additional concern that needs to be addressed explicitly. A revision that (a) builds an actual convergence-based detector or clearly reframes the claim, (b) adds baseline comparisons and proper evaluation, and (c) validates or softens the logit-lens assumption would make the contribution publishable. As it stands, the gap between the central claim and the experiments is too large for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the AUC of 0.9239 comes from final-layer cross-entropy with target ^y, which the paper's own appendix A.1 shows is just -log p(^y), the negative log likelihood of the model's top token. That number is available from the final softmax alone. The residual-stream dynamics—the 'rate and degree of convergence' promised in the abstract—are never used in the detector. The ROC uses one vertical slice; the trajectories in Figure 2 are shown but never quantified into a classifier or compared with final-layer confidence. So the central claim is not actually tested.\n\nCredit where it's due: the paper is clear, honest, and small in scope. The IIH plots (Figure 2b) do show a roughly monotonic decrease in loss with respect to ground truth, which is consistent with the iterative inference picture. The authors are upfront about limitations, including confidently incorrect outputs and the difficulty of separating prompt uncertainty from model uncertainty. The idiom dataset is a reasonable testbed for a preliminary study.\n\nThe soft spots are fixable but real. The AUC is computed on the same 330 samples used to observe the effect, with no cross-validation or confidence interval. There is no baseline—softmax entropy, for example, would likely do as well. The logit lens assumption, that intermediate residuals can be faithfully mapped through the final layernorm and unembedding, is plausible for GPT-2 but unvalidated; it matters for the IIH evidence but not for the final-layer AUC. The paper also conflates a static confidence score with a dynamic convergence measurement, which is the load-bearing weakness.\n\nBottom line: this is a workshop-level exploratory paper with an honest write-up and a misleading headline. The IIH evidence is suggestive, not definitive, and the uncertainty detector reduces to a known quantity. I would not cite it as a new method, but it might be referenced as an early empirical look at IIH. A serious referee could push the authors to actually test the convergence-rate claim, add held-out evaluation and baselines, and temper the abstract. I'd send it to peer review, expecting major revision.","headline":"The headline uncertainty detector is just final-layer -log p(^y); the residual-stream convergence story is never actually used, so the paper over-sells its central claim.","tokens_in":7104,"tokens_out":2466,"would_cite":false,"duration_ms":23844,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"How a transformer's layer-by-layer embedding trajectory predicts whether its next token is correct, with an AUC of 0.9239.","keywords":["iterative inference hypothesis","residual stream","cross-entropy","uncertainty detection","logit lens","next-token prediction","idiom completion","transformer interpretability"],"falsifier":"Take a held-out set of prompts with known next tokens, compute residual predictions at every layer via the final layer norm and unembedding head, and check whether median cross-entropy to the true token falls at nearly every layer and whether final output cross-entropy separates correct and incorrect generations; if either fails on a broad dataset, the paper's central mechanism is refuted.","tokens_in":6111,"feed_emoji":"🎯","tokens_out":9399,"duration_ms":86362,"temperature":0.7,"pith_summary":"The paper tries to show that a transformer's path toward its next-token prediction is visible inside its own residual stream and that the shape of that path reveals uncertainty. It reports that the embedding of the last input token moves through the model along a trajectory of decreasing loss, supporting the Iterative Inference Hypothesis, which treats each layer as an incremental correction toward the next token. It then defines a cross-entropy score between the model's layer-by-layer predictions and the single token it actually samples, and shows that this score separates correct from incorrect idiom completions: correct generations converge close to a one-hot distribution, incorrect ones stay diffuse, and the final score predicts correctness with an AUC of 0.9239. The motivation is practical as well as interpretive: a cheap, label-free measure of when a model is unsure could flag hallucinations before they are emitted.","feed_headline":"How embeddings settle predicts wrong next tokens, AUC 0.92","feed_subtitle":"A cheap, label-free cross-entropy score tracks confidence layer by layer and flags incorrect idiom completions.","key_machinery":"The central object is the residual stream, written as the recurrence $r_{i+1} = r_i + L_{i+1}(r_i)$, and specifically the trajectory of the $n$-th input token embedding as it is translated through token-embedding space by each layer. The central measurement is residual cross-entropy: after each layer, the intermediate embedding is passed through the model's output layer norm and unembedding head to form a residual prediction, and the negative log likelihood of a target token is computed. Two targets are used: the ground-truth next token $y$ (available in the idiom dataset, needed to test the IIH) and the deterministically sampled token $\\hat{y}$ (available at inference time, used for the uncertainty detector). The choice of $\\hat{y}$ implicitly penalizes high-entropy output distributions, which is what lets the final-layer score separate correct from incorrect generations.","core_discovery":"On the paper's own terms, the discovery is that transformer inference can be watched as a convergence process in the residual stream: the representation of the last input token is updated by every layer, and reading each intermediate representation through the final layer norm and unembedding head yields a per-layer prediction whose cross-entropy with the ground-truth next token falls nearly monotonically. The rate and degree to which these residual predictions settle into a stable output representation tracks the correctness of the sampled token. When the model is right, the final output cross-entropy against the sampled token $\\hat{y}$ is sharply concentrated near zero (mean 0.43); when it is wrong, the distribution is broad (mean 1.91), and the resulting separation yields an AUC of 0.9239 on the idiom dataset. The paper presents this as evidence that correctness and, plausibly, the amount of certainty for the next generated token can be measured from the convergence behavior of embeddings, not just from the final softmax.","pith_inferences":["Beyond the paper: the anecdotal spike pattern on the 'Alan Turing' generation suggests the metric may flag factual errors in open-ended text, but the paper only shows one example; testing on a labeled hallucination benchmark would turn this into a claim.","Beyond the paper: because the target is the model's own argmax token rather than the true token, part of the AUC may reflect the metric penalizing high-entropy outputs by construction; separating 'uncertainty' from 'wrongness' would require comparing correct low-entropy and incorrect low-entropy cases.","Beyond the paper: the monotone decreasing loss may be partly an artifact of the logit-lens readout, since later layers are closer to the final output head; a control using random or shuffled layer norms would help isolate genuine iterative refinement.","Beyond the paper: the cross-entropy per token could be aggregated over a multi-token generation to produce a document-level confidence score, but the paper only measures single-token completions."],"forward_implications":["Output cross-entropy against the sampled token can be computed during generation with no ground-truth label, so it is a candidate online flag for low-confidence or hallucinated tokens.","The per-layer trajectories give an interpretability handle: nearly every layer update reduces loss on average, with layers 20-38 doing the most work on correct idiom completions, so analysis of where convergence stalls can localize prediction failures.","If the result transfers beyond idioms, the same metric could be used as a threshold-based classifier; on this dataset, a final output cross-entropy above roughly 1.5 is very likely to be incorrect.","The IIH support means intermediate residual predictions carry real signal, so monitoring them layer by layer is a legitimate way to study and debug generation."],"supporting_citations":[{"why":"Establishes the embedding-space analysis of transformers that motivates reading intermediate residual embeddings as predictions.","marker":"[1]"},{"why":"Introduces the claim that residual networks perform unrolled iterative estimation, the root of the Iterative Inference Hypothesis.","marker":"[4]"},{"why":"Supplies the theoretical basis that residual connections encourage iterative inference in a direction of decreasing loss.","marker":"[7]"},{"why":"Provides the logit-lens technique used to turn intermediate residual embeddings into per-layer token distributions.","marker":"[9]"},{"why":"Defines the GPT-2 XL model whose 48 layers supply all residual-stream trajectories in the experiments.","marker":"[11]"},{"why":"Supplies the EPIE idiom dataset from which the 330 single-token idiom completions are constructed.","marker":"[12]"},{"why":"Frames autoregressive transformer inference as an iterative optimization process, motivating the convergence interpretation.","marker":"[14]"}],"fun_headline_variants":["Residual stream settling rate reveals when transformers generate wrong tokens","Embedding convergence dynamics spot incorrect token completions at AUC 0.92","Wrong next token predicted by how quickly residual embeddings settle","Layer-wise embedding convergence signals token errors at AUC 0.92"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on the assumption that feeding an intermediate residual embedding through the model's final layer norm and unembedding head yields a faithful distribution of what the model predicts at that layer; if that logit-lens reading is distorted, the per-layer loss trajectories and the convergence signal are not measuring what they appear to measure.","fun_headline_variants_meta":{"raw":{"variants":["Residual stream settling rate reveals when transformers generate wrong tokens","Embedding convergence dynamics spot incorrect token completions at AUC 0.92","Wrong next token predicted by how quickly residual embeddings settle","Layer-wise embedding convergence signals token errors at AUC 0.92"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000565,"raw_usage":{"total_tokens":2623,"prompt_tokens":835,"completion_tokens":1788,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":1716}},"tokens_in":451,"tokens_out":1788,"duration_ms":12215,"temperature":1.0,"reasoning_tokens":1716,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:23:00.337471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of prompts with known next tokens, compute residual predictions at every layer via the final layer norm and unembedding head, and check whether median cross-entropy to the true token falls at nearly every layer and whether final output cross-entropy separates correct and incorrect generations; if either fails on a broad dataset, the paper's central mechanism is refuted.","supporting_citations":[{"cited_title":"Analyzing transformers in embedding space","cited_arxiv_id":null,"evidence_quote":"Establishes the embedding-space analysis of transformers that motivates reading intermediate residual embeddings as predictions."},{"cited_title":"Srivastava, and Jürgen Schmidhuber","cited_arxiv_id":null,"evidence_quote":"Introduces the claim that residual networks perform unrolled iterative estimation, the root of the Iterative Inference Hypothesis."},{"cited_title":"Interpreting gpt: The logit lens., 2020","cited_arxiv_id":null,"evidence_quote":"Provides the logit-lens technique used to turn intermediate residual embeddings into per-layer token distributions."},{"cited_title":"Language models are unsupervised multitask learners","cited_arxiv_id":null,"evidence_quote":"Defines the GPT-2 XL model whose 48 layers supply all residual-stream trajectories in the experiments."},{"cited_title":"Epie dataset: A corpus for possible idiomatic expressions","cited_arxiv_id":null,"evidence_quote":"Supplies the EPIE idiom dataset from which the 330 single-token idiom completions are constructed."},{"cited_title":"Transformers learn in-context by gradient descent","cited_arxiv_id":null,"evidence_quote":"Frames autoregressive transformer inference as an iterative optimization process, motivating the convergence interpretation."}],"review_version":1}