{"id":"e1ce4590-e981-4592-88c5-cea04055b16a","arxiv_id":"2507.06722","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across 5 LLMs and 11 datasets, layer-wise probability trajectories for correct and incorrect predictions are largely aligned, so uncertainty appears to have little effect on when models commit to an answer.","lead":"This paper uses a Tuned Lens to watch how five language models build up their multiple-choice answers layer by layer, and finds that correct and incorrect answers follow similar probability trajectories. The result suggests that measuring layer-wise confidence alone may not be enough to detect when a model is uncertain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table 1 shows a systematic positive shift in prediction depth for incorrect answers, yet the central claim of 'largely aligned' dynamics rests on averaged trajectories that can conceal such shifts; no quantitative alignment statistic is provided.","rationale":"The reader's CONDITIONAL verdict is directionally right: the paper is honest, exploratory, and its proxy for uncertainty is questionable. I partially agree with the reader's weakest assumption, but I identify a more load-bearing, internal issue: the paper's own PD correlation results contradict the strong alignment claim. Table 1 consistently shows that incorrect answers correlate with later commitment layers, often with correlations of 0.3–0.5. That is not 'largely independent of uncertainty'; it is a systematic dynamic difference, albeit a modest one. The trajectory plots and PD histograms are aggregated in a way that can hide this shift, and the paper provides no quantitative test of trajectory alignment. Because the central claim is a negative result about inference dynamics, the evidence must show that the quantities being compared are actually equivalent—not merely that their averages look similar. The concrete test proposed here would settle whether the alignment conclusion survives a per-question analysis. If the shift is real, the paper's conclusion should be weakened to 'commitment-layer shifts are small and dataset-dependent' rather than 'uncertainty does not affect inference dynamics'. If the shift disappears under proper statistical testing, the original claim would be better supported. Given that the inconsistency is addressable with a re-analysis rather than fundamental invalidity, I recommend CONDITIONAL rather than REJECT: the paper should be accepted only if the quantitative alignment test supports the visual impression. This is consistent with the reader's verdict but for a sharper reason, and the concrete test is more specific than merely adding error bars.","tokens_in":13340,"tokens_out":1988,"duration_ms":26139,"concrete_test":"Compute a per-question alignment statistic instead of averaging. For each question, record the PD for the final top token and the full probability trajectory. Then: (1) test whether the per-dataset mean PD difference (incorrect − correct) is significantly nonzero, with bootstrap confidence intervals; Table 1 already suggests many such differences are positive and significant. (2) Compute the L2 or area distance between the average correct and incorrect top-token probability trajectories per layer, and compare it against a permutation null where correctness labels are shuffled within each dataset. (3) Check the distribution of per-question PD differences for bimodality or opposing signs across datasets.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that correct and incorrect prediction trajectories are 'largely aligned', implying uncertainty does not affect inference dynamics. The reader's weakest-assumption flag—that incorrect answers proxy epistemic uncertainty—is valid, but there is a more direct internal tension. The paper's own Table 1 reports Pearson correlations between answer incorrectness and prediction depth (PD): 97% of correlations are positive, with many above 0.3 (e.g., ARC-Easy for Llama-3-8B at 0.449, SciQ at 0.480, ANLI-R2 for Mistral-Nemo at 0.464, RACE for Mistral variants around 0.37–0.40). This is a systematic, statistically significant tendency for incorrect answers to commit at later layers; it is exactly the kind of layer-wise dynamic difference the abstract denies. The alignment conclusion (Figures 1–2) comes from averaging probability trajectories and PD histograms over thousands of questions. If commitment shifts later by even a fraction of a layer for incorrect answers, the averaged trajectory and the aggregate PD distribution can still look nearly identical, especially when the probability rise is steep and within-group variance is large. The paper reports no quantitative trajectory-alignment metric, no confidence intervals on the difference between correct and incorrect curves, and no check on whether opposing per-dataset shifts cancel in the aggregate. Thus the central negative result is not established by the displayed evidence; it is potentially an artifact of pooling data in a way that is insensitive to the systematic PD shift the authors themselves report. Specifically, a consistent later-commitment shift for uncertain outputs is a real dynamic difference, even if it does not change the shape of the averaged probability curve.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether epistemic uncertainty changes the layer-wise inference dynamics of LLMs, using Tuned Lens to extract token-probability trajectories and prediction depth (PD) across 11 multiple-choice datasets and 5 models. The authors report that correct and incorrect predictions show 'largely aligned' probability trajectories and similar PD distributions, concluding that uncertainty does not seem to affect inference dynamics, while also presenting evidence that more competent models may show greater adaptive dynamics.","tokens_in":13588,"tokens_out":3888,"duration_ms":41525,"significance":"If the central negative result were firmly established, the paper would be a useful cautionary finding for the interpretability and uncertainty-detection communities, showing that simple layer-wise commitment behavior does not reliably encode epistemic uncertainty. The study's breadth—5 models and 11 datasets—and its use of the established Tuned Lens method are strengths. However, the reported evidence does not yet support the strong claim in the abstract: the paper's own Table 1 shows a systematic positive correlation between incorrectness and PD, and the alignment claims rest on visual inspection of averaged curves without error bars or statistical tests. The paper also has an incomplete presentation of Figure 3, which undermines the secondary claim about model competence. These issues are fixable with re-analysis and reframing, but they are load-bearing for the paper's main conclusions.","major_comments":[{"comment":"The abstract claims that 'uncertainty does not seem to affect inference dynamics' and that trajectories are 'largely aligned,' but Table 1 reports Pearson correlations between answer incorrectness and prediction depth that are 97% positive across model-dataset pairs, with many values above 0.300 (e.g., ARC-Easy at 0.449 for Llama-3-8B, SciQ at 0.480, ANLI-R2 at 0.464 for Mistral-Nemo-Instruct). This is a systematic tendency for incorrect answers to commit at later layers, which is itself an effect of the studied variable on layer-wise dynamics. The paper needs to reconcile this quantitative result with the negative claim, either by providing a quantitative trajectory-alignment metric that controls for the PD shift, or by substantially softening the conclusion.","section":"Abstract and Table 1"},{"comment":"The claim that correct and incorrect probability trajectories are 'strongly aligned' and that PD distributions 'show similar peaks' is based entirely on visual inspection of averaged curves. No confidence intervals, error bars, or statistical tests for trajectory similarity or distribution equality are provided. Because the figures aggregate over thousands of questions, a small but systematic per-question shift in commitment layer could be invisible in the average while still being well captured by the positive correlations in Table 1. The authors should report per-layer bootstrap confidence intervals on the difference between correct and incorrect trajectories, and an alignment statistic (e.g., mean absolute difference, area between curves) with its uncertainty, both pooled and per dataset.","section":"Section 3, Figures 1 and 2"},{"comment":"Figure 3 displays only three of the five models (Llama-3-8B, Llama-3-8B-Instruct, Mistral-7B-Instruct), while the text states that 'two out of the five models ... show statistically significant positive correlations' and that 'other larger models do not observe significant trends.' The figure caption references 'other larger models' but no such panels are shown. This incomplete presentation makes it impossible to evaluate the RQ2 evidence about model competence, especially because Vicuna-13B and Mistral-Nemo-Instruct are absent. Either show all five models in the figure or clearly explain why the other two are omitted.","section":"Figure 3 and Section 2.2"},{"comment":"The paper explicitly equates incorrect answers with high epistemic uncertainty ('the model exhibits epistemic uncertainty when it incorrectly answers a question'). This assumption is load-bearing because the entire comparison is between correct and incorrect predictions. Incorrect answers could be confidently wrong (e.g., due to systematic bias), and correct answers could be the result of lucky guesses with low confidence. The paper provides no independent validation of the uncertainty proxy. At minimum, the conclusions should be reframed as being about correctness rather than uncertainty, or a supplementary analysis should condition on model confidence (e.g., final-layer probability bins) to show that the alignment holds within similar confidence levels.","section":"Section 1"}],"minor_comments":[{"comment":"The definition of prediction depth is slightly ambiguous: 'the layer at which a model's top prediction is different from the previous layer's and is maintained for subsequent layers' should clarify whether the first layer can be a prediction depth and how ties are broken.","section":"Section 2.2"},{"comment":"The figure captions refer to 'correct' and 'incorrect' without stating that these are defined by whether the model's final top label matches the ground-truth answer; this should be defined in the captions or in Section 2.","section":"Figures 1 and 2"},{"comment":"The values in parentheses are labeled 'standard error rates'; it would be clearer to state explicitly that these are standard errors of the correlation coefficient, and to report the number of samples used for each correlation (which vary widely per Table 2).","section":"Table 1"},{"comment":"The sentence 'PD correlations ... show weak positive correlations across models and datasets, with 80% below 0.300. However, the results are 97% positive' is internally awkward; a significance test on the proportion of positive correlations would be a more informative way to characterize the consistency.","section":"Section 3"},{"comment":"There is a typo in Table 3: 'OPTIMZIER' should be 'OPTIMIZER', and 'TOKENS PER STEP 2 18' is missing a superscript or spacing, making the value unclear.","section":"Appendix B.3"},{"comment":"The filtering to questions answered with a single label token may introduce selection bias, especially for models that often produce verbose answers; the paper should discuss how this filtering could affect the representativeness of the results.","section":"Appendix B.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is short and exploratory, and the authors already acknowledge in the conclusion that further testing is needed. The main issue is that the central negative claim is not supported by the displayed evidence, and the paper's own Table 1 actually points in the opposite direction. I believe this is fixable: the authors could either strengthen the analysis with quantitative alignment tests and rephrase the abstract to a more nuanced claim, or, if the quantitative analysis reveals a real effect, change the main contribution to documenting that effect. Given the breadth of the experimental setup, I see value in the work after major revision, but as it stands the conclusions outrun the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful empirical study: Tuned Lens analysis of token probability trajectories and prediction depth across 11 datasets and 5 models, with trained lenses for two Mistral models and full reproducibility details. The systematic measurement that correct and incorrect trajectories look similar in aggregate is new and extends Jiang et al. Second, the central claim is not fully supported by the paper's own evidence. Table 1 reports Pearson correlations between answer incorrectness and prediction depth; 97% are positive, many between 0.3 and 0.5. That is a consistent, statistically significant tendency for incorrect answers to commit at later layers. A difference in commitment layers is exactly a difference in inference dynamics. The aggregated trajectory plots and PD histograms can easily conceal such a shift, especially when the rise is steep and variance is large. The paper calls these correlations weak, but a systematic 0.2–0.5 correlation across dozens of model–dataset pairs is not nothing. The abstract's claim that uncertainty 'does not seem to affect inference dynamics' overstates what is shown.\n\nThe soft spots are addressable. A quantitative trajectory-similarity statistic with confidence intervals, or a per-dataset breakdown of the divergence, would fix the main gap. The incorrectness-as-epistemic-uncertainty proxy is lossy; the paper acknowledges the exploratory framing, but it should be validated against a proper uncertainty measure or at least discussed more. The Kappa–PD analysis is preliminary, and the paper correctly notes only two of five models reach significance. None of these are fatal; they are standard revision material.\n\nWho is this for? Anyone working on uncertainty detection, early exiting, or interpretability of LLMs. The negative result—that simple depth-based signals don't cleanly separate certain from uncertain outputs—is worth knowing even if the alignment is partial. It deserves a serious referee. My recommendation: send to review, but require the authors to either provide a proper quantitative alignment analysis or soften the abstract to match the actual effect sizes.","headline":"A useful empirical study whose headline claim overreaches: the paper's own Table 1 shows a systematic layer-wise shift for incorrect answers, so the 'largely aligned' conclusion needs quantitative backing.","tokens_in":14158,"tokens_out":2206,"would_cite":true,"duration_ms":24073,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across five models and eleven datasets, LLMs commit to their final answer at the same layers whether the answer is certain or uncertain.","keywords":["epistemic uncertainty","inference dynamics","Tuned Lens","prediction depth","layer-wise analysis","large language models","interpretability","multiple-choice question answering"],"falsifier":"Run the same trajectory analysis but label uncertainty with an independent signal, such as self-consistency over sampled generations or ensemble disagreement, instead of answer correctness; if high-uncertainty and low-uncertainty examples show visibly different commitment layers or trajectory shapes, the alignment claim would be overturned. A simpler check is to find any model-dataset pair where the average top-label probability trajectory for incorrect answers rises at a clearly different layer than for correct answers.","tokens_in":13126,"feed_emoji":"🧠","tokens_out":5219,"duration_ms":48741,"temperature":0.7,"pith_summary":"This paper tests whether large language models process predictions differently when they are uncertain. Using the Tuned Lens to read out token probabilities at every layer, it compares the layer-by-layer trajectories of correct and incorrect answers across five models and eleven multiple-choice datasets. The trajectories are largely aligned: both correct and incorrect outputs show a sharp increase in confidence at similar layers, so the model seems to decide on its output at the same depth regardless of uncertainty. The paper also finds a weak but consistent tendency for incorrect answers to commit slightly later, and a positive correlation between a dataset's accuracy score and the size of that commitment gap, suggesting that more competent models may adapt their inference to uncertainty to a greater degree.","feed_headline":"LLMs commit to answers at the same layers, certain or not","feed_subtitle":"Across 5 models and 11 datasets, correct and incorrect answers show aligned layer-wise probability trajectories.","key_machinery":"Tuned Lens (Belrose et al., 2023): a per-layer affine probe that converts a frozen transformer's residual-stream hidden states into vocabulary logits, giving a read-off of what the model believes at each layer. The paper uses these layer-wise logits to compute probability trajectories for the answer-label tokens and the prediction depth (Baldock et al., 2021), the first layer at which the top prediction stabilizes. The comparison of these trajectories and depth distributions for correct versus incorrect answers carries the argument.","core_discovery":"The central claim is that uncertainty, defined as incorrectness on multiple-choice questions, does not change the layer-wise inference dynamics of large language models. The probability of the final prediction token rises abruptly at the same layers for correct and incorrect answers, and the distribution of prediction depths—the layer at which the model locks in its top choice—is nearly identical for the two groups. The paper also reports that the prediction depth for incorrect answers is weakly, positively correlated with incorrectness in 97% of model-dataset pairs, and that datasets where a model is more accurate (higher Cohen's Kappa) tend to show larger prediction-depth differences between correct and incorrect answers, which the authors read as preliminary evidence that adaptability to uncertainty may emerge with competence.","pith_inferences":["If uncertainty does not change when the model commits, then any uncertainty signal must live in the content of the hidden states rather than in the timing of decisions, pointing toward representation-level detectors rather than depth-based ones.","Because correctness is a noisy proxy for epistemic uncertainty, a direct test using self-consistency or ensemble disagreement might reveal larger dynamic differences than this paper observes; the alignment result bounds the contribution of depth dynamics under the correctness proxy, not necessarily under all uncertainty definitions.","The Kappa-to-prediction-depth-difference correlation is observational; a causal test could manipulate task difficulty or add deliberate noise to questions to see whether commitment-layer shifts track induced uncertainty within the same model.","Early-exit systems that use intermediate confidence to decide when to stop might still work, since they do not require uncertainty to shift the commitment layer, only that intermediate confidence is predictive of final correctness; the alignment result does not rule that out."],"forward_implications":["Layer-wise commitment timing alone cannot serve as a simple uncertainty detector, since certain and uncertain outputs commit at nearly the same depths.","Inference is characterized by an abrupt decision event at a specific layer, largely unaffected by the model's uncertainty.","The weak positive correlation between incorrectness and prediction depth suggests a small, consistent tendency for models to commit slightly later on uncertain questions.","The positive correlation between dataset accuracy (Kappa) and the correct-versus-incorrect prediction-depth gap suggests that more competent models may develop adaptive layer usage with respect to uncertainty.","The Tuned Lens trajectory method can be used to study how uncertainty affects inference across diverse tasks and models."],"supporting_citations":[{"why":"Supplies the Tuned Lens method used to extract layer-wise probabilities from hidden states.","marker":"Belrose et al., 2023"},{"why":"Defines prediction depth, the commitment-layer measure the paper analyzes.","marker":"Baldock et al., 2021"},{"why":"Provides the epistemic-uncertainty framing and prior evidence that hidden states encode uncertainty, the contrast the paper tests.","marker":"Ahdritz et al., 2024"},{"why":"Shows that models' self-reports correlate with accuracy, a black-box counterpart to the layer-wise analysis.","marker":"Kadavath et al., 2022"},{"why":"Earlier study of inference dynamics in hallucination contexts that this work extends to epistemic uncertainty across tasks and models.","marker":"Jiang et al., 2024"}],"fun_headline_variants":["Uncertainty doesn't shift when LLMs commit","Incorrect answers lock in at the same layers as correct ones","Certainty and uncertainty show aligned layer-wise trajectories","LLMs commit at the same depth whether right or wrong"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper equates giving a wrong answer with being epistemically uncertain, so if models are sometimes confidently wrong or uncertain-but-correct, the comparison measures confidence rather than uncertainty.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty doesn't shift when LLMs commit","Incorrect answers lock in at the same layers as correct ones","Certainty and uncertainty show aligned layer-wise trajectories","LLMs commit at the same depth whether right or wrong"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001449,"raw_usage":{"total_tokens":5797,"prompt_tokens":869,"completion_tokens":4928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":4863}},"tokens_in":485,"tokens_out":4928,"duration_ms":33447,"temperature":1.0,"reasoning_tokens":4863,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:55:32.252671+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same trajectory analysis but label uncertainty with an independent signal, such as self-consistency over sampled generations or ensemble disagreement, instead of answer correctness; if high-uncertainty and low-uncertainty examples show visibly different commitment layers or trajectory shapes, the alignment claim would be overturned. A simpler check is to find any model-dataset pair where the average top-label probability trajectory for incorrect answers rises at a clearly different layer than for correct answers.","supporting_citations":[{"cited_title":"Deep learning through the lens of example difficulty","cited_arxiv_id":null,"evidence_quote":"Defines prediction depth, the commitment-layer measure the paper analyzes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the epistemic-uncertainty framing and prior evidence that hidden states encode uncertainty, the contrast the paper tests."}],"review_version":1}