{"id":"f3b38b02-6da9-4a02-b7ba-bf13d8400eff","arxiv_id":"2505.16894","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Context injection shifts LLM hidden states and attention, and the paper claims this drift tracks and eventually \"locks in\" hallucination behavior, but the reported data contain internal inconsistencies.","lead":"An empirical study tracks how incremental context injection shifts hidden states and attention in LLMs, linking that drift to rising hallucination rates across six open-source models. The paper proposes an \"attention-locking\" threshold as a marker of when hallucinations become resistant to correction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported JS-Drift saturation at ~0.69 matches the mathematical limit of Jensen-Shannon divergence between zero-padded distributions with disjoint supports, so the attention-locking threshold is likely an artifact of Eq. (14).","rationale":"The reader's weakest assumption identified exactly the same load-bearing concern: the JS/Spearman convergence in Table 2 may be an artifact of zero-padding in Eq. (14). I agree, and the numerical agreement of the asymptotic JS-Drift with log 2 is a particularly strong telltale sign. The claim that ~0.69 marks an attention-locking threshold is central to the abstract, Section 4.3, and Corollary 4, so if the saturation is a mathematical property of the metric, the paper's headline conclusion collapses. The near-identical values across six models in Table 2 further support the artifact interpretation. A simple length-invariant recomputation or null-model control would settle the question. Since the paper as presented does not rule out this artifact and provides no correction-resistance experiment, the reader's rejection remains appropriate.","tokens_in":14369,"tokens_out":3601,"duration_ms":29478,"concrete_test":"Recompute Table 2 with a length-invariant comparison: restrict both attention distributions to the intersection of token positions, or use an alignment that does not pad with epsilon, while also running a null control with random attention vectors of the same growing lengths through Eqs. (13)-(14). If the null control also saturates near 0.69 and Spearman near 0, the threshold is a padding artifact. Separately, test correctability: apply a corrective prompt after round 3 and after round 10 and measure correction success; if correction success does not drop after JS/Spearman saturate, Corollary 4 fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract, Section 4.3, Corollary 4) is that convergence of JS-Drift to ~0.69 and Spearman-Drift to ~0 marks an attention-locking threshold beyond which hallucinations solidify and resist correction. This threshold is not supported because the metric itself plausibly produces the saturation. Equation (14) zero-pads attention distributions of different lengths and renormalizes. As context grows each round, the overlap between the baseline support and the current support shrinks. The JS divergence between two distributions with nearly disjoint supports tends to log 2 = 0.6931; Table 2's asymptotic values (~0.6913) are numerically consistent with this limit rather than with any model-specific attention lock. Spearman-Drift on a padded vector with a growing block of epsilon entries likewise tends to ~0 as tied ranks dominate the correlation. This also explains why Table 2 shows near-identical JS/Spearman values across six different models: the artifact is model-independent. No correction-resistance experiment is reported, so Corollary 4's 'solidified and resistant to correction' is asserted without direct evidence. The attention-locking threshold is therefore not empirically established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how incremental context injection affects hallucination rates and internal representations in six open-source LLMs. It constructs two 16-round tracks per TruthfulQA question (relevant but partially flawed snippets versus irrelevant misleading snippets), measures QA-level and sentence-level hallucination rates with a tri-perspective detector, and tracks cosine, entropy, JS, and Spearman drifts of hidden states and attention maps. The central claim is that hallucination rates grow monotonically and saturate once an 'attention-locking threshold' (JS-Drift ≈ 0.69, Spearman-Drift ≈ 0) is reached, after which hallucinations become resistant to correction. The paper also reports a seesaw between semantic assimilation and attention diffusion that varies with model size.","tokens_in":14658,"tokens_out":8217,"duration_ms":56203,"significance":"If the central claims were supported, the paper would supply an interpretable internal-state signature for predicting when hallucinations resist correction, with practical implications for mitigation. The controlled dual-track design across six models and the attempt to couple overt error rates with covert representation drift are appropriate and potentially useful. However, the principal quantitative claims are not supported by the reported data: the headline monotonicity is violated in Table 1, and the attention-locking threshold is numerically consistent with an artifact of the zero-padding procedure in Eq. (14). The absence of any correction-resistance experiment means the abstract's central behavioral conclusion is asserted rather than demonstrated. The paper does not ship code, data, or machine-checked artifacts, and key tables are available only on request, further limiting verification.","major_comments":[{"comment":"The claim that hallucination rates 'rise monotonically and saturate' is contradicted by the paper's own table. For Qwen2.5-7B relevant QA-HallucRate, the reported values are 0.86 (round 1), 0.90 (round 3), 0.94 (round 5), 0.92 (round 7), 0.92 (round 9), 0.90 (round 11), and 0.92 (round 15); for Llama3-8B relevant, they are 0.90, 0.86, 0.88, 0.88, 0.90, 0.90, 0.90. These trajectories are non-monotonic and include decreases of 0.02–0.04 between consecutive reported rounds. The abstract and Section 5 state monotonic growth, so this load-bearing claim needs to be revised or the data reanalyzed before the paper's behavioral conclusions can be accepted.","section":"§5, Table 1"},{"comment":"The attention-locking threshold is not established because the JS-Drift metric, as defined, is dominated by the padding/support-size artifact. Zero-padding distributions of different lengths and renormalizing compares a baseline distribution over the initial token set with a current distribution over a growing token set; as the overlap of effective supports shrinks, the JS divergence approaches log 2 ≈ 0.6931, which is numerically consistent with the reported saturation values (0.6913–0.6916) across all six models. The near-identical JS-Drift values across models (e.g., Qwen2.5-7B and Qwen2.5-1.5B have identical rounded values 0.6861, 0.6886, 0.6897, 0.6903, 0.6907, 0.6913 from round 3 onward) and the Spearman-Drift values tending to zero are exactly what one expects from a padded-vector comparison with a growing block of near-zero entries, rather than from model-specific attention locking. The paper needs an alignment or renormalization procedure that separates genuine attention change from support growth, or an independent validation of the threshold.","section":"§3.3, Eq. (14); Table 2"},{"comment":"No experiment tests correction resistance. Corollary 4 asserts that once JS-Drift saturates and Spearman-Drift approaches zero, hallucinations become solidified and conventional dialogue correction methods become ineffective, but the manuscript reports no measurement of correction effectiveness before versus after the threshold. The sentence in Section 4.3 that 'the effectiveness of injected corrective information significantly decreases' is an interpretation, not a result. Since the abstract's third main claim is exactly this correction-resistance property, it requires a direct experiment, for example injecting corrective statements at pre-threshold and post-threshold rounds and measuring whether the model's answers change.","section":"§4.3, Corollary 4"},{"comment":"Several correlational and threshold claims are circular or under-validated. The drift metrics in Table 2 are computed on the same six models and the same rounds that are then used to define the 'seesaw' correlation (ρ ≈ −0.71, p < 0.01), with no independence, confidence intervals, or correction for multiple comparisons across the many reported correlations. Corollary 3 equates high metric variance with 'high-confidence self-consistent hallucinations,' but no confidence or self-consistency measurement is reported. In addition, the hallucination detector uses θsem = 0.7 described as 'empirically chosen' (Eq. 4), with no sensitivity analysis, no human validation of the logical-OR label, and no reported sample size N for the rates in Table 1. These issues affect the paper's behavioral metrics as well as its internal-state claims.","section":"§4.3, Corollaries 3 and 6; §3.2, Eq. (4)"}],"minor_comments":[{"comment":"The metric is defined as Intra-HallucRate in Eq. (9) but is called inter-HallucRate in Section 4.2 and Table 1; please standardize the terminology.","section":"§3.2, Eq. (9); §4.2; Table 1"},{"comment":"Equation (14) is ambiguous: it sets Ppad(i) = P(i) for i < |P| and epsilon otherwise, then says 'followed by renormalization'; it should specify whether epsilon is added to every position before renormalization or only to the padded positions, and how ties are handled for the Spearman rank computation.","section":"§3.3, Eq. (14)"},{"comment":"Figures 2 and 3 show detailed drift curves only for Llama3-8B and Llama3.2-1B; the claims about trends across all six models would be easier to verify if the same plots were provided for the other four models or if the per-round values were released.","section":"§4.3, Figures 2–3"},{"comment":"The complete tables are 'available upon request from the authors'; for reproducibility, the data and generation code should be released alongside the paper.","section":"§4.2, Table 1"}],"recommendation":"reject","confidential_remarks":"The stress-test concern about Eq. (14) is, in my reading, well-founded: the reported JS-Drift saturation is numerically compatible with the log 2 limit of the zero-padding procedure, and the near-identical values across different models point to a metric artifact rather than a model property. The monotonicity claim is also directly contradicted by Table 1, and the correction-resistance conclusion is not tested. These are load-bearing issues that cannot be fixed by local editing; the authors would need to reanalyze the data with token-aligned metrics and add a direct correction experiment before the central claims could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline result—an 'attention-locking threshold' at JS-Drift ≈ 0.69—is almost certainly an artifact of Eq. (14). Zero-padding the baseline attention vector with ϵ = 10⁻¹² and renormalizing forces the JS divergence toward log 2 = 0.6931 as the context grows and the supports become nearly disjoint. Table 2's asymptotic values (0.6913/0.6916) sit right on that limit. The identical Spearman-Drift columns across different models (e.g., Llama3-8B and Llama3.2-1B) confirm the metric is not capturing model-specific attention; the growing block of padded ϵ entries washes out rank information.\n\nWhat is genuinely new here is the 16-round relevant/irrelevant context titration protocol, and the idea of watching both overt hallucination rates and internal drift over rounds. That is a reasonable design, and the paper is transparent about citing prior internal-state work (Azaria & Mitchell, Duan et al., Wang et al.). The writing is clear.\n\nBut the load-bearing claims do not survive contact with the paper's own tables. Table 1 shows non-monotonic hallucination rates: Qwen2.5-7B relevant goes 0.94 (round 5), 0.92 (round 7), 0.90 (round 11); Llama3-8B relevant goes 0.90 (round 1), 0.86 (round 3). So 'monotonic growth' is false. Corollary 4's 'resistant to correction' is asserted without any correction-resistance experiment. The detector threshold θ_sem = 0.7 is uncalibrated, and the full data are withheld 'upon request.' The seesaw correlation (ρ ≈ −0.71) is computed on the same rounds that define the drift metrics, so it is partly circular.\n\nThese are load-bearing flaws, not nits. The protocol could be salvageable if the authors recompute drift on overlapping token sets or with a proper projection, and if they actually test correction resistance. But as it stands, the quantitative core is not evidence for the proposed mechanism.\n\nMy take: reject in current form. It is a useful reading-group example of how padding artifacts can manufacture signals, but I would not cite it. If the authors fix the metric and release the data, the protocol deserves another look. I would not send this to peer review as-is; the artifact is diagnosable in a sentence, and the data are not available for verification.","headline":"The 'attention-locking threshold' is a padding artifact: JS-Drift saturates at log 2 because Eq. (14) zero-pads and renormalizes, and the paper's own Table 1 contradicts the claimed monotonic rise in hallucination rates.","tokens_in":15153,"tokens_out":4266,"would_cite":false,"duration_ms":33206,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM hallucinations harden once attention distributions stop changing, and two drift statistics mark that threshold.","keywords":["hallucination","large language models","attention drift","internal representation","TruthfulQA","context perturbation","hallucination detection","attention locking"],"falsifier":"Run the same 15-round titration with content-free or randomly shuffled context snippets while keeping token counts identical to the real tracks; if JS-Drift still converges to about 0.69 and Spearman-Drift to about zero across all six models, the attention-locking threshold is an artifact of support-size change and padding rather than a cognitive-state signature. A cheaper check is to compute the drift metrics against a fixed-length reference distribution without zero-padding and see whether the universal saturation values persist.","tokens_in":14176,"feed_emoji":"🎯","tokens_out":5135,"duration_ms":39835,"temperature":0.7,"pith_summary":"The paper sets out to show that hallucinations in large language models are not random failures but the visible endpoint of a measurable internal process: as context is added round by round, hidden representations and attention distributions drift, hallucination rates climb, and then the system settles into a state the authors call “attention locking.” At that point, two statistics—JS-Drift near 0.69 and Spearman-Drift near zero—stop changing, and the errors they accompany become resistant to correction. This matters because it offers a candidate internal signature for predicting when a model is about to hallucinate and when correction will no longer work, which could support detection and mitigation before errors solidify.","feed_headline":"Drift metrics mark when LLM hallucinations harden","feed_subtitle":"Hallucination rates saturate as JS and Spearman drift converge across six open-source models.","key_machinery":"The load-bearing object is the pair of attention-drift statistics computed from zero-padded attention distributions. JS-Drift is the Jensen–Shannon divergence between baseline and context-injected attention, bounded in $[0,1]$; Spearman-Drift is the Spearman rank correlation between the same distributions, bounded in $[-1,1]$. Zero-padding with renormalization lets the authors compare attention over different token sets as context grows. Convergence of JS-Drift near $0.69$ and Spearman-Drift near zero is read as attention locking, the point where attention re-ranking stabilizes and the model’s error pattern hardens; together with cosine and entropy drift it forms the internal-state monitoring pipeline that links overt hallucination flags to hidden-state dynamics.","core_discovery":"Across six open-source transformer models answering TruthfulQA questions, the authors build two 16-round “titration” tracks per question: one appends relevant but partially flawed snippets and the other injects misleading content. They report that overt hallucination rates rise monotonically with context accumulation and plateau after roughly five to seven rounds, while four drift metrics—cosine and entropy drift of hidden states, plus Jensen–Shannon and Spearman drift of attention distributions—also increase and stabilize. The central discovery is the synchronous convergence of JS-Drift (about $0.69$) and Spearman-Drift (about zero) across all models, which is interpreted as an “attention-locking” threshold: once attention topology stops changing, hallucinations solidify and conventional dialogue-based correction becomes ineffective. The paper also distinguishes two error modes—high-confidence, self-consistent hallucinations under relevant context and topic-drift hallucinations under irrelevant context—and reports a negative correlation between assimilation capacity and attention diffusion across model sizes.","pith_inferences":["If the threshold is a genuine state property, it suggests a practical early-warning rule: monitor JS-Drift and Spearman-Drift during context construction and stop injecting before convergence, rather than attempting corrections after lock-in.","The near-identical saturation values across models and contexts raise the possibility that they are set by the zero-padding and renormalization procedure rather than by the model’s attention state; a direct test is to run the same protocol with randomly shuffled or content-free context snippets.","The same measurement could transfer to retrieval-augmented generation, where appended retrieved passages form a natural titration track; the paper does not test this setting, but its protocol would apply directly.","Because the hallucination detector uses empirically chosen thresholds such as a BERTScore cutoff of 0.7, the reported rates might shift under other judges; cross-detector replication would tighten the mapping from drift to error."],"forward_implications":["Hallucination frequency rises monotonically with accumulated context and saturates after 5–7 rounds, so the first few injected snippets carry disproportionate risk.","Relevant but partially flawed context drives deep semantic assimilation, producing high-confidence, self-consistent hallucinations, whereas irrelevant context produces topic-drift errors anchored by attention re-routing.","The attention-locking threshold (JS-Drift near 0.69, Spearman-Drift near 0) marks the point where hallucinated content becomes resistant to conventional dialogue correction.","The negative correlation between assimilation capacity and attention diffusion implies larger models absorb biased context more thoroughly while smaller models compensate by scattering attention and producing looser patchwork errors.","Internal drift metrics are candidate intrinsic precursors for hallucination prediction and context-aware mitigation."],"supporting_citations":[{"why":"Supplies the TruthfulQA benchmark and question set that the controlled-context injection protocol is built on.","marker":"[20]"},{"why":"Establishes that LLM hidden states carry signals about truthfulness, motivating the internal-state drift analysis.","marker":"[1]"},{"why":"Shows trajectory differences between correct and hallucinated answers, supporting representation drift as a hallucination indicator.","marker":"[9]"},{"why":"Demonstrates that mid-layer attention and feed-forward activations can predict hallucinations, grounding the attention-monitoring approach.","marker":"[29]"},{"why":"Shows attention backtracking detects and mitigates conflict-type hallucinations, supporting attention topology as a mechanism.","marker":"[8]"},{"why":"Links high entropy and sampling variance to hallucination risk, justifying the attention-entropy drift metric.","marker":"[17]"},{"why":"Provides evidence that context influence and sequence perturbation affect hallucination, supporting the context-injection design.","marker":"[13]"}],"fun_headline_variants":["Attention locking marks when LLM hallucinations harden","JS and Spearman drift converge as hallucinations plateau","Context titration reveals when LLM errors become uncorrectable","Drift thresholds predict solidification of LLM hallucinations","Relevant context breeds confident, hardened hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes that the convergence of the two attention-drift measures reflects the model’s internal state, not just the arithmetic of padding and renormalizing attention vectors of different lengths.","fun_headline_variants_meta":{"raw":{"variants":["Attention locking marks when LLM hallucinations harden","JS and Spearman drift converge as hallucinations plateau","Context titration reveals when LLM errors become uncorrectable","Drift thresholds predict solidification of LLM hallucinations","Relevant context breeds confident, hardened hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1355,"prompt_tokens":976,"completion_tokens":379,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":306}},"tokens_in":592,"tokens_out":379,"duration_ms":3128,"temperature":1.0,"reasoning_tokens":306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:53:16.471746+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 15-round titration with content-free or randomly shuffled context snippets while keeping token counts identical to the real tracks; if JS-Drift still converges to about 0.69 and Spearman-Drift to about zero across all six models, the attention-locking threshold is an artifact of support-size change and padding rather than a cognitive-state signature. A cheaper check is to compute the drift metrics against a fixed-length reference distribution without zero-padding and see whether the universal saturation values persist.","supporting_citations":[{"cited_title":"In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","cited_arxiv_id":null,"evidence_quote":"Supplies the TruthfulQA benchmark and question set that the controlled-context injection protocol is built on."},{"cited_title":"In: Findings of the Association for Computational Linguistics: EMNLP 2023","cited_arxiv_id":null,"evidence_quote":"Establishes that LLM hidden states carry signals about truthfulness, motivating the internal-state drift analysis."},{"cited_title":"What are Models Thinking about? Understanding Large Language Model Hallucinations \"Psychology\" through Model Inner State Analysis","cited_arxiv_id":"2502.13490","evidence_quote":"Demonstrates that mid-layer attention and feed-forward activations can predict hallucinations, grounding the attention-monitoring approach."},{"cited_title":"In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing","cited_arxiv_id":null,"evidence_quote":"Shows attention backtracking detects and mitigates conflict-type hallucinations, supporting attention topology as a mechanism."},{"cited_title":"IEEE Transactions on Software Engineering 51(2), 413–429 (2025)","cited_arxiv_id":null,"evidence_quote":"Links high entropy and sampling variance to hallucination risk, justifying the attention-entropy drift metric."},{"cited_title":"arXiv preprint arXiv:2410.03026 (2024)","cited_arxiv_id":null,"evidence_quote":"Provides evidence that context influence and sequence perturbation affect hallucination, supporting the context-injection design."}],"review_version":1}