{"id":"dd4ed5f0-edbb-4468-b2a2-901f0c655c06","arxiv_id":"2504.15630","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CaLE improves context-faithful QA by amplifying an intermediate transformer layer that carries the most V-usable contextual information.","lead":"The paper proposes CaLE, a method that finds the LLM layer where a context's answer is most detectable and amplifies that layer's hidden states during inference. It reports improved exact-match scores on question answering with conflicting or unknown context across Llama, Mistral, and Gemma models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CaLE's formal guarantee requires the context-correct token to be logit-lens top-1 at the chosen layer; the paper admits this is unguaranteed, leaving the stated enhancement mechanism unverified for many examples.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: CaLE's theoretical guarantee depends on the correct answer being the argmax of the logit-lens distribution at the selected layer, a condition that is never verified and is explicitly disclaimed in the paper. My stress-test agrees with this diagnosis. I considered whether the concern should move the verdict further toward REJECT, but the paper's contribution is primarily empirical: CaLE consistently improves EM/F1 over strong baselines across multiple models and datasets, and a weak or incomplete theoretical justification does not by itself invalidate those measurements. The empirical claim remains plausible and testable, but the absence of code, error bars, and per-example analysis makes the mechanism unverified. Therefore the conditional verdict should stand unchanged. The proposed concrete test directly settles the disputed premise by stratifying gains according to whether the selected layer's logit lens ranks the context answer first, and a secondary LayerNorm control isolates the ignored final normalization. If the test shows gains only under the argmax condition, the theory is supported; if gains appear without it, the paper's stated mechanism would need to be revised even though the intervention may still work.","tokens_in":18743,"tokens_out":4234,"duration_ms":45808,"concrete_test":"Run logit-lens analysis on the exact models, datasets, and selected layers of Tables 1, 3, and 4. For each test example, before applying CaLE, record the rank and probability of the gold/context-answer token at the chosen layer l, and record whether k = argmax_j v_j holds. Then stratify the EM/F1 gain of CaLE-A and CaLE-R relative to original decoding by this condition. If gains are concentrated on examples where k is already top-1 or top-5 at l and are near zero or negative otherwise, the proposed mechanism is confirmed. If gains persist on examples where k is not top-ranked, Proposition 3.1's premise is not necessary and the explanation must change. As a secondary check, compare CaLE-A against directly scaling the final hidden state immediately before the final LayerNorm; if the gains vanish, the ignored LayerNorm in Appendix C is materially involved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CaLE's central claim is that amplifying a layer identified by V-usable information or KL divergence enriches context-correct information in the final layer. The only formal support, Proposition 3.1, requires k = argmax_j v_j at the chosen layer, and it also ignores the final LayerNorm (Appendix C). The paper explicitly disclaims the premise: \"we cannot guarantee that vk will achieve the maximum probability proportion at a specific layer.\" Neither the supervised validation selection (Eq. 11) nor the unsupervised KL_c criterion (Eq. 18) checks the per-example rank of the context-correct answer token at the selected layer. If the correct answer is not top-ranked at layer l, amplifying alpha*v sharpens the wrong token, so the mechanism should actively hurt context-faithfulness. The reported aggregate EM/F1 gains therefore remain unexplained at the per-example level: they could come only from examples where the answer is already top-ranked at l, or from generic entropy reduction rather than from \"amplifying contextual information.\" The KL_c approximation in Eqs. 14-17 measures a distributional shift induced by context, not the rank of the answer token, so it cannot rescue the missing premise. This is an internal gap between the stated theory and the mechanism claimed in the abstract, not merely a disagreement with an external consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CaLE, a training-free inference-time intervention for improving context-faithful QA. CaLE identifies a 'context-aware layer' using either a supervised validation-accuracy criterion (Eq. 11) or an unsupervised KL-divergence criterion (Eq. 18), then enhances that layer's hidden states by amplifying them (CaLE-A) or adding a residual path (CaLE-R) before continuing the forward pass. The method is evaluated on CounterFact, NQ, NQ-Swap, SQuAD, and StrategyQA across Llama, Mistral, and Gemma models, reporting consistent EM/F1 improvements over original decoding, early exit, and IRCAN, with further gains when combined with contrastive decoding methods. The theoretical support (Proposition 3.1) claims that the final-layer conditional V-entropy tends to zero when the chosen layer's logit-lens distribution has the correct answer token at its maximum.","tokens_in":18985,"tokens_out":9398,"duration_ms":83683,"significance":"The empirical finding is potentially useful: if the reported gains hold, CaLE offers a simple, plug-in intervention that improves context faithfulness without fine-tuning, and its orthogonality to contrastive decoding is a practical advantage. Strengths include the breadth of the evaluation (five datasets, five model families), the ablation over alpha and intervention location, and the layer-consistency analysis for supervised selection. However, the paper's stated mechanism is not established: the formal guarantee hinges on a condition the authors admit is unverifiable, and the correlation supporting the unsupervised metric is not quantified. The independent EM/F1 evidence is the main support, and it could be made stronger with error bars and a clearly described hyperparameter selection protocol.","major_comments":[{"comment":"Proposition 3.1 requires k = arg max_j v_j at the chosen layer, a condition that the paper explicitly states cannot be guaranteed in Section 3.1.1, and neither the supervised layer selector in Eq. (11) nor the unsupervised KL_c selector in Eq. (18) checks the rank of the context-correct answer token at the selected layer. For any example where the correct token is not top-1 at layer l, amplifying by alpha > 1 sharpens the wrong token, so the stated mechanism predicts degradation rather than improvement; the aggregate gains in Tables 1 and 3 therefore remain unexplained at the per-example level. I ask the authors to report the frequency with which the argmax condition holds at the selected layers, or to replace the global theoretical claim with a conditional statement and empirical diagnostics.","section":"Section 3.1.1 (Prop. 3.1; App. E)"},{"comment":"The proof of Proposition 3.1 is incomplete as written. Appendix C ignores the final LayerNorm, and Appendix E asserts that the terms u_j(alpha v_j) - u_k(alpha v_k) are negligible without providing a bound; footnote 5 states 'Since alpha approaches zero' when the limit taken is alpha -> infinity. Because the subsequent-layer contributions u(alpha v) may themselves grow with alpha, the claimed exponential decay of the j != k terms is not established. The authors should provide a rigorous treatment of the u terms, for example by bounding |u_j(alpha v) - u_k(alpha v)| relative to alpha(v_j - v_k), or state the additional assumptions needed for the proof to go through.","section":"Appendix C and Appendix E"},{"comment":"The hyperparameters alpha1 and alpha2 are listed as fixed values (5/3 for CounterFact and 3/1 for other datasets) without any statement that they were selected on a held-out set. The ablation in Figure 6 shows that performance on CounterFact varies substantially with alpha, and the chosen alpha1 = 5 sits at the upper edge of the best-performing range, suggesting that the test set was used to select alpha. If so, the results in Tables 1 and 3 are optimistically biased. Please report the selection procedure explicitly or provide a per-dataset sensitivity analysis demonstrating that the improvements are robust across a range of alpha values.","section":"Section 4.1, footnote; Figure 6"},{"comment":"The main empirical claims are made without error bars, confidence intervals, or significance tests. Some of the improvements over the strongest baseline are modest (e.g., Table 3, SQuAD for Llama3.1-8B: supervised CaLE-A EM 67.38 vs. IRCAN 64.58), and the reader cannot tell whether these differences are consistent across examples. Reporting bootstrap intervals or paired tests over the test items would materially strengthen the paper's central claim.","section":"Tables 1-3"}],"minor_comments":[{"comment":"There are numerous typos: 'Unkown' (Appendix F.1), 'almostly' (Section 4.3), 'negative effective' (Section 1), 'lay' instead of 'layer' (Section 3.2.1), and 'faithfulnes' (Section 3).","section":"Throughout"},{"comment":"The caption says '-HV' but the axis label is '-H(Y|h)'; please unify the notation.","section":"Figure 2"},{"comment":"The symbol v is reused in Eq. (7) for the logits at layer l after being defined in Eq. (4) for a generic layer; clarify the notation to avoid confusion.","section":"Eq. (7)"},{"comment":"The unsupervised KL_c criterion in Eq. (18) is evaluated on the test set inputs; this transductive use of test inputs should be acknowledged, and the paper should state whether the layer would be selected from a separate unlabeled set in a deployment scenario.","section":"Section 3.2.2"},{"comment":"The footnote in the proof says 'Since alpha approaches zero' but the limit is alpha -> infinity; this appears to be a typo and should be corrected.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical results are likely to be of interest, but the theoretical framing is oversold. The authors should be asked to either verify the argmax condition empirically or substantially weaken the theoretical claim. The lack of a clear hyperparameter selection protocol is a particular concern for the multi-dataset results; I would request the code or at least a complete sensitivity analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper: it reports consistent EM gains from a cheap, training-free intervention across several model families and QA datasets, and it explains those gains with a mechanism that is not actually proven. The intervention, CaLE, picks a middle layer using V-usable information (or a KL surrogate) and then either amplifies that layer's hidden state or adds a residual connection from it to later layers. The gains are real-looking—up to several EM points over early exit and IRCAN, especially on conflicting-knowledge subsets like NQ-Swap.\n\nThe genuinely new piece is the layer-selection signal combined with the two enhancement operations. The observation that contextual information is non-monotonic across layers, with a peak in the middle then degradation, is concrete and shown in their Figure 2. The paper is clearly written, and the ablations over alpha, intervention layer, and component type (hidden state vs attention vs MLP) are useful. It's also honest: they explicitly admit they cannot guarantee that the target token is top-1 at the chosen layer, and they say the final LayerNorm is ignored for analytical simplicity.\n\nThat admission cuts against their own Proposition 3.1, which requires k = argmax_j v_j. If the target token isn't top-ranked at that layer, amplifying alpha*v sharpens whatever is on top—often the parametric wrong answer. The KL_c metric used for unsupervised selection measures distributional shift, not the rank of the answer token, so it doesn't rescue the premise. In short, the stated mechanism—amplifying contextual information to enrich the final layer—is not established per example. The gains could come from cases where the answer is already top-1 at the selected layer, or from generic sharpening.\n\nOther soft spots: no error bars or significance tests, per-dataset alpha values chosen after inspecting ablations, and no code shipped. These are fixable but real. The core empirical claim is still plausible; the method consistently helps, and the paper should be read as an empirical contribution, not a mechanistic proof.\n\nI'd send this to peer review. The results deserve scrutiny and replication, and the authors have been transparent about the theoretical gap. With error bars, a fixed hyperparameter protocol, and perhaps a per-example analysis of when the intervention helps or hurts, this could be a solid workshop or main-conference paper. The audience is people working on RAG faithfulness, decoding interventions, and interpretability.","headline":"A simple, training-free layer intervention that consistently improves context-faithful QA, but the stated mechanism is shakier than the results.","tokens_in":19558,"tokens_out":2032,"would_cite":true,"duration_ms":20332,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Amplifying one mid-layer hidden state makes LLMs answer from context, not memory.","keywords":["context-faithfulness","V-usable information","layer enhancement","knowledge conflicts","hidden state intervention","logit lens","question answering","large language models"],"falsifier":"Take a CounterFact example whose logit lens at the layer CaLE selects ranks the parametric wrong answer above the context-correct answer, then apply CaLE-A at $\\alpha = 2,4,8,16$; if the final softmax probability of the wrong answer rises with $\\alpha$, the premise that the correct token is the layer's argmax is violated and the claimed mechanism is refuted for that case.","tokens_in":1650,"feed_emoji":"🧠","tokens_out":1670,"duration_ms":78142,"temperature":0.7,"pith_summary":"Large language models often answer from memory instead of the supplied context, and existing fixes work at the decoding level without touching how context moves through the model. This paper argues that the context-correct answer is actually present in an intermediate layer's hidden state, then gets diluted in deeper layers, and introduces CaLE, an inference-time intervention that finds that layer and amplifies it. Using V-usable information to pick the layer, CaLE scales the hidden state (or adds a residual connection) so the final representation is enriched with contextual knowledge. On QA benchmarks with unknown or conflicting contexts, CaLE reports exact-match improvements over original decoding, early exit, and IRCAN across multiple Llama, Mistral, and Gemma models, and it stacks with contrastive decoding strategies.","feed_headline":"Amplify one hidden layer to override LLM parametric memory","feed_subtitle":"CaLE finds the layer holding the context answer, scales it up, and beats decoding baselines on conflicting QA.","key_machinery":"The machinery is V-usable information estimated by a logit lens: $I_V(h_l\\to Y)=H_V(Y)-H_V(Y|h_l)$, where the conditional V-entropy is computed from a softmax over logit-lens logits at layer $l$. This identifies an optimal layer before contextual information plateaus or decays. The intervention uses the residual-stream decomposition of final logits as $v+u(v)$, where $v=h_l W_U$ and $u(v)$ collects contributions from later layers; amplifying to $\\alpha h_l$ turns the final logits into $\\alpha v+u(\\alpha v)$. Proposition 3.1 states that if $k=\\arg\\max_j v_j$, then $\\lim_{\\alpha\\to\\infty} H_{V,f}(\\alpha)\\approx 0$, so the final distribution collapses onto the context-correct token. The residual-connection variant is shown to behave like an effective $(\\alpha+1)$ scaling.","core_discovery":"The core claim is that context-faithfulness is impaired not by a lack of contextual information but by its attenuation in deeper layers: the logit-lens distribution at some intermediate layer already ranks the context-correct answer near the top, and later layers dilute it. CaLE therefore locates that layer, using V-usable information in a supervised setting and a KL-divergence proxy in an unsupervised setting, and amplifies its hidden states (CaLE-A) or adds a residual connection from it (CaLE-R). The paper proves that if the correct token is the argmax of the layer-l logit vector, then as the amplification factor grows, the conditional V-entropy at the final layer tends to zero, so the final softmax concentrates on the context answer. Experiments on CounterFact, NQ, NQ-Swap, SQuAD, and StrategyQA show exact-match, F1, or accuracy gains over original decoding, early exit, and IRCAN across Llama, Mistral, and Gemma models, with the largest gains when the context conflicts with parametric knowledge.","pith_inferences":["An extension the paper leaves implicit is a per-instance layer selector: compute the KL-based proxy on each input and amplify only when the context-correct token is already near the top of that layer's logit-lens distribution, making the intervention adaptive rather than dataset-wide.","Because CaLE is orthogonal to decoding strategies, a natural next test is combining it with neuron-reweighting methods such as IRCAN rather than only with contrastive decoding; the paper tests CaLE with CAD, CD, and COIECD but not with IRCAN plus CaLE.","If the mechanism holds, it offers a diagnostic: an LLM that ignores context may have the correct answer localized at a specific layer and then diluted, so probing that layer could predict which inputs will be answered from memory rather than from the supplied evidence."],"forward_implications":["CaLE-A and CaLE-R both improve exact match over original decoding, early exit, and IRCAN on CounterFact, with the largest gains on the subset where context contradicts parametric knowledge.","On NQ-Swap, a dataset built entirely from conflicting contexts, CaLE raises exact match substantially across Llama, Mistral, and Gemma models, showing the intervention specifically helps when context should override memory.","Unsupervised layer selection via KL divergence matches or approaches supervised selection and still beats all baselines, so CaLE does not require labeled validation data.","CaLE composes with contrastive decoding methods such as CAD, CD, and COIECD, yielding cumulative gains over either intervention alone.","The best intervention layer is consistently in the middle-to-deep range, and amplifying hidden states helps while amplifying attention or MLP outputs does not."],"supporting_citations":[{"why":"Introduces V-usable information, the quantity CaLE uses to measure how much a layer's hidden state helps predict the answer.","marker":"[Xu et al., 2020]"},{"why":"Provides the V-entropy estimators used to compute $I_V$ layer by layer.","marker":"[Ethayarajh et al., 2022]"},{"why":"Logit lens, the decoding of hidden states into vocabulary distributions that underlies the layer-wise analysis.","marker":"[nostalgebraist, 2020]"},{"why":"Residual-stream decomposition that turns final logits into $v+u(v)$, the linear form the amplification proof relies on.","marker":"[Elhage et al., 2021]"},{"why":"Reports that intermediate layers can be more informative than the final layer, motivating intervention before information degrades.","marker":"[Skean et al., 2024]"},{"why":"IRCAN, the neuron-reweighting baseline CaLE is compared against.","marker":"[Shi et al., 2024a]"},{"why":"CounterFact dataset used for the information-flow analysis and the main evaluation.","marker":"[Meng et al., 2022a]"},{"why":"Natural Questions dataset, one of the open-domain QA benchmarks.","marker":"[Kwiatkowski et al., 2019]"},{"why":"NQ-Swap, the entity-swapped conflicting-context benchmark where CaLE's gains are largest.","marker":"[Longpre et al., 2021]"},{"why":"StrategyQA, the reasoning benchmark with implicit multi-step evidence.","marker":"[Geva et al., 2021a]"}],"fun_headline_variants":["Amplify the right layer to beat LLM memory bias","CaLE finds the context layer, scales it up, wins QA","One hidden layer: the switch to context over priors","Targeted layer boost boosts context-faithful answers","Context wins when you amplify the layer that has it"],"cache_read_input_tokens":21632,"weakest_assumption_plain":"CaLE assumes the layer it amplifies already ranks the context-correct answer at or near the top of its logit-lens distribution and that the scaled hidden state reaches the final logits linearly; when the parametric wrong answer is on top, amplification strengthens the wrong prediction.","fun_headline_variants_meta":{"raw":{"variants":["Amplify the right layer to beat LLM memory bias","CaLE finds the context layer, scales it up, wins QA","One hidden layer: the switch to context over priors","Targeted layer boost boosts context-faithful answers","Context wins when you amplify the layer that has it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000295,"raw_usage":{"total_tokens":1694,"prompt_tokens":905,"completion_tokens":789,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":707}},"tokens_in":521,"tokens_out":789,"duration_ms":6606,"temperature":1.0,"reasoning_tokens":707,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:21:58.656142+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a CounterFact example whose logit lens at the layer CaLE selects ranks the parametric wrong answer above the context-correct answer, then apply CaLE-A at $\\alpha = 2,4,8,16$; if the final softmax probability of the wrong answer rises with $\\alpha$, the premise that the correct token is the layer's argmax is violated and the claimed mechanism is refuted for that case.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the V-entropy estimators used to compute $I_V$ layer by layer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports that intermediate layers can be more informative than the final layer, motivating intervention before information degrades."}],"review_version":1}