{"id":"69ef421c-6ee5-436e-95fa-cd81a459bafb","arxiv_id":"2504.12585","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLMs often compute correct answers to simple deterministic tasks in their internal representations, and light fine-tuning of early layers lets them access that information by suppressing a learned prior over outputs.","lead":"This paper shows that LLM failures on simple deterministic tasks (counting, ciphers, acronyms) can be reduced by prompting the model to ignore prior knowledge, and even more by lightly fine-tuning early layers. It suggests models often internally contain the correct answer but are steered toward common tokens by a learned prior.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's 'Finetune-layer' row selects the best of 32 layers on the validation split, inflating the reported 72.3–99.9% accuracies; without a nested test split or a fixed/prior-localized layer control, the headline held-out accuracy and the localization story are not established.","rationale":"I focused on the finetune-layer protocol because it is the quantitative backbone of the paper's two strongest claims: that high held-out accuracy can be achieved by 'lightweight finetuning of these layers,' and that the prior is localized in specific layers. The in-context prompting results are real evidence that some task-relevant knowledge is latently present (especially for counting and acronyms), and I do not dispute those improvements. But the finetuning claims are the ones that carry the causal and localization story, and the reported numbers are validation-selected maxima. A nested split would settle whether the numbers are artifacts of selection; adding a fixed/prior-localized layer control would settle whether the selection is doing the work. I also note the paper honestly reports failed steering techniques and the stratified probing failure, which shows the authors are not hiding negative results; those failures are consistent with a more cautious reading in which the paper demonstrates effective interventions but does not yet demonstrate that the prior is causally encoded at the layers they fine-tune. The reader's verdict of CONDITIONAL is appropriate; my concern sharpens the condition under which the mechanistic claim should be accepted.","tokens_in":12742,"tokens_out":11563,"duration_ms":131074,"concrete_test":"Run the finetune-layer experiments with a nested three-way split: train LoRA for each of the 32 layers on the training split, select the best layer on the validation split, and evaluate only that layer on a separate test split whose answer tokens are disjoint from both training and validation answers. In the same setup, also evaluate a fixed reference layer (e.g., layer 0) and the layer predicted by the logit-lens correlation analysis (not by validation) on the test split. If the validation-selected layer's test accuracy drops materially below the Table 1 values, or if the reference layers perform comparably, then the headline held-out accuracy and the prior-localization explanation need to be revised; if the selected layer retains its advantage, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that finetuning removes a localized prior rests on Table 1's 'Finetune-layer' results. By the paper's own definition, this is 'performance obtained by only finetuning the best-performing layer,' and Figure 2 confirms the layer is chosen as the best layer in each task. Because selection is made after seeing validation accuracy, the validation split has already been used for model (layer) selection; reporting accuracy on that same split is selection-on-validation and overstates expected held-out accuracy. With 32 layers, even a null model can produce a spuriously high maximum. The abstract's 'lightweight finetuning of these layers' also implies the layers are the ones identified by the logit-lens prior-localization analysis, but no experiment shows the best-performing layer coincides with those prior-correlated layers; this severs the mechanistic link between the localization result and the mitigation result. The comparison with prior-insensitive tasks is additionally confounded by baseline accuracy: tasks that start near chance have more room to improve, so a larger absolute gain on prior-dominated tasks does not by itself demonstrate that the intervention specifically removes prior influence. The prompting results are genuine behavioral evidence that some latent competence exists, but the finetune-layer evidence for the stronger 'encoded and localized prior' claim is not yet clean.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies three deterministic tasks—counting letters, shift ciphers, and acronym construction—on Llama 3 8B, where model outputs are biased toward high-probability token sequences. Using a logit-lens analysis, the authors report that individual layers show an all-or-none pattern of Spearman correlation with prior logits, suggesting localized encoding of the prior. They then test two interventions: adding 'do not rely on your prior knowledge' to the prompt, and LoRA fine-tuning either the full model or a single selected layer under a stratified train/validation split that excludes validation answer tokens from training. They report that prompting improves accuracy on counting and acronyms, and that fine-tuning the best-performing layer reaches 72.3–99.9% validation accuracy on the three tasks, with larger gains than on two 'prior-insensitive' tasks (multiplication and make-letters). They interpret these results as evidence that the correct answers are already encoded in the residual stream and that the prior's influence can be localized and removed.","tokens_in":12971,"tokens_out":5644,"duration_ms":54602,"significance":"If the central claim holds, the paper makes a useful contribution: it provides a concrete setting in which a pretrained LLM's failures on simple deterministic tasks can be reduced by lightweight interventions, and it offers a template for separating 'known but not expressed' knowledge from genuine task inability. The use of a stratified split to prevent answer-token memorization is a thoughtful design choice, and the probing failure result is an honest negative result that strengthens the field's understanding of probe-based claims. The paper also ships code and uses an open model, which supports reproducibility. However, the current evidence does not yet establish the stronger mechanistic claims: the headline fine-tuning numbers are obtained by selecting the best of 32 layers on the validation split, and no experiment connects the fine-tuned layers to the layers identified as prior-correlated in the localization analysis. The comparison against prior-insensitive tasks is also confounded by floor effects. These are addressable with additional experiments, so the paper is on the right track but needs revision.","major_comments":[{"comment":"The 'Finetune-layer' row reports the accuracy of the best-performing layer selected on the validation split. With 32 layers, taking the maximum validation accuracy inflates the reported numbers relative to the expected accuracy of a fixed layer on a truly held-out test set; the manuscript does not report a nested split in which layer selection is made on training/validation data and the selected layer is evaluated on a disjoint test set. As a result, the headline held-out accuracies (72.3–99.9%) and the claim that finetuning is 'lightweight' are not yet established. Please report test-set accuracy of the selected layer and include a null baseline such as finetuning the same layer with shuffled labels.","section":"Table 1; Figure 2; Section 4.2"},{"comment":"The abstract says 'lightweight finetuning of these layers,' referring to the layers identified as prior-correlated via logit lens, but the experiments select the best-performing layer independently of the correlation analysis. No experiment shows that the best-performing layer coincides with or is even related to the layers with high prior correlation in Figure 3. The mechanistic link between localization and mitigation is therefore missing. Please report finetuning results for fixed layers chosen from the prior-correlation analysis (e.g., a layer with high correlation, a layer with low correlation, and a random layer), evaluated on a disjoint test set.","section":"Abstract; Section 4.1; Section 4.2"},{"comment":"The comparison between prior-dominated and prior-insensitive tasks is confounded by baseline accuracy. Multiplication and make-letters start near 0% accuracy, so they have more room for absolute improvement; larger absolute gains on prior-dominated tasks do not by themselves show that the intervention removes prior influence. Table 1 includes only the three prior-dominated tasks, so the reader cannot see the actual baselines and gains for the two control tasks. Please report all numbers and use a baseline-adjusted measure (e.g., improvement relative to the maximum possible improvement) or match tasks on baseline accuracy.","section":"Section 4.3; Figure 2; Table 1"},{"comment":"The central claim that 'the information required to produce a correct response is contained within the representations' is supported primarily by the fact that finetuning improves accuracy, since the probing evidence fails in the stratified setup. Finetuning can also learn the task from the training data rather than eliciting pre-existing representations. To support the 'already encoded' interpretation, please add a control in which the same finetuning recipe is applied with randomized answer labels (or to a held-in but non-prior task) and show that accuracy does not reach the reported levels; alternatively, demonstrate that the correct answer is linearly decodable from the selected layer under the stratified split with an appropriate control task.","section":"Section 4.2; Section 3.4.2"},{"comment":"The abstract claims that 'the error after finetuning is no longer correlated with the prior,' but the only supporting evidence is Figure 4, which shows counting accuracy by sequence length for one task. Please provide a quantitative test of the correlation between post-finetuning errors and prior token probabilities for all three prior-dominated tasks, with a significance test and, ideally, a comparison to the pre-finetuning correlation.","section":"Abstract; Figure 4; Section 4.3"}],"minor_comments":[{"comment":"The caption says 'six tasks' but the figure shows five; please correct the count.","section":"Figure 2 caption"},{"comment":"The 'all-or-none' pattern is based on a p<0.05 threshold without multiple-comparison correction; please report effect sizes or adjusted p-values to support the claim.","section":"Section 4.1"},{"comment":"The color legend is ambiguous ('dots colored in red have more answer logits with negative correlation'); please clarify the exact criterion for red versus blue.","section":"Figure 3"},{"comment":"'We take motivation from this approach an ablate out the unigram direction' contains a typo ('an' should be 'and'), and Table 3 reports no standard errors for the steering results.","section":"Appendix A; Table 3"},{"comment":"The entries 'Finetune' and 'Finetune-layer' are not defined in the table; please state whether 'Finetune' refers to whole-model LoRA or all-layer LoRA and specify the layer-selection protocol.","section":"Table 1"},{"comment":"The URL for torchtune has a typo ('https//github.com' should be 'https://github.com').","section":"Reproducibility Statement"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a preprint under review. The main issues are about experimental design rather than the core idea. I would be willing to review a revised version that addresses the selection-on-validation problem and adds the control experiments. I do not see grounds for rejection, but the current evidence overstates the mechanistic claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper shows something real: a simple \"do not rely on your prior knowledge\" prompt improves accuracy on counting, shift-cipher, and acronym tasks, and the model's internal logits show a clean all-or-none layer pattern for prior correlation. The stratified train/validation split is a thoughtful design—it prevents the probe and fine-tuner from memorizing answer tokens—and the honest report that probing fails under stratification is a useful negative result. Code is public and implementation details are sufficient to reproduce the main numbers.\n\nThe soft spots are real, but not fatal. The headline \"finetune-layer\" results in Table 1 are reported for the best of 32 layers selected on the validation split. That inflates expected held-out accuracy; with 32 candidates, even a null model can produce a high maximum. The paper does not show that the best-performing layer is one of the prior-correlated layers identified in Figure 3, so the localization and mitigation results are not actually linked. A fixed layer chosen a priori (e.g., an early layer) or a nested split would address this. The comparison against prior-insensitive tasks is also confounded by baseline accuracy: tasks starting near chance have more room to improve, so a larger absolute gain on prior-dominated tasks does not by itself prove the intervention specifically removes prior influence.\n\nThat said, the prompting result is direct behavioral evidence of latent competence, and the fine-tuning gains are large even before layer selection. The mechanistic claim—that the correct answer is encoded and the prior blocks access—remains plausible but indirect; the paper's own probing failure shows how hard this is to demonstrate. The citation pattern is fair and builds sensibly on McCoy et al. and Minder et al.\n\nThis is a paper for practitioners and interpretability researchers. It deserves a serious referee and likely revision; the core empirical contribution is real, but the headline numbers need a cleaner evaluation protocol before the strongest claims can stand. If I were the editor, I would send it out and ask for a nested validation plus a control condition.","headline":"A useful empirical paper with a clean prompt result and honest reporting of probing failures, but the layer-selection procedure inflates the headline fine-tuning numbers and the localization story needs a matched control.","tokens_in":13536,"tokens_out":2293,"would_cite":false,"duration_ms":23118,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that Llama 3 often computes the correct answer to deterministic tasks even when its learned prior steers output away from it, and that lightweight finetuning of one early layer exposes that knowledge.","keywords":["large language models","prior distribution","mechanistic interpretability","logit lens","LoRA finetuning","hallucination","residual stream","stratified evaluation"],"falsifier":"Run the paper's one-layer finetuning procedure with training labels randomly permuted: if held-out accuracy still rises substantially, the gain is not prior removal. Alternatively, switch off the layers identified by the correlation scan and check whether the common-token bias disappears; if it persists, the correlation is not the mechanism.","tokens_in":12517,"feed_emoji":"🧠","tokens_out":6536,"duration_ms":63869,"temperature":0.7,"pith_summary":"The paper tries to establish that an LLM's failure on deterministic tasks—counting, shift ciphers, and acronyms—is often not a failure to compute the answer, but a failure to output it because a learned prior over token sequences pulls generation toward common strings. On Llama 3, it shows the answer information is present in the residual stream and that the prior is localized to identifiable layers via logit-lens correlations. Interventions that target that influence—a one-line prompt and, more strongly, lightweight finetuning of a single early layer—substantially improve held-out accuracy, with held-out answers excluded from training so gains cannot be memorization. The same finetuning helps far less on tasks without a strong prior over answers, and after finetuning errors no longer track answer commonness. The result matters because it suggests some hallucinations are prior-driven and may be mitigable without retraining the whole model.","feed_headline":"Light finetuning surfaces answers Llama 3 already computes","feed_subtitle":"Counting, shift-cipher, and acronym accuracy climbs from 11–32% to 72–99.9% after one-layer finetuning.","key_machinery":"The machinery has three parts. First, logit lens: read each layer's hidden embedding through the unembedding matrix to approximate per-layer answer logits, and correlate those with prior logits elicited from a no-input prompt; this localizes the prior to certain layers in an all-or-none pattern. Second, a stratified train/validation split in which every answer token in validation is absent from training, so high accuracy cannot come from memorizing answer–question pairs. Third, low-rank (LoRA) finetuning applied to a single early layer selected by validation performance. The argument runs: if the correct answer can be read from hidden states but the final output follows the prior, then a narrow intervention that weakens prior encoding should let the encoded answer through.","core_discovery":"The paper's central claim is that Llama 3 8B, on three deterministic tasks where it fails—counting letter sequences, shifting ciphertext by one, and forming acronyms—stores the information needed for the correct answer in its residual stream, but its learned prior over token sequences steers decoding toward common tokens. Evidence comes from layer-wise logit-lens correlations with the prior, which are high in some layers and absent in others; from a prompt that says \"do not rely on your prior knowledge,\" which lifts accuracy on counting from 11.5% to 49.2% and on acronyms from 12.8% to 46.3% but barely on shift ciphers; and from lightweight LoRA finetuning of a single early layer on a stratified split, which reaches 72.3%, 99.4%, and 99.9% respectively on the three tasks while seeing no held-out answers during training. The same finetuning gives much smaller gains on multiplication and make-letters tasks, which the paper classifies as prior-insensitive, and after finetuning errors track question difficulty rather than token frequency. A linear probe fails on the stratified split, and two existing linear steering techniques fail, suggesting the prior's influence is not a simple linear direction that can be subtracted away.","pith_inferences":["If the localization is causal, the same layer-targeted finetuning could be tested on other failure modes with a known probabilistic bias, such as fact retrieval where high-frequency answers win, though the paper does not test that.","The authors did not ablate the identified prior-correlated layers to show they are necessary; that ablation would be the test that turns localization into causation.","The stratified design could be extended to other open-weight models to see whether the early-layer locus is architecture-specific or a general feature of transformer language models.","The failure of linear steering methods suggests a nonlinear, layer-specific representation-editing method beyond finetuning might reproduce the effect at inference time, which the paper leaves open."],"forward_implications":["Failures that look like hallucinations on counting, shift ciphers, and acronyms can be reduced by interventions that add no task-specific information, only reduce reliance on the prior.","Stratified held-out evaluation is necessary: random-split probing overstates what a model knows, while the stratified setup shows near-zero probe accuracy.","On prior-dominated tasks, one-layer finetuning with 2,000–8,000 sequences and 50 epochs is enough for high held-out accuracy.","Post-finetuning error correlates with question difficulty, not answer commonness, indicating the prior bias has been removed rather than masked.","The gains from finetuning are substantially larger on prior-dominated than prior-insensitive tasks, so the intervention is specifically prior removal.","The failure of linear steering methods while nonlinear finetuning succeeds suggests that prior influence is not a single linear direction in the residual stream."],"supporting_citations":[{"why":"Documents the probability-sensitivity phenomenon that defines prior-dominated tasks and supplies the task designs.","marker":"McCoy et al., 2023"},{"why":"Supplies the logit-lens method used to localize the prior across layers.","marker":"Belrose et al., 2023"},{"why":"Provides the LoRA method used for the lightweight finetuning experiments.","marker":"Hu et al., 2021"},{"why":"Provides prior evidence that LLMs know more than they show, which this paper extends to free-response tasks.","marker":"Orgad et al., 2024"},{"why":"Defines the Llama 3 model that all experiments run on.","marker":"Grattafiori et al., 2024"},{"why":"Provides the context-versus-prior steering subspace that the paper tests and finds ineffective.","marker":"Minder et al., 2025"},{"why":"Identifies the unigram prior direction that the paper ablated and found ineffective.","marker":"Stolfo et al., 2024"}],"fun_headline_variants":["Llama 3 already computes answers; one layer unleashes them","Prior bias hides LLM knowledge: a single-layer finetune fixes it","Flip a layer to fix counting, ciphers, acronyms in Llama 3","Ignore-prior prompt helps; one-layer finetune does more","Light finetuning exposes answers Llama already holds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the layers whose internal predictions correlate with the model's prior probabilities are the causal site of prior influence, not just correlated bystanders; the paper also chooses which layer to finetune by looking at validation performance and does not compare against a randomly labeled control.","fun_headline_variants_meta":{"raw":{"variants":["Llama 3 already computes answers; one layer unleashes them","Prior bias hides LLM knowledge: a single-layer finetune fixes it","Flip a layer to fix counting, ciphers, acronyms in Llama 3","Ignore-prior prompt helps; one-layer finetune does more","Light finetuning exposes answers Llama already holds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1765,"prompt_tokens":1091,"completion_tokens":674,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":707,"completion_tokens_details":{"reasoning_tokens":579}},"tokens_in":707,"tokens_out":674,"duration_ms":7750,"temperature":1.0,"reasoning_tokens":579,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:26:38.818329+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's one-layer finetuning procedure with training labels randomly permuted: if held-out accuracy still rises substantially, the gain is not prior removal. Alternatively, switch off the layers identified by the correlation scan and check whether the common-token bias disappears; if it persists, the correlation is not the mechanism.","supporting_citations":[],"review_version":1}