{"id":"ef70f631-ea24-46e7-b42c-e4f8c017ebd1","arxiv_id":"2607.14111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Small LLMs can be fine-tuned to localize activation-steering perturbations, raising Llama-1B accuracy from 9.6% to 60.6% and generalizing to a strength-comparison task.","lead":"The paper shows that small language models can learn to report when one sentence among several has been artificially altered by activation steering, with accuracy well above chance, and that focused fine-tuning dramatically improves this ability. It also argues that a simpler yes/no test of such 'introspection' is confounded by an affirmative response bias in small models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IFT evaluation shares the same fixed 100-sentence bank between training and testing, so the reported 9.6→60.6% localization gain may reflect sentence-bank memorization rather than a general introspective mechanism.","rationale":"The reader's concern about causal bypassing is valid and related, but the most load-bearing weakness is more specific: the train/test sentence-bank overlap. The reader's weakest_assumption mentions 'learned positional shortcuts' and the absence of an ablation that removes the injection, but does not pinpoint that the same 100 sentences are used in both training and evaluation. This is a concrete experimental-design leak that directly threatens the external validity of the headline IFT result. If the evaluation used held-out sentences, the causal-bypassing concern would still matter, but the sentence-memorization shortcut is arguably an even more immediate threat. I therefore agree partially with the reader: the verdict should remain CONDITIONAL, but for an additional, sharper reason. I also note the internal contradiction flagged by the reader: Appendix B's concluding sentence says the model's 'Yes' responses reflect genuine sensitivity, while §2.3.1 concludes the opposite (mean net signal ≈ 0, r=0.999 with a factual-no control). This inconsistency further supports CONDITIONAL, though it is secondary to the sentence-bank issue. The proposed concrete test—re-evaluating on a held-out sentence set—would settle whether the 6× improvement is a general introspective gain or an artifact of the fixed 100-sentence pool.","tokens_in":11718,"tokens_out":6534,"duration_ms":74894,"concrete_test":"Re-run the IFT evaluation while holding all hyperparameters fixed, but replace the 100-sentence bank with a freshly generated, disjoint set of 100 neutral sentences for evaluation only (same length/style distribution, never seen during training or validation for epoch selection). Compute the Llama-1B Random·Semantic average localization and the zero-shot strength-comparison accuracy from Table 2. If accuracy stays well above chance (e.g., ≥40–50% for localization, ≥55% for strength), the claimed trained introspection generalizes beyond the training sentences. If it falls to near chance (≈10% localization, ≈50% strength), the headline gains are an artifact of sentence-bank memorization and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that IFT trains a general introspective ability rests on Table 2, especially Llama-1B localization rising from 9.6% to 60.6% and zero-shot strength comparison from 30.2% to 52.2%. However, §2.3.2, §4.1, and Appendix D.5–D.6 show that both training and evaluation draw sentences from the same fixed pool of 100 neutral sentences. The model is trained on localization trials built from this pool and evaluated on trials built from the same pool; the zero-shot strength-comparison transfer is also measured on the same pool. Consequently, the model could learn sentence-specific signatures of the steering vector (e.g., how injection interacts with the particular wording of sentence 37) rather than a general mechanism for reading residual-stream perturbations. The paper reports held-out concepts but no held-out sentences, so the reported gains cannot distinguish a general introspective capacity from overfitting to the 100-sentence bank. This is a more concrete failure mode than the missing causal-bypassing ablation: even with perfect causal grounding, the evaluation design allows memorization to masquerade as introspection. The paper's own limitation section acknowledges that transfer to 'naturalistic settings' is open, but that is a much weaker caveat than the fact that the controlled setting itself shares training and test sentences.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether small language models can detect and report injected activation-steering perturbations in their own residual streams. It argues that the standard binary yes/no detection paradigm is confounded in small models by a global affirmative bias, and proposes two alternative metrics: sentence localization (which of N sentences was perturbed) and strength comparison (which of two sentences received a stronger perturbation). Across Llama-3.2 1B–8B and Gemma-4 2B–26B, the authors report that introspection emerges above chance at 2B+ scale and generally improves with scale. For Llama-1B, which performs at/below chance, they introduce Introspection Fine-Tuning (IFT): supervised fine-tuning on sentence-localization examples built from the model's own perturbed forward passes. They report that IFT raises Llama-1B localization accuracy from 9.6% to 60.6%, transfers zero-shot to strength comparison (30.2% to 52.2%), and largely preserves MMLU/Winogrande performance. The central claim is that introspective ability is not fixed by scale alone and can be directly trained.","tokens_in":12013,"tokens_out":5337,"duration_ms":62126,"significance":"If the main result holds, the paper would make a useful contribution to interpretability and self-monitoring: it provides a concrete, chance-referenced paradigm for measuring activation-level introspection in small models, a systematic scale curve, and evidence that fine-tuning can unlock a representational substrate for detecting residual-stream perturbations. The zero-shot transfer from localization to strength comparison is an interesting and nontrivial result. The authors also ship code, which supports reproducibility. However, the strength of the central claim is currently undermined by an internal contradiction about the binary-detection confound and by the use of a fixed 100-sentence bank in both training and evaluation, which leaves open a memorization-based alternative explanation for the reported gains.","major_comments":[{"comment":"The paper contains a direct internal contradiction about the binary-detection control. §2.3.1 (p.3) concludes that the increase in 'Yes' logits under injection is 'entirely explained by a content-independent upward shift in affirmative responding, not by any sensitivity to the injected concept.' Appendix B (p.11) concludes the opposite: the near-perfect correlation with the factual-no control 'confirms that the model's Yes responses ... reflect genuine sensitivity to the injected activation pattern, not a general affirmative bias.' These cannot both be true. Since the confound diagnosis motivates the two new metrics, this contradiction is load-bearing and must be resolved with a clear statement of which interpretation is correct and with the corresponding evidence.","section":"§2.3.1 vs Appendix B"},{"comment":"The central claim of a generalizable introspective mechanism is not supported by the evaluation design because training and evaluation draw sentences from the same fixed 100-sentence bank. Appendix D.3 and D.6 show that training examples sample from this pool and that evaluation, including the 'zero-shot' strength-comparison transfer, uses the same pool. The claim in D.5 that a fixed pool 'ensures that any improvement in localisation accuracy cannot be attributed to the model memorising specific sentence co-occurrences with concepts' addresses only co-occurrence memorization, not the more plausible memorization of sentence-specific interaction signatures between a sentence's wording and the injected steering vector. With held-out concepts but no held-out sentences, the reported 9.6%→60.6% localization gain could reflect the model learning how each of the 100 sentences responds to perturb","section":"§2.3.2, §4.1, Appendix D.5–D.6"},{"comment":"There is an unexplained discrepancy between the best-case baseline numbers in Table 2 and the scale-study results in §3/Figure 2. Table 2 reports pre-IFT baseline Best Loc = 100.0% and Best Str = 100.0% for both Llama-3B and Llama-8B, while §3 states that Llama-3B reaches ≈65% and Llama-8B ≈88% for sentence localization. If the baseline already attains 100% at some (alpha, layer) cell, then the claim in §4.3 that IFT 'unlocks introspective capacity that pre-training alone does not elicit' is misleading at best for these models; the improvement would be in average-case reliability, not in peak ability. This needs clarification and, if the 100% baseline values are correct, a revised interpretation of the IFT gains.","section":"Table 2 vs §3, §4.3"},{"comment":"The Gaussian-noise control results are reported inconsistently and sometimes appear worse than chance in ways that are not discussed. Table 5 shows Llama-3.1-8B Fixed·Gaussian and Random·Gaussian with average strength-comparison accuracy of 10.0%, far below the 50% chance level, yet §4.3 describes Gaussian noise as a control whose gains 'reflect learning to detect the presence and magnitude of any hidden-state perturbation.' A below-chance strength-comparison score indicates a systematic response bias, not merely the absence of learning, and calls for explanation. The near-identical baseline and Random·Gaussian rows for Llama-3B in Table 2 also need clarification, as they suggest the Gaussian condition may not have trained at all.","section":"§4.3, Table 2, Table 5"}],"minor_comments":[{"comment":"Typographical and formatting issues: 'stregth-comparison' (§4.2), 'localisation' vs 'localization' are used inconsistently, Table 5 contains '260.647.8' as a single cell value, and the Gemma model designation 'E2B ≈2B' is unexplained. Please standardize.","section":"Throughout"},{"comment":"The sentence beginning 'Using a fixed pool ensures...' is phrased in a way that overstates the control provided by the fixed pool. As noted in the major comments, it rules out co-occurrence memorization but not sentence-specific perturbation signatures. Please rephrase to accurately state what the fixed pool does and does not control for.","section":"Appendix D.5"},{"comment":"The 40-point correlation r=0.999 is reported without a confidence interval or per-point residuals. Given that the same data are used to support two opposite conclusions (§2.3.1 and Appendix B), the analysis would benefit from reporting the mean difference between the introspection and control deltas with a proper uncertainty estimate, rather than only the correlation.","section":"Figure 1 / Appendix B"},{"comment":"In the strength-comparison task, the phrase 'This matched-pairs design cancels positional preferences' should clarify that the same sentence pair is tested with swapped injection strengths; as written, a reader might think the strengths are swapped within a single trial rather than across two sub-trials.","section":"§2.3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting question, and the IFT result could be a meaningful contribution if the fixed-sentence-bank concern is addressed. I would encourage the editor to require the authors to either add a held-out-sentence evaluation or substantially weaken the generalization claims. The contradiction between §2.3.1 and Appendix B should be fixed before any further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"To be candid, the most valuable thing here is the confound diagnosis, not the IFT result. The binary detection paradigm in small models is genuinely confounded by affirmative bias, and the r=0.999 correlation with a factual-no control is convincing. The localization and strength-comparison tasks are well-designed in principle—matched pairs, no global bias—and the scale study across two families is a useful data point.\n\nThe IFT idea is also interesting: training a model to report on injected activations via SFT is not in the cited prior work. But the main claim—that introspective ability is not fixed by scale and can be trained—is not supported by the evidence as presented. The stress-test note is right: both training and evaluation draw sentences from the same fixed 100-sentence bank. Even with held-out concepts, the model can learn sentence-specific signatures of the perturbation. The appendix even says the fixed pool is supposed to prevent memorization, but it does the opposite—it enables it. So the 9.6% to 60.6% gain could be memorization, not a general mechanism. The zero-shot transfer to strength comparison is also measured on the same sentences, so it doesn't rescue the claim.\n\nThere are also smaller issues: Appendix B concludes the detection signal reflects genuine sensitivity, while §2.3.1 says it's entirely affirmative bias. That's a direct contradiction. No error bars or multiple seeds, best-epoch and peak-grid numbers are emphasized, and the capability numbers are mostly flat but with some oddities (e.g., Fixed·Gaussian at 10% on both tasks for 8B looks like a bug). These are addressable.\n\nIf the authors rerun with a held-out sentence pool and report variance, this could be a solid paper. As it stands, the central result is not established. Still, it deserves peer review—the confound in binary detection is important, and the IFT idea deserves scrutiny. I would not cite the IFT numbers, but I would cite the confound diagnosis and the tasks as a caution for the field.","headline":"Solid confound diagnosis and clean metric design, but the main IFT result is undermined by shared train/eval sentences and an internal contradiction.","tokens_in":12480,"tokens_out":3494,"would_cite":true,"duration_ms":36470,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 1-billion-parameter language model can be trained to detect and locate injected thoughts in its own activations, jumping from near-chance to 60.6% accuracy on a sentence-localization task, with gains transferring to a held-out strength-co","keywords":["introspection","activation steering","fine-tuning","self-monitoring","small language models","interpretability","confound-free evaluation","sentence localization"],"falsifier":"Run the same sentence-localization evaluation with the steering vector replaced by a zero vector (no perturbation) while keeping the prompt identical; if a model trained with IFT still localizes at substantially above the 10% chance level, the reports are not causally grounded in the injection. Alternatively, permute the sentence order in the prompt: if accuracy drops to chance, the model is using position rather than content.","tokens_in":11590,"feed_emoji":"🧠","tokens_out":4079,"duration_ms":37087,"temperature":0.7,"pith_summary":"This paper claims that introspective ability in language models—the capacity to detect and report on perturbations to their own internal activations—is not fixed by model scale but can be directly trained. The authors first show that the standard yes/no detection test is confounded in small models because activation steering inflates affirmative responses regardless of question content. They introduce two relative-judgment tests immune to that bias: sentence localization (which of N sentences carries an injected concept vector) and strength comparison (which of two carries the stronger injection). Using these tests, they show that models from 2B parameters upward introspect above chance, while a 1B model performs at or below chance. They then apply supervised fine-tuning on sentence-localization examples drawn from the model's own perturbed forward passes—Introspection Fine-Tuning (IFT)—raising the 1B model from 9.6% to 60.6% localization accuracy, with zero-shot transfer to strength comparison (30.2% to 52.2%).","feed_headline":"1B model learns to detect its own internal injections","feed_subtitle":"Supervised fine-tuning on the model's own perturbed forward passes raises sentence-localization accuracy from 9.6% to 60.6% and transfers to","key_machinery":"The central mechanism is the relative-judgment evaluation paired with activation steering. A concept vector is added to the residual stream at chosen token positions; the model must pick the sentence whose hidden state was perturbed by reading the final-position digit logits. Sentence localization and strength comparison are constructed so that a uniform shift in affirmative logits cannot privilege any position, making them confound-free. Introspection Fine-Tuning uses these same sentence-localization prompts as supervised training signal, sampling the injection layer at random, so the model learns a layer-agnostic strategy for detecting and localizing perturbations.","core_discovery":"Even a 1-billion-parameter language model contains a trainable representational substrate for reading its own residual-stream perturbations. When fine-tuned on the model's own perturbed forward passes with the sentence-localization task, a model that performed at or below chance localizes the injected concept vector at 60.6% accuracy, and the learned ability transfers zero-shot to a different relative-judgment task (strength comparison), indicating a general mechanism for computing functions of internal activations rather than a task-specific heuristic. Gains also appear at 3B and 8B scales, with negligible degradation on standard benchmarks. The paper argues from these results that introspe","pith_inferences":["A natural extension would be training on naturalistic internal anomalies (e.g., contradictory prompts or internal conflict) rather than engineered steering vectors; the paper leaves that open.","The fixed background sentence pool shared between training and evaluation could allow a shortcut: if the model memorizes sentence–concept co-occurrences, localization could rise without true introspective grounding. A held-out sentence bank would settle this.","If the trained signal truly reads residual-stream perturbations, IFT could be combined with activation decoders to give models a self-report channel for monitoring their own computations.","The causal-bypassing concern remains: without an ablation that removes the perturbation while holding the prompt fixed, part of the reported accuracy could reflect positional or template cues rather than the injected content."],"forward_implications":["If IFT works as described, introspective monitoring can be trained into small models without sacrificing general capability (MMLU and Winogrande scores are roughly preserved).","The zero-shot transfer to strength comparison implies the trained mechanism is general, not a memorized heuristic tied to the training task.","The confound diagnosis implies that binary detection results from prior work on small models cannot be trusted unless affirmative bias is controlled.","IFT improves introspection across scales, suggesting a recipe for self-monitoring that is complementary to existing interpretability methods."],"fun_headline_variants":["Fine-tuning teaches 1B LLM to introspect on its own activations","1B LLM jumps from chance to 61% on injection localization","From 9.6% to 60.6%: training introspection in 1B models","Self-monitoring: fine-tuning makes 1B models introspect","Even 1B LLMs can learn to read their activation vectors"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The accuracy numbers assume that the model's final-position digit logits reflect the perturbed internal state rather than a learned association between the prompt template and sentence positions; the paper does not include an ablation that removes the injection while holding the prompt fixed.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning teaches 1B LLM to introspect on its own activations","1B LLM jumps from chance to 61% on injection localization","From 9.6% to 60.6%: training introspection in 1B models","Self-monitoring: fine-tuning makes 1B models introspect","Even 1B LLMs can learn to read their activation vectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001299,"raw_usage":{"total_tokens":5218,"prompt_tokens":908,"completion_tokens":4310,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":4208}},"tokens_in":652,"tokens_out":4310,"duration_ms":31061,"temperature":1.0,"reasoning_tokens":4208,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T14:36:11.218268+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same sentence-localization evaluation with the steering vector replaced by a zero vector (no perturbation) while keeping the prompt identical; if a model trained with IFT still localizes at substantially above the 10% chance level, the reports are not causally grounded in the injection. Alternatively, permute the sentence order in the prompt: if accuracy drops to chance, the model is using position rather than content.","supporting_citations":[],"review_version":1}