REVIEW 4 major objections 4 minor 13 references
Introspection Fine-Tuning (IFT): Training Small LLMs to Introspect
T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read A 1-billion-parameter language model can be trained to detect and locate injected thoughts in its own activations, jumping from near-chance to 60.6% accuracy on a sentence-localization task, with gains transferring to a held-out strength-co
desk verdict Solid confound diagnosis and clean metric design, but the main IFT result is undermined by shared train/eval sentences and an internal contradiction. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the relative-judgment evaluation paired with activation steering. A concept vector is added to the residual stream at chosen token positions; the model must pick the sentence whose hidden state was perturbed by reading the final-position digit logits. Sentence localization and strength comparison are constructed so that a uniform shift in affirmative logits cannot privilege any position, making them confound-free. Introspection Fine-Tuning uses these same sentence-localization prompts as supervised training signal, sampling the injection layer at random, so the model learns a layer-agnostic strategy for detecting and localizing perturbations.
What would settle it
Run the same sentence-localization evaluation with the steering vector replaced by a zero vector (no perturbation) while keeping the prompt identical; if a model trained with IFT still localizes at substantially above the 10% chance level, the reports are not causally grounded in the injection. Alternatively, permute the sentence order in the prompt: if accuracy drops to chance, the model is using position rather than content.
Extended reading notes
Core claim
Even a 1-billion-parameter language model contains a trainable representational substrate for reading its own residual-stream perturbations. When fine-tuned on the model's own perturbed forward passes with the sentence-localization task, a model that performed at or below chance localizes the injected concept vector at 60.6% accuracy, and the learned ability transfers zero-shot to a different relative-judgment task (strength comparison), indicating a general mechanism for computing functions of internal activations rather than a task-specific heuristic. Gains also appear at 3B and 8B scales, with negligible degradation on standard benchmarks. The paper argues from these results that introspe
Load-bearing premise
The accuracy numbers assume that the model's final-position digit logits reflect the perturbed internal state rather than a learned association between the prompt template and sentence positions; the paper does not include an ablation that removes the injection while holding the prompt fixed.
Editorial extensions
If this is right
- If IFT works as described, introspective monitoring can be trained into small models without sacrificing general capability (MMLU and Winogrande scores are roughly preserved).
- The zero-shot transfer to strength comparison implies the trained mechanism is general, not a memorized heuristic tied to the training task.
- The confound diagnosis implies that binary detection results from prior work on small models cannot be trusted unless affirmative bias is controlled.
- IFT improves introspection across scales, suggesting a recipe for self-monitoring that is complementary to existing interpretability methods.
Reading between the lines
- A natural extension would be training on naturalistic internal anomalies (e.g., contradictory prompts or internal conflict) rather than engineered steering vectors; the paper leaves that open.
- The fixed background sentence pool shared between training and evaluation could allow a shortcut: if the model memorizes sentence–concept co-occurrences, localization could rise without true introspective grounding. A held-out sentence bank would settle this.
- If the trained signal truly reads residual-stream perturbations, IFT could be combined with activation decoders to give models a self-report channel for monitoring their own computations.
- The causal-bypassing concern remains: without an ablation that removes the perturbation while holding the prompt fixed, part of the reported accuracy could reflect positional or template cues rather than the injected content.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether small language models can detect and report injected activation-steering perturbations in their own residual streams. It argues that the standard binary yes/no detection paradigm is confounded in small models by a global affirmative bias, and proposes two alternative metrics: sentence localization (which of N sentences was perturbed) and strength comparison (which of two sentences received a stronger perturbation). Across Llama-3.2 1B–8B and Gemma-4 2B–26B, the authors report that introspection emerges above chance at 2B+ scale and generally improves with scale. For Llama-1B, which performs at/below chance, they introduce Introspection Fine-Tuning (IFT): supervised fine-tuning on sentence-localization examples built from the model's own perturbed forward passes. They report that IFT raises Llama-1B localization accuracy from 9.6% to 60.6%, transfers zero-shot to strength comparison (30.2% to 52.2%), and largely preserves MMLU/Winogrande performance. The central claim is that introspective ability is not fixed by scale alone and can be directly trained.
Significance. If the main result holds, the paper would make a useful contribution to interpretability and self-monitoring: it provides a concrete, chance-referenced paradigm for measuring activation-level introspection in small models, a systematic scale curve, and evidence that fine-tuning can unlock a representational substrate for detecting residual-stream perturbations. The zero-shot transfer from localization to strength comparison is an interesting and nontrivial result. The authors also ship code, which supports reproducibility. However, the strength of the central claim is currently undermined by an internal contradiction about the binary-detection confound and by the use of a fixed 100-sentence bank in both training and evaluation, which leaves open a memorization-based alternative explanation for the reported gains.
major comments (4)
- [§2.3.1 vs Appendix B] The paper contains a direct internal contradiction about the binary-detection control. §2.3.1 (p.3) concludes that the increase in 'Yes' logits under injection is 'entirely explained by a content-independent upward shift in affirmative responding, not by any sensitivity to the injected concept.' Appendix B (p.11) concludes the opposite: the near-perfect correlation with the factual-no control 'confirms that the model's Yes responses ... reflect genuine sensitivity to the injected activation pattern, not a general affirmative bias.' These cannot both be true. Since the confound diagnosis motivates the two new metrics, this contradiction is load-bearing and must be resolved with a clear statement of which interpretation is correct and with the corresponding evidence.
- [§2.3.2, §4.1, Appendix D.5–D.6] The central claim of a generalizable introspective mechanism is not supported by the evaluation design because training and evaluation draw sentences from the same fixed 100-sentence bank. Appendix D.3 and D.6 show that training examples sample from this pool and that evaluation, including the 'zero-shot' strength-comparison transfer, uses the same pool. The claim in D.5 that a fixed pool 'ensures that any improvement in localisation accuracy cannot be attributed to the model memorising specific sentence co-occurrences with concepts' addresses only co-occurrence memorization, not the more plausible memorization of sentence-specific interaction signatures between a sentence's wording and the injected steering vector. With held-out concepts but no held-out sentences, the reported 9.6%→60.6% localization gain could reflect the model learning how each of the 100 sentences responds to perturb
- [Table 2 vs §3, §4.3] There is an unexplained discrepancy between the best-case baseline numbers in Table 2 and the scale-study results in §3/Figure 2. Table 2 reports pre-IFT baseline Best Loc = 100.0% and Best Str = 100.0% for both Llama-3B and Llama-8B, while §3 states that Llama-3B reaches ≈65% and Llama-8B ≈88% for sentence localization. If the baseline already attains 100% at some (alpha, layer) cell, then the claim in §4.3 that IFT 'unlocks introspective capacity that pre-training alone does not elicit' is misleading at best for these models; the improvement would be in average-case reliability, not in peak ability. This needs clarification and, if the 100% baseline values are correct, a revised interpretation of the IFT gains.
- [§4.3, Table 2, Table 5] The Gaussian-noise control results are reported inconsistently and sometimes appear worse than chance in ways that are not discussed. Table 5 shows Llama-3.1-8B Fixed·Gaussian and Random·Gaussian with average strength-comparison accuracy of 10.0%, far below the 50% chance level, yet §4.3 describes Gaussian noise as a control whose gains 'reflect learning to detect the presence and magnitude of any hidden-state perturbation.' A below-chance strength-comparison score indicates a systematic response bias, not merely the absence of learning, and calls for explanation. The near-identical baseline and Random·Gaussian rows for Llama-3B in Table 2 also need clarification, as they suggest the Gaussian condition may not have trained at all.
minor comments (4)
- [Throughout] Typographical and formatting issues: 'stregth-comparison' (§4.2), 'localisation' vs 'localization' are used inconsistently, Table 5 contains '260.647.8' as a single cell value, and the Gemma model designation 'E2B ≈2B' is unexplained. Please standardize.
- [Appendix D.5] The sentence beginning 'Using a fixed pool ensures...' is phrased in a way that overstates the control provided by the fixed pool. As noted in the major comments, it rules out co-occurrence memorization but not sentence-specific perturbation signatures. Please rephrase to accurately state what the fixed pool does and does not control for.
- [Figure 1 / Appendix B] The 40-point correlation r=0.999 is reported without a confidence interval or per-point residuals. Given that the same data are used to support two opposite conclusions (§2.3.1 and Appendix B), the analysis would benefit from reporting the mean difference between the introspection and control deltas with a proper uncertainty estimate, rather than only the correlation.
- [§2.3.2] In the strength-comparison task, the phrase 'This matched-pairs design cancels positional preferences' should clarify that the same sentence pair is tested with swapped injection strengths; as written, a reader might think the strengths are swapped within a single trial rather than across two sub-trials.
Circularity Check
No significant circularity: IFT transparently optimizes localization and its novel evidence is zero-shot strength-comparison transfer.
full rationale
This is an empirical training study rather than a derivation, and no load-bearing step reduces to its own inputs by construction. IFT's objective (Section 4.1, L=-log p_theta(i|prompt)) is explicitly the sentence-localization task, so the localization improvement in Table 2 is a direct, transparent consequence of supervised fine-tuning on that task; the paper does not present localization as an unpredicted discovery. The central generalization claim is the zero-shot strength-comparison gain (30.2% to 52.2%), and there is no construction by which that output equals a training input: the paper states in Section 4.1 that 'the model never sees any strength-comparison labels at training time', and strength comparison requires comparing two injection magnitudes rather than reproducing the single-position localization supervision. No load-bearing self-citation occurs: Lindsey [2026], Macar et al. [2026], and Godet [2025] are external prior work, and the paper invokes no self-derived uniqueness theorem. The shared 100-sentence bank between training and evaluation (Appendix D.5-D.6) raises a legitimate memorization/external-validity concern, but that is not circularity: the zero-shot task still computes something not contained in the training labels, and a data-leakage failure mode does not make the output equivalent to the input by definition. The authors' own limitation that naturalistic transfer remains open (Section 6) is a scope caveat, not a circular step. Accordingly, no circular step meets the evidentiary bar; score 0.
Assumptions & free parameters
free parameters (3)
- Injection strength coefficients α =
{1,2,5,10,50,250} (swept, with peak reported)
- Injection layer l =
every third layer (fixed layer 3 in IFT training; random layers in random-layer mode)
- IFT hyperparameters =
LoRA rank 16, scaling 32, LR 2e-4, batch 8, 3 epochs
assumptions (4)
- domain assumption A steering vector v computed as mean activation contrast (concept minus baseline, or P minus N) is a faithful direction for the target concept and produces a detectable internal perturbation when added at the same layer.
- domain assumption The model's digit-token logits at the final position are a faithful readout of its judgment about the injected sentence, and relative judgments eliminate affirmative bias.
- domain assumption Fine-tuning with LoRA on 950 concepts and a fixed 100-sentence pool produces a general mechanism rather than memorization of sentence/concept pairs.
- domain assumption GPT-4.1-mini-generated sentences and concept pairs are valid stimuli for steering vectors.
Cite this review
Pith. "Pith review of Introspection Fine-Tuning (IFT): Training Small LLMs to Introspect." pith.science (2026). https://pith.science/paper/ON5CM22Z
@misc{pith2026260714111,
author = {Pith},
title = {Pith review of: Introspection Fine-Tuning (IFT): Training Small LLMs to Introspect},
year = {2026},
howpublished = {\url{https://pith.science/paper/ON5CM22Z}},
note = {Machine review of arXiv:2607.14111}
}
abstract
Can small language models detect and report on perturbations their own internal activations? We investigate this question through the lens of activation steering: injecting concept vectors into a model's residual stream and measuring whether the model can accurately report on the perturbation. We first show that the binary detection paradigm used in prior work -- prompting the model to answer Yes'' or No'' to whether it detects an injected thought -- is confounded in small models, as steering biases the model toward affirmative responses regardless of the question content. We therefore propose two confound-free evaluation paradigms: sentence localization (identifying which of $N$ sentences was perturbed, chance $= 1/N$) and strength comparison (identifying which of two sentences received a stronger injection, chance $= 50\%$). Evaluating across six models from two families (Llama-3.2 and Gemma-4), we find that models as small as 2B parameters introspect reliably well above chance, and that introspective ability generally increases with scale. Llama-1B, however, performs at or below chance. We then introduce \emph{Introspection Fine-Tuning} (IFT): supervised fine-tuning on sentence-localization examples constructed from the model's own perturbed forward passes. IFT raises Llama-1B sentence-localization accuracy from $9.6\%$ to $60.6\%$ (a $6\times$ improvement), with gains generalizing zero-shot to the held-out strength-comparison task ($30.2\% \to 52.2\%$). IFT also improves introspection for 3B and 8B models, while inducing negligible degradation on standard capability benchmarks. Our results suggest that introspective ability is not fixed by scale alone: it can be directly trained, and doing so unlocks latent self-monitoring capacity with implications for AI transparency and alignment. Our code is \href{https://anonymous.4open.science/r/IFT-introspection-2092/README.md}{here}.
Figures
Reference graph
Works this paper leans on
-
[1]
Felix Binder, James Chua, Tomasz Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, and Owain Evans. Looking inward: Language models can learn about themselves by introspection.arXiv preprint arXiv:2410.13787,
-
[7]
Uzay Macar, Li Yang, Atticus Wang, Peter Wallich, Emmanuel Ameisen, and Jack Lindsey
URL https: //arxiv.org/abs/2601.01828. Uzay Macar, Li Yang, Atticus Wang, Peter Wallich, Emmanuel Ameisen, and Jack Lindsey. Mecha- nisms of introspective awareness,
-
[8]
URLhttps://arxiv.org/abs/2603.21396. Meta AI. Llama 3.2: Connecting multimodal with edge and mo- bile devices, September
-
[9]
Meta AI Blog
URL https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/. Meta AI Blog. Adam Morris and Dillon Plunkett. Tests of LLM introspection need to rule out causal bypassing. Less- Wrong, November
2024
-
[12]
14 Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J
URLhttps://arxiv.org/abs/2501.01558. 14 Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering.arXiv preprint arXiv:2308.10248,
-
[13]
URL https://arxiv.org/abs/2305. 18153. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena.arXiv preprint arXiv:2306.05685,
-
[2019]
URL https://arxiv.org/abs/1907. 10641. Dylan Sam, Marc Finzi, and J. Zico Kolter. Predicting the performance of black-box llms through follow-up queries,
1907
-
[2021]
URL https://arxiv. org/abs/2009.03300. Vincent Huang, Dami Choi, Daniel D. Johnson, Sarah Schwettmann, and Jacob Steinhardt. Pre- dictive concept decoders: Training scalable end-to-end interpretability assistants.arXiv preprint arXiv:2512.15712,
arXiv 2009
Show all 13 references
-
[2022]
Activation oracles: Training and evaluating LLMs as general-purpose activation explainers.arXiv preprint arXiv:2512.15674,
Adam Karvonen, James Chua, Clément Dumas, Kit Fraser-Taliente, Subhash Kantamneni, Julian Minder, Euan Ong, Arnab Sen Sharma, Daniel Wen, Owain Evans, and Samuel Marks. Activation oracles: Training and evaluating LLMs as general-purpose activation explainers.arXiv preprint arX...
-
[2023]
Aditi Panickssery, Samuel R
URLhttps://arxiv.org/abs/2309.15840. Aditi Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. InAdvances in Neural Information Processing Systems, volume 37, pages 68772–68802,
-
[2024]
Teaching models to express their uncertainty in words.arXiv preprint arXiv:2205.14334,
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words.arXiv preprint arXiv:2205.14334,
-
[2025]
Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221,
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221,
-
[2026]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt
Accessed: 2026-05-07. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding,
2026
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.