Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

From Emergence to Control: Probing and Modulating Self-Reflection in Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Self-reflection in LLMs is a latent pretraining capability that can be dialed up or down by steering a single activation-space direction, improving accuracy by up to 12 points or cutting output length by over 32 percent without retraining.

desk verdict Solid steering results, but the paper's central claim that it modulates self-reflection is not actually verified—it never measures whether steering changes reflection-keyword frequency. read the letter →

arxiv 2506.12217 v1 pith:WHLC7VVV submitted 2025-06-13 cs.LG cs.AIcs.CL

Xudong Zhu , Jiachen Jiang , Mohammad Mahdi Khalili , Zhihui Zhu This is my paper
classification cs.LGcs.AIcs.CL
keywords self-reflectionlargelanguagemodelsactivationsteeringdifference-of-meanslinearrepresentationhypothesisreasoningefficiencyemergentbehaviortest-timecompute
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish two things. First, self-reflective reasoning—pausing with tokens like "wait" to reconsider—is not created by reinforcement learning with verifiable rewards; it already exists, rarely, in pretrained models and can be elicited by inserting reflection-triggering reasoning traces, raising its frequency from 0.6% to 18.6% in Qwen2.5. Second, the decision to reflect is encoded as a linear direction in the model's residual-stream activations, and adding or subtracting this "self-reflection vector" at a middle layer controllably enhances or suppresses reflection. If true, model internals provide a training-free knob for trading reasoning quality against compute: enhancing the direction raises Pass@1 by up to 12 percentage points, while suppressing it shortens outputs by over 32% with little accuracy loss. A sympathetic reader would take the main contribution to be evidence that a complex emergent behavior reduces to a single manipulable internal feature.

What carries the argument

The central object is the self-reflection vector $v^{(\ell)}$, the difference-of-means between hidden states $H_{\text{reflect}}^{(\ell)}$ and $H_{\text{non-reflect}}^{(\ell)}$ at layer $\ell$ (Eq. 5), applied as a residual-stream intervention $\tilde{h}^{(\ell)} = h^{(\ell)} + \alpha v^{(\ell)} \langle h^{(\ell)}, v^{(\ell)} \rangle$ (Eq. 6). The reflection-inducing tokens are the tokens immediately preceding an explicit reflection marker ("wait"), and the non-reflection tokens are identical surface forms that are not followed by reflection within 100 tokens. This contrast controls for token identity so that the direction is argued to capture the model's internal transition into reflective reasoning; the scalar $\alpha$ is the control knob. The machinery also includes reflection-inducing probing, which injects pre-reflection chains-of-thought from a fine-tuned model into a pretrained model to elicit latent reflection, and UMAP visualizations showing separation between the two hidden-state sets.

What would settle it

Run the SR Enhanced and SR Suppressed interventions on a held-out set and count how often the model emits "wait" or any of the other reflection keywords. If the frequency does not move in the expected direction with $\alpha$, the accuracy and length effects are not caused by modulating reflection. As a control, build a difference-of-means vector from the same hidden states with random reflective/non-reflective labels; if it produces comparable accuracy and length effects, the discovered direction is generic next-token steering rather than a reflection-specific control.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that self-reflection is not an artifact of RLVR fine-tuning but a latent representational feature that appears during pretraining, and that the degree of reflection is modulated by a single direction in activation space. Constructing the direction as the difference-of-means between hidden states of reflection-inducing tokens (tokens immediately preceding "wait") and identical surface-form tokens in non-reflective contexts, and applying the linear intervention in Eq. (6), the authors report bidirectional control: positive alpha enhances reflection and improves accuracy by up to 12 percentage points on MATH-500 (e.g., Qwen2.5 7B from 44.6% to 56.6%), while negative alpha suppresses reflection and reduces average response length by over 32% (e.g., DeepSeek-R1-7B from 3585 to 2439 tokens) with minor accuracy cost. The direction transfers from GPQA Diamond to MATH-500 and differs from the embedding of "wait", which the authors read as evidence that it encodes reflective state rather than surface token cues.

Load-bearing premise

The load-bearing premise is that the difference-of-means vector in Eq. (5), computed between hidden states that immediately precede a "wait" token and states of identical surface-form tokens that do not, encodes self-reflection rather than the trivial fact that the model is about to emit different next tokens; the paper never tests whether the intervention changes the frequency of "wait" or other reflection keywords.

Editorial extensions

If this is right

  • Enhancing the self-reflection vector at a middle layer improves reasoning accuracy by up to 12 percentage points on MATH-500 and also on AIME 2024 and GPQA Diamond across Qwen2.5, DeepSeek-R1, and Llama 3.1 8B models, with no additional training.
  • Suppressing the vector shortens generations substantially, by about 32% on DeepSeek-R1-7B MATH-500 and often by more than 50% on other model-task pairs, while preserving most accuracy and offering a compute-cost dial.
  • Self-reflection vectors extracted from one domain (GPQA Diamond) transfer to another (MATH500) with similar gains, suggesting a shared, task-agnostic internal reflection mechanism.
  • Pretrained models that rarely show reflection on their own, such as Qwen2.5 and Llama 3.1, can be steered to reflect—Qwen2.5 7B jumps from 44.8% to 56.8% Pass@1 on MATH-500—so reflection control does not require RLVR-distilled models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the vector truly isolates reflective state rather than surface form, steering should also raise the frequency of other reflection markers such as "re-check" or "reconsider" even though the vector was built only from "wait" contexts; measuring that would confirm the mechanism generalizes.
  • The reported trade-off suggests an adaptive controller: estimate per-question difficulty from hidden-state geometry and set $\alpha$ dynamically, letting the model reflect more on hard problems and less on easy ones; the authors explicitly leave autonomous adjustment as future work.
  • Because the method needs hidden-state access, an API-only variant might estimate the reflection direction from log-probability differences over candidate next tokens; testing whether such a proxy reproduces the accuracy and length effects would extend the control to closed models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that self-reflection in LLMs is a latent capability that emerges during pretraining, not solely from RLVR fine-tuning. It supports this with a keyword-based frequency analysis (0.6% for a pretrained model, rising to 18.6% after injecting fine-tuned chain-of-thought), a UMAP visualization of hidden-state separation between reflection-inducing and non-reflection-inducing tokens, and a difference-of-means 'self-reflection vector' (Eq. 5) used in a linear activation intervention (Eq. 6). The intervention is reported to improve Pass@1 by up to 12 percentage points and reduce output length by over 32% across reasoning benchmarks, with cross-domain transfer. The central claim is that this vector modulates self-reflection and enables a performance-efficiency trade-off without additional training.

Significance. If the central claim held, the paper would make a meaningful contribution: it would show that a complex meta-cognitive behavior like self-reflection is linearly encoded and controllable, with practical implications for test-time reasoning efficiency. The paper also provides a useful comparison across model families and benchmarks, and its experiments are reproducible in principle. However, the evidence as presented does not yet support the claim that the intervention specifically modulates self-reflection rather than generic token-level behavior. The paper ships no code or checklists in the text, but the methodology is sufficiently standard that the main gap is experimental verification, not reproducibility.

major comments (3)
  1. [§3.2, §4.1, §4.3 (Eq. 5, Table 1)] The self-reflection vector v^(ℓ) in Eq. (5) is defined as the difference between hidden states immediately preceding a 'wait' token and hidden states of the same surface-form tokens that do not precede 'wait'. This contrast is definitionally aligned with next-token prediction: the two sets of states differ precisely because the model is about to emit different next tokens. The paper never shows that the separation in Figure 3 is not merely an artifact of this next-token identity, nor does it demonstrate that the intervention selectively alters self-reflection rather than producing generic token-level steering. A direct test would be to report, under SR Enhanced/Suppressed, the frequency of the self-reflection keywords defined in Appendix B. Table 1 reports only Pass@1 and LEN; without this behavioral check, the accuracy and length effects could arise from any stylistic shift, and the claim of 'modulating self-reflection' is not internally verified.
  2. [§3.1, Figure 1] The emergence claim rests on a comparison between a 0.6% spontaneous reflection rate and an 18.6% rate under Reflection-Inducing Probing. This comparison lacks error bars, significance testing, and a control condition. The probing method injects pre-reflection chain-of-thought from the fine-tuned model into the pretrained model and then measures whether the pretrained model emits a reflection token 'at the appropriate point'. The baseline, by contrast, is natural generation with no injection. The two settings differ in context length, prompt format, and the presence of fine-tuned reasoning traces, any of which could explain the frequency difference. A control condition injecting non-reflective chain-of-thought of comparable length is needed to support the claim that the increase reflects a latent capacity for self-reflection.
  3. [§4.3, Table 1, Appendix C] The paper selects the steering strength α and injection layer using a validation set (10% of the data) and reports test results averaged over five runs, but it does not specify whether a single α is used across all test prompts, how sensitive the results are to the validation split, or whether the reported best results involve per-dataset tuning that could overfit. More importantly, there is no ablation against a control direction, such as a random unit vector or the difference-of-means direction for a non-reflective token like 'therefore' or 'however'. Such a control is necessary to establish that the observed improvements and length reductions are specific to the self-reflection contrast rather than generic properties of activation steering along any mean-difference direction.
minor comments (5)
  1. [§3 (nomenclature)] In Section 3, the text says 'we compare DeepSeek-R1-Distill-Qwen-1.5B (Aft), a model fine-tuned from Qwen2.5-1.5B (Apt)' and then later refers to 'both pretrained models Aft and fine-tuned ones Apt', swapping the subscripts. This is confusing and contradicts Section 2.2; the definitions should be stated once and used consistently throughout.
  2. [Eq. (6)] The intervention in Eq. (6) includes a projection term ⟨h^(ℓ), v^(ℓ)⟩ that scales the added vector by the norm of the current hidden state along v^(ℓ). This differs from the standard contrastive activation addition (CAA) formulation, which adds the steering vector directly. The authors do not justify this scaling choice, and it makes the effective steering strength dependent on the varying norm of h^(ℓ), which complicates the interpretation of α. A brief justification or a comparison with the unscaled version would improve clarity.
  3. [Figure 1] The left panel of Figure 1 does not show the numerical values 0.6% and 18.6% on the plot itself; the reader must rely on the text. The figure should be self-contained, for instance by annotating the bars with the percentages. Also, the y-axis is labeled 'Percentage' but the caption could specify that it is the percentage of generations containing at least one self-reflection keyword.
  4. [Appendix B] The paper states that the analytical framework can generalize beyond 'wait' to any token with an analogous reflective role, but all experiments use 'wait' as the sole reflection-inducing token for vector construction. It would strengthen the paper to show at least one additional marker (e.g., 'let me double-check') in the vector construction or the intervention analysis, to substantiate the claimed generality.
  5. [Appendix C] The ablation in Appendix C fixes α=0.01 for all layers, but the layer-selection procedure is described as involving a grid search over α for each candidate layer. The relationship between the per-layer α search and the fixed-α layer comparison should be clarified: are the results in Figure 7 obtained with α=0.01 only, or with the best α per layer?

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the self-reflection vector is an empirical contrastive fit, and the reported accuracy and length effects are measured outcomes rather than the fitted quantity by construction.

full rationale

The paper's derivation chain is not circular. The self-reflection vector v^(l) in Eq. (5) is computed as the difference of means between hidden states that immediately precede a 'wait' token and hidden states of the same surface tokens that do not precede 'wait'. This is a supervised contrastive fit, not a prediction. The intervention in Eq. (6) shifts activations along this fitted direction, and its effects are evaluated on held-out Pass@1 and output length (Table 1). Those outcomes are not the same quantity as the contrast used to build v^(l); the accuracy gains and length reductions are therefore empirical findings rather than consequences of the vector's definition. The paper does not directly measure whether steering changes the frequency of the Appendix B reflection keywords, which is a construct-validity gap: the claim that the intervention modulates 'self-reflection' rather than generic token-level behavior is under-supported. Specifically, Section 4.3 reports only Pass@1 and LEN, not keyword frequency. However, under-support is not circularity. No load-bearing step invokes a self-citation or a uniqueness theorem from the authors' prior work; the cited difference-in-means and linear-representation methods [16,32,33] are external and independent. The vector's construction, the reflection-inducing probing experiment, and the transfer experiments are all self-contained empirical procedures. Accordingly, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central contributions rest on two load-bearing assumptions: (1) keyword presence is a valid measure of self-reflection, and (2) the difference-of-means direction is specific to self-reflection rather than to next-token prediction of the keyword. Neither is independently verified. The free parameters, including alpha, injection layer, keyword list, and the 100-token window, are chosen to maximize validation performance or are hand-set without sensitivity analysis.

free parameters (4)
  • alpha (steering strength) = not reported
    Selected via grid search over [-1, 1] on a 10% validation set for each model, dataset, and direction. The chosen values are not listed in the paper.
  • injection layer(s) = not reported
    Layer 14 for DeepSeek-R1, multiple layers for Qwen, selected based on validation performance (Appendix C).
  • non-reflection window = 100 tokens
    Hand-chosen threshold used to label tokens as non-reflection-inducing (Section 3.2).
  • reflection keyword list = 17 phrases including 'wait', 'rethink', 'check again'
    Hand-curated list; detection threshold is the presence of any keyword (Appendix B).
assumptions (5)
  • domain assumption The presence of tokens like 'wait', 'rethink', etc. is a valid and sufficient indicator of self-reflection behavior.
    Used throughout to detect reflection instances (Appendix B); no human validation or precision/recall analysis is reported, and 'wait' can appear in non-reflective contexts such as 'wait for the result'.
  • domain assumption Self-reflection is encoded as a linear direction in residual stream activation space.
    Motivated by prior work on the linear representation hypothesis [32,33]; the paper does not test this assumption for self-reflection specifically, yet it is the basis for the difference-of-means vector in Eq. (5).
  • ad hoc to paper Hidden states preceding a reflection token are directly comparable to hidden states of the same surface-form tokens that do not precede a reflection token, and their mean difference isolates the self-reflection signal.
    This assumption is introduced in Section 3.2 and is confounded by the fact that hidden states encode the next-token distribution, so states before 'wait' and before other tokens must differ.
  • domain assumption The hidden state of the final token in r1 adequately represents the model's transition into self-reflection.
    Section 3.2 states the final token is a 'reflection-inducing token' and its hidden state captures the entire context; this is a simplifying assumption about where reflection decisions are localized.
  • standard math Standard decoder-only transformer architecture and residual stream update equations.
    Section 2.1 Eq. (1) uses standard transformer layer definitions.
invented entities (1)
  • Self-reflection vector v^(ℓ)
    purpose: A direction in activation space used to enhance or suppress self-reflection behavior via linear intervention (Eq. 6).
    The vector is empirically computed from data as a difference of means, not predicted from theory; its interpretation as specifically encoding self-reflection, rather than next-token 'wait' prediction, is not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Emergence to Control: Probing and Modulating Self-Reflection in Language Models." pith.science (2026). https://pith.science/paper/WHLC7VVV

@misc{pith2026250612217,
  author       = {Pith},
  title        = {Pith review of: From Emergence to Control: Probing and Modulating Self-Reflection in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WHLC7VVV}},
  note         = {Machine review of arXiv:2506.12217}
}
read the original abstract

Self-reflection -- the ability of a large language model (LLM) to revisit, evaluate, and revise its own reasoning -- has recently emerged as a powerful behavior enabled by reinforcement learning with verifiable rewards (RLVR). While self-reflection correlates with improved reasoning accuracy, its origin and underlying mechanisms remain poorly understood. In this work, {\it we first show that self-reflection is not exclusive to RLVR fine-tuned models: it already emerges, albeit rarely, in pretrained models}. To probe this latent ability, we introduce Reflection-Inducing Probing, a method that injects reflection-triggering reasoning traces from fine-tuned models into pretrained models. This intervention raises self-reflection frequency of Qwen2.5 from 0.6\% to 18.6\%, revealing a hidden capacity for reflection. Moreover, our analysis of internal representations shows that both pretrained and fine-tuned models maintain hidden states that distinctly separate self-reflective from non-reflective contexts. Leveraging this observation, {\it we then construct a self-reflection vector, a direction in activation space associated with self-reflective reasoning}. By manipulating this vector, we enable bidirectional control over the self-reflective behavior for both pretrained and fine-tuned models. Experiments across multiple reasoning benchmarks show that enhancing these vectors improves reasoning performance by up to 12\%, while suppressing them reduces computational cost, providing a flexible mechanism to navigate the trade-off between reasoning quality and efficiency without requiring additional training. Our findings further our understanding of self-reflection and support a growing body of work showing that understanding model internals can enable precise behavioral control.

Figures

Figures reproduced from arXiv: 2506.12217 by the authors.

Figure 1
Figure 1. Left: Frequency distribution of self-reflection behaviors for pretrained model Apt, fine-tuned model Aft, and Apt with reflection-inducing probing by injecting CoT from Aft, evaluated on the MATH500 dataset. Right: A representative example of spontaneous self-reflection in Apt, demonstrating that this capability emerges naturally during pretraining, albeit with different self-reflection tokens than those typically o… view at source ↗
Figure 2
Figure 2. Hidden State Selection Methodology: We identify tokens immediately preceding "wait" tokens as reflection-inducing tokens and extract their hidden states. For comparison, we collect hidden states of identical tokens appearing in non-reflective contexts. This contrastive approach enables us to analyze the neural signatures associated with self-reflection in language models. instances are characterized by explicit reco… view at source ↗
Figure 3
Figure 3. UMAP visualization of hidden state representations of (Left) Aft and (Right) Apt. Both models show separation between H (ℓ) reflect and H (ℓ) non−reflect. 4 Controlling Self-Reflection in Language Models In this section, we introduce our approach for identifying and manipulating self-reflection vectors in LLMs. Building on our finding in the last section that hidden representations distinctly separate self-reflectiv… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Effect of α on performance and response length on Math500 Dataset (reflection vector is in￾jected on layer 14). Ablation study on α To illustrate the effect of the lin￾ear intervention method for controlling self-reflection, we conduct an ablation study by varying the …
Figure 5
Figure 5. Figure 5: Left: Cosine similarity of self-reflection vectors and the “wait” token across MATH500 and GPQA datasets. The green curve shows similarity between vectors from MATH500 and GPQA. Blue and orange curves show similarity with the “wait” token. Right: Performance on MATH500…
Figure 6
Figure 6. Figure 6: Left: List of self-reflection keywords used in our analysis. Right: Frequency distribution of these keywords across model-generated outputs. To validate this detection method, we applied it to model outputs on the MATH500 benchmark using the DeepSeek-R1-Distill-Qwen-1.…
Figure 7
Figure 7. Figure 7: Effect of injecting the self-reflection vector [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: UMAP visualization of hidden states from the Qwen2.5-1.5B model across 14 layers. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: UMAP visualization of hidden states from the DeepSeek-R1-Distill-Qwen-1.5B model across 14 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.

Reference graph

Works this paper leans on

60 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Towards large reasoning models: A survey of reinforced reasoning with large language models,

    F. Xu, Q. Hao, Z. Zong, J. Wang, Y. Zhang, J. Wang, X. Lan, J. Gong, T. Ouyang, F. Meng,et al., “Towards large reasoning models: A survey of reinforced reasoning with large language models,”arXiv preprint arXiv:2501.09686, 2025

    arXiv 2025

  2. [2]

    Self-reasoning language models: Unfold hidden reasoning chains with few reasoning catalyst,

    H. WANG, D. Cai, W. Zhong, S. Huang, J. Z. Pan, Z. Liu, and K.-F. Wong, “Self-reasoning language models: Unfold hidden reasoning chains with few reasoning catalyst,” inWorkshop on Reasoning and Planning for Large Language Models, 2025

    work page 2025

  3. [3]

    Reinforcement learning with verifiable rewards: Grpo’s effective loss, dynamics, and success amplification,

    Y. Mroueh, “Reinforcement learning with verifiable rewards: Grpo’s effective loss, dynamics, and success amplification,”arXiv preprint arXiv:2503.06639, 2025

    arXiv 2025

  4. [4]

    R1-omni: Explainable omni-multimodal emotion recognition with rein- forcement learning,

    J. Zhao, X. Wei, and L. Bo, “R1-omni: Explainable omni-multimodal emotion recognition with rein- forcement learning,”arXiv preprint arXiv:2503.05379, 2025

    arXiv 2025

  5. [5]

    Reasoning beyond limits: Advances and open problems for llms,

    M. A. Ferrag, N. Tihanyi, and M. Debbah, “Reasoning beyond limits: Advances and open problems for llms,”arXiv preprint arXiv:2503.22732, 2025

    arXiv 2025

  6. [6]

    Expanding rl with verifiable rewards across diverse domains,

    Y. Su, D. Yu, L. Song, J. Li, H. Mi, Z. Tu, M. Zhang, and D. Yu, “Expanding rl with verifiable rewards across diverse domains,”arXiv preprint arXiv:2503.23829, 2025

    arXiv 2025

  7. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi,et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  8. [8]

    Understanding r1-zero-like training: A critical perspective,

    Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin, “Understanding r1-zero-like training: A critical perspective,”arXiv preprint arXiv:2503.20783, 2025

    arXiv 2025

Show all 60 references
  1. [9]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild,

    W. Zeng, Y. Huang, Q. Liu, W. Liu, K. He, Z. Ma, and J. He, “Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild,”arXiv preprint arXiv:2503.18892, 2025

    2025 arXiv

  2. [10]

    Ttrl: Test-time reinforcement learning,

    Y. Zuo, K. Zhang, S. Qu, L. Sheng, X. Zhu, B. Qi, Y. Sun, G. Cui, N. Ding, and B. Zhou, “Ttrl: Test-time reinforcement learning,”arXiv preprint arXiv:2504.16084, 2025

    2025 arXiv

  3. [11]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?,

    Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, S. Song, and G. Huang, “Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?,”arXiv preprint arXiv:2504.13837, 2025. 10

    2025 arXiv

  4. [12]

    Self-reflection outcome is sensitive to prompt construction,

    F. Liu, N. AlDahoul, G. Eady, Y. Zaki, B. AlShebli, and T. Rahwan, “Self-reflection outcome is sensitive to prompt construction,”arXiv preprint arXiv:2406.10400, 2024

    2024 arXiv

  5. [13]

    Dynamic early exit in reasoning models,

    C. Yang, Q. Si, Y. Duan, Z. Zhu, C. Zhu, Z. Lin, L. Cao, and W. Wang, “Dynamic early exit in reasoning models,”arXiv preprint arXiv:2504.15895, 2025

    2025

  6. [14]

    Self-reflection in llm agents: Effects on problem-solving performance,

    M. Renze and E. Guven, “Self-reflection in llm agents: Effects on problem-solving performance,”arXiv preprint arXiv:2405.06682, 2024

    2024 arXiv

  7. [15]

    Stop overthinking: Asurveyonefficientreasoningforlargelanguagemodels,

    Y.Sui,Y.-N.Chuang,G.Wang,J.Zhang,T.Zhang,J.Yuan,H.Liu,A.Wen,S.Zhong,H.Chen,etal.,“Stop overthinking: Asurveyonefficientreasoningforlargelanguagemodels,”arXivpreprintarXiv:2503.16419, 2025

    2025 arXiv

  8. [16]

    Steering llama 2 via contrastive activation addition,

    N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. Turner, “Steering llama 2 via contrastive activation addition,” inProceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers)(L.-W. Ku, A. Martins, and V. Srik...

    2024

  9. [17]

    Generating wikipedia by summarizing long sequences,

    P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, and N. Shazeer, “Generating wikipedia by summarizing long sequences,”arXiv preprint arXiv:1801.10198, 2018

    2018 arXiv

  10. [18]

    Beyond accuracy: Evaluating the reasoning behavior of large language models - a survey,

    P. Mondorf and B. Plank, “Beyond accuracy: Evaluating the reasoning behavior of large language models - a survey,” inFirst Conference on Language Modeling, 2024

    2024

  11. [19]

    Oat: A research-friendly framework for llm online alignment,

    Z. Liu, C. Chen, C. Du, W. S. Lee, and M. Lin, “Oat: A research-friendly framework for llm online alignment,” 2024

    2024

  12. [20]

    Self- consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self- consistency improves chain of thought reasoning in language models,” inThe Eleventh International Conference on Learning Representations, 2023

    2023

  13. [21]

    X-reasoner: Towards generalizable reasoning across modalities and domains,

    Q.Liu,S.Zhang,G.Qin,T.Ossowski,Y.Gu,Y.Jin,S.Kiblawi,S.Preston,M.Wei,P.Vozila,T.Naumann, and H. Poon, “X-reasoner: Towards generalizable reasoning across modalities and domains,” 2025

    2025

  14. [22]

    Reinforcement learning enhanced llms: A survey,

    S. Wang, S. Zhang, J. Zhang, R. Hu, X. Li, T. Zhang, J. Li, F. Wu, G. Wang, and E. Hovy, “Reinforcement learning enhanced llms: A survey,”arXiv preprint arXiv:2412.10400, 2024

    2024 arXiv

  15. [23]

    Absolute zero: Reinforced self-play reasoning with zero data,

    A. Zhao, Y. Wu, Y. Yue, T. Wu, Q. Xu, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang, “Absolute zero: Reinforced self-play reasoning with zero data,”arXiv preprint arXiv:2505.03335, 2025

    2025 arXiv

  16. [24]

    When hindsight is not 20/20: Testing limits on reflective thinking in large language models,

    Y. Li, C. Yang, and A. Ettinger, “When hindsight is not 20/20: Testing limits on reflective thinking in large language models,”arXiv preprint arXiv:2404.09129, 2024

    2024 arXiv

  17. [25]

    Demystifyinglongchain-of-thoughtreasoninginLLMs,

    E.Yeo,Y.Tong,X.Niu,G.Neubig,andX.Yue,“Demystifyinglongchain-of-thoughtreasoninginLLMs,” inICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025

    2025

  18. [26]

    Openai o1 system card,

    A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney,et al., “Openai o1 system card,”arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  19. [27]

    2 olmo 2 furious,

    T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, et al., “2 olmo 2 furious,”arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  20. [28]

    Qwen3technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. L...

    2025

  21. [29]

    Measuring mathematical problem solving with the math dataset,

    D.Hendrycks,C.Burns,S.Kadavath,A.Arora,S.Basart,E.Tang,D.Song,andJ.Steinhardt,“Measuring mathematical problem solving with the math dataset,”NeurIPS, 2021

    2021

  22. [30]

    Qwen2.5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y....

    2024 arXiv

  23. [31]

    Umap: Uniform manifold approximation and projection,

    L. McInnes, J. Healy, N. Saul, and L. Grossberger, “Umap: Uniform manifold approximation and projection,”The Journal of Open Source Software, vol. 3, no. 29, p. 861, 2018

    2018

  24. [32]

    Refusal in language models is mediated by a single direction,

    A. Arditi, O. B. Obeso, A. Syed, D. Paleka, N. Rimsky, W. Gurnee, and N. Nanda, “Refusal in language models is mediated by a single direction,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  25. [33]

    Axbench: Steering llms? even simple baselines outperform sparse autoencoders,

    Z. Wu, A. Arora, A. Geiger, Z. Wang, J. Huang, D. Jurafsky, C. D. Manning, and C. Potts, “Axbench: Steering llms? even simple baselines outperform sparse autoencoders,”arXiv preprint arXiv:2501.17148, 2025

    2025 arXiv

  26. [34]

    GPQA: A graduate-level google-proof q&a benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman, “GPQA: A graduate-level google-proof q&a benchmark,” inFirst Conference on Language Modeling, 2024

    2024

  27. [35]

    The llama 3 herd of models,

    A.Grattafiori,A.Dubey,A.Jauhri,A.Pandey,A.Kadian,A.Al-Dahle,A.Letman,A.Mathur,A.Schelten, A. Vaughan,et al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  28. [36]

    s1: Simple test-time scaling,

    N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. Hashimoto, “s1: Simple test-time scaling,” 2025

    2025

  29. [37]

    Discovering latent knowledge in language models without supervision,

    C. Burns, H. Ye, D. Klein, and J. Steinhardt, “Discovering latent knowledge in language models without supervision,”ArXiV, 2022

    2022

  30. [38]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,”arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  31. [39]

    A language model’s guide through latent space,

    D. Von Rütte, S. Anagnostidis, G. Bachmann, and T. Hofmann, “A language model’s guide through latent space,”arXiv preprint arXiv:2402.14433, 2024

    2024 arXiv

  32. [40]

    Improvingactivationsteeringinlanguagemodels with mean-centring,

    O.Jorgensen,D.Cope,N.Schoots,andM.Shanahan,“Improvingactivationsteeringinlanguagemodels with mean-centring,”arXiv preprint arXiv:2312.03813, 2023

    2023 arXiv

  33. [41]

    Finding alignments between interpretable causal variables and distributed neural representations,

    A. Geiger, Z. Wu, C. Potts, T. Icard, and N. Goodman, “Finding alignments between interpretable causal variables and distributed neural representations,” inCausal Learning and Reasoning, pp. 160–187, PMLR, 2024

    2024

  34. [42]

    Generative agents: Interactivesimulacraofhumanbehavior,

    J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactivesimulacraofhumanbehavior,”inInthe36thAnnualACMSymposiumonUserInterfaceSoftware and Technology (UIST ’23), UIST ’23, (New York, NY, USA), Association for Computing ...

    2023

  35. [43]

    Man is to computer programmer as woman is to homemaker? debiasing word embeddings,

    T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai, “Man is to computer programmer as woman is to homemaker? debiasing word embeddings,”Advances in neural information processing systems, vol. 29, 2016

    2016

  36. [44]

    Sparseautoencodersfindhighlyinter- pretable features in language models,

    R.Huben,H.Cunningham,L.R.Smith,A.Ewart,andL.Sharkey,“Sparseautoencodersfindhighlyinter- pretable features in language models,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  37. [45]

    Gold doesn‘t always glitter: Spectral removal of linear and nonlinear guardedattributeinformation,

    S. Shao, Y. Ziser, and S. B. Cohen, “Gold doesn‘t always glitter: Spectral removal of linear and nonlinear guardedattributeinformation,”inProceedingsofthe17thConferenceoftheEuropeanChapteroftheAssociation for Computational Linguistics(A. Vlachos and I. Augenstein, eds.), (Dubr...

    2023

  38. [46]

    LEACE: Perfect linear concept erasure in closed form,

    N. Belrose, D. Schneider-Joseph, S. Ravfogel, R. Cotterell, E. Raff, and S. Biderman, “LEACE: Perfect linear concept erasure in closed form,” inThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  39. [47]

    Monitoring latent world states in language models with proposi- tional probes,

    J. Feng, S. Russell, and J. Steinhardt, “Monitoring latent world states in language models with proposi- tional probes,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  40. [48]

    Let’s verify step by step,

    H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  41. [49]

    Self-refine: Iterative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, “Self-refine: Iterative refinement with self-feedback,” inThirty-seventh Conference on...

    2023

  42. [50]

    Fine-tuning with divergent chains of thought boosts reasoning through self-correction in language models,

    H. Puerto, T. Chubakov, X. Zhu, H. T. Madabushi, and I. Gurevych, “Fine-tuning with divergent chains of thought boosts reasoning through self-correction in language models,” 2024

    2024

  43. [51]

    STar: Bootstrapping reasoning with reasoning,

    E. Zelikman, Y. Wu, J. Mu, and N. Goodman, “STar: Bootstrapping reasoning with reasoning,” in Advances in Neural Information Processing Systems(A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, eds.), 2022

    2022

  44. [52]

    Let’s verify step by step,

    H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” inThe Twelfth International Conference on Learning Representations, 2023

    2023

  45. [53]

    Llms can easily learn to reason from demonstrations structure, not content, is what matters!,

    D. Li, S. Cao, T. Griggs, S. Liu, X. Mo, E. Tang, S. Hegde, K. Hakhamaneshi, S. G. Patil, M. Zaharia,et al., “Llms can easily learn to reason from demonstrations structure, not content, is what matters!,”arXiv preprint arXiv:2502.07374, 2025

    2025 arXiv

  46. [54]

    Shorterbetter: Guidingreasoningmodelstofindoptimalinferencelengthforefficient reasoning,

    J.YiandJ.Wang,“Shorterbetter: Guidingreasoningmodelstofindoptimalinferencelengthforefficient reasoning,”arXiv preprint arXiv:2504.21370, 2025

    2025

  47. [55]

    Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought,

    Q. Chen, L. Qin, J. WANG, J. Zhou, and W. Che, “Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  48. [56]

    Kimi k1. 5: Scaling reinforcement learning with llms,

    K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao,et al., “Kimi k1. 5: Scaling reinforcement learning with llms,”arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  49. [57]

    Rethinking reflection in pre-training,

    D. J. Shah, P. Rushton, S. Singla, M. Parmar, K. Smith, Y. Vanjani, A. Vaswani, A. Chaluvaraju, A. Hojel, A. Ma,et al., “Rethinking reflection in pre-training,”arXiv preprint arXiv:2504.04022, 2025

    2025 arXiv

  50. [58]

    Reflexion: Languageagentswithverbal reinforcement learning,

    N.Shinn, F.Cassano, A.Gopinath, K.Narasimhan, andS.Yao, “Reflexion: Languageagentswithverbal reinforcement learning,”Advances in Neural Information Processing Systems, vol. 36, pp. 8634–8652, 2023

    2023

  51. [59]

    Rein- forcement learning for reasoning in large language models with one training example,

    Y. Wang, Q. Yang, Z. Zeng, L. Ren, L. Liu, B. Peng, H. Cheng, X. He, K. Wang, J. Gao,et al., “Rein- forcement learning for reasoning in large language models with one training example,”arXiv preprint arXiv:2504.20571, 2025

    2025 arXiv

  52. [60]

    There may not be aha moment in r1-zero-like training — a pilot study

    Z. Liu, C. Chen, W. Li, T. Pang, C. Du, and M. Lin, “There may not be aha moment in r1-zero-like training — a pilot study.”https://oatllm.notion.site/oat-zero, 2025. Notion Blog. 13 A UMAP Visualization of Self-Reflection States To investigate how self-reflective states are in...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.