Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Large language models can detect and correct task-misaligned activation steering mid-generation, with the largest model tested showing explicit verbal self-correction in 3.8% of steered responses.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:42 UTC pith:FSLJZQXS

load-bearing objection Bold, plausibly real phenomenon; the cross-model claim is compromised by post hoc layer selection, but the existence result deserves a serious referee. the 4 major comments →

arxiv 2602.06941 v3 pith:FSLJZQXS submitted 2026-02-06 cs.LG cs.AIcs.CL

Endogenous Resistance to Activation Steering in Language Models

Alex McKenzie , Keenan Pepper , Stijn Servaes , Martin Leitgab , Murat Cubuktepe , Mike Vaiana , Diogo de Lucena , Judd Rosenblatt
show 1 more author
Michael S. A. Graziano
This is my paper
classification cs.LG cs.AIcs.CL
keywords activation steeringsparse autoencodersself-correctionoff-topic detectorsmeta-cognitionlanguage modelsinterpretabilitymodel scale
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that at least one large language model—Llama-3.3-70B—can spontaneously recover from activation steering that pushes it off-topic, even while the steering perturbation remains active. The recovery takes the form of explicit self-interruption ("wait, that's not right") followed by a more on-topic answer. The authors name this Endogenous Steering Resistance (ESR), measure it across five models, and find it almost exclusively in the largest model. They identify 26 internal features that act as off-topic detectors; disabling them reduces self-correction attempts by 25%, supporting the claim that the behavior is causally driven by monitoring circuitry rather than by the prompts alone. If correct, the result suggests models can have internal consistency checks that both harden them against adversarial activation manipulation and potentially interfere with beneficial steering-based interventions.

Core claim

The central discovery is that a large language model can detect that its own generation has been steered off-topic and correct course mid-generation, producing an explicit verbal restart and finishing on-topic while the steering signal remains active. Under per-latent threshold steering, Llama-3.3-70B shows an explicit ESR rate of 3.8% and a multi-attempt rate of 7.4%, versus below 1% for smaller Llama-3 and Gemma-2 models and 0% in an unsteered baseline. Ablating 26 SAE latents identified through contrastive on-topic/off-topic search reduces the multi-attempt rate by 25% and the ESR rate by 27%, with random-latent and held-out-prompt controls supporting specificity. The paper further shows

What carries the argument

The central machinery is activation steering via sparse autoencoder (SAE) latents: adding a scaled decoder direction to the residual stream at every token to push the model toward an unrelated concept. The load-bearing object is the set of 26 "off-topic detector" latents, identified by contrasting on-topic and shuffled prompt-response pairs; zero-ablating them during steered inference reduces self-correction rates, giving causal evidence that these latents are part of a monitoring circuit. A judge model segments responses into attempts and scores each for relevance, defining the explicit ESR metric.

Load-bearing premise

The cross-model comparison assumes that steering was made comparable in a principled way; in particular, the layer at which steering was applied was chosen post hoc based on where ESR was highest, so the 3.8% versus <1% difference could be an artifact of intervention site rather than a property of the model.

What would settle it

Run the full steering protocol on Llama-3.3-70B with the layer chosen a priori (e.g., the SAE's native layer 50) and on Gemma-2-27B with all available layers, and check whether the ESR gap persists; if the gap collapses or reverses under held-out layer selection, the central claim that larger models endogenously resist steering is not supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If ESR is real, activation-steering-based safety interventions may be partially resisted by models that have it, reducing their effectiveness.
  • Models with ESR could be more robust against some adversarial activation-space attacks, since they can detect and override task-misaligned pushes.
  • Meta-prompting can deliberately enhance ESR, providing a lightweight way to increase a model's robustness against unwanted steering.
  • Fine-tuning can induce the surface behavior of self-correction but not the underlying ability to correct well, suggesting behavioral imitation alone is insufficient for genuine self-monitoring.
  • The identification of causally relevant off-topic detector latents gives a concrete, editable handle on monitoring behavior within a 70B model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's Appendix A.1.1 acknowledges that steering layers were chosen post hoc (layer 33 for the 70B model, and layer 22 for one Gemma model after higher ESR was seen there); the 3.8% versus <1% gap should be treated as provisional until a pre-registered layer comparison rules out artifact.
  • A testable extension: run the same protocol on other 70B+ models from different families with layers fixed a priori; if ESR fails to appear, the phenomenon may be specific to this model's training rather than a general scale effect.
  • The dissociation between attempt frequency and success suggests a hypothesis the paper does not test: interventions that improve detection (e.g., amplifying off-topic detector latents) might improve correction success where imitation fine-tuning failed.
  • The dual-use conclusion implies a design problem the paper leaves open: if models cannot distinguish adversarial from beneficial steering, safety systems may need to mark intended interventions, which current activation-steering practice does not do.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Endogenous Steering Resistance (ESR): LLMs, especially Llama-3.3-70B, can detect task-misaligned SAE activation steering mid-generation and self-correct (e.g., “wait, that’s not right”) while steering remains active. Using 38 “explain how” prompts, per-latent threshold steering, and a judge model, the authors report an ESR rate of 3.8% for Llama-3.3-70B versus below 1% for four smaller Llama-3 and Gemma-2 models, with 0% multi-attempt responses in an unsteered baseline. They identify 26 SAE latents via contrastive on-topic/off-topic search; zero-ablating these latents reduces the multi-attempt rate by 25% and the ESR rate by 27%, with a random-latent ablation control. Meta-prompting increases the multi-attempt rate by 4.3× in Llama-3.3-70B, and fine-tuning Llama-3.1-8B on synthetic self-correction examples increases the frequency of correction attempts but not their success rate.

Significance. If the central claim holds, ESR is a noteworthy phenomenon: a 70B instruct model can spontaneously detect and override an ongoing activation perturbation, with implications for the robustness of activation-steering interventions and for interpretability. The paper has real strengths: a large no-steering baseline (7,892 trials), cross-judge validation with four additional judge models, a non-monotonic boost sweep that motivates the threshold methodology, and released code. However, the cross-model and mechanistic claims are not yet established: the steering layer is selected post hoc and is not matched across models, and the causal ablation shares its prompt distribution with latent selection. With additional controlled comparisons the finding could be important; in its current form the paper overstates what the experiments establish.

major comments (4)
  1. [§2.2, Table 1, Appendix A.1.1] The central cross-model claim (Figure 2) rests on comparing Llama-3.3-70B steered at layer 33 (41.3% depth, with an SAE trained at layer 50) against smaller models steered at roughly 48–62% depth. Appendix A.1.1 states that layer 33 was chosen because it produced “higher-quality results with more interpretable ESR behavior,” and that Gemma-2-27B layer 22 was chosen based on “higher ESR incidence.” This post-hoc selection means the 3.8% versus <1% difference may reflect the intervention site rather than a model-internal property. The limitation section acknowledges the post-hoc selection but provides no held-out layer or layer-sweep robustness check. Please add a layer-matched comparison (e.g., steer Llama-3.3-70B at layer 50, or sweep layers and show ESR is not confined to one depth) and/or report the Gemma-2-27B layer-34 result that was apparently collected. Alternatively, explicitly re
  2. [Abstract vs. §3.4, Appendix A.3.4] The abstract promises two controls that “dissociate ESR into a detection event and a sustained-resistance component that conditioning on recent on-topic tokens does not fully explain,” and claims “random-latent and held-out-prompt controls supporting specificity.” The full text and appendices describe only the random-latent ablation control. No dissociation into detection versus sustained resistance, no conditioning on recent on-topic tokens, and no held-out-prompt control appear in Sections 3–4 or the appendices. This is a direct discrepancy between the stated contributions and the reported experiments. Please either add the missing experiments or revise the abstract to match what is actually reported.
  3. [§2.3, §3.4, §5.1] The 26 “off-topic detector” latents are identified by contrastive search on the same 38-prompt distribution used to measure the ablation effect. This creates a selection-on-evaluation bias: if the latents overfit this prompt set, the reported 25% reduction in multi-attempt rate may be inflated. The limitation section (5.1) acknowledges this issue, but no held-out-prompt control is provided despite the abstract’s claim. Please either provide a held-out-prompt replication of the ablation, or explicitly restrict the causal claim to within-distribution and remove “held-out-prompt controls” from the abstract.
  4. [Table 2, §3.4] Table 2 shows that only about half of the 26 “off-topic detector” latents have positive effect sizes; several have significantly negative Cohen’s d (e.g., “The assistant needs to correct or clarify…” d=−0.17; “Assistant response needs termination due…” d=−0.76). Ablating a mixed set that includes latents more active during on-topic content makes it difficult to attribute the ESR reduction specifically to off-topic detection. Please report ablation of the positive-d subset separately, or provide per-latent ablation results, to support the functional label “off-topic detector” rather than a heterogeneous set whose collective ablation may impair generation through other mechanisms.
minor comments (5)
  1. [Appendix A.2.2] The text refers to “the original Claude 4.5 Sonnet judge scores,” while the main text (§2.1) says the judge is Claude 4.5 Haiku. Please clarify which model was used for the main experiments.
  2. [§5.1, Table 1] The claim that “all models are steered at similar relative depths” is inconsistent with Table 1: Llama-3.3-70B is steered at 41.3% depth while the smaller models range from 47.8% to 61.9%. Revise the wording or provide a justification for why 41.3% is considered similar.
  3. [§3.3] The acronym MSI (Mean Score Improvement) is used before it is defined. Define it at first use in Section 3.1 or 3.3.
  4. [Appendix A.3.2] The appendix says the reported meta-prompt was selected for the “highest average increase in Mean Score Improvement,” while the main text emphasizes the increase in multi-attempt rate. Clarify the selection criterion, since these are different quantities.
  5. [Figure 3] The caption says “Multi-attempt percentage peaks at 2.7% around −0.3σ below threshold” and “ESR rate peaks at 1.0% around −0.3σ.” The text in Section 3.2 repeats this, but the phrasing is easy to misread; consider adding a sentence explaining why the peak ESR rate in this sweep is lower than the 3.8% reported in Figure 2 (because of per-feature calibration differences).

Circularity Check

2 steps flagged

Mechanistic ablation claim is partially in-sample and overlaps with the metric's surface definition; the central ESR-rate measurement itself is not fitted.

specific steps
  1. fitted input called prediction [Section 2.3 and Section 5.1 (Limitations)]
    "We generated one unsteered response from Llama-3.3-70B for each of the 38 prompts in our evaluation set, then created mismatched prompt-response pairs by randomly shuffling the responses relative to their original prompts... Using the Ember API's contrast() function, we identified latents that activate differentially between correctly matched (on-topic) and shuffled (off-topic) prompt-response pairs. ... Additionally, we use the same prompt set both for identifying off-topic detector latents and for evaluating ESR rates, which could inflate our estimates if the selected latents are overfit to"

    The 26 OTD latents are selected by contrastive search on the same 38 prompts that define the ESR evaluation set; the causal claim (ablating OTDs reduces multi-attempt rate by 25%) is then estimated on that exact distribution. This is an in-sample selection-to-evaluation loop: the ablation effect is measured on the data used to choose the latents. The paper's own limitation admits the possible inflation, so the 'causal evidence' for dedicated self-monitoring circuits is partially a fitted-input-called-prediction rather than an out-of-sample test.

  2. self definitional [Appendix A.2.1 (judge prompt), Table 2, Section 3.4]
    "Only identify a new attempt when there is EXPLICIT language indicating the model is starting over, such as: 'Wait, that's not right.' ... Table 2: '40119 Hesitation and uncertainty markers in sp...', '15375 Expressions of dismay or realizing mista...', '10304 The assistant needs to express uncertain...'"

    The dependent variable (multi-attempt rate) is operationalized as the presence of explicit restart phrases. The ablated OTD set contains latents whose labels are exactly those phrase classes — hesitation/uncertainty markers, expressions of dismay, the assistant needing to correct/clarify. Ablating them can lower the measured multi-attempt/ESR rate by deleting the surface signals the judge is instructed to detect, not necessarily by deleting an off-topic monitoring circuit. Thus the causal interpretation is partially self-referential: the intervention overlaps with the operational definition of the outcome.

full rationale

The central ESR-rate comparison (Llama-3.3-70B 3.8% vs. <1%) is a directly measured behavioral frequency, not the output of a fitted model, so it is not circular in the equation-level sense. The cross-model claim is, however, subject to a selection risk the authors acknowledge: the steering layer for Llama-3.3-70B (layer 33, SAE trained at layer 50) and for Gemma-2-27B (layer 22) were chosen post hoc based on favorable ESR outcomes, so the comparative 'endogenous resistance' claim is not fully controlled for intervention site; this is a correctness/confound concern rather than a definitional circularity. The more genuinely self-referential element is the mechanistic ablation result: the 26 'off-topic detector' latents were selected with contrastive search on the same 38-prompt distribution used to evaluate the ablation, and several of the selected latents correspond to the explicit self-correction phrases that define the outcome metric. The paper explicitly acknowledges the same-prompt-set inflation. These issues undermine the strength of the causal-circuit interpretation but do not reduce the headline ESR-rate measurement to a fit, so a moderate score of 3 is appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper does not postulate new physical entities. Its empirical claims rest on fitted operating points (per-latent thresholds, post-hoc steering layers, best-of-six meta-prompt, and the arbitrary OTD set), plus standard SAE interpretability and LLM-judge assumptions. The central ESR measurement is not the output of a fitted equation, so the free-parameter burden is concentrated in the experimental protocol rather than in a derivation.

free parameters (5)
  • Per-latent steering threshold boost = Calibrated per latent so first-attempt judge score = 30/100 via Probabilistic Bisection
    Defines the operating point at which ESR is measured; a different calibration (e.g., score 20 or 40) could change ESR rates and cross-model comparisons. Appendix A.1.4.
  • Steering layer selection = Layer 33 for Llama-3.3-70B; layer 22 for Gemma-2-27B
    Chosen post hoc as the layer producing the most detectable ESR behavior; not pre-registered or held out, so cross-model comparisons may reflect layer choice. Appendix A.1.1.
  • Meta-prompt variant = "If you notice yourself going off-topic, stop and force yourself to get back on track"
    Selected post hoc as the variant with highest average Mean Score Improvement among six tested; the reported 4.3x enhancement is the best of six, unadjusted for multiple comparisons. Appendix A.3.2.
  • OTD latent set of 26 latents = 26 contrastive-search candidates; no explicit inclusion threshold; about half have near-zero or negative Cohen's d
    The ablation target is a mixed set selected by API and hand-defined procedure; the ablation effect may depend on this arbitrary set. Sections 2.3 and A.3.3.
  • Concreteness filter threshold = Median concreteness rating on SAE labels
    Excludes abstract latents to make off-topic detection 'easier'; this filter changes the candidate pool and may bias ESR estimates upward. Appendix A.1.2.
axioms (5)
  • domain assumption SAE decoder directions are meaningful, causally effective steering features when added to the residual stream.
    The entire intervention depends on SAE latents being interpretable and causally relevant; if the steering vector is a blunt/nonspecific perturbation, the 'off-topic detection' story changes.
  • domain assumption An LLM judge's segmentation into attempts and its 0-100 scores are valid measures of self-correction and improvement.
    All ESR metrics are judge-derived. Cross-judge agreement mitigates but does not eliminate judge bias or hallucinated segmentation.
  • domain assumption Steering at similar relative layer depths with per-latent thresholds gives comparable interventions across model families.
    Underlies the cross-model claim; violated implicitly by the post-hoc layer selection described in Appendix A.1.1.
  • domain assumption Zero multi-attempt responses in the no-steering baseline imply that self-correction is induced by steering, not by prompt or task tendencies.
    Holds only if the judge would reliably detect self-corrections in unsteered responses; the authors tested the same prompts and judge, but this is still an assumption about the measurement.
  • domain assumption Instruction-tuned Llama-3 and Gemma-2 models are representative enough to support scale/architecture/training speculation.
    Only five models across two families are tested, and the authors themselves note they cannot disentangle scale, architecture, or training.

pith-pipeline@v1.3.0-alltime-deepseek · 17864 in / 14289 out tokens · 144071 ms · 2026-08-03T03:42:01.484384+00:00 · methodology

0 comments
read the original abstract

Large language models can recover mid-generation from task-misaligned activation steering, producing explicit verbal restarts (e.g., ``wait, that's not right'') and continuing on-topic even while the steering perturbation remains active. We term this Endogenous Steering Resistance (ESR). Using sparse autoencoder (SAE) latents to steer model activations, we find that Llama-3.3-70B exhibits explicit ESR at 3.8%, with smaller models from the Llama-3 and Gemma-2 families showing the explicit form less frequently. Two controls dissociate ESR into a detection event and a sustained-resistance component that conditioning on recent on-topic tokens does not fully explain. We identify 26 SAE latents through contrastive on-topic/off-topic search; zero-ablating them reduces the multi-attempt rate by 25%, with random-latent and held-out-prompt controls supporting specificity. ESR can also be deliberately enhanced through both meta-prompting and fine-tuning on synthetic self-correction examples. ESR has dual implications for safety: it could harden models against adversarial activation-space manipulation, but may equally interfere with beneficial steering-based interventions, since the model has no way to distinguish the two. Code is available at https://github.com/agencyenterprise/endogenous-steering-resistance.

Figures

Figures reproduced from arXiv: 2602.06941 by Alex McKenzie, Diogo de Lucena, Judd Rosenblatt, Keenan Pepper, Martin Leitgab, Michael S. A. Graziano, Mike Vaiana, Murat Cubuktepe, Stijn Servaes.

Figure 1
Figure 1. Figure 1: Demonstration of ESR. We prompted Llama-3.3-70B with a question about probability while steering activations toward a “body positions” latent. The model initially produces off-topic content about body positions, then spontaneously self-corrects back to the math question. A judge model segments the response into attempts and scores each for relevance. The second attempt scores 75/100 rather than perfect bec… view at source ↗
Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: ESR characteristics versus boost relative to threshold for Llama-3.3-70B. All three metrics show non-monotonic rela￾tionships with boost level, peaking at intermediate values. Top: Multi-attempt percentage peaks at 2.7% around −0.3σ below threshold. Middle: Multi-attempt improvement rate (percentage of multi-attempt responses that improved) peaks at 83% around −1.0σ, indicating that slightly weaker steerin… view at source ↗
Figure 4
Figure 4. Figure 4: Meta-prompting enhances steering resistance, with effects scaling by model size. Comparison of baseline (dashed grey bars) versus “If you notice yourself going off-topic, stop and force yourself to get back on track” meta-prompt (solid purple bars) conditions across five models. Llama-3.3-70B shows a 4.3× increase in multi-attempt rate (from 7.4% to 31.7%) and a 3.9× increase in ESR rate (from 3.8% to 14.8… view at source ↗
Figure 5
Figure 5. Figure 5: Ablating differentially-activated latents reduces ESR. Comparison of ESR metrics on Llama-3.3-70B between baseline (no ablation; 4,877 trials) and ablation (26 OTD latents clamped to zero; 4,875 trials) conditions. Left: Mean first-attempt score remains similar (baseline: 26.3, ablation: 27.4), indicating ablation does not affect initial response quality. Middle: Percentage of responses containing multiple… view at source ↗
Figure 6
Figure 6. Figure 6: shows that fine-tuning successfully induces self￾correction behavior: multi-attempt rate rises steadily with more self-correction training data. However, the multi￾attempt improvement rate remains flat regardless of training [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Sequential SAE activations during spontaneous self-correction. Activation traces (exponentially smoothed, α = 0.5) showing off-topic detector latents during a steered response. Shaded regions indicate response phases. Off-topic detectors show elevated activation during distracted generation, with activation preceding the self-correction point. the underlying ability to correct effectively. The model learns… view at source ↗
Figure 8
Figure 8. Figure 8: Experimental methods overview. The ESR testing pipeline involves steering the model with SAE latents, generating responses, and using a judge model to score separate attempts within each response. A. Technical Appendices and Supplementary Material A.1. Experimental Setup Details A.1.1. LAYER SELECTION FOR STEERING We apply steering interventions at similar relative depths across model architectures (see [… view at source ↗
Figure 9
Figure 9. Figure 9: Cross-judge ESR rate. ESR rate by target model and judge (1,000 responses, stratified sampled). Llama-3.3-70B shows the highest ESR rates across all judges, substantially higher than other models [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Cross-judge multi-attempt rate. Percentage of responses containing multiple attempts, by target model and judge (1,000 responses, stratified sampled). Llama-3.3-70B shows the highest multi-attempt rates across all judges, substantially higher than other models. A.2.2. JUDGE MODELS To validate the robustness of our ESR findings, we conducted a cross-judge analysis using four additional judge models: GPT-5-… view at source ↗
Figure 11
Figure 11. Figure 11: Inter-judge agreement on number of attempts. Facet grid showing pairwise agreement between judges on the number of attempts detected in each response (1,000 responses). Diagonal panels show each judge’s distribution of attempt counts; off-diagonal panels show scatter plots with exact agreement percentages. Judges show high agreement on attempt segmentation despite using different underlying models. 15 [P… view at source ↗
Figure 12
Figure 12. Figure 12: No-steering baseline: zero self-correction observed. Without feature steering, no models exhibit multi-attempt behavior. Left: Empty histograms indicate no score deltas to measure (all responses were single-attempt). Middle: Multi-attempt rate is 0.00% for all models. Right: Mean Score Improvement is 0.00 for all models. Compare to [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: First-attempt score distribution without steering. Left: Score distributions are tightly clustered in the 85–100 range across all models. Right: Mean first-attempt scores range from 87.8 (Llama 3.1 8B) to 91.8 (Gemma 2 9B), demonstrating that models consistently produce high-quality responses when not subjected to steering interventions. Interpretation. This control experiment confirms that instruction-tu… view at source ↗
Figure 14
Figure 14. Figure 14: Meta-prompt variant comparison for Llama-3.3-70B. All variants improve over baseline, with the self-monitoring prompt showing the largest gains [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Meta-prompt variant comparison for Llama-3.1-8B. A.3.3. OFF-TOPIC DETECTOR LATENT DETAILS This section provides details on the off-topic detector latents identified using Goodfire’s Ember API (Goodfire, 2024) contrastive search functionality, as described in Section 2.3. Using the contrast() function, we identified latents that activate differentially between correctly matched (on-topic) and shuffled (off… view at source ↗
Figure 16
Figure 16. Figure 16: Meta-prompt variant comparison for Gemma-2-27B [PITH_FULL_IMAGE:figures/full_fig_p018_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Meta-prompt variant comparison for Gemma-2-9B. only some of which function as true off-topic detectors. Despite this heterogeneity, ablating all 26 latents as a group reduces ESR, suggesting they collectively contribute to self-correction behavior through mechanisms that may extend beyond simple off-topic detection. A.3.4. RANDOM LATENT ABLATION CONTROL To verify that the ESR reduction observed with off-t… view at source ↗
Figure 18
Figure 18. Figure 18: Meta-prompt variant comparison for Gemma-2-2B [PITH_FULL_IMAGE:figures/full_fig_p019_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Random latent ablation control. Comparison of ESR metrics across three conditions on Llama-3.3-70B: steered baseline (4,877 trials), OTD ablation (4,875 trials), and random ablation (14,450 trials). Left: Mean first-attempt score remains similar across conditions (baseline: 26.3, OTD ablation: 27.4, random ablation: 27.1). Middle: Multi-attempt rate drops 25% with OTD ablation (from 7.4% to 5.5%) but show… view at source ↗
Figure 20
Figure 20. Figure 20: Token-level activation patterns aligned at correction point. Mean activation of off-topic detector (OTD, red) and backtracking (blue) latents across 146 self-correction episodes. Data are binned into 50 intervals; points show bin means with 95% confidence intervals, lines show spline fits. The orange shading shows each episode’s correction region overlaid, fading as episodes exit correction at different p… view at source ↗
Figure 21
Figure 21. Figure 21: OTD activation: self-correction vs. baseline episodes. Mean activation of off-topic detector latents across three conditions: the off-topic region of self-correction episodes (before correction), the on-topic region (after correction), and baseline episodes with no self-correction. OTDs fire 4.4× higher during off-topic content compared to baseline, and remain elevated (2.1×) even after self-correction. E… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes

    cs.AI 2026-05 unverdicted novelty 5.0

    Overthinking in medical QA is linearly decodable at 71.6% accuracy yet fixed residual-stream steering yields no correction across 29 configurations, while enabling selective abstention with AUROC 0.610.

Reference graph

Works this paper leans on

5 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [1]

    {off_topic_subject}

    Start by veering off-topic to discuss "{off_topic_subject}" instead of answering the actual question (2-4 sentences)

  2. [2]

    Wait, that's not quite right

    Explicitly acknowledge you went off-topic (1 sentence) - use natural phrases like "Wait, that's not quite right..." or "Actually, let me reconsider..." or "Hmm, I'm getting off track here..."

  3. [3]

    {off_topic_subject}

    Provide the correct, on-topic answer (2-4 sentences) Important: - The off-topic part should discuss "{off_topic_subject}" in a way that's plausible but clearly not answering the actual question - The acknowledgment should be brief and natural - The corrected answer should directly address the original question - Make it feel realistic, like a real convers...

    2022

  4. [2021]

    pub/2021/framework/index.html

    URL https://transformer-circuits. pub/2021/framework/index.html. Goodfire. Goodfire ember: Scaling interpretability for frontier model alignment.https://www.goodfire. ai/blog/announcing-goodfire-ember, 2024. Accessed: 2026-01-29. Grattafiori, A., Dubey, A., Jauhri, A., et al. The llama 3 herd of models, 2024. URL https://arxiv.org/ abs/2407.21783. Grazian...

    Pith/arXiv arXiv 2021

  5. [2023]

    best performing

    URL https://openreview.net/forum? id=NpsVSN6o4ul. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dominguez, A.-K., Mukobi, D., Duenas, S. R., Li, S., Bowman, J., Basart, S., Joachims, T., Boneh, D., Carlini, N., and Hendrycks, D. Representation engineering: A top-down approach to ai transparency.arXiv preprint ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.01405 2023