Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Instruction-tuned models often lock in their answer before the chain of thought begins.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:37 UTC pith:HQTVMHZZ

load-bearing objection The pre-CoT decoding result holds up; the abstract's causal 'pre-committed answer' claim runs ahead of the steering evidence. the 3 major comments →

arxiv 2603.01437 v2 pith:HQTVMHZZ submitted 2026-03-02 cs.AI

Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers

Kyle Cox , Darius Kianersi , Adri\`a Garriga-Alonso This is my paper
classification cs.AI
keywords chain-of-thought faithfulnesspost-hoc reasoninglinear probingactivation steeringpre-committed answersconfabulationnon-entailmentinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that chain-of-thought reasoning in instruction-tuned models is often post hoc: the model settles on its final answer before generating the rationale. Using linear probes on residual-stream activations at the token just before the reasoning text, the authors decode the eventual answer with AUC above 0.9 on most of their factual, social, and sports tasks. They then show the decoded direction is causally involved: adding that direction during decoding flips answers at rates well above norm-matched random directions. When the flip produces a wrong answer, the model either confabulates false premises or states correct premises and draws an unsupported conclusion. If true, this matters because it means CoT text is not a faithful record of the decision process in these models—an answer can be fixed before any reasoning is written.

Core claim

The central claim is that the final answer is represented in the residual stream before the first reasoning token is generated, and that this representation is both linearly readable and causally functional. The authors test three hypotheses: the answer is linearly decodable from pre-CoT activations; the decodable direction, when added during generation, flips the answer far more than an orthogonal perturbation; and the resulting incorrect chains show recognizable pathologies—confabulated false premises and non-entailing conclusions. On tasks where CoT provides little benefit, decoding accuracy is high; on a task where CoT is genuinely useful, decodability drops, which the authors read as co

What carries the argument

The load-bearing object is a difference-of-means probe computed at t0, the last pre-CoT token: the mean residual-stream activation for examples whose final answer is yes minus the mean for no, normalized. A held-out example's score is the cosine similarity between its t0 activation and this direction. The same direction is then used for steering via contrastive activation addition—adding a scaled multiple of it at every decoding position after t0. A second machinery is the two-dimensional classification of reasoning traces by premise truthfulness and logical entailment, which yields confabulation, non-entailment, and hallucination. The probe connects prediction (H1) to causation (H2); the cl

Load-bearing premise

The causal interpretation rests on assuming that adding the probe direction at every post-prompt token manipulates a stable 'pre-committed answer' feature, rather than an upstream feature that flips the answer by altering the content of the reasoning; the probe is fit to the very answer labels it later predicts, so predictiveness alone cannot settle that.

What would settle it

Steer only at the last pre-CoT token and leave all later positions untouched; if flip rates fall to the orthogonal-baseline level, the effect is carried by changes to the CoT rather than by a pre-existing answer feature. The paper's own intervention adds the direction at every decoding position, so the collapse would disconfirm the causal reading.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Chain-of-thought text should not be treated as a faithful transcript of how the answer was reached: on these tasks the verdict is typically fixed before the rationale is written.
  • Pre-CoT activations are a practical early-warning signal for the final answer, reaching >0.9 AUC on most model–task pairs—so monitoring does not have to wait for the CoT to finish.
  • The probe direction is not a correlation artifact: adding it during generation flips answers at rates well above equal-magnitude orthogonal perturbations, so the pre-committed answer is causally manipulable.
  • When forced to the wrong answer, models do not simply break down; they generate coherent-looking but unfaithful rationales, falling into two identifiable patterns—fabricated premises and non-entailing conclusions.
  • The effect is task-dependent: pre-CoT decoding is weakest on a task where CoT most improves accuracy, so CoT is doing genuine work precisely where probes fail.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Left implicit in the paper: if pre-committed answers are common, any safety system that audits only the verbalized CoT can be misled, since the reasoning can be generated after and adapted to a fixed verdict.
  • A testable extension: steering at the single pre-CoT position t0, without adding the direction to later tokens, would separate a stable answer feature from an effect that propagates by changing the CoT content.
  • The appendix's finding that a reasoning model is decodable on one task but hard to steer suggests that reinforcement-trained deliberation may shift the causal burden into the CoT; whether that holds broadly is an open question the paper only begins to address.
  • If the causal reading generalizes, this is a dual-use result: the same steering that induces confabulation could be used adversarially to implant false beliefs with plausible rationales, so pre-CoT circuits may need their own safeguards.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper asks whether instruction-tuned LLMs commit to a final answer before generating chain-of-thought (CoT). The authors train difference-of-means linear probes on residual-stream activations at the last prompt token before CoT (t0), evaluate held-out AUC, and then steer generation by adding the probe direction at every decoding token after t0. They report that pre-CoT probes reach >0.9 AUC on most model–task pairs, that steering along the probe direction flips answers at rates substantially above norm-matched orthogonal baselines, and that flipped answers display two failure modes: confabulation and non-entailment. They interpret these results as evidence that the probe direction is a causal representation of a pre-committed answer, and they discuss implications for CoT faithfulness.

Significance. If the causal interpretation holds, this is a significant mechanistic contribution: it would show that, for instruction-tuned models on several tasks, the final answer is linearly decodable and causally manipulable in the residual stream before any reasoning token is generated. The paper has real strengths: probe AUC is reported on held-out data; the steering baseline is norm-matched and orthogonal; results span two model families and four tasks; a reasoning-model negative control is included; parse-failure rates are discussed in an appendix; and the failure-mode taxonomy is accompanied by consistency checks. However, the causal part of the central claim currently overreaches the experimental design. The steering intervention is applied throughout CoT, not at t0, and the qualitative reply to CoT-mediated causation is not a control experiment. The paper is strong evidence for representational pre-commitment (H1) and a useful behavioral characterization, but additional controls are needed before the causal wording is warranted.

major comments (3)
  1. [§2.4 and §3.4, Eq. (3)] The central causal claim — that the answer is determined before CoT — is not entailed by the steering experiment as run. The intervention adds αw at every decoding position t > t0, including the first CoT token and the answer position. This permits an alternative mechanism: the added direction changes the distribution of early CoT tokens, and the altered reasoning (or the altered hidden state at the answer position) flips the final answer. The §4.1 reply "Against CoT-mediated causation" is not a control: non-entailment traces show only that the final answer can contradict the written CoT after steering, not that the flip was caused by a pre-CoT representation rather than by the same additive perturbation acting during CoT decoding. Please add a t0-only condition (steer once at the last prompt token, then decode with no further edits) and/or a clamped-CoT condition (steer at t0, then forc
  2. [§3.4 and Appendix G] The flip-rate estimates and their comparison to the orthogonal baseline may be biased by the data-selection rule. The text notes that any α with fewer than 20 parsed generations is omitted and that sweeps terminate early when no examples parse. If the probe direction and the orthogonal direction produce different parse-failure rates at large |α|, then the reported comparison is conditional on different subsets of examples for the two conditions. Please report flip rates under two conservative codings — treating parse failures as non-flips and treating them as flips — and give the parse-failure rate by condition across the full sweep. This is central to H2 because the claim "steering with the probe was more effective than steering with orthogonal vectors" could in principle be an artifact of differential exclusion.
  3. [§3.3, Table 3] Probe AUCs are reported as point estimates with no uncertainty. With 500 held-out test examples, bootstrap confidence intervals or standard errors would let the reader evaluate the ">0.9 AUC on most tasks" claim and the cross-task comparisons (e.g., Logical Deduction AUCs of 0.688–0.878). This is a reporting gap rather than a fatal flaw, but it should be fixed for the quantitative headline to be fully assessed.
minor comments (6)
  1. [§3.5, Table 1] The category "Hallucination" for the cell where both dimensions are false is potentially confusing given that "confabulation" is defined as false premises. Consider naming this cell something like "incoherent/unfounded reasoning" to avoid overlap in the taxonomy.
  2. [§3.4, Figure 3] The x-axis is described as the absolute value of the steering coefficient while the text explains that the sign of α pushes toward "yes" or "no." A legend or axis annotation should make the sign convention and the Syes/Sno subset definitions explicit, especially because the curves are truncated by the parse-failure rule.
  3. [Appendix H, Figure 11] The logit-lens figure appears to be corrupted: the displayed entries are sequences like "/uni00000003/uni0000006d..." rather than readable tokens. If this is a rendering issue, the figure needs to be regenerated; as it stands, the reader cannot verify the claimed semantic content of the unembedded probe directions.
  4. [Abstract and §3.4] The abstract states that steering "flips model answers in over 50% of cases," but the body more cautiously says "often flips" and Figure 3 shows substantial variation across model–task pairs and α. Please align the abstract with the quantitative summary, or state the aggregate statistic that supports the 50% figure.
  5. [§2.5 and Appendix C.1] The classification prompt for GPT-5-mini is described at a high level but the full prompt is not included. Given that the failure-mode analysis and the argument in §4.1 rely on these labels, please include the exact prompt used for reproducibility.
  6. [§3.2 and Appendix D] The CoT sensitivity interventions are run on 50 correct generations per model–task pair. This selection may overstate robustness, since incorrect baseline generations might be more sensitive to CoT perturbation. Please state whether the conclusions are robust to including incorrect baseline examples, or add a sensitivity check.

Circularity Check

0 steps flagged

No circularity: probes are evaluated out-of-sample and steering is compared against orthogonal baselines; the causal gap is an identification issue, not a definitional reduction.

full rationale

The derivation chain (P0 -> H1 -> H2 -> H3) does not contain a load-bearing step that reduces to its own inputs. In §2.3 the probe direction is w^(ℓ) = μ_yes − μ_no, computed from training examples partitioned by final answer, but the reported AUC is evaluated on held-out test examples via cosine similarity: "For a held-out test example j, we compute the cosine similarity score ... and compute AUC". This is an out-of-sample generalization check, not the training-separating direction restated as a prediction. H2 is tested on held-out Syes/Sno subsets against a per-example norm-matched orthogonal baseline (§2.4), so the observed flip-rate dominance is empirical rather than forced by the definition of w. H3 classifies traces with GPT-5-mini using entailment/truthfulness rubrics (§2.5), which are independent of the probe construction. There are no load-bearing author self-citations, and no uniqueness or ansatz theorem is imported from the authors' prior work. The paper also explicitly acknowledges remaining uncertainty about whether a discrete "pre-committed answer" feature exists (§4.1, "Remaining uncertainty"). The main weakness is causal identification, not circularity: the steering edit is applied at every decoding position t > t0, so a CoT-mediated pathway remains possible; the §4.1 reply is argumentative rather than a control experiment. Similarly, selecting ℓ⋆ as the layer with the highest probe AUC may involve test-label leakage if that AUC is computed on the test set, but this is a statistical validity concern, not an equation-level identity. No step in the claimed derivation is equivalent to its own input.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No physically or computationally novel entities are postulated: the 'pre-committed answer feature' is an interpretation of a fitted probe direction, and the paper itself questions whether such a discrete feature exists (§4.1 'Remaining uncertainty'). The free parameters above are the fitted/selected quantities the empirical claims depend on; the axioms are the methodological premises that would need to fail for the interpretation to collapse.

free parameters (3)
  • probe direction w^(ℓ) = μ_yes − μ_no = vector per model/layer/dataset; not reported in text
    Fit on 500 training examples partitioned by the model's final answer; the central decoding object. Held-out AUC is the honest test, so this is a fitted model, not an input constant.
  • steering coefficient α = sweep 0, ±2, ..., ±20; terminated early on parse failures
    Chosen by hand; the flip-rate comparison vs. the norm-matched orthogonal baseline at matched α is the causal test, so α is load-bearing for H2.
  • steering layer ℓ⋆ = layer with highest probe AUC
    Selected by maximizing train/validation AUC; a model-selection choice that could in principle inflate steering efficacy if the high-AUC layer happens to be off-manifold-sensitive.
axioms (4)
  • domain assumption Final-answer classes are linearly decodable in the residual stream at position t0
    The difference-of-means probing method in §2.3 presupposes a linear structure; high AUC validates it empirically for these tasks, but the probing position and linearity are assumed, not derived.
  • domain assumption GPT-5-mini judgments of 'false premises' and 'conclusion follows' constitute valid ground truth for the failure-mode taxonomy
    §2.5 and Appendix C.3 check only run-to-run consistency (81.5% overall; 92.0% false-premises; 89.5% entailment); no human-labeled ground truth is provided.
  • domain assumption Parse-failure exclusions and early sweep termination do not systematically bias flip-rate estimates
    §2.4 and Appendix G: sweeps stop when unparsable generations rise, and flip rates are computed over parsed generations only; the exclusion is disclosed but could be answer-correlated.
  • domain assumption Norm-matched orthogonal random directions are an adequate control for generic perturbation effects
    §2.4: the control's flip rate saturates near 0.5 as expected from random guessing, which the authors interpret as supporting the control's validity rather than as evidence of a confound.

pith-pipeline@v1.3.0-alltime-deepseek · 28631 in / 15395 out tokens · 160734 ms · 2026-08-02T19:37:46.271599+00:00 · methodology

0 comments
read the original abstract

As chain of thought (CoT) has become central to scaling reasoning capabilities in large language models (LLMs), it has also emerged as a promising tool for interpretability, suggesting the opportunity to understand model decisions through verbalized reasoning. However, the utility of CoT toward interpretability depends upon its faithfulness---whether the model's stated reasoning reflects the underlying decision process. We provide mechanistic evidence that instruction-tuned models often determine their answer before generating CoT. Training linear probes on residual stream activations at the last token before CoT, we can predict the model's final answer with >0.9 AUC on most tasks. We find that these directions are not only predictive, but also causal: steering activations along the probe direction often flips model answers, with flip rates substantially exceeding norm-matched orthogonal baselines across most model-dataset pairs. When steering induces incorrect answers, we observe two distinct failure modes: confabulation (fabricating false premises) and non-entailment (stating correct premises but drawing unsupported conclusions). While post-hoc reasoning may be instrumentally useful when the model has a correct pre-CoT belief, these failure modes suggest it can result in undesirable behaviors when reasoning from a false belief.

Figures

Figures reproduced from arXiv: 2603.01437 by Adri\`a Garriga-Alonso, Darius Kianersi, Kyle Cox.

Figure 1
Figure 1. Figure 1: Example prompt from the Sports Understanding task. The model generates its response starting from “Let’s think step by step:”. 2.2. Testing for CoT Sensitivity These experiments establish the empirical premise (P0) that models often exhibit post-hoc reasoning on our tasks. We evaluate how sensitive the model is to chain-of-thought in two ways: Accuracy improvement due to CoT. We evaluate model accuracy wit… view at source ↗
Figure 2
Figure 2. Figure 2: Example of activation steering causing confabulation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Answer flip rates under steering across models and datasets. ing out this interpretation. When the model states largely correct premises but reaches a non-sequitur conclusion, the answer changes without being implied by the written rea￾soning. If the CoT mediated the steering effect, we would expect its content to change in ways that support the new answer. Instead, the CoT can remain largely correct while… view at source ↗
Figure 4
Figure 4. Figure 4: CoT classification results across models and datasets on examples where steering flipped the answer. Examples from Syes and Sno are aggregated for a given steering setting. OpenAI et al., 2024; Yang et al., 2025; Anthropic, 2024). In these systems, the CoT is optimized as a latent that con￾tributes to task reward, which may change the faithfulness￾usefulness trade-off. In Appendix F, we perform probe and s… view at source ↗
Figure 5
Figure 5. Figure 5: CoT classification results on examples from Syes [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: CoT classification results on examples from Sno. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Probe AUC across layers for each model–dataset pair. Higher AUC indicates stronger linear decodability of the final answer from pre-CoT activations at that layer. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Probe AUCs over layer for GPT-OSS 20B [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Answer flip rates under steering for GPT-OSS 20B. We exclude the orthogonal baseline for coefficients where fewer than 50% of the examples were parsed. We hypothesize that most of the load-bearing reasoning ability for LRMs occurs in their chain-of-thought compared to instruction-tuned models. This would explain why the pre-committed answer direction prior to CoT is not well represented across most dataset… view at source ↗
Figure 10
Figure 10. Figure 10: reports steering flip rates alongside the corresponding parse-failure rate (proportion of generations we could not parse) over the α sweep for all model–dataset pairs [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Tokens corresponding to five highest logits after unembedding the task probe for each model–dataset pair, after filtering for English-alphabetical tokens. We filter to only include alphabetical tokens to increase the probability that each token has interpretable semantic content, and is common to English (and thus more interpretable to the authors). While some tokens are incomprehensible, or appear to der… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Two Regimes of Chain-of-Thought Unfaithfulness: Behavioral Detection Fails Where Models Are Wrong

    cs.CL 2026-07 conditional novelty 6.5

    Answer correctness splits CoT unfaithfulness detection into two regimes: behavioral signals work only on correct answers, and fail at chance on incorrect answers where most unfaithfulness lives.

  2. Committed Before Reasoning: Behavioral Reproduction and Preliminary Activation-Level Evidence of Answer Pre-Commitment in an Open-Weight LLM

    cs.CL 2026-07 conditional novelty 6.0

    On a car-wash task, Qwen3-8B commits to 'walk' in 85–100% of rollouts, and a training-free activation oracle reads 'walk' from pre-answer hidden states even in rollouts that finally answer 'drive'.

  3. Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes

    cs.AI 2026-05 unverdicted novelty 5.0

    Overthinking in medical QA is linearly decodable at 71.6% accuracy yet fixed residual-stream steering yields no correction across 29 configurations, while enabling selective abstention with AUROC 0.610.

Reference graph

Works this paper leans on

3 extracted references · 2 linked inside Pith · cited by 3 Pith papers

  1. [1419]

    During the Allied bombardment of the beaches of Iwo Jima, Ralph spoke loudly into his iPhone

    URL https://aclanthology.org/N19- 1419/. Jacovi, A. and Goldberg, Y . Towards faithfully interpretable NLP systems: How should we define and evaluate faith- fulness? In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.),Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 4198–4205, Online, July 2020. ...

    Pith/arXiv arXiv 2020

  2. [2022]

    pub/2022/toy_model/index.html

    URL https://transformer-circuits. pub/2022/toy_model/index.html. Forbes, M., Hwang, J. D., Shwartz, V ., Sap, M., and Choi, Y . Social chemistry 101: Learning to reason about social and moral norms, 2021. URL https://arxiv.org/ abs/2011.00620. Gao, L. Shapley value attribution in chain of thought, Apr 2023. URL https://www.lesswrong. com/posts/FX5JmftqL2j...

    Pith/arXiv arXiv 2022

  3. [2024]

    Accessed: 2025-08-21

    URL https://assets.anthropic.com/ m/785e231869ea8b3b/original/claude- 3-7-sonnet-system-card.pdf . Accessed: 2025-08-21. Arad, D., Mueller, A., and Belinkov, Y . Saes are good for steering – if you select the right features, 2025. URL https://arxiv.org/abs/2505.20063. 8 Post-hoc Reasoning in Chain-of-Thought: Evidence from Pre-CoT Probes and Activation St...

    arXiv 2025