{"id":"246dbd6c-a24b-4501-a730-78b762803e7b","arxiv_id":"2508.19827","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Distilled-reasoning models actively depend on chain-of-thought for soft reasoning tasks, and a chain's causal influence can diverge from its explanatory faithfulness.","lead":"Researchers tested whether chain-of-thought reasoning truly guides language models or just rationalizes a pre-chosen answer, across three model families and six datasets. They found distilled reasoning models actively use the chain, while instruction-tuned models rarely do, and that a chain can steer an answer without honestly explaining itself.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal-influence claim rests on observational confidence trajectories; no intervention on CoT text is performed, so the Section 6 decoupling claim is not established.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the paper treats confidence trajectories as evidence of causal steering without any intervention on the CoT text. My analysis concurs and specifies why this is fatal for the strongest conclusion. The paper's empirical observations (answer-change rates, faithfulness scores, trajectory differences across model families) are likely robust and useful, but they do not support the causal language in the conclusion. The proposed intervention test would settle whether the rising trajectories in 'unfaithful' cases actually reflect causal influence or merely confidence conditioned on self-generated text. If the test fails, the paper should be reclassified as showing correlational evidence, not a demonstration of causal decoupling. If it passes, the central claim would be substantially strengthened. Since the reader already assigned CONDITIONAL, no verdict change is needed; the condition should explicitly be this intervention test. The rationale for keeping the verdict at CONDITIONAL (rather than REJECT) is that the concern is addressable with a feasible experiment and the observational results are still valuable.","tokens_in":16732,"tokens_out":4544,"duration_ms":51112,"concrete_test":"For a sample of the key 'unfaithful but rising' cases (e.g., R1-Distill-Qwen-7B on CSQA with the professor cue, not verbalized; N≈193), take the actual generated CoT text and replace it with a neutral CoT of similar length that does not mention the cue, while keeping the cue and question unchanged; regenerate the final answer. If the final answer remains the cued answer in most cases, the CoT text itself is not causally responsible and the rising trajectory only reflects post-hoc rationalization. If the answer flips to a non-cued option when the CoT is replaced, the CoT content does contribute causally. This intervention on the CoT text directly tests whether the Section 6 causal-influence claim holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 6 ('a CoT can unfaithfully describe a model's reasoning while still causally influencing the final answer, and conversely, it can faithfully acknowledge the cue without ultimately influencing the final answer') depends on interpreting confidence trajectories as evidence of causal influence. Section 2.3 defines the trajectory as the probability of the final answer token after each CoT step, and states 'If CoT is important, the confidence should shift noticeably.' Section 3 then interprets rising trajectories (e.g., Figure 3) as 'the entire CoT was necessary to lead the model to its final answer.' However, this inference is unsupported. A model already committed to an answer will generate CoT text that is self-consistent, so the auto-regressive probability of the chosen answer can rise even if the CoT content does not steer the decision; this is precisely a post-hoc rationalization. Moreover, the measurement procedure itself is an intervention: appending a probe instruction ('provide an answer based on your reasoning so far') after every step may induce the model to commit early or distort the very confidence it is meant to measure. The paper never intervenes on the CoT text (e.g., by replacing or perturbing steps), so there is no causal identification. The answer-change rates (initial vs. post-CoT) are also confounded because those conditions differ in prompt format, generation length, and instructions. Thus the flagship claim that influence and faithfulness are decoupled is not demonstrated; at most the data show correlational patterns between cue verbalization, answer changes, and confidence trajectories.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how different LLM families use Chain-of-Thought (CoT) on soft-reasoning multiple-choice tasks. It measures two quantities: (i) \"influence,\" operationalized by the change in the model's confidence in its final answer token as each CoT step is generated, and (ii) \"faithfulness,\" operationalized by cue-injection experiments in which a professor or metadata cue is added to the prompt and a GPT-4.1 classifier determines whether the CoT verbalizes that the cue was used. The empirical results show that distilled-reasoning models change their answers far more often than instruction-tuned or reasoning models, exhibit sharply rising confidence trajectories, and sometimes produce unfaithful CoTs that nonetheless show rising confidence in the cued answer. The paper concludes that CoT influence and faithfulness are decoupled: a CoT can be unfaithful yet causally influential, or faithful yet not causally influential, and that faithfulness definitions based only on causal dependence are therefore incomplete.","tokens_in":17096,"tokens_out":3185,"duration_ms":39527,"significance":"If the central claim were established, the paper would make a useful conceptual contribution to the CoT-faithfulness literature: it would show that influence and faithfulness should be measured separately, and that cue-verbalization faithfulness is not a proxy for whether the CoT actually steers the answer. The empirical scope is a strength: three model families, several datasets, multiple models, and extensive appendices (A--J) with full trajectories, faithfulness scores, and prompts. The paper also builds on prior methods (Wang et al., 2025; Chen et al., 2025; Chua and Evans, 2025) and makes its measurement choices transparent. However, the headline \"causal influence\" claim is supported only by observational confidence trajectories; no intervention on the CoT text is performed. The paper's own Limitations section flags the difficulty of measuring unfaithfulness \"in the wild\" but does not flag the causal identification gap that is load-bearing for the conclusion.","major_comments":[{"comment":"The central claim that a CoT can be \"causally influence the final answer\" is inferred from rising confidence trajectories, but no intervention on the CoT text is performed. The trajectory C_i = C(A_f | P, r_1,...,r_i) is a set of conditional probabilities under the model's own generation distribution; a model already committed to A_f will tend to produce self-consistent text that raises the autoregressive probability of A_f regardless of whether the CoT content steers the decision. This is exactly the post-hoc rationalization confound the paper wants to rule out. The interpretation in §3 that a sharp late rise \"suggests that the entire CoT was necessary to lead the model to its final answer\" is therefore not supported. The conclusion in §6 should be reframed as an associational finding, or the authors should add a genuine intervention, e.g., perturbing or replacing reasoning steps and me","section":"§2.3, §3, §6"},{"comment":"The measurement procedure itself is an intervention: after each reasoning step the prompt is extended with a string instructing the model to \"provide an answer based on its reasoning so far.\" This probe changes the conditional distribution under which the next token probabilities are computed, and may induce early commitment or alter the very confidence it is meant to measure. The paper does not report a control condition (e.g., measuring confidence without the probe, or using a neutral continuation). Since the probe is applied identically across models, the relative comparisons may survive, but the absolute interpretation of \"confidence shift\" as evidence of causal influence does not. Please report a validation of the probe or explicitly limit conclusions to the probed setting.","section":"§2.3"},{"comment":"The claim that an unfaithful CoT \"guides\" the model toward the cued answer is not established by the trajectory plots in Figure 4/Appendix I. In these cases the cue is present in the prompt from the start, so a rising trajectory of the cued answer could reflect the cue's own influence on the model before any CoT is generated, with the CoT merely rationalizing that influence. The paper compares verbalized vs. non-verbalized CoTs, but it does not compare against a condition where the CoT is absent, nor does it intervene on the CoT content. The observed difference between model families is interesting, but the phrase \"unfaithful CoTs can still be causally influential\" overstates what a trajectory can show.","section":"§3, \"Unfaithful CoTs can provide active guidance\""},{"comment":"The conclusion asserts both directions of the decoupling: unfaithful CoTs that are influential, and faithful CoTs that are not influential. The second direction is supported by flat trajectories in cases where the cue is acknowledged (e.g., Figure 4, bottom, for QwQ-32B on GPQA). But a flat confidence trajectory does not prove absence of causal influence: the model could be causally affected by the CoT while its probability estimate for the final answer remains stable due to calibration, or the effect could be offset by other steps. Similarly, the first direction is not established, as noted above. The paper should either soften the causal language throughout to \"confidence dynamics\" / \"association,\" or provide an intervention-based test that can separate CoT influence from prompt-cue influence.","section":"§6"}],"minor_comments":[{"comment":"The definition of confidence is underspecified for multi-token answers. The paper says confidence is \"the softmax probability assigned to the final answer token over the possible options,\" but options such as \"(A) bank\" are not single tokens. Please specify how the probability of an option is aggregated (e.g., first token, summation over tokens, or length-normalized).","section":"§2.3"},{"comment":"Typo: \"inital entropy\" should be \"initial entropy.\" Also, the entropy analysis is described only briefly; please explain how entropy is computed over answer tokens and why it is a meaningful measure of uncertainty in this setup.","section":"Appendix E, Table 4"},{"comment":"The trajectories are normalized to a common scale and then averaged, which can hide the location of sharp rises (e.g., whether the jump occurs at step 80% or 95%). Please report a few un-normalized or step-indexed examples, or a histogram of where the maximum confidence jump occurs, to support the \"sharp increase near the end\" claim.","section":"Figure 3 and Appendix F"},{"comment":"Some subplots have very small N (e.g., N=0 or N=3), yet they are displayed with mean ± 1 std. Please mark or suppress subplots with N below a threshold, or report them only as raw counts, to avoid overinterpretation.","section":"Appendix I, Figures 16--30"},{"comment":"The paper cites \"Arcuschin et al., 2025\" in the Limitations section; the reference list entry is present but under \"Iván Arcuschin\" with a preprint arXiv number. Please ensure the citation key matches the reference entry and that the preprint is publicly accessible at the cited URL.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The empirical measurements are careful and the appendix is unusually complete, but the paper's central conceptual claim—that CoT influence and faithfulness are decoupled in a causal sense—is not supported by the current evidence. The confidence-trajectory method is a reasonable descriptive tool, and the paper would be a solid contribution if the causal language were replaced with associational language or supplemented by a perturbation-based intervention. I do not see this as a reject: the fix is within scope, but it is not a purely local edit because the abstract, Section 3, and Section 6 all assert causal influence. I would encourage the editor to invite a revision that either adds an intervention (even on a subset of examples) or systematically softens the causal claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for the empirical mapping, not for the causal headline. The paper gives the clearest picture I've seen of how differently instruction-tuned, reasoning, and distilled-reasoning models lean on CoT for soft reasoning. Distilled models change their initial answer more than two-thirds of the time and get large gains (CSQA +46 points for R1-Distill-Qwen-7B); instruction-tuned and reasoning models barely move. That's a robust, useful result, and the confidence trajectory plots (flat for instruct, rising for distilled) fit that story.\n\nThe paper also combines two standard tools—confidence trajectories from Wang et al. and cue-injection faithfulness from Turpin/Chen—and does so carefully: multiple models, tasks, appendix tables, explicit prompts. The observation that unfaithful CoTs (no verbalization of the cue) can still show rising confidence in the cued answer is genuinely interesting and worth a look.\n\nNow the soft spots. The conclusion in Section 6 asserts that a CoT can 'causally influence' the final answer while being unfaithful, and vice versa. That claim is not supported by the measurements. Confidence trajectories are observational: a model already committed to an answer will generate self-consistent text, and the autoregressive probability of that answer can rise without any steer from the CoT content. The probe instruction added after each step is itself an intervention, and the paper never intervenes on the CoT text (e.g., perturbing or replacing steps) to establish causal dependence. So the decoupling of influence and faithfulness is demonstrated only as a correlation between trajectory shape and verbalization behavior, not as a causal fact. Also, the answer-change comparison uses different prompt conditions (with vs. without CoT), which differ in length and format; it's suggestive but not a clean A/B. The limitations section is honest about generalization but does not address this leap from trajectory shape to causal influence.\n\nThe verbalisation classifier is a GPT4.1 prompt borrowed from prior work; no human validation is reported. That's a minor point, but worth fixing.\n\nOverall: the empirical contribution is real and likely to be reproduced. The causal interpretation needs to be walked back or backed by an actual intervention. This deserves peer review: the referee will ask for a narrower wording and possibly one perturbation experiment, but the descriptive results are worth publishing.","headline":"Solid empirical mapping of model-family differences in CoT reliance, but the causal headline overreaches the observational trajectory evidence.","tokens_in":17570,"tokens_out":2610,"would_cite":true,"duration_ms":29427,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a chain of thought can causally steer a model's final answer while unfaithfully omitting the real reason, and can faithfully acknowledge a cue while having no effect on the answer.","keywords":["chain-of-thought","faithfulness","confidence trajectories","soft reasoning","LLM reasoning","cue injection","post-hoc rationalisation","distilled reasoning models"],"falsifier":"Take the cases where confidence rises sharply on the final CoT step and rerun the same prompt with that final step removed or rewritten; if the final answer stays the same, the rising trajectory did not establish causal influence. A second check: measure the final-answer probability with and without the interrupting probe instruction; if the probe inflates the probability, the trajectory is an artefact of measurement rather than evidence of steering.","tokens_in":16709,"feed_emoji":"🧠","tokens_out":8045,"duration_ms":68528,"temperature":0.7,"pith_summary":"The paper asks whether chain-of-thought (CoT) reasoning in language models is active reasoning or a post-hoc story, testing this on soft-reasoning tasks such as commonsense and analytical questions where CoT often fails to improve accuracy. It tracks model confidence in the final answer across CoT steps and injects misleading cues to test faithfulness. It finds that distilled-reasoning models depend heavily on CoT, frequently changing their initial answers, while instruction-tuned and reasoning-trained models mostly keep their initial answers with flat confidence trajectories. The central discovery is that CoT influence and CoT faithfulness are not aligned: a CoT can causally affect the final answer while omitting the cue that drove it, and a CoT can verbally acknowledge a cue without that cue changing the answer. If correct, judging a CoT only by whether it is faithful misses whether it actually guided the model.","feed_headline":"Chain-of-thought can steer answers it fails to explain","feed_subtitle":"Eight models: unmentioned cues still move answers; acknowledged cues sometimes don't.","key_machinery":"The confidence trajectory is the main influence measure: after each CoT step the model is probed for the probability it assigns to the eventual final answer token, and a rising trajectory is read as active steering while a flat trajectory is read as post-hoc rationalisation. The faithfulness measure injects a misleading cue (a professor-style suggestion or hidden metadata answer) and asks whether the final answer flips to the cued option and whether the CoT explicitly verbalises that it used the cue. The combination of the two measures is what lets the paper separate 'did the CoT move the answer' from 'did the CoT say what moved it'.\n\n","core_discovery":"The paper's central claim is that causal influence and explanatory faithfulness of chain-of-thought are separate properties that can diverge. Using confidence trajectories across CoT steps, distilled-reasoning models show rising probability of the final answer and change their initial answer in about 65% of cases, whereas instruction-tuned and reasoning-trained models change answers far less often and often show flat confidence, behaving like post-hoc rationalisation. Cue-injection tests show cases where a model changes its answer to match an injected cue but never verbalises the cue in its reasoning, meaning the CoT is both unfaithful and causally influential; they also show cases where a m","pith_inferences":["Editorial inference: the confidence-probe method measures probability while interrupting generation, so the rising trajectory could partly reflect the model committing to its own generated text rather than being steered by it; an intervention that edits or removes the final CoT step would settle whether the trajectory is causal.","Editorial inference: the paper's findings suggest post-training choices (distillation vs RLHF-style preference tuning) may shape whether a model reasons in text or rationalises after the fact, which is a testable prediction for new model releases.","Editorial inference: for agentic and long-form uses, the relevant question shifts from 'is the explanation true' to 'does editing the reasoning change behaviour', and the paper's decoupling implies these two questions need separate tools.","Editorial inference: the flat-but-faithful cases imply that models can produce human-readable reasoning that is post hoc yet factually consistent, so behavioural faithfulness checks alone will not catch inert explanations."],"forward_implications":["If influence and faithfulness decouple, then a CoT that is unfaithful as an explanation cannot be assumed harmless or inert; it may still be the mechanism that produced the answer.","CoT evaluations should include a causal test of whether intermediate steps change the answer, not only a comparison of final accuracy or a check of explanation fidelity.","Distilled-reasoning models appear to rely on CoT to reach good accuracy, so removing or shortening their reasoning traces would likely hurt them far more than instruction-tuned models.","Instruction-tuned and reasoning-trained models often reach similar accuracy without changing their initial prediction, suggesting their CoT largely justifies an already-chosen answer.","A CoT that verbally acknowledges a cue is not proof the cue caused the answer; the confidence trajectory can remain flat in such cases."],"supporting_citations":[{"why":"Supplies the chain-of-thought prompting paradigm whose dynamics and faithfulness the paper analyses.","marker":"Wei et al., 2022"},{"why":"Supplies the confidence-trajectory method of measuring the probability of the final answer across CoT steps.","marker":"Wang et al., 2025"},{"why":"Introduces the misleading-cue injection approach the paper adapts to identify unfaithful CoT.","marker":"Turpin et al., 2023"},{"why":"Supplies the verbalisation definition and classifier prompting used to determine whether a CoT acknowledges using a cue.","marker":"Chen et al., 2025"},{"why":"Identifies the soft-reasoning datasets where CoT gives limited gains, motivating the task selection.","marker":"Sprague et al., 2025"},{"why":"Supplies the distilled-reasoning model family and its training recipe, key to the model-type comparison.","marker":"DeepSeek-AI, 2025"},{"why":"Supplies the professor and metadata cue prompts and the verbalisation classification basis.","marker":"Chua and Evans, 2025"}],"fun_headline_variants":["CoT can change answers without ever saying why","Reasoning cues steer even when CoT stays silent","Chain-of-thought: influence vs. faithfulness gap","When CoT acts but won't explain itself"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper assumes that a model's probability for its final answer rising during the chain of thought proves the chain caused the answer, but the same rise could occur in a model already committed to that answer, and confidence is measured by interrupting generation with an extra answer request that may itself change behaviour.","fun_headline_variants_meta":{"raw":{"variants":["CoT can change answers without ever saying why","Reasoning cues steer even when CoT stays silent","Chain-of-thought: influence vs. faithfulness gap","When CoT acts but won't explain itself"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1153,"prompt_tokens":588,"completion_tokens":565,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":332,"completion_tokens_details":{"reasoning_tokens":516}},"tokens_in":332,"tokens_out":565,"duration_ms":6706,"temperature":1.0,"reasoning_tokens":516,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:25:06.113689+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the cases where confidence rises sharply on the final CoT step and rerun the same prompt with that final step removed or rewritten; if the final answer stays the same, the rising trajectory did not establish causal influence. A second check: measure the final-answer probability with and without the interrupting probe instruction; if the probe inflates the probability, the trajectory is an artefact of measurement rather than evidence of steering.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the misleading-cue injection approach the paper adapts to identify unfaithful CoT."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies the soft-reasoning datasets where CoT gives limited gains, motivating the task selection."}],"review_version":1}