{"id":"8d9821d5-39ec-4564-8a94-4f60cd7e071e","arxiv_id":"2508.16599","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Humans cannot reliably identify which steps in AI reasoning text causally influence the model's answer, performing near chance even on high-agreement questions.","lead":"This paper tests whether people can tell which steps in an AI model's reasoning text actually matter to the model's final answer. Participants scored 29% on a counterfactual causal-influence task, barely above random 25%, and only 42% on the most agreed-upon questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Counterfactual task may not measure true causal influence; validity needs synthetic ground-truth validation.","rationale":"I read the claim in good faith: the paper asks whether humans can identify which reasoning steps causally influence later steps, using counterfactual measurements. The headline number—29% accuracy versus 25% chance—is striking, and if the task is valid, the conclusion is important. However, the entire argument hinges on the construct validity of the counterfactual task. The abstract does not describe how counterfactuals were generated, how 'causal influence' was defined, how ambiguity was controlled, or how the ground truth was established. This is precisely the reader's weakest_assumption. I agree with it and see it as load-bearing. My proposed test is a synthetic-control validation: apply the same pipeline to traces with known causal structure. If humans perform well there, the task is measuring something real; if not, the low accuracy on real model traces is uninterpretable. Because the full text is unavailable and the key methodological details are absent, I do not see grounds to change the reader's UNVERDICTED verdict. The concern does not require rejection; it requires evidence that the task measures what it claims. I also credit the abstract's falsifiable framing and the use of majority-vote agreement as a secondary check, but those do not resolve the validity question.","tokens_in":609,"tokens_out":2241,"duration_ms":25152,"concrete_test":"Run a control condition using synthetic reasoning chains with known ground-truth causal dependencies (e.g., a scripted generator where step B's value is computed only from step A, and other steps are irrelevant). Feed the same counterfactual question-generation pipeline to these chains and recruit participants. If accuracy on synthetic known-structure items is high (say >75%) while real-model accuracy is 29%, the task is valid; if accuracy is similarly near chance on known-structure items, low accuracy is a measurement artifact. Additionally, compute inter-annotator agreement and per-question accuracy stratified by perturbation type to check whether only a subset of items drives the effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the counterfactual questions being a faithful operationalization of causal influence in the model's reasoning. The abstract does not establish this. In autoregressive LLMs, replacing or modifying a reasoning step can alter continuation distributions through surface statistics, coherence, or sampling, not necessarily through the step's actual causal role. Conversely, a step can be genuinely load-bearing yet its counterfactual replacement produce no answer change if the model is insensitive to that particular perturbation. Without evidence that (a) perturbations isolate single steps, (b) the causal ground truth is well-defined and validated, and (c) low accuracy is not driven by ambiguous questions or arbitrary modifications, the 29% figure can reflect task artifacts rather than human misunderstanding. The reader's weakest_assumption identifies exactly this: the counterfactual questions must faithfully capture the causal structure. The abstract alone provides no methods to rule out these artifacts, so the verdict remains unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical study of whether humans can identify which steps in an AI model's step-by-step reasoning text causally influence later steps. The authors construct counterfactual questions based on measurement of causal influence and test human participants. They report that participant accuracy was only 29%, barely above the 25% chance level, and remained low (42%) on questions with high agreement. They conclude that there is a fundamental gap between human interpretation of reasoning text and the model's actual computational process, arguing that reasoning text should be treated as an artifact to be investigated rather than taken at face value. This review is based on the abstract only, as no full text was provided.","tokens_in":832,"tokens_out":1790,"duration_ms":19293,"significance":"If the finding is robust, it has direct implications for the interpretability and transparency of chain-of-thought reasoning in AI systems, where reasoning text is often presented as a human-readable window into model computation. The paper's framing of a necessary condition for correspondence — humans' ability to identify causally influential steps — is a sensible and testable starting point. The counterfactual approach is a plausible empirical strategy. However, the abstract alone provides insufficient methodological detail to assess the validity of the measurement, and the central claim depends entirely on that validity. The significance would be high if the full paper establishes a validated causal ground truth and statistically sound human evaluation.","major_comments":[{"comment":"The abstract reports accuracy figures (29%, 42%) but gives no sample size, participant demographics, statistical test, confidence interval, or p-value. With a binary/4-option task, the difference between 29% and 25% could be within noise for a modest sample. A rigorous report must include these quantities; without them the central empirical claim is not evaluable.","section":"Abstract, results sentence"},{"comment":"The abstract says questions were 'composed based on counterfactual measurements' but does not specify how causal influence was operationalized or validated. In autoregressive LLMs, replacing a reasoning step can change answer distributions through surface statistics, coherence, or sampling effects, not necessarily through the step's actual causal role. Conversely, a genuinely load-bearing step might not change the output under a particular perturbation. Without evidence that (a) the perturbations isolate single steps, (b) the causal ground truth is well-defined and validated, and (c) low accuracy is not driven by ambiguous or arbitrary counterfactual modifications, the 29% figure may reflect task artifacts rather than human misunderstanding. This is the load-bearing validity concern for the paper's main claim.","section":"Abstract, method description"},{"comment":"The 42% accuracy on 'questions with high agreement' is used to argue that the effect persists even under favorable conditions. However, the abstract does not define how agreement was measured (inter-participant agreement? model agreement?) or what threshold was used. If high-agreement questions were selected in a way that correlates with question simplicity or with the counterfactual modification being more natural, the interpretation changes. This needs a concrete definition and sensitivity analysis.","section":"Abstract, 'high agreement' clause"}],"minor_comments":[{"comment":"The phrase 'majority vote on questions' is ambiguous: does it mean majority voting over participants' answers per question, or majority voting over multiple question versions? If the former, the 42% is the per-question accuracy after aggregating participant responses; if the latter, the description needs clarification.","section":"Abstract, method description"},{"comment":"The phrase 'how models use it' should likely be 'how models use them' (referring to reasoning texts). Minor wording issue.","section":"Abstract, conclusion"},{"comment":"The term 'barely above chance' is qualitative; please report effect size (e.g., Cohen's h or odds ratio) and uncertainty. This is important for interpreting the magnitude of the discrepancy.","section":"Abstract, results sentence"}],"recommendation":"major_revision","confidential_remarks":"This is an abstract-only review. The core idea is interesting and the necessary-condition framing is a good contribution. However, the abstract alone does not provide enough information to verify the central claim. The main risk is construct validity of the counterfactual task; if the full paper includes a synthetic ground-truth validation (e.g., controlled tasks with known causal structure), the evidence could be convincing. I recommend the editor obtain the full manuscript before making a final decision. Also, please ensure the authors provide full statistical reporting and task details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: I've only seen the abstract, but the headline result is genuinely interesting. Participants identified causally influential reasoning steps in LLM chain-of-thought text only 29% of the time, barely above the 25% chance level, and stayed low (42%) even on high-agreement questions. If the measurement is sound, this is a real finding that weakens the assumption that CoT text is a transparent window into model computation.\n\nWhat's new: the paper applies counterfactual causal-influence questions to human comprehension of model reasoning. That is a fresh operationalization, not a restatement of prior interpretability work. The paper also frames the task as testing a necessary condition for correspondence, which is honest and non-circular. No self-citation games or invented machinery here.\n\nWhere the soft spots are: they're big, but they're the soft spots of an abstract, not necessarily of the underlying study. No sample size, no participant demographics, no model specifications, no task construction details, no statistical tests. The biggest conceptual concern is the counterfactual operationalization itself. The stress-test note is right: replacing a reasoning step can change the continuation distribution through surface statistics or sampling noise, not necessarily through the step's true causal role. Conversely, a genuinely load-bearing step might not change the answer if the model is insensitive to that perturbation. Without evidence that (a) the counterfactual modifications isolate single steps, (b) the causal ground truth is validated—ideally against synthetic tasks where the causal structure is known—and (c) the questions aren't simply ambiguous, the 29% figure could reflect task artifacts rather than human misunderstanding.\n\nThat said, I can't conclude the concern is fatal without reading the methods. The abstract alone doesn't rule it out or confirm it. The paper deserves a serious referee because the question matters and the result is sharp. The referee should demand full task details, a validation of the causal ground truth, and a robustness analysis that rules out perturbation artifacts.\n\nBottom line: this is a paper for interpretability and AI safety readers who worry about CoT transparency. It deserves peer review, not desk rejection, but it needs substantial methodological scrutiny before the headline claim can be trusted.","headline":"Abstract-only review: the 29%-vs-chance result on causal CoT understanding is striking and worth refereeing, but the counterfactual task's validity is unestablished and must be substantiated.","tokens_in":1214,"tokens_out":1258,"would_cite":false,"duration_ms":14033,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Humans perceive a wrong narrative from AI reasoning text, the paper claims: readers cannot reliably tell which steps actually drive a model's conclusion.","keywords":["chain-of-thought reasoning","interpretability","human-AI alignment","causal influence","counterfactual evaluation","reasoning text","transparency","large language models"],"falsifier":"A controlled study with reasoning texts whose causal structure is objectively known, such as simple multi-step arithmetic or logic derivations where changing step A necessarily changes or removes step B, would settle the claim: if humans can identify the causally influential steps at high accuracy on such texts, the paper's low accuracy is specific to language-model reasoning text; if humans still fail there, the failure is a general property of how people read stepwise reasoning.","tokens_in":563,"feed_emoji":"🤖","tokens_out":1641,"duration_ms":18143,"temperature":0.7,"pith_summary":"The paper asks whether the step-by-step reasoning text produced by AI models can serve as a genuine window into how those models compute. It argues that a necessary condition for that window to work is that humans can identify which reasoning steps causally influence later steps. Using counterfactual questions derived from the model's own reasoning, the paper finds that people pick the causally influential step only 29% of the time, barely above 25% chance, and remain near chance (42%) even on questions where experts strongly agree. The authors conclude that reasoning text is not a simple interpretability tool and should be treated as an artifact to investigate rather than a faithful record of computation.","feed_headline":"People can't spot which AI reasoning steps matter","feed_subtitle":"In counterfactual tests, humans picked the causally decisive step only 29% of the time, near chance.","key_machinery":"The central machinery is the counterfactual causal question: take a reasoning text, alter one step (or set of steps), and ask whether a subsequent step's content would change. The authors treat this as the ground-truth measure of which steps are causally influential, then test whether human readers can identify those influential steps by answering the same counterfactual questions. A majority-vote aggregation over participants supplies a second layer, the 'high agreement' subset, which tests whether the failure is merely individual noise or a systematic collective misreading.","core_discovery":"The central claim is that humans lack the ability to recover the causal influence structure of AI-generated reasoning text. The authors construct questions from counterfactual measurements: they modify a particular reasoning step and ask whether a later step would change, which is the ground-truth definition of causal influence. Aggregating across participants, accuracy in identifying the causally decisive steps was 29%, essentially indistinguishable from the 25% guessing baseline. Even when the authors restricted to questions with high inter-participant agreement, accuracy rose only to 42%. This demonstrates a systematic mismatch between the narrative humans construct when reading reasoning","pith_inferences":["The paper's counterfactual test could be repurposed as a training or interface intervention: if humans were shown multiple altered versions of the same reasoning trace, their causal attributions might improve, testing whether the failure is a fixed perception limit or a learnable skill.","The 29% accuracy may partly reflect the strange, non-narrative way models chain ideas; comparing human performance on synthetic reasoning traces with known causal structure (e.g., arithmetic proofs) would isolate whether the problem is human causal reasoning in general or specifically AI text.","The results connect to the broader phenomenon of post-hoc rationalization: models may produce plausible-sounding text that narrates a conclusion without tracking the computation that produced it, implying that interpretability claims based on textual coherence are suspect.","A practical extension would measure whether giving humans full access to the model's internal token probabilities or attention weights alongside the text improves causal attribution, testing whether richer signals can bridge the perceived-vs-actual gap."],"forward_implications":["If humans cannot reliably identify causally influential reasoning steps, then chain-of-thought text cannot be trusted as a standalone transparency mechanism for auditing or debugging model behavior.","Reasoning text should be studied as a separate artifact, not equated with the model's internal computation, opening a research direction into how models' language use diverges from human narrative conventions.","Human oversight of AI systems that relies on reading reasoning traces likely misses critical errors, so safety mechanisms should be built on other evidence, such as behavior or external verifiers.","The near-chance performance on one-step causal attribution suggests that even simple interpretability tasks over reasoning text are unreliable, not just complex ones."],"supporting_citations":[],"fun_headline_variants":["Human readers can't follow AI's causal reasoning","AI reasoning text misleads humans, study finds","Humans near chance at spotting AI decisive steps","People see wrong narratives in AI reasoning","AI's step-by-step text doesn't explain its logic"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The counterfactual questions perfectly capture which steps actually cause later steps in the model's reasoning; if the questions are ambiguous or the modifications alter the model's behavior in unexpected ways, the near-chance human scores could be an artifact of the test rather than evidence of a fundamental interpretation gap.","fun_headline_variants_meta":{"raw":{"variants":["Human readers can't follow AI's causal reasoning","AI reasoning text misleads humans, study finds","Humans near chance at spotting AI decisive steps","People see wrong narratives in AI reasoning","AI's step-by-step text doesn't explain its logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1051,"prompt_tokens":685,"completion_tokens":366,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":294}},"tokens_in":429,"tokens_out":366,"duration_ms":4014,"temperature":1.0,"reasoning_tokens":294,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:21:43.680864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study with reasoning texts whose causal structure is objectively known, such as simple multi-step arithmetic or logic derivations where changing step A necessarily changes or removes step B, would settle the claim: if humans can identify the causally influential steps at high accuracy on such texts, the paper's low accuracy is specific to language-model reasoning text; if humans still fail there, the failure is a general property of how people read stepwise reasoning.","supporting_citations":[],"review_version":1}