{"id":"9830a470-d1c9-4dea-a324-e27a72aed19d","arxiv_id":"2605.25891","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs encode causal direction internally via probes but revert to commonsense in Yes/No outputs on anti-commonsense items, showing output accuracy alone does not measure causal understanding.","lead":"Large language models internally encode correct answers to anti-commonsense causal questions in their hidden states, recoverable by a linear probe at ~0.97 accuracy. Their direct Yes/No outputs instead default to commonsense answers at ~0.5 accuracy, creating a gap the authors term Causal Tongue-Tie.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Linear probe may recover spurious correlations rather than causal encoding","rationale":"The identified concern is identical to the reader’s weakest assumption. Because the review was abstract-only and the full text is now available, the concrete_test above is the minimal check that would falsify or confirm the assumption; until performed the UNVERDICTED status is appropriate.","tokens_in":1664,"tokens_out":290,"duration_ms":18056,"concrete_test":"Retrain the same linear probe architecture on the identical hidden-state activations but with randomly permuted evidence-supported labels; if accuracy remains >0.7, the original probe is not isolating causal information.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The claim that hidden states encode the evidence-supported causal answer (probe acc ~0.97) while output does not rests on the probe extracting genuine causal direction. On anti-commonsense CLadder items this could instead reflect any feature in the residual stream that is merely statistically aligned with the ground-truth label (e.g., specific token patterns, negation markers, or graph-description statistics that differ between commonsense and anti-commonsense subsets). No mention is made of label-shuffled controls, causal interventions on the model, or probes trained on matched non-causal tasks that would rule this out. If such a feature exists, the “internal signal vs. verbal interface” decomposition does not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript claims that LLMs internally encode the evidence-supported causal direction on anti-commonsense CLadder items (recoverable via a fixed linear probe at ~0.97 accuracy from hidden states) but their Yes/No outputs fail to express it, defaulting to the commonsense answer (~0.5 accuracy). This 'Causal Tongue-Tie' is presented as two separable failure modes, with implications that output-only causal benchmarks are insufficient to assess whether models have understood causal structure.","tokens_in":1805,"tokens_out":476,"duration_ms":23307,"significance":"If the central empirical separation between probe and output holds after controls, the result would be moderately significant for causal reasoning evaluations in LLMs: it would demonstrate that output accuracy alone cannot distinguish internal encoding failures from verbal-interface failures. The approach of using probes on hidden states to surface evidence-supported answers offers a useful diagnostic lens beyond verbal responses, though its value depends on ruling out non-causal confounds.","major_comments":[{"comment":"Abstract: the central claim rests on a linear probe achieving ~0.97 accuracy recovering the evidence-supported answer, yet the abstract (and by extension the reported method) provides no details on probe training procedure, feature selection, regularization, statistical significance tests, or cross-validation; without these, it is impossible to assess whether the number supports genuine causal encoding or reflects overfitting or post-hoc selection.","section":"Abstract"},{"comment":"Abstract / implied methods: on anti-commonsense items the probe is said to recover the 'evidence-supported' answer while output reverts to commonsense, but no label-shuffled controls, probes trained on matched non-causal tasks, or causal interventions on the residual stream are described to rule out spurious correlations (e.g., token patterns or graph-description statistics that differ between subsets); this is load-bearing for the 'internal signal vs. verbal interface' decomposition.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract uses approximate figures ('approximately 0.97', 'approximately 0.5') without reporting exact values, confidence intervals, or dataset sizes; adding these would improve precision.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the thoughtful and constructive comments. We address each major point below, indicating where we will revise the manuscript to incorporate additional details and controls while defending the core empirical separation presented.","responses":[{"response":"We agree that the abstract is highly condensed and omits key methodological parameters. The full manuscript describes the probe as a linear logistic regression classifier applied to the final-layer residual stream activations, trained with L2 regularization (strength selected via inner cross-validation) on an 80/20 train/test split per item set, with 5-fold cross-validation used to compute mean accuracy and standard deviation. Feature selection was not applied; the full hidden-state dimension was used. Statistical significance against chance was evaluated with permutation tests (p < 0.001). To address the concern directly, we will revise the abstract to include a concise clause on the probe type, cross-validation, and regularization, and we will add a short methods paragraph summarizing these choices with exact hyperparameter values and significance results.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the central claim rests on a linear probe achieving ~0.97 accuracy recovering the evidence-supported answer, yet the abstract (and by extension the reported method) provides no details on probe training procedure, feature selection, regularization, statistical significance tests, or cross-validation; without these, it is impossible to assess whether the number supports genuine causal encoding or reflects overfitting or post-hoc selection."},{"response":"The anti-commonsense construction itself provides partial protection against simple commonsense or token-frequency confounds, because the probe recovers the evidence-supported direction (opposite to commonsense) at high accuracy while the verbal output does not. Nevertheless, we acknowledge that explicit controls would make the separation more robust. In revision we will add (i) label-shuffled baselines showing probe accuracy collapsing to chance (~0.5) and (ii) a matched non-causal probe (e.g., on syntactic subject-verb agreement) to demonstrate that the high accuracy is not an artifact of any linear probe on these activations. Full causal interventions on the residual stream (e.g., activation patching) lie beyond the current experimental scope and would require substantial additional compute; we will therefore note this as a limitation rather than claim to have performed them.","revision_made":"partial","referee_comment":"[Abstract] Abstract / implied methods: on anti-commonsense items the probe is said to recover the 'evidence-supported' answer while output reverts to commonsense, but no label-shuffled controls, probes trained on matched non-causal tasks, or causal interventions on the residual stream are described to rule out spurious correlations (e.g., token patterns or graph-description statistics that differ between subsets); this is load-bearing for the 'internal signal vs. verbal interface' decomposition."}],"tokens_in":1343,"tokens_out":601,"duration_ms":23552,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that on anti-commonsense CLadder questions a linear probe hits roughly 0.97 accuracy on the evidence-supported answer from hidden states, while the model's spoken yes/no drops to about 0.5 and follows commonsense instead. They name the gap Causal Tongue-Tie and argue it splits into two distinct problems: no internal signal versus a signal the output layer cannot express.\n\nWhat the work does is separate those two failure modes and show why output accuracy alone can mislead when judging causal reasoning. That distinction is worth keeping in mind for anyone running causal benchmarks.\n\nThe soft spot is the probe itself. Nothing in the abstract or stress-test note indicates label-shuffled controls, interventions, or matched non-causal probes that would rule out the probe latching onto token patterns or graph statistics rather than causal direction. Without those checks the 0.97 number does not yet establish that the model encodes the causal fact internally.\n\nThe paper is aimed at people who build or critique LLM evaluation suites. Anyone measuring causal ability from yes/no outputs would find the framing useful to think about.\n\nIt should go to peer review. The observation is straightforward and the implication for benchmarks is real, but the methods need the usual controls before the central claim can be taken as settled.","headline":"The paper flags a probe-output gap on anti-commonsense causal items but the probe's claim to recover genuine causal direction rests on thin controls.","tokens_in":2305,"tokens_out":337,"would_cite":false,"duration_ms":19502,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"LLMs internally encode the evidence-based causal answer but output the commonsense yes/no instead on conflicting questions.","keywords":["causal reasoning","large language models","linear probes","hidden states","commonsense bias","CLadder dataset","yes/no outputs"],"falsifier":"A new collection of anti-commonsense causal questions on which the same linear probe no longer achieves high accuracy at recovering the evidence answer, or on which probe accuracy falls to match the verbal output accuracy.","tokens_in":2558,"feed_emoji":"","tokens_out":635,"duration_ms":15531,"temperature":0.7,"pith_summary":"The paper demonstrates a mismatch on anti-commonsense causal questions from the CLadder dataset: a fixed linear probe on hidden states recovers the answer supported by the given evidence at roughly 97 percent accuracy, yet the model's spoken yes/no answer follows the conflicting commonsense at roughly 50 percent accuracy. This gap is labeled Causal Tongue-Tie and is decomposed into two separable issues—one where no internal signal exists and one where a signal exists but cannot be expressed verbally. The authors conclude that single accuracy numbers from output-only benchmarks are insufficient to determine whether models have grasped causal direction, because a correct output need not indicate understanding and an incorrect output need not indicate inability.","feed_headline":"LLMs encode causal answers internally but say the wrong yes or no","feed_subtitle":"On anti-commonsense questions a probe reads the evidence answer from hidden states while the verbal output follows commonsense instead","key_machinery":"A fixed linear probe applied to the model's hidden states that extracts the evidence-supported causal direction, set against the verbal yes/no generation interface that fails to express it.","core_discovery":"On anti-commonsense CLadder items, hidden-state representations contain the evidence-supported causal answer even when the model's verbal yes/no output reverts to the commonsense answer, producing an accuracy gap of approximately 0.5 between probe recovery and spoken response.","pith_inferences":["Similar internal-versus-output gaps may exist for other structured reasoning tasks that pit evidence against prior patterns.","Evaluation protocols could combine output accuracy with lightweight probes on the same items to separate encoding failures from expression failures.","Interventions that alter hidden-state representations might be tested to see whether they shift verbal outputs toward the probed answer."],"forward_implications":["A benchmark answer labeled correct does not establish that the model has internally represented the causal relation.","A benchmark answer labeled incorrect does not establish that the model lacks the relevant causal representation.","Causal reasoning claims drawn from yes/no accuracy alone require separate checks on internal representations.","The verbal output channel can mask encoded causal knowledge that remains accessible via probing."],"fun_headline_variants":["LLMs encode causal answers internally but fail to voice them in yes no","Hidden states reveal evidence supported causality despite wrong verbal output","Gap shows LLMs understand causality but cannot say the correct yes no","Probe finds causal direction in LLM internals while response reverts to bias","LLMs internally know causal evidence answer but verbalize commonsense instead"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The linear probe is recovering genuine causal knowledge encoded in the hidden states rather than some other correlated but non-causal pattern, and the independently identified evidence-supported answer is the correct target.","fun_headline_variants_meta":{"raw":{"variants":["LLMs encode causal answers internally but fail to voice them in yes no","Hidden states reveal evidence supported causality despite wrong verbal output","Gap shows LLMs understand causality but cannot say the correct yes no","Probe finds causal direction in LLM internals while response reverts to bias","LLMs internally know causal evidence answer but verbalize commonsense instead"]},"model":"grok-4.3","cost_usd":0.00633,"raw_usage":{"total_tokens":2929,"prompt_tokens":579,"num_sources_used":0,"completion_tokens":87,"cost_in_usd_ticks":63299500,"prompt_tokens_details":{"text_tokens":579,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2263,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":579,"tokens_out":87,"duration_ms":19877,"temperature":1.0,"reasoning_tokens":2263,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T22:11:19.507473+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A new collection of anti-commonsense causal questions on which the same linear probe no longer achieves high accuracy at recovering the evidence answer, or on which probe accuracy falls to match the verbal output accuracy.","supporting_citations":[],"review_version":1}