{"id":"6a67a92a-fa5e-461f-8cf0-b9bba97289a2","arxiv_id":"2506.16151","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs distribute attention differently for Chinese and English causal sentences, lose more accuracy on reversed Chinese order, yet converge to similar representations when they reason correctly.","lead":"This paper studies whether language models reason differently in Chinese and English by building a bilingual causal-reasoning dataset and inspecting internal attention patterns. It finds Chinese inputs bias attention toward causes and sentence-initial connectives, and that reversed causal order hurts Chinese accuracy more than English accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on unverified corpus-frequency/naturalness assumption: Chinese reversed chains are asserted rare and unnatural, but no counts or human ratings support this; if false, the 76.5% accuracy drop is a stimulus artifact, not an internalized typological prior.","rationale":"Good faith reading: the paper is a careful descriptive study of attention/accuracy across EN/ZH causal chains, with a novel dataset and fine-grained analysis. It has real strengths: controlled bilingual construction, modular decomposition, multi-model appendices, and a falsifiable prediction (reversed Chinese should degrade if the prior is learned). The central inference, however, is the step from observed accuracy/attention differences to the claim that LLMs 'internalize the reasoning biases shaped by language.' That step depends on the corpus-frequency asymmetry being true. The reader flagged exactly this as the weakest assumption; I agree. I do not think this requires rejection: the paper can be made convincing by adding corpus counts and naturalness ratings, and by testing alternative Chinese reversed paraphrases. A second candidate concern is the absence of significance testing on the 76.5% vs 88.5% gap; that is important but secondary, because with n=400 per condition the gap is likely significant; the bigger question is what it means. A third candidate is that attention patterns are correlational, not causal; but the paper does not overclaim causation, and its 'rigidly apply' language is interpretive. The corpus-frequency/naturalness issue is the one that, if wrong, would undercut the headline conclusion rather than merely weaken a detail. Hence I recommend keeping the reader's CONDITIONAL verdict, so the verdict is UNCHANGED.","tokens_in":14651,"tokens_out":4808,"duration_ms":57827,"concrete_test":"Compute corpus frequency of the exact reversed causal constructions in matched Chinese and English corpora (e.g., Zh-C4/Wikipedia vs En-C4/Wikipedia) for the templates 'A，是由于B，而这源自C' vs 'A, due to B, which originates from C', and collect native-speaker naturalness ratings (Likert 1-7) from 20+ speakers per language on 50 matched forward/reversed items. Also run the same BICAUSE reversed-chain evaluation with 2-3 alternative idiomatic Chinese reversed paraphrases (e.g., 'A之所以B，是因为C' constructions). If the alternative paraphrases close the accuracy gap, the original drop is a template artifact; if not, the cause-first prior reading is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of the paper's strongest claim is that reversed causal constructions are genuinely rare and marked in Chinese but common in English, so the model's degraded accuracy on Chinese reversed chains (76.5% vs 88.5% English reversed) reflects an internalized cause-first prior. This premise is asserted in Section 5.2 ('they are rare in native usage and likely underrepresented in the training corpus') and in Figure 1, but no corpus frequency counts or native-speaker naturalness ratings are provided. The specific Chinese reversed template '...是由于...而这源自...' is a formal, literary construction that may be a calque of the English 'due to... which originates from...'; if it is less idiomatic than the English template, the 12-point accuracy gap could be a stimulus-construction artifact. The central behavioral finding therefore does not uniquely support the linguistic-relativity/internalization reading without measurement of the assumed corpus asymmetry and template naturalness. This is the key unverified link between observed attention/accuracy patterns and the claim that LLMs internalize language-specific causal expression preferences.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BICAUSE, a bilingual Chinese-English causal reasoning dataset with semantically aligned forward and reversed causal chains across eight domains. Using the RCAR attention metric (Eq. 1-2), SVCCA, and hidden-state cosine similarities on Qwen1.5-1.8B-Chat plus several larger models, it reports three findings: LLMs allocate attention in typologically aligned ways (more subject/connective focus in Chinese, more verb/result focus in English); they internalize a cause-first ordering preference for Chinese that degrades accuracy on reversed Chinese chains (Table 2: Qwen1.5-1.8B 76.8% Chinese vs 88.5% English); and successful reasoning is accompanied by cross-language convergence of hidden representations. The authors interpret these results as evidence that LLMs internalize language-specific causal reasoning biases, echoing linguistic relativity.","tokens_in":14829,"tokens_out":5260,"duration_ms":61130,"significance":"If the claims hold, the paper provides a new kind of evidence for linguistic relativity in LLMs and a reusable dataset and analytical framework for cross-lingual interpretability. The dataset is publicly released, the RCAR metric is explicitly defined, and the main attention analyses are replicated across several model families and sizes, which are concrete strengths. The central limitation is that the behavioral asymmetry is attributed to an unmeasured corpus-frequency asymmetry, so the current evidence does not yet uniquely support the linguistic-relativity reading. The representation-convergence claim is also weakened by selection on correct predictions. These issues are addressable, however, and the paper's overall direction is sound.","major_comments":[{"comment":"The claim that reversed causal chains are 'rare in native usage and likely underrepresented in the training corpus' is the load-bearing premise for explaining the Chinese reversed-chain accuracy drop as an internalized cause-first prior. The manuscript provides no corpus counts, n-gram frequencies, independent citations, or human naturalness ratings for the reversed Chinese template such as '香气扩散，是由于面包烤熟，而这源自面包机加热'. Without such measurements, the 12-point accuracy gap in Table 2 could be a stimulus-construction artifact, e.g., if the Chinese reversed template is a formal or literary calque of the English 'due to ... which originates from'. I request a concrete test: report corpus frequencies of the forward and reversed construction types in large Chinese and English corpora, and/or collect native-speaker naturalness ratings, and show that model accuracy tracks this measured asymmetry.","section":"Section 5.2, Figure 1, Table 2"},{"comment":"The central quantitative comparisons, including RCAR differences up to 1.585 in Figure 3 and the Chinese reversed accuracy drop, are presented without confidence intervals, bootstrap estimates, or significance tests, despite the dataset containing only 400 samples (50 per domain). For instance, Table 2 reports Qwen1.5-1.8B at 88.5% (English) versus 76.8% (Chinese) on reversed chains, but no uncertainty is attached and the domain-level values show high variance (Qwen1.5-1.8B Chinese ranges from 64% to 86% across domains). Please report per-condition confidence intervals and pairwise tests across models before claiming a systematic Chinese-specific degradation.","section":"Section 4.2-4.3, Figures 2-3, Tables 1-2"},{"comment":"The claim that successful reasoning converges to shared, language-agnostic representations is based on cosine similarity computed only over pairs where both predictions are correct. This selection on the dependent variable can inflate similarity, because correct cases may be systematically easier or less noisy, so the convergence could reflect task difficulty rather than language-agnostic abstraction. Please report the same analysis on all samples or on matched correct/incorrect subsets, and compare against a shuffled-token baseline, to show that the convergence is not an artifact of filtering.","section":"Section 6, Figure 5"},{"comment":"The typological interpretation of attention divergence is plausible but confounded with surface word order and the connective inventory. For example, 'once' appears sentence-initially in both languages in the forward template, while Chinese and English connectives have different positional and co-occurrence distributions. A stronger test would include English stimuli with Chinese-like preposed connectives or Chinese stimuli with English-like postverbal connectives, to separate typological bias from template-specific surface effects.","section":"Section 4.3"}],"minor_comments":[{"comment":"The text reports the Chinese reversed-chain accuracy as 76.5%, but Table 2 lists 76.8%; please reconcile the numbers.","section":"Section 5.2 and Table 2"},{"comment":"Table 2 is labeled 'paraphrased data,' while Section 3 consistently calls these examples 'reversed causal chains'; please unify the terminology and clarify whether the 400-sample total refers to forward chains, reversed chains, or chain semantics shared across both forms.","section":"Section 3 and Appendix A"},{"comment":"The caption contains a typo, 'LLayerwise,' and the figure label in Section 5.1 also says 'lanuages'; please correct these.","section":"Figure 4 caption"},{"comment":"Several reference entries are malformed or inconsistent, including 'Steven Piantadosi' as the author of 'Meaning without reference in large language models' (should be Piantadosi and Hill), the formatting of the Feng-Fu citation, and 'V oita' in the Voita et al. entry; please check the bibliography.","section":"References"},{"comment":"The bars appear sorted by magnitude, but the sorting criterion is not stated; please describe the ordering and add error bars or a significance annotation.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a provocative and timely thesis, but the central behavioral result rests on an unverified corpus-frequency assumption and the statistical reporting is thin. I see the revision as feasible: adding corpus counts or naturalness ratings, confidence intervals, a non-selected representation analysis, and the template-control experiment would materially strengthen the paper. I would not reject on the current evidence, but I also would not accept before those points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core is BICAUSE and the RCAR decomposition. The controlled forward/reversed causal chains in Chinese and English are a good testbed, and the attention divergence between the two languages is a real descriptive finding. The reversed-Chinese accuracy drop on Qwen1.5-1.8B (76.5% vs 88.5%) is concrete. But the paper's central interpretation—that LLMs internalize a language-specific cause-first prior—rests on an assumption that is asserted, not measured. Section 5.2 says reversed Chinese forms are 'rare in native usage and likely underrepresented in the training corpus'; Figure 1 says the same. No corpus counts, no frequency statistics, no human naturalness ratings. If the reversed Chinese template is simply stilted or a calque, the accuracy gap is a stimulus artifact, not evidence about internalized reasoning biases. The stress-test note lands.\n\nWhat is genuinely new: BICAUSE is carefully aligned semantically and syntactically, and the RCAR metric is clearly defined and reasonable for component-level attention. Running eight models is better than most papers do. The finding that attention tracks word order and connective position across languages is worth having.\n\nThe soft spots are real. Four hundred samples total, no confidence intervals, no significance tests; the 12-point gap on the 1.8B model could easily be noise. More seriously, the effect does not generalize across the appendix models: Qwen1.5-32B gets 94.0 on Chinese reversed versus 90.5 on English reversed; 14B and 72B show almost no gap. The text says the conclusion is generalizable, but the tables contradict it. The representation-convergence analysis (Figure 5) is restricted to samples where both inputs are correct, which biases the cosine similarities upward and makes the convergence claim close to circular. The typological explanation is plausible but post hoc; it is not tested against the possibility that the specific templates, or tokenization, drive the differences.\n\nThis paper is for people working on multilingual interpretability and cross-lingual evaluation. The dataset and analysis framework deserve engagement, but the Whorfian claim outruns the evidence. A serious referee should send it back for significance testing, corpus frequency counts for the reversed Chinese constructions, naturalness ratings, and an honest treatment of the model-size inconsistency. I would not desk-reject it, and I'd be willing to referee it myself, but I wouldn't cite the internalization claim in its current form.","headline":"New dataset and attention-analysis lens, but the headline claim about internalized causal priors rests on an unmeasured corpus-frequency assumption and thin statistics.","tokens_in":15393,"tokens_out":3260,"would_cite":false,"duration_ms":35787,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models internalize language-specific causal habits — attending to sentence-initial causes in Chinese — and the habit degrades reasoning when Chinese input presents effects first.","keywords":["linguistic relativity","causal reasoning","large language models","attention allocation","cross-lingual representations","Chinese-English typology","multilingual interpretability","BICAUSE"],"falsifier":"Count effect-first causal constructions (with connectives such as '是由于...而这源自') in the pretraining corpora of the tested models and compare the rates with their English counterparts; if reversed Chinese forms occur at comparable or higher frequency than English 'X happened, because Y' forms and the 76.5% versus 88.5% gap persists, the internalized-rarity explanation fails. As a second check, have native Chinese readers rate the naturalness of the reversed BICAUSE sentences: ratings at the level of translationese would indicate the stimuli, not a learned prior, drive the drop.","tokens_in":14451,"feed_emoji":"🧠","tokens_out":10305,"duration_ms":94928,"temperature":0.7,"pith_summary":"The paper tests whether large language models inherit the reasoning habits of the languages they are trained on — a machine analogue of the linguistic-relativity idea that language shapes thought. To do this it introduces BICAUSE, 400 semantically aligned Chinese and English three-step causal chains, each written in a forward (cause-first) and a reversed (effect-first) order with an inference question. The authors find that models attend differently in the two languages, favor causes and sentence-initial connectives in Chinese, apply this cause-first habit so rigidly that reversed Chinese chains score 76.5% accuracy versus 88.5% for reversed English, and yet converge on nearly identical hidden representations whenever the reasoning succeeds. If the findings hold, they give the first internal-structure evidence that LLMs encode language-specific causal conventions as stable attention strategies, with measurable behavioral cost when input departs from the convention.","feed_headline":"Reversed Chinese causal chains trip up LLMs at 76.5%","feed_subtitle":"A bilingual causal test shows models inherit a cause-first habit from training data, yet succeed on shared abstractions.","key_machinery":"Two instruments carry the argument. BICAUSE is a dataset of 400 semantically and syntactically aligned three-step causal chains in Chinese and English, each in forward and reversed order with a QA inference question, decomposed into 13 syntactic components and 3 causal components ([cause], [intermediate effect], [final effect]). The Relative Component Attention Ratio (RCAR) measures the attention each component receives from later tokens, normalized per token so comparisons across languages are fair; SVCCA (Singular Vector Canonical Correlation Analysis) scores the similarity of the 24-layer attention trajectories across the four input variants, and layerwise cosine similarity of hidden vectors at the final causal token tracks whether successful reasoning shares a representation. The forward/reversed contrast is what makes the internalized prior visible: identical semantics, identical model, different word order, and the attention pattern moves exactly as a typological habit would predict.","core_discovery":"The paper's central claim is that LLMs do not merely process surface syntax: they internalize language-specific habitual patterns of causal expression as stable attention-allocation strategies, and those habits shape downstream reasoning. Because Chinese prefers forward causal order and topic-initial structure, the model learns a 'sentence-initial component is the cause' prior and locks attention onto the first event; applied to a grammatically valid but atypical reversed chain, the prior misfires, the model anchors on the final effect, and accuracy falls to 76.5%. Because English routinely uses effect-first constructions such as 'X happened, because Y', the same model keeps attention flexible and accuracy high (88.5%) under both orders. The paper further argues that the destination of reasoning is language-agnostic: correctly reasoned samples from all four conditions converge on nearly identical hidden representations, so the divergence is in the habitual path, not in the shared understanding reached at the end.","pith_inferences":["The same measurement could be run on language pairs with other typological contrasts — head-final languages, for example — to predict which constructions will degrade and which will transfer easily.","A causal test of the attention-as-mechanism reading: redirect attention toward the true cause in reversed Chinese chains by activation patching and check whether accuracy recovers to the English level; if attention is only a correlate, the intervention should do nothing.","The rarity premise is directly checkable: corpus statistics on effect-first causal constructions in native Chinese, plus native-speaker naturalness ratings for the reversed BICAUSE sentences, would confirm or refute the claim that the drop comes from an underrepresented pattern rather than from awkward stimuli.","If the habit is learned from corpus statistics, fine-tuning on reversed Chinese causal chains should shrink the 76.5% versus 88.5% gap — a cheap intervention test with practical value for aligning multilingual models."],"forward_implications":["Chinese reversed causal chains will keep underperforming English reversed chains across model families and sizes, because the cause-first habit is a property of the pretraining distribution, not of one architecture.","Answer accuracy on a causal question is predictable from attention placement: the SVCCA similarity between a structure and the canonical forward form tracks accuracy, from 0.73 with roughly 91% down to 0.46 with 76.5%.","Multilingual benchmarks that test only canonical word order will overstate cross-lingual reasoning ability; atypical constructions expose the language-specific ceiling.","Because successful reasoning converges on a shared abstraction, cross-lingual semantic alignment is attainable in representation space even while surface processing remains language-specific."],"supporting_citations":[{"why":"defines the primary model (Qwen1.5-1.8B-Chat) whose attention and representations carry every analysis in the paper.","marker":"(Bai et al., 2023)"},{"why":"supplies the typological claim that Chinese is topic-prominent, the explanation offered for subject-focused attention in Chinese.","marker":"(Li and Thompson, 1976)"},{"why":"supplies the contrastive claim that English is verb-centric, the explanation offered for verb-focused attention in English.","marker":"(Halliday and Matthiessen, 2004)"},{"why":"provides SVCCA, the method used to compare attention trajectories across languages and structures.","marker":"(Raghu et al., 2017)"},{"why":"grounds the linguistic-relativity framing the study is testing in language models.","marker":"(Whorf, 1956)"},{"why":"the prior result that models decide in an English-like representational space, which the paper's convergence finding extends to causal reasoning.","marker":"(Schut et al., 2025)"},{"why":"the prior result that shared morphosyntactic concepts are encoded across languages, supporting the shared-abstraction reading.","marker":"(Brinkmann et al., 2025)"}],"fun_headline_variants":["Chinese cause-first habit drops LLM accuracy to 76.5% on reversed chains","LLMs inherit language-specific causal order biases from training","Bilingual study: LLMs reason alike despite different attention habits","Why LLMs trip on Chinese causal order but ace English reversed forms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that effect-first causal sentences are genuinely rare in the Chinese text the models were trained on; if they are actually common, or if the authors' reversed Chinese sentences read as unnatural templates rather than native usage, the accuracy gap would reflect the test items rather than an internalized cause-first habit.","fun_headline_variants_meta":{"raw":{"variants":["Chinese cause-first habit drops LLM accuracy to 76.5% on reversed chains","LLMs inherit language-specific causal order biases from training","Bilingual study: LLMs reason alike despite different attention habits","Why LLMs trip on Chinese causal order but ace English reversed forms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00064,"raw_usage":{"total_tokens":2946,"prompt_tokens":946,"completion_tokens":2000,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":1925}},"tokens_in":562,"tokens_out":2000,"duration_ms":15874,"temperature":1.0,"reasoning_tokens":1925,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:44:11.601338+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count effect-first causal constructions (with connectives such as '是由于...而这源自') in the pretraining corpora of the tested models and compare the rates with their English counterparts; if reversed Chinese forms occur at comparable or higher frequency than English 'X happened, because Y' forms and the 76.5% versus 88.5% gap persists, the internalized-rarity explanation fails. As a second check, have native Chinese readers rate the naturalness of the reversed BICAUSE sentences: ratings at the level of translationese would indicate the stimuli, not a learned prior, drive the drop.","supporting_citations":[],"review_version":1}