{"id":"a7f66181-00cf-4bf5-9d40-2010972a509f","arxiv_id":"2605.27715","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"DATG framework diagnoses that non-English reasoning in Qwen3 models shows reduced mathematical anchor coverage and dependency fidelity, with Loop-Retry and Formula-Retry improving target-language accuracy.","lead":"The paper finds that language affects LLM math reasoning execution itself, not just input understanding, as shown by accuracy drops when forcing non-English reasoning even on English problems. A new graph framework diagnoses specific failures in anchor coverage and dependencies, leading to simple retry fixes that improve low-resource language performance.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"DATG mapping may embed language-specific artifacts from the trace-to-anchor LLM","rationale":"The reader's weakest_assumption directly identifies the same load-bearing point. Because the full manuscript is now available, the concern can be tested precisely against the DATG construction details rather than remaining at the abstract level; no other internal inconsistency appears more central to the execution-vs-understanding distinction.","tokens_in":1685,"tokens_out":311,"duration_ms":16094,"concrete_test":"Take 20 matched English/non-English reasoning traces for the same math problems; re-run the DATG mapping pipeline on both while holding the mapping model and prompt fixed; measure Jaccard overlap of extracted anchor sets and edge sets. If overlap is <0.85 for low-resource languages while human annotators judge the mathematical content equivalent, the language-independence assumption fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that language affects reasoning execution beyond input understanding—rests on DATG producing truly language-independent anchors and edges. The abstract states that traces are mapped to 'language-independent mathematical anchors,' but if this mapping is performed by an LLM (standard for such frameworks), the anchor extraction step itself can inherit language biases from the input trace. Non-English traces could then yield systematically different or incomplete anchor sets even when the underlying mathematical steps are identical, confounding the reported drops in 'anchor coverage' and 'dependency fidelity.' This directly weakens the diagnosis that observed accuracy gaps reflect execution differences rather than mapping artifacts.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that failures in multilingual mathematical reasoning by large reasoning models are not limited to input understanding but extend to reasoning execution itself. This is evidenced by experiments where English problem statements yield lower accuracy when the model is induced to reason in non-English languages. To diagnose this, the authors introduce the Directed Acyclic Trace Graph (DATG) framework, which maps reasoning traces to language-independent mathematical anchors and dependency edges, enabling metrics for anchor coverage, dependency fidelity, and avoidance of harmful actions. Experiments on the Qwen3 series across 12 languages show reduced coverage and fidelity in non-English reasoning, particularly low-resource languages; motivated by this, they propose Loop-Retry and Formula-Retry test-time interventions that improve performance.","tokens_in":1789,"tokens_out":557,"duration_ms":27938,"significance":"If the DATG construction produces truly language-independent anchors, the work provides a valuable diagnostic lens that moves beyond input-centric explanations of multilingual gaps and offers concrete, deployable test-time fixes. The multi-language empirical scope and the introduction of a trace-alignment framework are strengths that could influence future multilingual LRM evaluation. The practical improvements from the retry methods add applied value.","major_comments":[{"comment":"§3 (DATG construction): The central claim that observed accuracy gaps reflect reasoning-execution differences (rather than mapping artifacts) rests on the assertion that anchors and edges are language-independent. The manuscript provides no validation that the trace-to-anchor LLM produces equivalent anchor sets for semantically identical reasoning steps expressed in different languages; without inter-language anchor agreement statistics or human validation on a held-out set, differences in 'anchor coverage' could arise from the mapping step itself inheriting language biases from the input trace.","section":"§3"},{"comment":"§4 (Experiments and results): The reported accuracy reductions when controlling reasoning language (even with English inputs) are load-bearing for the 'beyond input understanding' thesis, yet the text supplies no details on the number of independent runs, statistical significance tests, variance across prompts, or explicit controls that isolate reasoning-language effects from prompt-format confounds. This absence prevents assessment of whether the effect sizes are robust.","section":"§4"}],"minor_comments":[{"comment":"The abstract would benefit from explicitly naming the 12 languages and the precise Qwen3 variants used, to allow immediate replication assessment.","section":"Abstract"},{"comment":"Figure captions for DATG visualizations should include the exact prompt template used for anchor extraction so readers can judge potential language leakage.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on the DATG framework and experimental reporting. The comments highlight important areas for strengthening the claims regarding language-independent anchors and result robustness. We address each major comment below and commit to revisions where needed.","responses":[{"response":"We agree that explicit validation of anchor language-independence is necessary to rule out mapping artifacts and support the core thesis. The anchors are designed as language-agnostic mathematical primitives (e.g., 'solve linear equation' or 'apply chain rule'), extracted via a fixed prompt template, but the manuscript indeed lacks inter-language agreement metrics or human validation. In the revised version, we will add: (1) automated anchor agreement rates across language pairs on a held-out set of 200 traces, and (2) human evaluation on a 50-trace sample per language to measure semantic equivalence, with results reported in a new subsection of §3.","revision_made":"yes","referee_comment":"§3 (DATG construction): The central claim that observed accuracy gaps reflect reasoning-execution differences (rather than mapping artifacts) rests on the assertion that anchors and edges are language-independent. The manuscript provides no validation that the trace-to-anchor LLM produces equivalent anchor sets for semantically identical reasoning steps expressed in different languages; without inter-language anchor agreement statistics or human validation on a held-out set, differences in 'anchor coverage' could arise from the mapping step itself inheriting language biases from the input trace."},{"response":"We acknowledge that the current experimental reporting lacks sufficient detail on reproducibility and controls, which limits evaluation of robustness. The manuscript reports point estimates without variance or significance testing. In the revision, we will expand §4 to include: results averaged over 5 independent runs with different seeds (reporting mean and standard deviation), paired t-tests for significance between English-input/English-reasoning vs. English-input/non-English-reasoning conditions, and an ablation varying prompt phrasing while holding reasoning language constant to isolate language effects from format confounds.","revision_made":"yes","referee_comment":"§4 (Experiments and results): The reported accuracy reductions when controlling reasoning language (even with English inputs) are load-bearing for the 'beyond input understanding' thesis, yet the text supplies no details on the number of independent runs, statistical significance tests, variance across prompts, or explicit controls that isolate reasoning-language effects from prompt-format confounds. This absence prevents assessment of whether the effect sizes are robust."}],"tokens_in":1439,"tokens_out":530,"duration_ms":25150,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main things to know are that even English inputs produce lower accuracy when the model is forced to reason in another language, and that the authors built DATG to turn traces into graphs anchored on language-independent math steps so they can measure coverage, dependency fidelity, and bad actions.\n\nDATG is the actual new piece. It lets them align non-English traces against reference DAGs and quantify where the reasoning breaks. The experiments run on Qwen3 across 12 languages and report worse anchor coverage and weaker edges in low-resource languages. They then test two simple retry methods aimed at the diagnosed problems and claim consistent gains.\n\nThe work is useful because it pushes past the usual input-understanding story and gives a concrete way to look at execution failures. The retry controls are straightforward and target the measured issues directly.\n\nThe soft spots sit in the DATG construction itself. The abstract gives no detail on how anchors are extracted or validated, so it is hard to rule out language-specific artifacts in the mapping step. The stress-test point about the trace-to-anchor process inheriting biases is reasonable and needs checking; if the anchors are not fully independent, the reported drops could partly reflect the diagnostic rather than pure execution differences. No statistical tests or prompt controls are mentioned either.\n\nThis is for people working on multilingual reasoning models and practical deployment. A reader who wants diagnostics for non-English math failures will get something from the framework and the failure-mode breakdown. It deserves serious referee time because the problem matters and the method is a fresh angle, even if the paper will need more on how DATG is built and tested.","headline":"The paper shows language affects math reasoning execution beyond input understanding, via controlled experiments and a new DATG diagnostic on Qwen3 across languages.","tokens_in":2309,"tokens_out":399,"would_cite":false,"duration_ms":27629,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Even with English problem statements, forcing non-English reasoning substantially lowers mathematical accuracy in large models.","keywords":["multilingual mathematical reasoning","directed acyclic trace graphs","reasoning language effects","low-resource languages","test-time interventions","trace alignment","mathematical anchors"],"falsifier":"If models reach the same accuracy on identical English math problems when forced to reason in English as when allowed to reason in their target language, the claim that language affects reasoning execution would be refuted.","tokens_in":2587,"feed_emoji":"","tokens_out":667,"duration_ms":23346,"temperature":0.7,"pith_summary":"Large reasoning models excel at math when working in English but perform worse in other languages. The paper shows this gap is not only about reading the question, because accuracy falls when the model is made to reason in a non-English language even if the problem itself is in English. To examine the effect, the authors build DATG, a graph that turns reasoning steps into language-free math anchors and the links between them. Experiments with Qwen3 models across twelve languages find that non-English reasoning covers fewer anchors, follows fewer required links, and produces more invalid steps, with the largest problems in low-resource languages. Two simple retry methods that fix the exposed errors raise performance in those languages.","feed_headline":"Reasoning language cuts math accuracy even on English problems","feed_subtitle":"DATG graphs show non-English traces miss anchors and dependencies; two retry methods raise low-resource results","key_machinery":"DATG, a Directed Acyclic Trace Graph that converts reasoning traces into language-independent mathematical anchors and dependency edges for alignment and error measurement.","core_discovery":"The central claim is that language shapes the execution of mathematical reasoning, not merely the understanding of the input. Using DATG to align target-language traces against reference DAGs built from English traces, the paper finds that non-English traces achieve lower coverage of required mathematical anchors, weaker fidelity to dependency edges, and higher rates of harmful actions, with the deficits most pronounced in low-resource languages. This diagnosis directly motivates two test-time controls, Loop-Retry and Formula-Retry, that target the identified failure modes and improve accuracy.","pith_inferences":["Models may need training signals that explicitly separate language from step-by-step math structure.","The same anchor-and-dependency analysis could diagnose reasoning shortfalls in domains such as code generation.","If the mapping from trace to anchors is reliable, similar graphs could serve as training targets for language-agnostic reasoning modules."],"forward_implications":["Non-English reasoning traces cover fewer required mathematical anchors than English traces do.","Dependency edges are respected less faithfully when the model reasons in the target language, especially low-resource ones.","Loop-Retry and Formula-Retry improve target-language performance by correcting the failures DATG identifies.","The accuracy gap appears consistently across the Qwen3 model series and twelve languages."],"fun_headline_variants":["Reasoning language affects math execution not input","DATG exposes anchor coverage failures in non-English traces","Low-resource languages suffer math dependency failures","Test-time retries boost target language math reasoning"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The graph method creates math steps and connections that stay the same no matter which language the model used to produce the trace.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning language affects math execution not input","DATG exposes anchor coverage failures in non-English traces","Low-resource languages suffer math dependency failures","Test-time retries boost target language math reasoning"]},"model":"grok-4.3","cost_usd":0.008354,"raw_usage":{"total_tokens":3782,"prompt_tokens":666,"num_sources_used":0,"completion_tokens":54,"cost_in_usd_ticks":83537000,"prompt_tokens_details":{"text_tokens":666,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3062,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":666,"tokens_out":54,"duration_ms":25515,"temperature":1.0,"reasoning_tokens":3062,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T17:54:52.507439+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If models reach the same accuracy on identical English math problems when forced to reason in English as when allowed to reason in their target language, the claim that language affects reasoning execution would be refuted.","supporting_citations":[],"review_version":1}