{"id":"65c3436a-0c07-4a00-b10c-876eff27f85c","arxiv_id":"2506.18341","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Multilingual unification learning, which augments a tiny English chain-of-thought seed set with translated and code-switched solutions, is claimed to match or exceed larger English-only test-time scaling recipes with less data and fewer inference tokens.","lead":"The paper tests whether training a reasoning language model on the same math questions translated into several languages, with solutions written partly in one language and partly in another, needs far less data and generates fewer inference tokens than English-only training.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'less data' claim rests on uncontrolled cross-paper baselines and within-noise differences; AIME24's 30-problem scale makes 2-3 point gaps statistically indistinguishable from chance.","rationale":"Fair reading: the paper proposes a multilingual data-augmentation and decoding-intervention recipe and claims it reduces both the required seed data and the number of inference tokens while keeping performance. For that central claim to hold, the empirical comparison must isolate the multilingual augmentation from pipeline and evaluation differences and must be statistically stable. Neither condition is met. Table 3's same-pipeline controls show small, inconsistent gains, and Table 1's headline comparison to s1 relies on external baseline numbers. AIME24 has only 30 items, so a 3.3-point gap is one question; no confidence intervals, seeds, or significance tests are reported. The MATH500 footnote describes manual correction of 8-12 items, affecting the exact reported tie and leaving no correction log for verification. Section 4.8 selects intervention parameters using AIME24 itself, so the reported 73.3% and the associated token curves risk being fitted to the test set. The translation-quality concern identified by the reader is real but secondary: even assuming perfect translations, the controlled comparison is too weak to establish the data-efficiency and token-efficiency claims. I therefore keep the reader's REJECT verdict unchanged, with the recommendation that the authors release artifacts and run the matched-baseline bootstrap check. If that check passes, a CONDITIONAL acceptance with open data and preregistered evaluation would be appropriate.","tokens_in":41649,"tokens_out":8787,"duration_ms":97096,"concrete_test":"Run the L2 ablation with code and data from the authors: train Qwen2.5-32B on (a) the 651-sample English-only s1 subset, (b) the 1k s1 English-only set, and (c) L2-32B-MS1^4_651, using the identical SFT recipe, vLLM settings, and evaluation code. Record AIME24, GPQA, and MATH500 scores over 5 random seeds, plus the mean accepted inference tokens per benchmark. Report bootstrap 95% confidence intervals and a paired McNemar or Fisher test for AIME24. If the L2-651 versus English-1k difference falls inside the confidence interval or does not replicate, the less-data claim fails; if token counts are not significantly lower on matched language prompts, the less-tokens claim also fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3's headline comparison (L2-32B-MS1^4_651 vs s1-32B, Table 1) uses published s1 numbers rather than a baseline run inside the L2 pipeline, so 'comparable with 651 samples' conflates method differences with evaluation and prompt differences. The paper's own matched-scale controls in Table 3 show multilingual gains of only 0.00 on AIME24, 0.04 on GPQA, and 0.00 on MATH500 at 651 samples, and 0.03/0.01/0.02 at 1k samples. Since AIME24 has 30 problems, a 3-point gap is one question, and no seeds, confidence intervals, or significance tests are reported. The MATH500 footnote in Section 4.3 additionally reports manual post-hoc correction of 8-12 answers, which affects the exact parity claim (93.0 vs 93.0) and is not reproducible without the correction log. Finally, Section 4.8 tunes the decoding intervention parameters on AIME24 itself and reports 73.3% for k=2/4, so the token-efficiency evidence is selected on the test set. This is more load-bearing than the translation-quality caveat in Section 3.2: even perfect translations would not fix the uncontrolled comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes L2, a multilingual unification learning method that augments a small seed of high-quality reasoning samples (6, 651, or 1,000 examples) with multilingual chain-of-thought data and step-wise code-switched data, then fine-tunes Qwen2.5-32B to improve reasoning accuracy while reducing inference tokens. A decoding intervention that boosts or suppresses language-boundary tokens is also introduced. The main claims are that multilingual learning reduces the required training data and the number of inference tokens while maintaining comparable performance, and that the approach is orthogonal to other data-efficient methods. The evaluation is on AIME24, MATH500, and GPQA Diamond, with additional pilot studies across nine languages.","tokens_in":41930,"tokens_out":8304,"duration_ms":75136,"significance":"The problem addressed, namely reducing both data annotation cost and test-time compute for long chain-of-thought reasoning, is timely and of broad interest, and the idea that multilingual data diversity could serve as a source of sample and token efficiency is worth investigating. However, the paper's central quantitative claims are not supported by its own tables: headline numbers in the abstract and introduction are inconsistent with the reported results, the matched-scale controls show essentially null multilingual gains, and the decoding-intervention evidence is selected on the test set. The paper does not ship code, data, machine-checked proofs, or parameter-free derivations, and the only detailed artifact is an appendix case study. If the claims could be established with proper controlled baselines and held-out validation, the contribution would be meaningful; in its current form, the evidence is not commensurate with the claims.","major_comments":[{"comment":"The abstract and introduction claim that only six samples improve long reasoning by 20% and that L2 reaches comparable 53% with 651 samples, but these values cannot be reconciled with Tables 1 and 2. In Table 1, L2−32B−Mo1^10_6 scores 23.3 AIME24, 87.4 MATH500, and 49.5 GPQA, while the base model Qwen2.5-32b* scores 26.7, 84.0, and 49.0; the claimed improvements of 16.6%, 18.2%, and 12% do not match these numbers, and AIME actually decreases. The '53%' figure does not correspond to any reported accuracy on AIME24, MATH500, or GPQA. Since the 'less data' claim rests on these numbers, this inconsistency is load-bearing.","section":"§4.3, Table 1; §1"},{"comment":"Table 2 shows L2−32b−Mo1^4_6 at 0.33 on AIME24 and L2−32b−Mo1^9_6 at 0.23, implying that increasing from four to nine languages hurts performance, yet Table 1 reports L2−32B−Mo1^10_6 at 23.3 on AIME24. This contradictory pattern is not discussed and undermines the paper's claim that more language families improve accuracy, as asserted around Figure 6. The discrepancy must be resolved because the language-scaling result is central to the method's motivation.","section":"§4.4, Table 2; §4.3, Table 1"},{"comment":"The matched-scale controls in Table 3 do not support the 'less data' claim. At 651 samples, L2−32b−MS1^4_651 gives 0.63 AIME24, 0.60 GPQA, and 0.93 MATH500, essentially identical to the single-language baseline Qwen2.5-32b-S1 651 (0.63, 0.56, 0.93). The headline comparison with s1-32B in Table 1 relies on published s1 numbers rather than an in-pipeline baseline, conflating evaluation differences with method differences. With only 30 AIME24 problems, a 3-point gap is one question, and no confidence intervals, seeds, or significance tests are reported.","section":"§4.6, Table 3"},{"comment":"The decoding intervention result of 73.3% on AIME24 is selected by tuning k∈{2,4} after evaluating on the test set itself, and α and β are free parameters adjusted on the same benchmark. This makes the token-efficiency and accuracy evidence circular; there is no held-out validation showing that the intervention generalizes. The paper should either pre-register these hyperparameters or evaluate the intervention on a separate benchmark.","section":"§4.8"},{"comment":"Footnote 3 reports that MATH500 numbers were manually corrected for 8–12 problems after format-based validation failed; without a complete correction log or the full list of corrected examples, the exact parity numbers (e.g., 93.0 vs 93.0 and 95.0 vs 93.0) cannot be reproduced. Additionally, Section 3.2 states that translation quality was 'not rigorously evaluated'; because the paper attributes the gains to cross-lingual diversity, the lack of any translation-quality validation leaves the proposed mechanism unsupported.","section":"§4.3 footnote 3; §3.2"},{"comment":"The abstract and introduction state that L2 is orthogonal to other data-efficient methods, but the experiments only apply L2 augmentation to data from s1 and Bespoke-Stratos; no experiment combines L2 with an existing data-selection or budget-forcing method, and Section 3.1 explicitly says the datasets are not combined. The orthogonality claim is therefore unsupported by the experiments as presented.","section":"§3.1; abstract; §4.6"}],"minor_comments":[{"comment":"There are formatting errors in the tables: Table 1 contains '1/93.0' in the MATH500 column, and Table 2 shows '0.490.85' without a separator.","section":"Table 1 and Table 2"},{"comment":"Figures 3 and 4 have identical captions ('Comparison of reasoning strategies: mixed Chinese-English reasoning...') despite displaying different content, which confuses the reader.","section":"Figures 3 and 4"},{"comment":"Section 4.2 mentions evaluation on Graduate Entrance Exam tasks, but no results for this dataset appear in Table 1 or elsewhere in the main text; please clarify or remove the reference.","section":"§4.2"},{"comment":"The paper introduces special language boundary tokens but never specifies how they are added to the vocabulary or initialized during fine-tuning; this detail is needed for reproducibility.","section":"§3.3"},{"comment":"No code, data release statement, random seeds, top-p sampling details, or hyperparameter ranges are provided, which hampers independent verification of the reported results.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The manuscript has potential topical appeal, but the internal numerical inconsistencies (e.g., the 16.6%/18.2%/12% claims vs Table 1, and the '53%' claim vs Table 1), the null matched-scale results in Table 3, and the test-set tuning of the decoding intervention are fundamental problems for the central claims. In my view these issues would require substantial new experiments and a reframing of the main claims rather than point fixes, which is why I recommend rejection rather than major revision. If the authors can provide controlled in-pipeline baselines, correct the internal contradictions, and validate the decoding intervention on held-out data, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth a look if you work on data-efficient reasoning SFT, but the headline claims should not be taken at face value. The core idea—multilingual/code-switched CoT augmentation plus a decoding intervention can reduce both training data and inference tokens—is plausible, and the pilot studies give it some credence. The construction of step-level code-switched data (translating selected reflection fragments and tagging language boundaries) is a genuinely new combination, as is applying it at the 6-sample extreme. I also credit the paper for including matched-scale controls in Table 3, which show multilingual augmentation helping modestly or not at all depending on the benchmark.\n\nThe soft spots are real and load-bearing. The introduction's numbers (20% improvement, 53% with 651 samples) don't line up with the tables; the 10-language 6-sample model in Table 1 (23.3 AIME) is worse than the 4-language variant in Table 2 (33.3), which is never explained. The main claim of being comparable to s1 with 651 samples uses published s1 numbers rather than a baseline run inside the same pipeline. The paper's own Table 3 matched controls show L2-MS1^4_651 vs the English-only S1_651 baseline: 63.3 vs 63.3 on AIME, 93 vs 93 on MATH500, and 60 vs 56 on GPQA. That is at most a 4-point gain on GPQA and a wash elsewhere; AIME has only 30 problems, so a 3.3-point gap is one question. No seeds, confidence intervals, or significance tests are reported. The MATH500 footnote admits manual post-hoc correction of 8-12 answers, which may be legitimate but is not reproducible from this paper. The decoding intervention result (73.3% on AIME24) is tuned on the test set via alpha, beta, and k, so it does not support the token-efficiency claim. Finally, translation quality is admitted to be unevaluated, leaving the proposed mechanism underdetermined.\n\nSo the qualitative observation—multilingual and code-switched CoT can help small-scale reasoning SFT—remains plausible and worth re-running. But the quantitative 'less data, less tokens' claim is not established, and the paper should not be cited as evidence in its current form. If I were an editor, I would still send it to peer review rather than desk reject, because the idea is useful and the flaws are fixable with released artifacts, controlled baselines, and honest reporting. Expect major revision.","headline":"Plausible idea, unreliable evidence: multilingual/code-switched CoT for data-efficient reasoning SFT is worth exploring, but the paper's numbers are internally inconsistent and the key comparisons are uncontrolled or fitted.","tokens_in":42469,"tokens_out":3024,"would_cite":false,"duration_ms":30721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that multilingual unification learning—training on whole-solution and step-wise mixed chains of thought in several languages—lets a reasoning LLM match or beat single-language baselines with fewer training samples and…","keywords":["multilingual reasoning","chain-of-thought","test-time scaling","data efficiency","token efficiency","code-switching","low-data fine-tuning","reasoning diversity"],"falsifier":"Keep the same 651 questions and the same training pipeline, but replace every translated chain of thought and every code-switched trace with a matched-length English paraphrase that preserves the underlying steps and contains no language alternation. If this English-control model reaches the same AIME24, MATH500, and GPQA scores with the same inference-token counts, the multilingual structure is not the cause; if it falls short, the multilingual data is doing the work.","tokens_in":1524,"feed_emoji":"🌐","tokens_out":1782,"duration_ms":81901,"temperature":0.7,"pith_summary":"The paper argues that the diversity of reasoning across languages is itself a training resource. It introduces L2, multilingual unification learning, which takes a handful of high-quality English reasoning traces, produces whole-solution chains of thought in additional languages, and mixes reflective steps across languages with language tags. Fine-tuning a 32-billion-parameter base model on such data, with as few as six source questions or 651 curated samples, matches or beats a 1,000-sample single-language recipe on AIME24, MATH500, and GPQA while emitting fewer reasoning tokens. If the central claim is right, data scarcity and inference cost at test time can both be attacked by choosing languages deliberately.","feed_headline":"Multilingual data cuts reasoning LLMs' data and token budgets","feed_subtitle":"A few high-quality questions in several languages match bigger single-language sets and use fewer inference tokens.","key_machinery":"The load-bearing mechanism is the multilingual unification corpus: for each source question, a solution-level chain of thought in the original language, a translated solution-level chain of thought, and a step-wise code-switched chain of thought in which reflective fragments (signaled by words like 'Wait' or 'Hmm') are randomly translated and enclosed in language tokens. Training on this mixture teaches the model to associate reflective reasoning with alternate languages, and the mixture is what converts language choice into a reusable source of reasoning diversity. A decoding intervention adjusts the logit values of language tokens at inference, allowing the trained model to be steered into a preferred reasoning language. The mechanism's work is to make language a controllable and data-generating dimension of long reasoning.","core_discovery":"The central claim is that multilingual chain-of-thought annotation is a data-efficient and token-efficient augmentation. Reasoning paths vary across languages, with different accuracy and token counts for the same question, and a model exposed to those varied paths learns the problem's structure more cheaply than a model given more English samples. Concretely, the multilingual unification model trained from 651 questions augmented with whole-solution CoTs in four languages plus step-wise code-switched traces scores 63.3 on AIME24, 93.0 on MATH500, and 60.0 on GPQA, while the single-language 1,000-sample baseline scores 56.0, 93.0, and 59.6. The paper also reports an inflection point near 30 curated questions and finds that training across more language families raises accuracy and lowers token use.","pith_inferences":["If the real driver is reasoning-trace diversity rather than language identity, then any way of diversifying traces—multi-prompt, multi-expert, or multi-format—should reproduce part of the effect; this is testable by replacing language switching with format switching.","Because translation quality was not rigorously checked, a portion of the gain may come from the regularizing noise of code-switched text rather than from semantically meaningful cross-lingual reasoning; the token-efficiency claim would need to survive a clean-translation control.","Tokenizer differences across languages could inflate or deflate the reported token savings; measuring reasoning length in characters or another language-neutral unit would separate semantic concision from tokenization artifacts.","The paper's orthogonality claim suggests multilingual augmentation could be stacked with other low-data selection strategies, and that combination is a natural next test."],"forward_implications":["With only six source questions rendered multilingually, long-reasoning performance rises by roughly 20% over the base model.","With 651 questions, the multilingual recipe matches or exceeds the 1,000-question single-language recipe on AIME24, MATH500, and GPQA while using fewer inference tokens.","Training across more language families yields higher accuracy and lower token usage than training within a single language family.","A decoding intervention can steer the model's reasoning into another language and reach higher AIME24 accuracy, making language a controllable inference-time variable.","The multilingual benefit shrinks as training data grows, so the method is most valuable in the low-data regime."],"supporting_citations":[{"why":"Supplies the 1,000-sample S1k pool and the single-language s1-32B baseline that the multilingual recipe must beat with fewer data and tokens.","marker":"(Muennighoff et al., 2025)"},{"why":"Provides the API-generated chain-of-thought annotation source used to build multilingual solutions, plus R1 as a top comparison model.","marker":"(DeepSeek-AI et al., 2025)"},{"why":"Provides the Bespoke-Stratos-17k pool from which a 500-question subset is drawn to test whether multilingual gains transfer across data sources.","marker":"(Labs, 2025)"},{"why":"Provides LIMO, the 817-sample low-data baseline that motivates the paper's less-data question and serves as a comparison point.","marker":"(Ye et al., 2025)"},{"why":"Source of the six official long-reasoning examples used as the minimal training seed, and of the o1 API baselines in the main table.","marker":"(OpenAI, 2024)"},{"why":"Defines the Qwen2.5-32B base model that all L2 variants are finetuned from, setting the pretraining context for multilingual reasoning.","marker":"(Qwen et al., 2025)"}],"fun_headline_variants":["Multilingual CoT: less data, fewer tokens","Multilingual CoT cuts both data and tokens","Fewer tokens, less data via multilingual CoT","One question in many languages beats many in one","Small multilingual data, big reasoning efficiency"],"cache_read_input_tokens":44544,"weakest_assumption_plain":"The load-bearing premise is that the translated chains of thought and step-wise code-switched mixtures faithfully encode genuine cross-lingual reasoning diversity; the paper itself notes that translation quality was not rigorously evaluated.","fun_headline_variants_meta":{"raw":{"variants":["Multilingual CoT: less data, fewer tokens","Multilingual CoT cuts both data and tokens","Fewer tokens, less data via multilingual CoT","One question in many languages beats many in one","Small multilingual data, big reasoning efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001071,"raw_usage":{"total_tokens":4472,"prompt_tokens":916,"completion_tokens":3556,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":3485}},"tokens_in":532,"tokens_out":3556,"duration_ms":23654,"temperature":1.0,"reasoning_tokens":3485,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:51:10.768361+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Keep the same 651 questions and the same training pipeline, but replace every translated chain of thought and every code-switched trace with a matched-length English paraphrase that preserves the underlying steps and contains no language alternation. If this English-control model reaches the same AIME24, MATH500, and GPQA scores with the same inference-token counts, the multilingual structure is not the cause; if it falls short, the multilingual data is doing the work.","supporting_citations":[],"review_version":1}