{"id":"4e22abf2-6969-4245-9c90-cb4df8e3bf97","arxiv_id":"2505.15508","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"MITT, a prefix-tuning method for multilingual test-time scaling, is evaluated on questions whose English reasoning was used for training, confounding the reported gains.","lead":"This paper studies how test-time scaling, generating longer reasoning traces, behaves across six languages, and proposes a lightweight fine-tuning trick to improve it. The proposed method trains on English reasoning prefixes from the same test questions, so the reported gains may come from memorization rather than transfer.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MITT is evaluated on the same AIME 2025 questions used to collect its English prefix training data; the reported gains may be question-level memorization rather than transferable inductive priors.","rationale":"The most load-bearing concern is the train/test overlap in the MITT evaluation: the model is fine-tuned on English reasoning prefixes extracted from the same AIME 2025 questions that are later used for evaluation in all six languages. This makes the headline result equally explainable by question-level memorization as by the claimed transferable inductive priors. The paper's own framing in Sections 1 and 4.5 says MITT 'transfers structured inductive priors across languages' and 'uses any high-resource language's reasoning priors,' which is a generalization claim; evaluating on the training questions cannot support it. A held-out split or an external benchmark is the minimal condition for the claim to stand. The reader's formal weakest_assumption — cross-lingual comparability of the wait-prompt and answer-extraction protocol — is a real secondary issue, but it is not the primary threat: even if the protocol were perfectly consistent, the within-distribution evaluation would still not establish transfer. I agree with the reader's overall REJECT verdict, but the specific weakest assumption I would emphasize is the missing held-out evaluation, which the reader mentions in the rationale though not in the weakest_assumption field. This critique is about the experimental design, not author intent; the paper may describe the procedure accurately, but the procedure as reported cannot validate the central claim.","tokens_in":14506,"tokens_out":3846,"duration_ms":35538,"concrete_test":"Run MITT with a held-out split: fine-tune E-3 on English prefixes from 15 of the 30 AIME 2025 questions (or, better, on a separate benchmark such as AIME 2024), then evaluate Base vs E-3 vs H-1 on the remaining held-out questions (or the separate benchmark) in all six languages under the identical wait-prompt and answer-extraction protocol at the same token limits (2000–8000). If the accuracy gains in Table 1 largely disappear on held-out questions, the reported boost is question memorization rather than transfer; if the gains persist, the leakage concern is resolved. Report per-language accuracy with the 30-question set; ideally also compute bootstrap confidence intervals.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim — that English prefix tuning provides broadly transferable inductive priors improving test-time scaling across languages (abstract; Section 4.5) — is not supported by the reported experiment because of train/test overlap. In Section 3.6, MITT is trained on 'all 100 initial reasoning prefixes (i.e., the first 32 tokens) sampled per question' from the same AIME 2025 questions used for evaluation, and the E-3 and H-1 rows of Table 1 are measured on those same questions in six languages. Since the model is fine-tuned on model-generated reasoning prefixes for the exact problems it is later asked to solve, any improvement could come from fitting question-specific reasoning starts, not from a language- or reasoning-general prior. This is especially acute for H-1, which pools prefixes from English, Italian, German, and Portuguese for the same questions. The paper provides no held-out evaluation, so the load-bearing inference from Table 1 to a transferable mechanism is unestablished. This is distinct from the protocol concerns in Section 3.2; even a perfectly consistent evaluation protocol cannot fix the train/test overlap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a multilingual study of test-time scaling on AIME 2025, translated into six Latin-script languages, using DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B. It reports that test-time scaling gains are uneven across languages and models, that models often switch to English mid-reasoning under monolingual prompts, and that low-resource languages have less similar and less consistent initial reasoning prefixes relative to English. The paper then proposes MITT, an unsupervised prefix-tuning method that fine-tunes DeepSeek-R1-Distill-Qwen-7B with LoRA on the first 32 tokens of reasoning prefixes sampled in English (E-3) or in four high-resource languages (H-1), and claims that this improves accuracy and test-time scaling across all six languages, especially low-resource ones. The central contribution, however, is evaluated on the same 30 questions used to collect the training prefixes, with no held-out split, no error bars, and an underspecified inference-time answer-detection protocol.","tokens_in":14709,"tokens_out":7203,"duration_ms":55396,"significance":"The descriptive findings (uneven scaling, language switching, prefix dissimilarity and consistency) are a useful contribution to a largely English-centric literature, and the paper extends AIME 2025 with Vietnamese and Tagalog translations. The two-method similarity analysis (multilingual embeddings and translation-plus-English-embeddings) is a reasonable attempt to control for embedding bias. That said, the paper's headline method, MITT, is not supported by the reported experiment: the training prefixes are drawn from the same questions used for evaluation, and accuracy differences are not quantified with uncertainty. The transferability claim in the abstract and Section 4.5 therefore requires a proper held-out evaluation before the method can be assessed. The paper does not ship machine-checked proofs or reproducible code in a verifiable form; the GitHub link is a placeholder.","major_comments":[{"comment":"MITT is fine-tuned on initial reasoning prefixes sampled from the same AIME 2025 questions that are used for evaluation. Section 3.6 states that \"we extract all 100 initial reasoning prefixes (i.e., the first 32 tokens) sampled per question,\" and Section 4.5 describes training on \"each of 100 English reasoning prefixes for each question for both datasets,\" while Table 1 reports accuracy on those same questions in six languages. No held-out split is described anywhere in the paper, including Section 7, which lists limitations but does not mention this overlap. Improvements in Table 1 could therefore be explained by memorization of question-specific reasoning starts rather than by transferable inductive priors. The claim in Section 4.5 that \"English prefix tuning provides broadly transferable inductive priors for reasoning across the multilingual spectrum\" is not supported by the reported experiment. The authors should retrain on a disjoint set of questions (or a disjoint reasoning dataset) and evaluate on the same test questions, and also report an ablation that trains on prefixes from one set of questions and evaluates on another.","section":"§3.6, §4.5, Table 1"},{"comment":"The test-time scaling protocol is underspecified at a load-bearing point. Section 3.2 says a language-specific wait prompt is inserted \"whenever M tries to produce an answer on its own,\" but the manuscript never states how this event is detected during inference. Appendix B describes how wait prompts were placed manually in the 1-shot demonstration, and the main text does not say whether the same mechanism is used for all six languages and for all strategies (Base, E-3, H-1). If the trigger or the placement of wait prompts differs across languages or runs, the cross-lingual scaling curves in Figure 2 and the MITT comparisons in Table 1 are not comparable. The authors should specify the automatic detection rule (for example, detecting an answer-like pattern or a stop token), and verify that it is applied identically across conditions.","section":"§3.2, Appendix B"},{"comment":"The reported accuracies are based on 30 questions per language (two AIME subsets of 15) and are presented without error bars, raw counts, or significance tests. Several differences that support the headline claim are differences of one to three questions; for example, English Base at 2000 tokens is 0.164 and E-3 is 0.257, and Tagalog Base at 2000 is 0.043 versus E-3 at 0.077. Conversely, the claim of a \"clear upward trajectory\" is not uniform across Table 1: German H-1 drops to 0.000 at 6000 and 8000 tokens, and English H-1 at 2000 and 4000 is below E-3 and sometimes below Base. Bootstrap confidence intervals and per-question results are needed before \"significantly boosts\" (abstract) can be assessed.","section":"Table 1, Figure 1"}],"minor_comments":[{"comment":"Several numeric entries in Table 1 are concatenated (e.g., \"0.2540.251\", \"0.3430.271\", \"0.2390.235\", \"0.1760.349\", \"0.0780.1080.1930.147\", \"0.2200.1930.237 0.209\"), making the table unreliable as printed.","section":"Table 1"},{"comment":"The model name is inconsistently capitalized as \"DeepSeek-R1-Distill-LLama-8B\" rather than \"Llama\" in several places, and Section 4.1 contains a typo \"Disill\" for \"Distill\".","section":"§3.1, Figure 2"},{"comment":"The caption is incomplete: it ends with a colon followed by a period, with no sentence following the colon.","section":"Figure 2 caption"},{"comment":"The GitHub repository link is a placeholder (\"Link\"), so the claimed release of dataset and code cannot be verified.","section":"Introduction"},{"comment":"Appendix B says wait prompts are manually placed in the demonstration; the main text should clarify whether the same prompts are also injected automatically during evaluation, and if so, how the trigger is detected.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The train/test overlap is the single most important issue: the current Table 1 cannot be used to support the abstract's claim that MITT significantly boosts reasoning performance. A held-out evaluation is feasible within the current pipeline and should be required. The descriptive parts (Sections 4.1-4.4) are more defensible, though they would also benefit from error bars. If the authors redo the MITT evaluation properly, the manuscript could become suitable; as is, the main contribution is unverified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper's descriptive study is genuinely new, but the main method claim (MITT) is built on a train/test overlap. Section 3.6 fine-tunes on 100 English reasoning prefixes per question from the same AIME 2025 questions used in Table 1 evaluation, in all six languages. So any improvement can be explained by memorizing question-specific reasoning starts, not by a transferable inductive prior. The abstract's claim about 'broadly transferable inductive priors' is not supported.\n\nWhat the paper does well: it is the first systematic look at test-time scaling in strictly monolingual multilingual settings, and the observations that gains are uneven across languages, that models (especially Llama-8B) switch to English mid-reasoning, and that low-resource languages show more variable initial reasoning are interesting and plausibly useful. The Vietnamese/Tagalog AIME translations are a small new resource, though they are GPT-4o translations without reported human verification, which is a minor concern.\n\nThe soft spots are real. The leakage is load-bearing. The accuracy numbers come from 30 questions per language, no error bars or significance tests, and the differences in Table 1 are often one to three questions, so even the descriptive 'uneven scaling' needs a re-analysis. The wait-prompt insertion protocol is manually placed and the detection of an attempted answer is unspecified; if that varies across languages, the scaling curves in Figure 2 are not directly comparable. The limitations section is honest about Latin-script and model coverage, but it does not acknowledge the training/evaluation overlap, which is the bigger issue.\n\nWho this is for: someone working on multilingual reasoning or test-time scaling who wants a map of open questions. The descriptive findings could stand after a proper evaluation, but the MITT contribution should be treated as unverified until held-out evaluation exists.\n\nRecommendation: this deserves referee time, not a desk reject, but my referee report would recommend rejection in current form, with a request for a held-out evaluation set, multiple seeds with error bars, and a fully specified inference protocol. If the descriptive findings survive that re-analysis, they would be a solid contribution on their own.","headline":"The descriptive multilingual test-time scaling study is new, but MITT's headline gains are uninterpretable due to train/eval overlap on the same 30 questions.","tokens_in":15260,"tokens_out":2290,"would_cite":false,"duration_ms":20297,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DeepSeek-R1-Distill-Qwen-7B's multilingual test-time scaling is restored by fine-tuning on English-only initial reasoning prefixes.","keywords":["test-time scaling","multilingual reasoning","initial thought transfer","prefix tuning","low-resource languages","cross-lingual transfer","chain-of-thought","DeepSeek-R1"],"falsifier":"Rerun the DeepSeek-R1-Distill-Qwen-7B evaluation with wait prompts removed entirely or held constant in English for every language, and check whether low-resource scaling curves still stay flat; if the gap disappears, the reported language disparity comes from the prompt mechanics rather than from model reasoning. A second decisive comparison would fine-tune the model on low-resource-language prefixes of equal length and show whether MITT's English prefixes beat them.","tokens_in":14273,"feed_emoji":"🧠","tokens_out":10369,"duration_ms":80449,"temperature":0.7,"pith_summary":"Test-time scaling does not benefit all languages equally, and this paper argues the gap has a concrete cause in the earliest reasoning thoughts. The paper evaluates DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B on challenging mathematics in English, Italian, German, Portuguese, Vietnamese, and Tagalog, finding that high-resource languages show clearer accuracy gains as generation length grows, while low-resource languages, especially for Qwen-7B, scale erratically or not at all. It also finds that models switch to English mid-reasoning under strict monolingual prompts, and that low-resource initial reasoning prefixes are less similar to English and less consistent across repeated generations. Building on that diagnosis, the paper proposes MITT, an unsupervised prefix-tuning method that fine-tunes on English or high-resource-language reasoning prefixes and restores steady test-time scaling across languages. If correct, English-only, unlabeled reasoning prefixes can act as transferable inductive priors, improving low-resource reasoning without target-language training data.","feed_headline":"A 32-token English prefix tune lifts multilingual reasoning","feed_subtitle":"Unsupervised English-only fine-tuning restores test-time scaling gains in low-resource languages.","key_machinery":"The carrying mechanism is the reasoning prefix: the first 32 tokens of a chain of thought, treated as a portable inductive prior. MITT collects 100 such prefixes per question in English, or in a pooled set of high-resource languages, and fine-tunes DeepSeek-R1-Distill-Qwen-7B with 4-bit quantized LoRA adapters on the query and value projections. The evaluation protocol that makes the claim visible appends an answer-extraction prompt every 32 generated tokens and inserts a language-specific wait prompt whenever the model attempts to answer on its own, so accuracy can be plotted against reasoning length. The paper's key observation is that after the English-prefix tune, this accuracy-versus-length curve turns from flat to clearly increasing for low-resource languages, which supports the transfer claim.","core_discovery":"The central discovery is that multilingual test-time scaling is deeply uneven and that the unevenness tracks the stability and similarity of the model's earliest reasoning. Test-time scaling gains are robust for high-resource languages but weak or absent for low-resource ones in DeepSeek-R1-Distill-Qwen-7B, while DeepSeek-R1-Distill-Llama-8B scales more steadily across languages. Low-resource languages produce initial reasoning prefixes, the first 32 tokens of a chain of thought, that diverge sharply from English and are more variable across repeated generations, which the paper interprets as less stable inductive reasoning priors. The proposed intervention, Multilingual Initial Thought Transfer (MITT), fine-tunes the model with LoRA on English-only reasoning prefixes and reports that this single unsupervised English tune raises accuracy at every tested token budget and turns a flat accuracy-versus-length curve into a clearly increasing one across high- and low-resource languages.","pith_inferences":["An untested extension is whether the same English-prefix transfer works for non-Latin scripts such as Arabic, Hindi, or Chinese; the paper only studies Latin-script languages, so the mechanism could be tied to shared tokenization or Roman-script similarity.","A sharper causal test would compare English-prefix tuning with prefix tuning on the same number of low-resource-language prefixes; if low-resource prefixes work equally well, the real bottleneck is consistency of early reasoning, not the English language itself.","If the effect is about structuring the start of a chain of thought rather than about English content, then even non-linguistic structured prefixes, such as a fixed planning template, might reproduce part of the gain—this is my inference, not the paper's claim.","The periodic answer-extraction and wait-prompt protocol itself may interact with the fine-tune; a practitioner should check whether MITT's gain persists when wait prompts are omitted before attributing it purely to reasoning quality."],"forward_implications":["If English prefix tuning provides transferable reasoning priors, an unsupervised English-only fine-tune can substitute for supervised multilingual reasoning data when improving low-resource test-time scaling.","Test-time scaling evaluations should report per-language curves, because language-averaged curves can hide the flat or erratic scaling of low-resource languages.","Mid-reasoning switches to English under monolingual prompts mean language fidelity is a separate failure mode that multilingual reasoning benchmarks should track alongside accuracy.","Models with code-heavy, English-skewed pretraining mixes may need targeted prefix adaptation before test-time scaling can help non-English languages.","The first-32-token prefix consistency of a language could serve as a cheap diagnostic for whether a model will benefit from longer reasoning in that language."],"supporting_citations":[{"why":"Supplies the AIME 2025 multilingual mathematics questions that the paper translates into Vietnamese and Tagalog to extend the benchmark.","marker":"(AIM, 2025)"},{"why":"Provides the two evaluated DeepSeek-R1-Distill models and defines the test-time scaling phenomenon under study.","marker":"(DeepSeek-AI et al., 2025)"},{"why":"Supplies the forced-long-reasoning wait prompt technique the paper adapts for its scaling evaluation.","marker":"(Muennighoff et al., 2025)"},{"why":"Motivates MITT with the finding that early reasoning tokens shape long-horizon reasoning performance.","marker":"(Ji et al., 2025a)"},{"why":"Provides the multilingual embedding model used to measure cross-lingual similarity of initial reasoning segments.","marker":"(Reimers and Gurevych, 2020)"},{"why":"Provides the langid classifier used to label reasoning windows and detect English switching.","marker":"(Lui and Baldwin, 2011)"},{"why":"Establishes the limited cross-lingual transferability of English chain-of-thought prompts that MITT works against.","marker":"(Qin et al., 2023)"},{"why":"Provides GPT-4o, the zero-shot translator used to create the Vietnamese and Tagalog dataset and demonstration.","marker":"(OpenAI et al., 2024a)"}],"fun_headline_variants":["English prefix tune lifts low-resource reasoning","MITT: one English prefix boosts all languages","32-token English prefix fixes multilingual gaps","Multilingual reasoning fixed by English prefix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that inserting an answer-extraction prompt every 32 tokens and manually placed language-specific wait prompts measures reasoning quality fairly and equivalently in all six languages; if these prompts help or hinder different languages differently, the cross-lingual scaling curves are not comparable and the 'uneven scaling' finding collapses.","fun_headline_variants_meta":{"raw":{"variants":["English prefix tune lifts low-resource reasoning","MITT: one English prefix boosts all languages","32-token English prefix fixes multilingual gaps","Multilingual reasoning fixed by English prefix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000367,"raw_usage":{"total_tokens":1967,"prompt_tokens":936,"completion_tokens":1031,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":976}},"tokens_in":552,"tokens_out":1031,"duration_ms":7324,"temperature":1.0,"reasoning_tokens":976,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:16:19.711383+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the DeepSeek-R1-Distill-Qwen-7B evaluation with wait prompts removed entirely or held constant in English for every language, and check whether low-resource scaling curves still stay flat; if the gap disappears, the reported language disparity comes from the prompt mechanics rather than from model reasoning. A second decisive comparison would fine-tune the model on low-resource-language prefixes of equal length and show whether MITT's English prefixes beat them.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the langid classifier used to label reasoning windows and detect English switching."}],"review_version":1}