{"id":"edb06c4a-c3de-4417-ab7c-5fafc18d299e","arxiv_id":"2608.05802","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"OPD^2 improves multilingual math reasoning over standard distillation, narrows the English-Korean gap, but English-only training shifts output language to English.","lead":"This paper applies a newer AI training method, On-Policy Delta Distillation (OPD^2), to math reasoning in English, Korean, and Japanese. It finds OPD^2 beats the earlier method, and that English-only training can boost Korean and Japanese scores while causing models to answer in English.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified; main claims are internally consistent and supported by reported tables","rationale":"The reader's weakest_assumption focuses on the omitted OPD2 method details and the unspecified language detector. I partially agree: these are the most plausible verification gaps. However, I do not elevate them to a load-bearing concern because the paper explicitly states that all experiments used the official OPD2 implementation and the training recipe from [4], and the language-shift finding has a clear quantitative demonstration in Table 1 with a consistent direction (English-only training reduces target-language rates by large margins in both modes and both languages). The internally reported tables are consistent with the abstract and Section 3 claims: for Qwen3-1.7B non-thinking English/Korean/Japanese, multilingual OPD2 values (65.7, 56.5, 65.4) all exceed the corresponding OPD values (61.7, 48.7, 62.9), and the same ordering holds for Qwen3-8B (75.3 vs 74.0; 72.2 vs 67.1; 75.2 vs 73.4). The English-only transfer results in Appendix A.2 match the main text's statement that English-only OPD2 improves Korean and Japanese non-thinking scores to 59.3 and 66.5, comparable to multilingual OPD2 (56.5 and 65.4). The language-shift claim is internally consistent because accuracy gains on translated benchmarks can reflect English-composed reasoning that is then translated or partially mixed, and the paper explicitly measures response language to separate these. The only genuine limitation is the lack of variance/error bars and the deferral of OPD2 details to [4]; these are external reproducibility concerns. Since the task asks whether the central argument has a load-bearing soft spot, I conclude the argument holds up under scrutiny, with the caveat that independent verification of the official implementation remains necessary, which is already reflected in the CONDITIONAL verdict. Therefore, no verdict change is needed.","tokens_in":9027,"tokens_out":1759,"duration_ms":14228,"concrete_test":"Independently run the multilingual OPD and OPD2 training recipes from the linked repository (https://github.com/naver-ai/opd2) on Qwen3-1.7B with a fixed seed, compute the English/Korean/Japanese PolyMath+Global-MGSM non-thinking averages over at least three seeds, and compare with Figures 1 and Tables 2–4; also independently measure target-language response rates on PolyMath outputs with a second language detector to confirm Table 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that OPD2 consistently outperforms OPD in multilingual settings—is supported by the reported aggregate tables (Tables 2–10) across two model sizes, three languages, and multiple benchmarks. The claim about English-only OPD2 improving Korean/Japanese accuracy while shifting response language is also internally consistent: Appendix A.2 shows large non-thinking Korean/Japanese gains (e.g., 59.3 and 66.5) that are comparable to or above the multilingual OPD2 numbers, and Table 1 quantifies the drop in target-language response rates (e.g., Korean 90.5%→48.3%, Japanese 90.9%→29.6% in non-thinking mode). The main residual risk is not an internal inconsistency but an external one: the OPD2 implementation, including reward centering and the conditioning criterion, is referenced to a prior paper [4] and the linked repository rather than specified in this manuscript. If the released implementation deviates from the described reward (log π* − log π*_base), the comparison between OPD and OPD2 could be confounded. However, the paper explicitly states it uses the official OPD2 implementation and follows its recipe, and the reader's concern about the unspecified language detector is also an external verifiability issue rather than a flaw in the reported argument. These are verification gaps, not identified errors, so I do not find a load-bearing internal concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper experimentally compares On-Policy Distillation (OPD) and its delta variant OPD^2 for multilingual mathematical reasoning. Using Qwen3-1.7B and Qwen3-8B as students, a Qwen3-30B-A3B-2507 teacher, 100K multilingual (English/Korean/Japanese 1:1:1) and English-only training sets, and evaluations on PolyMath, Global-MGSM, HRM8K, and MAWPS, the authors report that OPD^2 outperforms OPD on average in all three languages and in both thinking and non-thinking modes, narrows the English–Korean gap on most benchmarks, and that English-only OPD^2 transfers accuracy to Korean and Japanese while reducing target-language response rates. The paper positions OPD^2 as a drop-in post-training upgrade and argues that benchmark gains from English-only training do not imply that the model will respond in the user's language.","tokens_in":9309,"tokens_out":7852,"duration_ms":77897,"significance":"If the empirical claims hold, the paper is a useful extension of OPD research to East Asian languages and a clear demonstration that cross-lingual accuracy transfer and target-language preservation are separable. The explicit research questions, the multilingual data construction, the English–Korean gap analysis, and the response-language metric are valuable, and the linked official OPD^2 implementation is a reproducibility asset. The reported tables are internally consistent with the central claims. However, the absence of variance reporting or significance testing, together with incomplete specification of the OPD^2 objective and of the language-detection procedure, means the paper currently supports directional findings rather than a strong quantitative conclusion.","major_comments":[{"comment":"The claim that OPD^2 'consistently outperforms' OPD is based on single runs with no seeds, confidence intervals, or significance tests reported. Several advantages are small, such as the Qwen3-8B English non-thinking PolyMath+Global-MGSM average of 70.5 vs. 69.8 in Figure 1(b), and Table 8 shows a per-benchmark exception (GMGSM-EN: 98.4 for OPD vs. 98.0 for OPD^2). Without repeated runs or a statistical procedure, the central quantitative claim is not established. Please add variance information or explicitly reframe the result as point estimates.","section":"Section 3.2, Figure 1, Tables 2–10"},{"comment":"The OPD^2 reward in Eq. (4) is not the complete training objective: reward centering and the conditioning criterion are mentioned but deferred to the authors' prior paper [4] ('we omit their detailed formulations due to space constraints'). Since the paper's main comparison is OPD vs. OPD^2, the manuscript should include the full objective or an appendix with the exact formulas for both components. Without this, the comparison is not self-contained and any deviation between the intended method and the released implementation cannot be detected by a reader.","section":"Section 2.1, Eq. (4)"},{"comment":"The target-language response rate is load-bearing for the language-shift claim, but its measurement is underspecified. The paper does not state what language-identification method or threshold is used, how code-switched responses are classified, or how the after-</think> segmentation is performed. Because Table 1 reports large drops (e.g., non-thinking Japanese from 90.9% to 29.6%), the detector's accuracy is material to the conclusion. Please specify the metric, the detector, and its treatment of mixed-language outputs.","section":"Section 3.4, Table 1"}],"minor_comments":[{"comment":"Several tables contain malformed numeric entries with missing spaces or collapsed columns, e.g., Table 5 base row 'Qwen3-1.7B36.3 93.7 90.592.652.8 55.371.670.4' and Table 8 OPD row '41.298.493.3 95.2 63.9 67.258.974.0'. Please repair the formatting so that every value is readable.","section":"Appendix A, Tables 2, 4, 5, 6, 8, 9, 10"},{"comment":"The paper does not state whether any decontamination or overlap check was performed between the Nemotron training samples and the PolyMath, Global-MGSM, HRM8K, and MAWPS evaluation sets. Even though the OPD vs. OPD^2 comparison would be less affected, the absolute transfer numbers in Section 3.4 could be inflated by overlap; please report the overlap analysis or discuss the risk.","section":"Section 3.1"},{"comment":"Decoding and evaluation details are missing: the paper does not specify greedy vs. sampled decoding, temperature, number of samples, or the answer-extraction procedure for thinking-mode outputs. Please provide these details for reproducibility.","section":"Section 3.1"},{"comment":"The notation D_KL(πθ(y|x)‖π*(y|x)) may confuse readers because Eq. (2) defines the reward as log π* − log πθ, which is the negative of the KL gradient. A one-sentence explanation connecting Eq. (1), Eq. (2), and Eq. (3) would improve readability.","section":"Section 2, Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Candid take on arXiv:2608.05802. The paper does one thing and does it adequately: it runs the authors' own OPD^2 recipe on multilingual math reasoning (English/Korean/Japanese) with Qwen3-1.7B and 8B, and reports that OPD^2 beats OPD across languages, narrows the English-Korean gap in most benchmarks, and—the genuinely new finding—that English-only OPD^2 transfers accuracy to Korean/Japanese while shifting output language toward English.\n\nThe empirical pattern supports the main claims: OPD^2 numbers are higher than OPD in essentially every table, and the language-shift numbers in Table 1 are striking (Korean non-thinking response rate drops from 90.5% to 48.3% with English-only training). That dissociation between reasoning accuracy and target-language preservation is the real contribution—it tells practitioners that accuracy gains from English-only data do not imply the model will answer in the user's language.\n\nSoft spots are real but mostly about verifiability, not logic. There are no seeds, no error bars, no significance tests. Differences like the 0.7-point English gain on Qwen3-8B are within the range where a single run is thin evidence. The OPD^2 method details—reward centering and conditioning criterion—are not in the paper, only cited to the prior arXiv paper and the linked code. That is acceptable if the code is sound, but it makes the comparison hard to audit independently. The language detector is never specified, and for code-switched reasoning traces that matters. The English-only ablation is only on Qwen3-1.7B, so we don't know whether the language-shift pattern persists at 8B. None of this is fatal; the paper is transparent about what it measured.\n\nThe citation pattern is fine. Self-citation to OPD^2 is natural for an extension, and the African-languages self-distillation work is correctly positioned as a different setting.\n\nWho is this for? People doing multilingual post-training of reasoning models, and anyone who needs to know whether 'transfer' means 'accuracy' or 'language of expression.' It deserves a serious referee—I'd send it out, asking for variance estimates, a description of the language identification method, and ideally the 8B English-only ablation. Without those, the reported numbers are pointers rather than proof.\n\nI'd cite it if I were working on multilingual distillation. Bring to reading group? Maybe.","headline":"A solid empirical extension of OPD^2 to Korean/Japanese math reasoning, with the language-shift dissociation as the genuinely new finding; needs error bars and specification details before the quantitative claims are fully load-bearing.","tokens_in":9797,"tokens_out":2756,"would_cite":true,"duration_ms":24347,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Delta distillation consistently beats plain on-policy distillation for math reasoning in English, Korean, and Japanese—and English-only training transfers accuracy but not the target language.","keywords":["on-policy distillation","delta distillation","multilingual mathematical reasoning","cross-lingual transfer","target-language preservation","post-training","English-Korean gap"],"falsifier":"Run the released $OPD^{2}$ code on Qwen3-1.7B with the same multilingual dataset and measure PolyMath Japanese accuracy and the Japanese response rate in non-thinking mode; the paper predicts 52.0 average accuracy and 90.9 percent Japanese responses. If the code reproduces OPD's numbers rather than $OPD^{2}$'s, or if the response language detector labels English-heavy outputs as Japanese, the claimed advantage and the language-shift finding would not hold.","tokens_in":8872,"feed_emoji":"🧮","tokens_out":6757,"duration_ms":60220,"temperature":0.7,"pith_summary":"This paper asks whether on-policy distillation, a token-level alternative to reinforcement learning for post-training large language models, still works when the task is mathematical reasoning in English, Korean, and Japanese. It claims that a variant called On-Policy $\\Delta$ Distillation ($OPD^{2}$), which uses the probability gap between a post-trained teacher and its own base model as the training signal, consistently beats the original on-policy distillation in all three languages. The paper also claims that multilingual $OPD^{2}$ generally narrows the English–Korean accuracy gap. A third claim is that English-only $OPD^{2}$ can raise Korean and Japanese benchmark scores to nearly the level of multilingual training, but the model then often answers in English instead of the question language. That last finding matters because it separates transfer of reasoning ability from preservation of the target language in evaluation.","feed_headline":"Delta distillation beats plain distillation in 3 languages","feed_subtitle":"Using teacher-vs-base probability gaps outperforms plain distillation and narrows the English–Korean gap.","key_machinery":"The central object is the delta signal: for each sampled token, $OPD^{2}$ replaces the student log-probability in the ordinary OPD reward with the teacher's base-model log-probability, giving reward $R_t = \\log \\pi^*(y_t \\mid x, y_{<t}) - \\log \\pi^*_{\\mathrm{base}}(y_t \\mid x, y_{<t})$. This isolates capabilities acquired during post-training, particularly reasoning, from stylistic tendencies already present in the base model. Two auxiliary components, reward centering to obtain an advantage and a conditioning criterion based on the original OPD signal, are used in all experiments to fix the delta signal's convergence point; the paper cites its companion work for their formulas. The mechanism's work is to convert teacher rollout feedback into token-level rewards that can be applied through the on-policy update in the paper's Eq. 3.","core_discovery":"Using Qwen3-1.7B and Qwen3-8B as students with Qwen3-30B-A3B-2507 as teacher, and training on 100K questions in a 1:1:1 English/Korean/Japanese mix, the paper finds that $OPD^{2}$ outperforms OPD in every language and setting tested. For Qwen3-1.7B non-thinking average on PolyMath plus Global-MGSM, $OPD^{2}$ reaches 63.6 in English, 51.9 in Korean, and 52.0 in Japanese, versus 61.5, 48.8, and 48.0 for OPD. In thinking mode, $OPD^{2}$ cuts the English–Korean gap on six of seven benchmarks while leaving the seventh unchanged. The paper's additional claim is that English-only $OPD^{2}$ transfers accuracy to Korean and Japanese, averaging 52.6 and 53.4 non-thinking, close to multilingual $OPD^{2}$'s 51.9 and 52.0, but the target-language response rate drops sharply—for example, Korean from 90.5 percent to 48.3 percent and Japanese from 90.9 percent to 29.6 percent in non-thinking mode. Therefore benchmark accuracy alone overstates cross-lingual capability when the response language is not checked.","pith_inferences":["A testable extension is to apply English-only OPD^2 to other language pairs and measure both accuracy and target-language rate; the paper's numbers predict large accuracy transfer but small language-fidelity transfer.","Because the delta signal uses the base model as a stylistic baseline, the English-only transfer should depend on how much reasoning versus style the base model already encodes in the target language.","Since the paper omits the centering and conditioning formulas, a reimplementation from the text alone may not reproduce the gains; the linked code, not the paper, is the actual specification of OPD^2.","For deployment, adding a response-language filter or an explicit language instruction could close the gap exposed by the language-shift result."],"forward_implications":["OPD^2 can be used as a direct replacement for OPD in multilingual reasoning post-training, giving consistent gains across model sizes and languages.","Multilingual OPD^2 narrows the English–Korean gap on most benchmarks without sacrificing English accuracy, so cross-lingual disparity is partly a post-training problem.","English-only delta distillation transfers reasoning accuracy to Korean and Japanese, which means reasoning skills learned in English are not language-bound.","A model can score near-multilingual accuracy while answering in English, so evaluations that only report accuracy will miss this failure mode.","Multilingual training data remains necessary when the goal is to answer in the user's language, even if English-only training matches accuracy."],"supporting_citations":[{"why":"Defines the OPD^2 delta-signal reward and supplies the reward-centering and conditioning-criterion components used in all experiments.","marker":"[4]"},{"why":"Provides the Qwen3 model family used as students and teacher.","marker":"[1]"},{"why":"Supplies the Korean and Japanese training questions for the multilingual dataset.","marker":"[9]"},{"why":"Supplies the English training questions for both multilingual and English-only sets.","marker":"[10]"},{"why":"Provides the PolyMath benchmark used to evaluate multilingual math reasoning.","marker":"[11]"},{"why":"Supplies the Global-MGSM benchmark used for English, Korean, and Japanese evaluation.","marker":"[12]"},{"why":"Supplies the HRM8K Korean benchmark and its five subsets tracked in the gap analysis.","marker":"[14]"},{"why":"Supplies the MAWPS Japanese benchmark used in the Japanese evaluation.","marker":"[15]"}],"fun_headline_variants":["Delta distillation outshines plain in multilingual math","Probability gap distillation boosts Korean and Japanese math","English-only distillation shifts answers toward English","Delta distillation narrows English-Korean math gap","Probability gap beats teacher signal for multilingual math"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that the linked implementation actually performs the $OPD^{2}$ update described in the companion paper, and that the automatic language check correctly identifies whether answers are in Korean or Japanese.","fun_headline_variants_meta":{"raw":{"variants":["Delta distillation outshines plain in multilingual math","Probability gap distillation boosts Korean and Japanese math","English-only distillation shifts answers toward English","Delta distillation narrows English-Korean math gap","Probability gap beats teacher signal for multilingual math"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001087,"raw_usage":{"total_tokens":4551,"prompt_tokens":963,"completion_tokens":3588,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":3535}},"tokens_in":579,"tokens_out":3588,"duration_ms":24427,"temperature":1.0,"reasoning_tokens":3535,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T23:12:13.056197+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released $OPD^{2}$ code on Qwen3-1.7B with the same multilingual dataset and measure PolyMath Japanese accuracy and the Japanese response rate in non-thinking mode; the paper predicts 52.0 average accuracy and 90.9 percent Japanese responses. If the code reproduces OPD's numbers rather than $OPD^{2}$'s, or if the response language detector labels English-heavy outputs as Japanese, the claimed advantage and the language-shift finding would not hold.","supporting_citations":[{"cited_title":"Nemotron-sft-multilingual-v2","cited_arxiv_id":null,"evidence_quote":"Supplies the Korean and Japanese training questions for the multilingual dataset."},{"cited_title":"Polymath: Evaluating mathematical reasoning in multilingual contexts.Advances in Neural Information Processing Systems, 38, 2026","cited_arxiv_id":null,"evidence_quote":"Provides the PolyMath benchmark used to evaluate multilingual math reasoning."},{"cited_title":"Understand, solve and translate: Bridging the multilingual mathematical reasoning gap","cited_arxiv_id":null,"evidence_quote":"Supplies the HRM8K Korean benchmark and its five subsets tracked in the gap analysis."},{"cited_title":"Verification of chain-of-thought prompting in japanese","cited_arxiv_id":null,"evidence_quote":"Supplies the MAWPS Japanese benchmark used in the Japanese evaluation."}],"review_version":1}