{"id":"ed22f593-9575-460e-8290-c55c2ac60714","arxiv_id":"2608.08503","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":6,"one_line_summary":"In matched Bangla math training, chain-of-thought supervision does not beat answer-only training in-domain for strong backbones, but wins out-of-domain by 20 to 28 points, improving language adherence and auditable reasoning more than reasoning validity.","lead":"This paper tests whether teaching small Bangla math models to produce step-by-step reasoning, beyond just the answer, is worth the extra cost, using a tightly matched comparison. The benefit depends on the model and the test: reasoning supervision mainly helps language adherence and out-of-domain performance, not in-domain accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The matched protocol does not isolate rationale content: CoT targets are far longer than answer-only targets, so the OOD robustness claim could reflect token-count/length regularization rather than reasoning; a length-matched non-reasoning control is needed.","rationale":"The token-count confound is the most load-bearing concern because it targets the causal reading of the headline claim, not a peripheral statistic. The paper's novelty is precisely the matched answer-only comparison, and its main practical conclusion is that rationale supervision protects out-of-domain robustness. If that effect is reproduced by a length-matched non-reasoning target, the comparison still shows an empirical difference, but the 'rationale' label is misleading. No current analysis rules this out: Table 8 documents the inference-token asymmetry, but no experiment varies target length while holding content fixed. The other concerns noted by the reader—the GanitLLM common-subset calculation and co-author annotation—are real but secondary: they affect specific numbers (base vs fine-tuned for one model) and the human-study interpretation, not the general CoT-vs-AO OOD reversal or its attribution. The paper deserves credit for its transparent matched design and disclosure of limitations; this is an omitted control rather than an internal inconsistency. It therefore reinforces the existing CONDITIONAL verdict rather than changing it.","tokens_in":19362,"tokens_out":6953,"duration_ms":81159,"concrete_test":"Add a third matched condition to the protocol of §3.5–3.6: answer-only with a length-matched, reasoning-free target—the same final answer, but the CoT rationale replaced by non-mathematical Bangla text of the same token length (or the answer repeated with a fixed Bangla filler) so that token count, sequence length, loss masking, and epochs match the CoT condition. If this condition matches CoT's OOD accuracy and base-preservation on BanglaMATH, the rationale-content attribution fails; if it stays near the short answer-only condition, the attribution survives. Also report training-target token counts for all conditions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central attribution—that rationale supervision, not just longer training targets, drives the out-of-domain robustness benefit—rests on the claim that the two conditions differ only in the training target (§3.5–3.6). But the CoT target is a ~54-word Bangla trace (Table 1) while the answer-only target is a short answer; at inference the generated sequences are 15–52× longer (Table 8), and training tokens differ by a similar factor under fixed epochs. Longer targets change loss-averaging denominators, gradient magnitudes, and the total amount of Bangla language-modeling signal; they can act as regularization and as additional target-language supervision. These optimization-side effects, not the mathematical rationale, could explain why CoT preserves OOD accuracy while answer-only degrades below the base model. The human study (§4.7) compounds the problem: CoT does not improve reasoning-content criteria over the base model, so the observed OOD advantage has no demonstrated link to reasoning validity. The paper's own interpretation—'protecting against generalization loss'—is compatible with a pure length/language-regularization account. The strongest empirical finding (the OOD reversal) is therefore not uniquely attributable to rationale supervision without a length-matched control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper constructs MathShikkha, a Bangla mathematical reasoning dataset of 1,436 problems with GPT-5.4-generated chain-of-thought rationales, and fine-tunes four 4B--7B small language models under a matched protocol in which answer-only and CoT conditions share data splits, response-only loss masking, greedy decoding, and answer normalization, differing only in the training target. On a held-out in-domain test set, CoT supervision is statistically indistinguishable from answer-only supervision for three stronger backbones (paired bootstrap CIs include zero; McNemar p >= 0.17) and significantly better only for the weakest 4B backbone. On a contamination-audited external BanglaMATH benchmark, the pattern reverses: CoT outperforms answer-only supervision by 20--28 points for all four models, while answer-only fine-tuning degrades out-of-domain accuracy below the base model for three of four models. A human study of intermediate reasoning quality, with two co-author annotators and external adjudication, finds no significant CoT improvement over the base model on reasoning-content criteria; its measurable effect is target-language adherence and generation of inspectable reasoning. The paper concludes that rationale supervision's value depends on backbone capability and distribution shift, with benefits mainly in Bangla adherence, auditable reasoning, and out-of-domain robustness rather than improved in-domain reasoning validity.","tokens_in":19563,"tokens_out":4210,"duration_ms":45066,"significance":"If the results hold, this is a valuable controlled contribution to a low-resource-language reasoning problem. The matched protocol, paired bootstrap confidence intervals, exact McNemar tests, and detailed contamination audit are methodologically strong, and the finding that the value of CoT supervision reverses from in-domain to out-of-domain is a falsifiable, practically relevant claim. The paper also honestly documents a negative human-study result (CoT does not improve reasoning validity) and a 15.4% rate of correct answers with invalid reasoning, which is important for educational deployment. The central empirical claims are internally consistent with the reported statistics. However, the central attribution of the out-of-domain benefit to \"rationale supervision\" rather than to longer training targets is not established by the current design, and one external-evaluation comparison uses a selection-prone common subset; these issues require additional experiments or careful reframing before the paper's central claims can be accepted as stated.","major_comments":[{"comment":"The claim that the two conditions differ only in the training target is not strictly true: the CoT target is 15–52× longer than the answer-only target (Table 8), so under the response-only masked objective in Eqs. (4) and (6) the CoT condition has far more loss terms per example, different gradient magnitudes, and substantially more Bangla language-modeling signal. These optimization-side effects, rather than the mathematical rationale, could explain the out-of-domain robustness and the in-domain null result. The human study (§4.7) shows no improvement in reasoning-content criteria, so the observed OOD advantage has no demonstrated link to reasoning validity. The paper's interpretation that rationale supervision 'protects against generalization loss' is therefore not uniquely supported. A length-matched control (e.g., a non-reasoning extended Bangla target) is needed to attribute the OOD benefit to rationale content rather than to longer target sequences.","section":"§3.5–3.6; Table 8"},{"comment":"For GanitLLM, the out-of-domain comparison is computed on a common subset of 384 items after excluding 48 items for which the base model generated no output. This exclusion can bias the base accuracy estimate if the ungenerated items are systematically harder, which would inflate the reported base accuracy of 67.19% and thereby strengthen the claim that answer-only fine-tuning 'degrades below the base model.' The CoT-vs-answer-only gap is less directly affected because both conditions are scored on the same subset, but the paper should report the base accuracy on the full 432-item set (or the subset-specific base accuracy) and analyze the 48 excluded items to show the conclusion is robust to this selection.","section":"§4.2, Table 4 (GanitLLM row)"},{"comment":"The human study cannot currently separate 'target-language adherence' from the token-length effect, because the base condition is CoT-prompted (producing 98.97% English, Table 11) while the fine-tuned CoT condition is trained to produce Bangla. The paper acknowledges that the fluency difference is mechanical, but this means the human study provides no evidence about whether the out-of-domain benefit is due to reasoning content, to longer Bangla text, or to both. The joint outcome in Table 12 (15.4% correct answers with invalid reasoning) is informative, but it is computed only for base and CoT conditions, not for answer-only; adding a length-matched condition would make the table directly relevant to the paper's causal question.","section":"§4.7, Tables 10–12"}],"minor_comments":[{"comment":"The text says the benchmark is a 432-example clean set derived from 435 candidates, but the contamination funnel in Table C.1 reports 0 genuine contamination and one false positive; the relationship between 435 candidates, the 1 flagged item, and the final 432 items should be stated explicitly.","section":"§4.2"},{"comment":"The seed-robustness conclusion ('the direction never favors CoT') is based on only two seeds for only one model; this is a reasonable transparency choice, but the wording slightly overstates the generality of the robustness check.","section":"§4.1, Table 3"},{"comment":"The teacher model is referred to only as GPT-5.4; providing the exact model identifier, API version, generation temperature, and access date would improve reproducibility.","section":"§3.4 and §4.7"},{"comment":"The figure caption states 'Fine-tuned SLM reasoning succeeds' after distillation, but Section 4.7 shows that CoT fine-tuning does not improve reasoning validity over the base model; the figure should be qualified to avoid overstating what the human study found.","section":"Figure 1"},{"comment":"Equation (1) uses y = [z; a] but later text refers to y_i = [z_i; a_i]; the notation should be made uniform, and the definition of the concatenation operator should be stated once.","section":"Notation and formatting"},{"comment":"The token-efficiency table reports only in-domain inference tokens; reporting out-of-domain token counts would allow a direct comparison of the 20–28 point OOD gain against its actual inference cost.","section":"§4.5, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and reports an interesting, carefully analyzed negative result with a sharp OOD reversal. The main risk is the token-length confound: the 'matched' protocol does not isolate the rationale, and the paper's own human study undercuts the reasoning-validity interpretation. The authors should add a length-matched control (or clearly reframe the claims as being about extended target-language supervision rather than rationale content) and address the GanitLLM common-subset issue. These are fixable within the scope of a revision, so I recommend major revision rather than rejection. I do not see any problematic citation pattern; the single self-citation is not load-bearing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it actually does the matched comparison everyone else skips: same splits, same loss masking, same greedy decoding, same scorer, differing only in the assistant target. That makes the in-domain null result for strong backbones (CoT no better than answer-only, 95% CIs including zero) and the OOD reversal (20–28 point CoT advantage, answer-only degrading below base in three of four models) worth taking seriously. The contamination audit is careful, the paired bootstrap and exact McNemar tests are appropriate, and the human study at least tries to separate accuracy from reasoning quality. Credit where it is due: this is the first Bangla math distillation study I know of that gives answer-only a fair fight, and the authors are transparent about their annotator conflict and the 15.4% wrong-reasoning-correct-answer rate.\n\nThe soft spots are real but not fatal. The biggest is the token-count confound, which the stress test rightly pins down: CoT targets are 15–52× longer, so the OOD robustness could come from more Bangla language modeling or an implicit length-based regularization effect, not from the reasoning content. The human study makes this worse by showing CoT fine-tuning does not improve reasoning validity over the base model in-domain, so we have no demonstrated link between the rationales and the OOD accuracy gain. A length-matched control with non-reasoning Bangla text would fix this, and the paper currently lacks it. The GanitLLM OOD comparison on 384 rather than 432 items is a minor selection-bias risk, and the co-author annotators, while disclosed and adjudicated, are not a substitute for independent annotation. Also, the experimental appendix omits LoRA rank and alpha, which matters for reproduction.\n\nI would not call the paper's central claim false—the matched protocol is a real contribution, and the practical advice (answer-only is fine in-domain for strong backbones, but distills poorly out-of-domain) is useful—but I would insist the authors soften the attribution to \"rationale supervision\" or add the missing control before publication. As it stands, the precise mechanism is unproven, even though the empirical pattern is solid.\n\nWho is this for? People working on distillation evaluation, low-resource math reasoning, or CoT supervision generally. It deserves peer review and would likely get better with the control experiment. I would cite it, and I would bring it to a reading group for the matched-protocol design alone.","headline":"A genuinely careful matched comparison of answer-only vs. CoT supervision for Bangla math, whose main OOD claim is plausible but partially confounded by target length; worth a serious referee.","tokens_in":20156,"tokens_out":2049,"would_cite":true,"duration_ms":24806,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The value of Bangla chain-of-thought training depends entirely on where it is tested: out-of-domain it wins by 20–28 points, while in-domain it adds nothing for strong models.","keywords":["Bangla mathematical reasoning","chain-of-thought supervision","answer-only supervision","small language models","out-of-domain generalization","rationale distillation","low-resource languages"],"falsifier":"Train a third condition whose target is the same length as the CoT rationales but contains no mathematical reasoning (for example, repeated answer tokens or a scrambled rationale), keep the matched protocol identical, and compare out-of-domain accuracy: if the filler-length condition matches CoT's 20–28 point advantage, the paper's attribution to reasoning content is refuted.","tokens_in":19078,"feed_emoji":"🧮","tokens_out":7117,"duration_ms":61497,"temperature":0.7,"pith_summary":"This paper asks whether teaching a small language model to produce Bangla step-by-step rationales adds anything beyond ordinary answer-only fine-tuning on the same data. It constructs MathShikkha, a Bangla math dataset with teacher-generated rationales, and trains four 4B–7B models under a matched protocol where the two conditions share splits, loss masking, decoding, and scoring. In-domain, the rationale gives no significant accuracy gain for the three stronger backbones (confidence intervals include zero despite 15–52 times more tokens), while the weakest backbone improves by 18.56 points. Out-of-domain on a contamination-audited benchmark, the pattern reverses: rationale supervision beats answer-only supervision by 20–28 points for all four models, whereas answer-only fine-tuning drops below the base model for three. A human study finds the rationale's measurable in-domain effect is staying in Bangla and producing inspectable reasoning, not improving reasoning validity.","feed_headline":"Chain-of-thought training helps Bangla math only out-of-domain","feed_subtitle":"Matched study: CoT beats answer-only by 20–28 points externally; in-domain, strong models gain nothing.","key_machinery":"The load-bearing mechanism is a matched supervised fine-tuning protocol: the answer-only and CoT conditions share the identical train/validation split, response-only loss masking, greedy decoding, and type-safe answer-normalization pipeline, differing only in the assistant training target (the final answer alone versus rationale-plus-answer). On top of this, the paper uses paired bootstrap 95% confidence intervals and exact McNemar tests on the paired held-out items to decide when a difference is real, and a three-stage contamination audit (exact normalized-text overlap, near-duplicate character similarity, and number-masked template similarity) before treating the external benchmark as a valid generalization test. The human study scores intermediate reasoning on eight criteria with external adjudication and reports Cohen's κ to verify that the qualitative judgments are reliable.","core_discovery":"The paper's central claim is that the benefit of Bangla chain-of-thought supervision over matched answer-only supervision is not a fixed quantity: it depends on backbone capability and on whether evaluation is in- or out-of-domain. On the held-out test set, CoT supervision is statistically indistinguishable from answer-only fine-tuning for Qwen2.5-Math-7B-Instruct, DeepSeek-R1-Distill-Qwen-7B, and Mathstral-7B-v0.1 (paired bootstrap 95% confidence intervals include zero; exact McNemar p ≥ 0.17), despite generating 15–52× more tokens, and only the 4B GanitLLM backbone gains significantly (+18.56 points, p < 0.0001). On the larger, contamination-audited BanglaMATH benchmark, CoT supervision outperforms answer-only supervision for all four models by 20.1–28.1 points (all p < 0.0001), while answer-only fine-tuning reduces external accuracy below the non-fine-tuned base model for three of the four. A human study with two annotators, external-adjudication of disagreements, and Cohen's κ = 0.76–1.00 finds no significant CoT improvement over the base model on reasoning-content criteria; the measurable effects are target-language adherence and the production of inspectable reasoning. The practical conclusion is that in this low-resource setting rationale supervision's value is auditable reasoning, Bangla adherence, and out-of-domain robustness, not higher in-domain reasoning validity.","pith_inferences":["Editorial inference: Because the CoT target is 15–52× longer than the answer-only target, the two conditions differ in the total number of tokens each model trains on, so the out-of-domain advantage could in principle come from optimization-side effects rather than from the reasoning content itself.","Editorial inference: The same matched protocol could be applied to other low-resource languages, and if the pattern holds, CoT supervision's main benefit in low-resource settings would be protecting generalization while answer-only tuning suffices for narrow in-domain use.","Editorial inference: A natural follow-up experiment would train a third condition whose target matches the CoT rationales in length but contains no mathematical reasoning—for example, repeated answer tokens or scrambled rationale text—to test whether the 20–28 point out-of-domain gap requires reasoning content or just longer supervised sequences."],"forward_implications":["For deployments that will only see in-distribution problems, answer-only fine-tuning is cheaper and just as accurate as CoT supervision on strong backbones, so the 15–52× token overhead is not justified there.","For systems expected to face distribution shift, CoT supervision is the safer choice: it preserves or improves external accuracy for all four models, while answer-only fine-tuning degrades three of them below the base model.","Evaluation of rationale supervision that stops at in-domain accuracy will mislead; matched answer-only comparisons and out-of-domain tests are needed to see the real effect.","Even when final answers are correct, reasoning can be invalid: 15.4% of correct-answer items in the human study had reasoning judged invalid, so accuracy alone overstates reasoning quality in Bangla math models."],"supporting_citations":[{"why":"Establishes that chain-of-thought prompting improves multi-step reasoning in large models, the capability the paper extends to Bangla supervision.","marker":"Wei et al. 2022"},{"why":"Shows zero-shot CoT via simple instructions, framing the prompting-only baseline used in the paper's comparisons.","marker":"Kojima et al. 2022"},{"why":"Introduces Fine-tune-CoT, the distillation approach whose benefit over matched answer-only supervision the paper tests.","marker":"Ho et al. 2023"},{"why":"Provides Distilling Step-by-Step, evidence that label-rationale pairs give data-efficient supervision, the claim the paper scrutinizes with a matched baseline.","marker":"Hsieh et al. 2023"},{"why":"Introduces symbolic chain-of-thought distillation for smaller models, another precedent the matched comparison qualifies.","marker":"Li et al. 2023"},{"why":"Supplies BanglaMATH, the external benchmark the paper audits for contamination and uses for out-of-domain evaluation.","marker":"Prama et al. 2025"},{"why":"Documents the multilingual performance drop that motivates studying whether Bangla rationale supervision recovers reasoning in a low-resource language.","marker":"Shi et al. 2022"},{"why":"Identifies off-target language generation as a persistent multilingual reasoning failure, directly relevant to the paper's finding that CoT supervision keeps models in Bangla.","marker":"Luo et al. 2025"}],"fun_headline_variants":["CoT for Bangla math: out-of-domain win, in-domain only for weak model","Bangla math CoT: strong models gain nothing in-domain, 20+ pts out","CoT boosts Bangla math out-of-domain, but in-domain only weak model gains","In Bangla math, CoT pays off on external tests, not in-domain for strong"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the rationale is the only causally relevant difference between its two training conditions, but the CoT condition trains on 15–52× more tokens per example, so a difference in optimization signal—not reasoning content—could account for the out-of-domain gap.","fun_headline_variants_meta":{"raw":{"variants":["CoT for Bangla math: out-of-domain win, in-domain only for weak model","Bangla math CoT: strong models gain nothing in-domain, 20+ pts out","CoT boosts Bangla math out-of-domain, but in-domain only weak model gains","In Bangla math, CoT pays off on external tests, not in-domain for strong"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001849,"raw_usage":{"total_tokens":7410,"prompt_tokens":1233,"completion_tokens":6177,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":849,"completion_tokens_details":{"reasoning_tokens":6083}},"tokens_in":849,"tokens_out":6177,"duration_ms":40403,"temperature":1.0,"reasoning_tokens":6083,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:33:32.293735+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a third condition whose target is the same length as the CoT rationales but contains no mathematical reasoning (for example, repeated answer tokens or a scrambled rationale), keep the matched protocol identical, and compare out-of-domain accuracy: if the filler-length condition matches CoT's 20–28 point advantage, the paper's attribution to reasoning content is refuted.","supporting_citations":[],"review_version":1}