Pith. sign in

REVIEW 3 major objections 6 minor 21 references

MathShikkha: A Controlled Study of Answer-Only and Chain-of-Thought Supervision for Bangla Mathematical Reasoning in Small Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The value of Bangla chain-of-thought training depends entirely on where it is tested: out-of-domain it wins by 20–28 points, while in-domain it adds nothing for strong models.

desk verdict A genuinely careful matched comparison of answer-only vs. CoT supervision for Bangla math, whose main OOD claim is plausible but partially confounded by target length; worth a serious referee. read the letter →

arxiv 2608.08503 v1 pith:LTPKKVWH submitted 2026-08-09 cs.AI cs.CL

Rahma Simin Ali , Jawad Hossain This is my paper · ORCID
classification cs.AIcs.CL
keywords Banglamathematicalreasoningchain-of-thoughtsupervisionanswer-onlysmalllanguagemodelsout-of-domaingeneralizationrationaledistillationlow-resourcelanguages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether teaching a small language model to produce Bangla step-by-step rationales adds anything beyond ordinary answer-only fine-tuning on the same data. It constructs MathShikkha, a Bangla math dataset with teacher-generated rationales, and trains four 4B–7B models under a matched protocol where the two conditions share splits, loss masking, decoding, and scoring. In-domain, the rationale gives no significant accuracy gain for the three stronger backbones (confidence intervals include zero despite 15–52 times more tokens), while the weakest backbone improves by 18.56 points. Out-of-domain on a contamination-audited benchmark, the pattern reverses: rationale supervision beats answer-only supervision by 20–28 points for all four models, whereas answer-only fine-tuning drops below the base model for three. A human study finds the rationale's measurable in-domain effect is staying in Bangla and producing inspectable reasoning, not improving reasoning validity.

What carries the argument

The load-bearing mechanism is a matched supervised fine-tuning protocol: the answer-only and CoT conditions share the identical train/validation split, response-only loss masking, greedy decoding, and type-safe answer-normalization pipeline, differing only in the assistant training target (the final answer alone versus rationale-plus-answer). On top of this, the paper uses paired bootstrap 95% confidence intervals and exact McNemar tests on the paired held-out items to decide when a difference is real, and a three-stage contamination audit (exact normalized-text overlap, near-duplicate character similarity, and number-masked template similarity) before treating the external benchmark as a valid generalization test. The human study scores intermediate reasoning on eight criteria with external adjudication and reports Cohen's κ to verify that the qualitative judgments are reliable.

What would settle it

Train a third condition whose target is the same length as the CoT rationales but contains no mathematical reasoning (for example, repeated answer tokens or a scrambled rationale), keep the matched protocol identical, and compare out-of-domain accuracy: if the filler-length condition matches CoT's 20–28 point advantage, the paper's attribution to reasoning content is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the benefit of Bangla chain-of-thought supervision over matched answer-only supervision is not a fixed quantity: it depends on backbone capability and on whether evaluation is in- or out-of-domain. On the held-out test set, CoT supervision is statistically indistinguishable from answer-only fine-tuning for Qwen2.5-Math-7B-Instruct, DeepSeek-R1-Distill-Qwen-7B, and Mathstral-7B-v0.1 (paired bootstrap 95% confidence intervals include zero; exact McNemar p ≥ 0.17), despite generating 15–52× more tokens, and only the 4B GanitLLM backbone gains significantly (+18.56 points, p < 0.0001). On the larger, contamination-audited BanglaMATH benchmark, CoT supervision outperforms answer-only supervision for all four models by 20.1–28.1 points (all p < 0.0001), while answer-only fine-tuning reduces external accuracy below the non-fine-tuned base model for three of the four. A human study with two annotators, external-adjudication of disagreements, and Cohen's κ = 0.76–1.00 finds no significant CoT improvement over the base model on reasoning-content criteria; the measurable effects are target-language adherence and the production of inspectable reasoning. The practical conclusion is that in this low-resource setting rationale supervision's value is auditable reasoning, Bangla adherence, and out-of-domain robustness, not higher in-domain reasoning validity.

Load-bearing premise

The paper assumes the rationale is the only causally relevant difference between its two training conditions, but the CoT condition trains on 15–52× more tokens per example, so a difference in optimization signal—not reasoning content—could account for the out-of-domain gap.

Editorial extensions

If this is right

  • For deployments that will only see in-distribution problems, answer-only fine-tuning is cheaper and just as accurate as CoT supervision on strong backbones, so the 15–52× token overhead is not justified there.
  • For systems expected to face distribution shift, CoT supervision is the safer choice: it preserves or improves external accuracy for all four models, while answer-only fine-tuning degrades three of them below the base model.
  • Evaluation of rationale supervision that stops at in-domain accuracy will mislead; matched answer-only comparisons and out-of-domain tests are needed to see the real effect.
  • Even when final answers are correct, reasoning can be invalid: 15.4% of correct-answer items in the human study had reasoning judged invalid, so accuracy alone overstates reasoning quality in Bangla math models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because the CoT target is 15–52× longer than the answer-only target, the two conditions differ in the total number of tokens each model trains on, so the out-of-domain advantage could in principle come from optimization-side effects rather than from the reasoning content itself.
  • Editorial inference: The same matched protocol could be applied to other low-resource languages, and if the pattern holds, CoT supervision's main benefit in low-resource settings would be protecting generalization while answer-only tuning suffices for narrow in-domain use.
  • Editorial inference: A natural follow-up experiment would train a third condition whose target matches the CoT rationales in length but contains no mathematical reasoning—for example, repeated answer tokens or scrambled rationale text—to test whether the 20–28 point out-of-domain gap requires reasoning content or just longer supervised sequences.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper constructs MathShikkha, a Bangla mathematical reasoning dataset of 1,436 problems with GPT-5.4-generated chain-of-thought rationales, and fine-tunes four 4B--7B small language models under a matched protocol in which answer-only and CoT conditions share data splits, response-only loss masking, greedy decoding, and answer normalization, differing only in the training target. On a held-out in-domain test set, CoT supervision is statistically indistinguishable from answer-only supervision for three stronger backbones (paired bootstrap CIs include zero; McNemar p >= 0.17) and significantly better only for the weakest 4B backbone. On a contamination-audited external BanglaMATH benchmark, the pattern reverses: CoT outperforms answer-only supervision by 20--28 points for all four models, while answer-only fine-tuning degrades out-of-domain accuracy below the base model for three of four models. A human study of intermediate reasoning quality, with two co-author annotators and external adjudication, finds no significant CoT improvement over the base model on reasoning-content criteria; its measurable effect is target-language adherence and generation of inspectable reasoning. The paper concludes that rationale supervision's value depends on backbone capability and distribution shift, with benefits mainly in Bangla adherence, auditable reasoning, and out-of-domain robustness rather than improved in-domain reasoning validity.

Significance. If the results hold, this is a valuable controlled contribution to a low-resource-language reasoning problem. The matched protocol, paired bootstrap confidence intervals, exact McNemar tests, and detailed contamination audit are methodologically strong, and the finding that the value of CoT supervision reverses from in-domain to out-of-domain is a falsifiable, practically relevant claim. The paper also honestly documents a negative human-study result (CoT does not improve reasoning validity) and a 15.4% rate of correct answers with invalid reasoning, which is important for educational deployment. The central empirical claims are internally consistent with the reported statistics. However, the central attribution of the out-of-domain benefit to "rationale supervision" rather than to longer training targets is not established by the current design, and one external-evaluation comparison uses a selection-prone common subset; these issues require additional experiments or careful reframing before the paper's central claims can be accepted as stated.

major comments (3)
  1. [§3.5–3.6; Table 8] The claim that the two conditions differ only in the training target is not strictly true: the CoT target is 15–52× longer than the answer-only target (Table 8), so under the response-only masked objective in Eqs. (4) and (6) the CoT condition has far more loss terms per example, different gradient magnitudes, and substantially more Bangla language-modeling signal. These optimization-side effects, rather than the mathematical rationale, could explain the out-of-domain robustness and the in-domain null result. The human study (§4.7) shows no improvement in reasoning-content criteria, so the observed OOD advantage has no demonstrated link to reasoning validity. The paper's interpretation that rationale supervision 'protects against generalization loss' is therefore not uniquely supported. A length-matched control (e.g., a non-reasoning extended Bangla target) is needed to attribute the OOD benefit to rationale content rather than to longer target sequences.
  2. [§4.2, Table 4 (GanitLLM row)] For GanitLLM, the out-of-domain comparison is computed on a common subset of 384 items after excluding 48 items for which the base model generated no output. This exclusion can bias the base accuracy estimate if the ungenerated items are systematically harder, which would inflate the reported base accuracy of 67.19% and thereby strengthen the claim that answer-only fine-tuning 'degrades below the base model.' The CoT-vs-answer-only gap is less directly affected because both conditions are scored on the same subset, but the paper should report the base accuracy on the full 432-item set (or the subset-specific base accuracy) and analyze the 48 excluded items to show the conclusion is robust to this selection.
  3. [§4.7, Tables 10–12] The human study cannot currently separate 'target-language adherence' from the token-length effect, because the base condition is CoT-prompted (producing 98.97% English, Table 11) while the fine-tuned CoT condition is trained to produce Bangla. The paper acknowledges that the fluency difference is mechanical, but this means the human study provides no evidence about whether the out-of-domain benefit is due to reasoning content, to longer Bangla text, or to both. The joint outcome in Table 12 (15.4% correct answers with invalid reasoning) is informative, but it is computed only for base and CoT conditions, not for answer-only; adding a length-matched condition would make the table directly relevant to the paper's causal question.
minor comments (6)
  1. [§4.2] The text says the benchmark is a 432-example clean set derived from 435 candidates, but the contamination funnel in Table C.1 reports 0 genuine contamination and one false positive; the relationship between 435 candidates, the 1 flagged item, and the final 432 items should be stated explicitly.
  2. [§4.1, Table 3] The seed-robustness conclusion ('the direction never favors CoT') is based on only two seeds for only one model; this is a reasonable transparency choice, but the wording slightly overstates the generality of the robustness check.
  3. [§3.4 and §4.7] The teacher model is referred to only as GPT-5.4; providing the exact model identifier, API version, generation temperature, and access date would improve reproducibility.
  4. [Figure 1] The figure caption states 'Fine-tuned SLM reasoning succeeds' after distillation, but Section 4.7 shows that CoT fine-tuning does not improve reasoning validity over the base model; the figure should be qualified to avoid overstating what the human study found.
  5. [Notation and formatting] Equation (1) uses y = [z; a] but later text refers to y_i = [z_i; a_i]; the notation should be made uniform, and the definition of the concatenation operator should be stated once.
  6. [§4.5, Table 8] The token-efficiency table reports only in-domain inference tokens; reporting out-of-domain token counts would allow a direct comparison of the 20–28 point OOD gain against its actual inference cost.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims rest on held-out evaluation and an external contamination-audited benchmark, not on parameters fitted to the predicted outcomes.

full rationale

The paper's contribution is empirical rather than derivational. It constructs a Bangla CoT dataset, fine-tunes student models under a matched answer-only versus CoT protocol, and evaluates on a fixed held-out split (Section 3.3) and a contamination-audited external benchmark (Section 4.2). Accuracy is computed with Equation (8) on items never used for training or model selection, so no reported accuracy is a fitted parameter renamed as a prediction. The statement that answer-only supervision yields no inspectable reasoning is explicitly acknowledged as true by construction ('Answer-only outputs contain no reasoning trace, so they score at floor on reasoning-content criteria by construction') and is not used to support the causal claims; it is a definitional observation, not a derived result. The only self-citation (Hossain et al., 2026) appears in related work and is not load-bearing for the paper's central comparisons. The main methodological caveat, that CoT targets are 15–52x longer than answer-only targets and may introduce token-count or language-regularization effects, is a confound or validity threat rather than a circular reduction: the paper does not define the outcome in terms of the treatment, nor does it use evaluation data to fit the quantities it predicts. Under the hard rules requiring a specific reduction by construction, no circular step is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

This is an empirical ML paper; the ledger lists the modeling choices and domain assumptions the central claims rest on. No parameters are fitted to the evaluation data, and no new physical or conceptual entities are postulated. The main unexamined assumption is the token-count asymmetry between the two training conditions, which is also the weakest assumption of the paper.

free parameters (6)
  • Learning rate = 2e-4
    Chosen from standard QLoRA practice; shared across both conditions, so it does not confound the matched comparison but affects absolute accuracy and could interact with target length.
  • Number of epochs = 5
    Chosen by hand; checkpoint selection uses validation loss, so this is not fitted to the test data.
  • LoRA rank r = not reported
    Table B.1 omits the QLoRA rank, which controls adapter capacity; without it, exact reproduction is impossible. It is shared across conditions.
  • LoRA scaling alpha = not reported
    Also omitted from Table B.1; affects update magnitude and thus fine-tuning behavior.
  • Maximum sequence length (CoT) = 3072
    Set high enough to accommodate full rationales; this only applies to the CoT condition, so it is part of the target-length asymmetry.
  • Contamination audit thresholds = 0.85 (near-dup), 0.90 (masked)
    Hand-set similarity thresholds in Section 4.2; they affect the contamination claim but not the central model comparison.
assumptions (5)
  • domain assumption GPT-5.4 teacher rationales, after answer-consistency filtering and partial manual inspection, are high-quality enough to serve as CoT supervision.
    Section 3.4 states that teacher outputs with incorrect final answers are discarded or regenerated, and a subset is manually inspected, but exhaustive proof-level verification is not performed. If the rationales contain systematic reasoning errors, the CoT training signal would be flawed and the reasoning-quality conclusions in Section 4.7 would change.
  • domain assumption The BanglaMATH benchmark is a valid out-of-domain test, and the three-stage contamination audit is sufficient to rule out train/test leakage.
    Section 4.2 describes the audit; the paper treats the benchmark as uncontaminated. If the audit missed semantic paraphrases beyond number/template masking, the OOD comparison would not be truly external.
  • domain assumption Final-answer accuracy after string normalization is a valid primary measure of mathematical reasoning performance.
    Section 3.8 defines accuracy via normalized string equality. This ignores partially correct reasoning, but the paper supplements it with a human study; still, the headline OOD claims rest on this metric.
  • domain assumption For the out-of-domain comparison, computing accuracies on the common set of items for which all conditions produced output is unbiased with respect to missing generations.
    Table 4, note: GanitLLM uses n=384 instead of 432 because 48 base-model items were ungenerated. If non-generation correlates with difficulty, the common-set estimates could be biased.
  • domain assumption The matched protocol isolates supervision type: differences in training target length (and hence gradient statistics) do not themselves cause the observed in-domain and OOD differences.
    Sections 3.5-3.6 hold splits, masking, decoding, and scoring constant, but the CoT target is 15-52x longer than the answer-only target (Table 8). If the longer supervised sequences, rather than the reasoning content, drive the OOD robustness, the paper's attribution to 'rationale supervision' would be wrong.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MathShikkha: A Controlled Study of Answer-Only and Chain-of-Thought Supervision for Bangla Mathematical Reasoning in Small Language Models." pith.science (2026). https://pith.science/paper/LTPKKVWH

@misc{pith2026260808503,
  author       = {Pith},
  title        = {Pith review of: MathShikkha: A Controlled Study of Answer-Only and Chain-of-Thought Supervision for Bangla Mathematical Reasoning in Small Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LTPKKVWH}},
  note         = {Machine review of arXiv:2608.08503}
}
abstract

Mathematical reasoning remains challenging in low-resource languages such as Bangla. We study whether teacher-generated Bangla Chain-of-Thought (CoT) supervision provides benefits beyond ordinary supervised fine-tuning. We construct \textsc{MathShikkha}, a Bangla mathematical reasoning dataset with GPT-5.4-generated rationales, and fine-tune four 4B--7B student models under a matched protocol in which answer-only and CoT conditions share data splits, response-only loss masking, decoding, and scoring, differing only in the training target. In-domain, CoT provides no significant improvement over answer-only fine-tuning for three stronger backbones (paired bootstrap 95\% CIs include zero; exact McNemar $p \geq 0.17$), despite generating 15--52$\times$ more tokens, but significantly improves the weaker 4B model by 18.56 points ($p < 0.0001$). On the larger, contamination-audited BanglaMATH benchmark, this pattern reverses: CoT significantly outperforms answer-only supervision for all four models by 20.1--28.1 points (all $p < 0.0001$). Answer-only fine-tuning also reduces out-of-domain accuracy below the base model for three models, whereas CoT preserves or improves it for all four. A human study with two co-author annotators, external-expert adjudication, and Cohen's $\kappa = 0.76$--$1.00$ finds no significant CoT improvement over the base model on reasoning-content criteria; instead, its measurable effect is target-language adherence and producing inspectable reasoning. Overall, rationale supervision's value depends on backbone capability and distribution shift: in this setting, its main benefits are Bangla adherence, auditable reasoning, and out-of-domain robustness rather than improved in-domain reasoning validity.

Figures

Figures reproduced from arXiv: 2608.08503 by the authors.

Figure 1
Figure 1. Example reasoning comparison on a Bangla math problem. The Base SLM fails to solve the problem, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 8 canonical work pages

  1. [1]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  2. [2]

    Advances in neural information processing systems , volume=

    Large language models are zero-shot reasoners , author=. Advances in neural information processing systems , volume=

  3. [3]

    International Conference on Learning Representations , volume=

    To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning , author=. International Conference on Learning Representations , volume=

    work page

  4. [4]

    Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

    Large language models are reasoning teachers , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

  5. [5]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Symbolic chain-of-thought distillation: Small models can also “think” step-by-step , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  6. [6]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages=

    Teaching small language models to reason , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages=

  7. [7]

    arXiv preprint arXiv:2402.14830 , year=

    Orca-math: Unlocking the potential of slms in grade school math , author=. arXiv preprint arXiv:2402.14830 , year=

    arXiv

  8. [8]

    Findings of the Association for Computational Linguistics: ACL 2023 , pages=

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes , author=. Findings of the Association for Computational Linguistics: ACL 2023 , pages=

    2023

Show all 21 references
  1. [9]

    arXiv preprint arXiv:2407.10167 , year=

    Key-point-driven mathematical reasoning distillation of large language model , author=. arXiv preprint arXiv:2407.10167 , year=

    arXiv

  2. [10]

    IEEE Transactions on Neural Networks and Learning Systems , year=

    Distilling reasoning ability from large language models with adaptive thinking , author=. IEEE Transactions on Neural Networks and Learning Systems , year=

  3. [11]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  4. [12]

    Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025) , pages=

    Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap , author=. Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025) , pages=

    2025

  5. [13]

    Preprint , year=

    Mmath: A multilingual benchmark for mathematical reasoning , author=. Preprint , year=

  6. [14]

    International Conference on Learning Representations , volume=

    Mammoth: Building math generalist models through hybrid instruction tuning , author=. International Conference on Learning Representations , volume=

  7. [15]

    International Conference on Learning Representations , volume=

    Metamath: Bootstrap your own mathematical questions for large language models , author=. International Conference on Learning Representations , volume=

  8. [16]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    arXiv

  9. [17]

    arXiv preprint arXiv:2210.03057 , year=

    Language models are multilingual chain-of-thought reasoners , author=. arXiv preprint arXiv:2210.03057 , year=

    arXiv

  10. [18]

    Proceedings of The 3rd Workshop on Mathematical Natural Language Processing (MathNLP 2025) , pages=

    BanglaMATH: A Bangla benchmark dataset for testing LLM mathematical reasoning at grades 6, 7, and 8 , author=. Proceedings of The 3rd Workshop on Mathematical Natural Language Processing (MathNLP 2025) , pages=

    2025

  11. [19]

    Discover Artificial Intelligence , volume=

    BMWP: the first Bengali math word problems dataset for operation prediction and solving , author=. Discover Artificial Intelligence , volume=. 2025 , publisher=

    2025

  12. [20]

    arXiv preprint arXiv:2505.21354 , year=

    Leveraging large language models for bengali math word problem solving with chain of thought reasoning , author=. arXiv preprint arXiv:2505.21354 , year=

    arXiv

  13. [21]

    arXiv preprint arXiv:2604.12229 , year=

    Hintmr: Eliciting stronger mathematical reasoning in small language models , author=. arXiv preprint arXiv:2604.12229 , year=

    arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.