Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Leveraging Large Language Models for Bengali Math Word Problem Solving with Chain of Thought Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new 8,792-problem Bengali math dataset with human-written step-by-step solutions, translated from GSM8K, lets LLaMA-3.3 70B reach 88 percent accuracy with few-shot chain-of-thought prompting.

desk verdict Potentially useful first Bengali CoT MWP dataset, but the paper's central scale claim is contradicted by its own Section 4.3 and the evaluation details are too thin for the headline 88% to be taken at face value yet. read the letter →

arxiv 2505.21354 v2 pith:RUKMFS7M submitted 2025-05-27 cs.CL cs.LG

Bidyarthi Paul , Jalisha Jashim Era , Mirazur Rahman Zim , Tahmid Sattar Aothoi , Faisal Muhammad Shah This is my paper
classification cs.CLcs.LG
keywords Bengalimathwordproblemschain-of-thoughtpromptingSOMADHANdatasetlow-resourcelanguageNLPlargemodelsLoRAfine-tuningGSM8Ktranslationmathematicalreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Bengali, spoken by over 270 million people, has no human-annotated math word problem dataset with reasoning steps, which has left large language models unevaluated on mathematical reasoning in that language. This paper tries to close that gap by creating SOMADHAN, a dataset of 8,792 Bengali math word problems translated manually from the English GSM8K benchmark, each with a step-by-step Bengali solution and final answer. On a 1,000-problem evaluation subset, the paper reports that chain-of-thought prompting consistently beats standard prompting, and that LLaMA-3.3 70B reaches 88 percent accuracy with five-shot chain-of-thought prompting, the best result among the 15 models tested, which include GPT-4o, GPT-3.5, LLaMA variants, Deepseek, and Qwen. The outcome matters because it gives the Bengali NLP community its first reasoning benchmark and a transferable recipe for bringing math reasoning evaluation to low-resource languages.

What carries the argument

The central object is the SOMADHAN dataset itself: 8,792 grade-school math problems from GSM8K, translated into Bengali by five human translators under guidelines that preserve the original reasoning steps and numbers while replacing dollars with taka, English names with Bengali names, and English numerals with Bengali numerals. Each problem is stored with a step-by-step Bengali solution and a final answer, which is what makes chain-of-thought evaluation possible. The method that carries the experiments is chain-of-thought prompting, instructing the model to emit intermediate reasoning steps before the final answer, tested in two prompt styles, one asking for a step-by-step Bengali solution, the other additionally requiring the final numerical answer after '####', in zero-shot, five-shot, and fine-tuned configurations. The third piece is Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning scheme that freezes the pretrained weights and learns only two small matrices of rank $r$.

What would settle it

Take a random sample of 100-200 SOMADHAN problems, have independent Bengali-English translators back-translate them to English, and compare each recovered problem against the original GSM8K problem for preserved numbers, operations, and relations; a discrepancy rate above a few percent would mean the reported 88 percent accuracy rests on partly corrupted ground truth. As a second check, re-score the same 1,000-problem evaluation set with an automated final-answer extractor instead of manual checking and see whether the per-model accuracies reproduce.

Watch

Extended reading notes

Core claim

The central claim is that Bengali math word problem solving is blocked mainly by missing data, not missing model capability: SOMADHAN supplies 8,792 complex Bengali problems, each with a manually written chain of thought and a final answer, built by translating GSM8K's train and test splits into Bengali while preserving the mathematics and swapping in taka, Bengali names, and Bengali numerals. On a 1,000-problem evaluation subset drawn from the 3,200 test samples, the paper reports that chain-of-thought prompting consistently outperforms standard prompting, with the largest models benefiting most: LLaMA-3.3 70B reaches 88 percent with five-shot chain-of-thought prompting, GPT-4o reaches 83 percent, and smaller models such as LLaMA-3 8B lose accuracy when forced into Bengali chain-of-thought. The same models score far higher on PatiGonit, a simpler equation-based Bengali dataset where GPT-4o reaches 99 percent, which the paper reads as evidence that the difficulty lies in multi-step reasoning rather than in Bengali itself. Fine-tuning results are reported as secondary: LoRA-adapted LLaMA-3 8B reaches only 12-17 percent, and fine-tuned GPT-3.5 peaks at 23 percent, both far below the prompting-based results.

Load-bearing premise

The entire benchmark rests on the unverified assumption that the five human translators preserved each GSM8K problem's exact numbers and reasoning steps when rendering it into Bengali, because the paper reports no back-translation, no second-pass check, and no agreement measurement between translators.

Editorial extensions

If this is right

  • Bengali NLP gains its first human-annotated, reasoning-step math word problem dataset, enabling reasoning-focused evaluation and model development in a language with over 270 million speakers.
  • Chain-of-thought prompting transfers to Bengali: it consistently improves accuracy over standard prompting, with gains up to 10 points for LLaMA-3.3 70B and 19 points for Qwen-2.5 in zero-shot.
  • Model scale is decisive in low-resource reasoning: 70B-class models score 66-88 percent, while 8B and smaller models stay below 48 percent and often lose accuracy when chain-of-thought is added.
  • The hard part is multi-step reasoning, not the Bengali language: on the simple equation-based PatiGonit set, GPT-4o reaches 99 percent.
  • LoRA fine-tuning on a few hundred examples reaches only 12-17 percent on the test set, far below the prompting-based results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The abstract claims 8,792 problems with step-by-step solutions, but Section 4.3 of the paper states that only 4,000 samples are annotated so far and the rest are still being processed, so a reader should check which version is actually released before building on the dataset.
  • Because SOMADHAN is a translation of GSM8K, it inherits GSM8K's problem mix; a Bengali-authored set of original problems could behave differently, so the 88 percent figure is not evidence about a ceiling for original Bengali math.
  • Several models answered in English even when instructed in Bengali, so part of the measured difficulty is language alignment on top of arithmetic; scoring English-language answers separately from Bengali-language answers would separate the two failure modes.
  • The pipeline has no built-in verification step, so any port of this approach to another low-resource language should add back-translation checks and inter-annotator agreement from the start.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SOMADHAN, described in the abstract and Contribution 1 as a dataset of 8,792 complex Bengali math word problems with manually written, step-by-step solutions, built by manually translating the English GSM8K dataset into Bengali. The authors evaluate a range of large language models (GPT-3.5, GPT-4o, LLaMA variants, DeepSeek, Qwen) under zero-shot and few-shot prompting with and without chain-of-thought (CoT) reasoning, report that LLaMA-3.3 70B achieves the highest accuracy of 88% with few-shot CoT prompting, and also present LoRA and GPT-3.5 fine-tuning experiments. The central contributions claimed are the Bengali CoT MWP dataset, the prompting pipeline, and the comparative evaluation.

Significance. If the full 8,792-problem dataset were actually available with verified step-by-step Bengali solutions, it would be a valuable resource for a genuinely low-resource language, and the broad comparison across open and proprietary LLMs would be a useful reference point for the community. The manual annotation effort, even for the currently stated 4,000 problems, and the detailed prompt design are constructive contributions. However, the discrepancy between the abstract and Section 4.3, the missing data link in Section 4.4, and the undocumented sampling of the 1,000-problem evaluation subset mean that the headline claims are not currently supported by the manuscript as written. The paper would also benefit from acknowledging that several reported accuracy figures correspond to English-language model outputs, not Bengali, which complicates the interpretation of the benchmark results.

major comments (4)
  1. [§4.3 vs Abstract and Contribution 1] The abstract and Contribution 1 claim that SOMADHAN contains 8,792 complex Bengali MWPs with manually written, step-by-step solutions, but Section 4.3 states that 'Currently, the dataset includes 4,000 manually annotated samples' and that annotation of the remaining problems is ongoing. These statements are irreconcilable. Furthermore, Section 6.1 says the experiments use 3,200 test and 800 training examples from the annotated portion, and evaluation is limited to 1,000 test samples. Therefore, every accuracy figure in Tables 3 and 4, including the headline 88% for LLaMA-3.3 70B, characterizes at most the 4,000-problem annotated subset, not the advertised 8,792-problem benchmark. The abstract and contributions must be corrected to match the actual dataset size, or the remaining annotations must be completed and released before the 8,792-problem claim is made.
  2. [§6.1 and §5.1] The evaluation protocol is under-specified. The paper selects 1,000 of the 3,200 test samples for evaluation but does not document how those 1,000 samples were chosen, nor does it report any confidence intervals or significance tests for the accuracy numbers. The manual evaluation of final answers described in Section 5.1 is appropriate for a small study, but without a sampling protocol the representativeness of the 1,000-sample subset cannot be assessed. The authors should state the sampling method, report binomial confidence intervals (or equivalent), and ideally evaluate on the full test split to support the comparative claims made throughout Section 7.
  3. [§4.2 (Data Translation)] The dataset is a manual translation of GSM8K by five translators who evenly split the corpus, but no inter-annotator agreement, back-translation, or independent verification of numerical and reasoning fidelity is reported. The guidelines in Section 4.2.2 are reasonable, but without any quality verification the claim that SOMADHAN provides 'high-quality, human-annotated' reasoning data is unsupported. Since translation errors would directly affect both the gold answers and every measured accuracy, the authors should add a validation protocol, such as double-annotating a sample, back-translating a subset, or reporting an error analysis of the translated problems.
  4. [Tables 3 and 4] Several rows in Tables 3 and 4 carry an asterisk indicating that the model's responses were in English rather than Bengali, yet these accuracies are presented in the same table as Bengali-language results. For example, Llama-3 70B reaches 70% in zero-shot standard prompting in Table 3, and deepseek-r1-distill-qwen reaches 51% with CoT few-shot in Table 4, all in English. Because SOMADHAN is intended as a Bengali benchmark, accuracy scores for English-language responses are not directly comparable to Bengali-language scores; the paper should report these separately or include a language-adherence metric so that the comparative claims are not misleading.
minor comments (5)
  1. [§4.4] The data availability statement says the dataset is publicly available via 'Dataset of SOMADHAN (Original data) (Mendeley Data)' but no working URL or DOI is provided; please supply a complete link or repository identifier.
  2. [§4.2.1] The heading 'Translator Indentity' contains a typo and should read 'Translator Identity'.
  3. [§7.1 and Table 3] The text states that GPT-4o reaches '80.4%' in zero-shot CoT, but Table 3 lists 80.0%; similarly, Section 7.3 reports fine-tuned GPT-3.5 accuracies of 23.4% and 12.6% while Table 6 lists 23.0% and 12.0%. Please reconcile the text with the tables.
  4. [§5.4.2 and Table 5] The LoRA fine-tuning experiments in Table 5 do not state which base model was fine-tuned, making the reported 13–17% accuracies uninterpretable; please specify the base model and the training/validation split.
  5. [§5.1] The evaluation measures section says that for SOMADHAN only the final answer is checked, while for PatiGonit the predicted equation is matched manually; this distinction should be stated more clearly in the experimental setup since it affects how the accuracy numbers in Tables 3 and 4 should be read.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: central dataset and accuracy claims are grounded in external GSM8K source data and public model evaluations, not in fitted parameters or self-citations.

full rationale

The main claimed result—the SOMADHAN Bengali MWP benchmark and the 88% LLaMA-3.3 few-shot CoT accuracy—is derived by manually translating the externally sourced GSM8K train/test problems (Section 4.1) and evaluating public API models on a held-out test subset (Section 6.1). The correctness metric in Equation 1 compares the model's predicted final result with the translated ground-truth answer; this is an external check, not a fitted input renamed as a prediction. The only work by the same authors is cited for the secondary PatiGonit dataset (reference [22]), which is used only to contrast simple equation-based problems and is not needed to support the SOMADHAN accuracy claims. The internal inconsistency between the abstract's '8,792 manually written solutions' and Section 4.3's '4,000 manually annotated samples' is a data-quality and claims issue, not a circular derivation. No equation, fitted parameter, or uniqueness theorem is imported from the authors' prior work, and no benchmark conclusion is equivalent by construction to its own input.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is an empirical benchmark paper with no mathematical derivation and no invented physical or formal entities. The central claims rest on assumptions about translation quality, subset representativeness, and manual evaluation reliability. There are no fitted free parameters in the dataset construction or prompting pipeline.

assumptions (4)
  • domain assumption GSM8K problems, after cultural adaptation, are valid complex Bengali MWPs with preserved reasoning.
    Section 4.1 and 4.2 treat translated GSM8K problems as the foundation of SOMADHAN without independent validation that the resulting Bengali problems are natural or correctly reasoned.
  • domain assumption The five translators produced correct Bengali step-by-step solutions without independent verification.
    Section 4.2 and Table 2 describe the translators and guidelines but report no back-translation, inter-annotator agreement, or expert review of solution correctness.
  • ad hoc to paper The manually checked 1,000-sample subset is representative of the full test set.
    Section 6.1 states that only 1,000 samples were evaluated due to token costs, but no random sampling or stratification protocol is documented.
  • domain assumption Manual evaluation of final answers and PatiGonit equations is error-free and reproducible.
    Section 5.1 says evaluation was performed through manual checks, but no evaluation script, annotator agreement, or double-checking procedure is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Large Language Models for Bengali Math Word Problem Solving with Chain of Thought Reasoning." pith.science (2026). https://pith.science/paper/RUKMFS7M

@misc{pith2026250521354,
  author       = {Pith},
  title        = {Pith review of: Leveraging Large Language Models for Bengali Math Word Problem Solving with Chain of Thought Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RUKMFS7M}},
  note         = {Machine review of arXiv:2505.21354}
}
read the original abstract

Solving Bengali Math Word Problems (MWPs) remains a major challenge in natural language processing (NLP) due to the language's low-resource status and the multi-step reasoning required. Existing models struggle with complex Bengali MWPs, largely because no human-annotated Bengali dataset has previously addressed this task. This gap has limited progress in Bengali mathematical reasoning. To address this, we created SOMADHAN, a dataset of 8792 complex Bengali MWPs with manually written, step-by-step solutions. We designed this dataset to support reasoning-focused evaluation and model development in a linguistically underrepresented context. Using SOMADHAN, we evaluated a range of large language models (LLMs) - including GPT-4o, GPT-3.5 Turbo, LLaMA series models, Deepseek, and Qwen - through both zero-shot and few-shot prompting with and without Chain of Thought (CoT) reasoning. CoT prompting consistently improved performance over standard prompting, especially in tasks requiring multi-step logic. LLaMA-3.3 70B achieved the highest accuracy of 88% with few-shot CoT prompting. We also applied Low-Rank Adaptation (LoRA) to fine-tune models efficiently, enabling them to adapt to Bengali MWPs with minimal computational cost. Our work fills a critical gap in Bengali NLP by providing a high-quality reasoning dataset and a scalable framework for solving complex MWPs. We aim to advance equitable research in low-resource languages and enhance reasoning capabilities in educational and language technologies.

Figures

Figures reproduced from arXiv: 2505.21354 by the authors.

Figure 1
Figure 1. Chain-of-Thought (CoT) Prompting enables LLMs (Large Language Model)s to improve complex reasoning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pipeline for the development of the SOMADHAN dataset [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 5
Figure 5. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figures from the paper (12 more)
Figure 3
Figure 3. Figure 3: Challenges faced during the translation of BGSM8K dataset [PITH_FULL_IMAGE:figures/full_fig_p009_3.png]
Figure 6
Figure 6. Figure 6: Schematic diagram of our proposed prompting approach [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Instruction (Prompt-1) for SOMADHAN Dataset (Zero Shot Prompting) [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 11
Figure 11. Figure 11: Instruction (Prompt-3) for PatiGonit Dataset (Few (5) Shot) Prompting [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 13
Figure 13. Figure 13: Structure of JSON file format fine-tuning process while saving storage. As part of this process, we also integrated chain-of-thought examples into the fine-tuning dataset to encourage the model to generate logical, step-by-step reasoning [PITH_FULL_IMAGE:figures/full…
Figure 14
Figure 14. Figure 14: LoRA Fine-Tuning Process: During and After Training (LoRA, 2024)[65] [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: SOMADHAN dataset, Prompting and Output in OpenAI Playground [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]
Figure 16
Figure 16. Figure 16: PatiGonit dataset, Prompting and Output in OpenAI Playground [PITH_FULL_IMAGE:figures/full_fig_p028_16.png]
Figure 17
Figure 17. Figure 17: Response generated by LLaMA 3.3 model on a Bengali Math Word Problem using Chain-of-Thought [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]
Figure 18
Figure 18. Figure 18: Example of correct chains of thought produced by the GPT-3.5 for the SOMADHAN dataset for Few Shot [PITH_FULL_IMAGE:figures/full_fig_p030_18.png]
Figure 19
Figure 19. Figure 19: Example of correct chains of thought produced by the GPT-4o for the SOMADHAN dataset for Few Shot [PITH_FULL_IMAGE:figures/full_fig_p031_19.png]
Figure 20
Figure 20. Figure 20: Example of correct chains of thought produced by the GPT-3.5 for the SOMADHAN dataset for Fine Tuning [PITH_FULL_IMAGE:figures/full_fig_p032_20.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    The paper releases GSM-Plus-BN, a human-verified Bengali translation of the GSM-Plus perturbed math benchmark, and reports accuracy baselines for six open LLMs under standard and CoT prompting.

Reference graph

Works this paper leans on

63 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Enhancing multiple-choice question answering through sequential fine-tuning and curriculum learning strategies

    Gulsum Yigit and Mehmet Fatih Amasyali. Enhancing multiple-choice question answering through sequential fine-tuning and curriculum learning strategies. Knowledge and Information Systems, 65(11):5025–5042, 2023

    work page 2023

  2. [2]

    A seq2seq-based approach to question answering over knowledge bases

    Linjuan Wu, Peiyun Wu, and Xiaowang Zhang. A seq2seq-based approach to question answering over knowledge bases. In Semantic Technology: 9th Joint International Conference, JIST 2019, Hangzhou, China, November 25–27, 2019, Revised Selected Papers 9, pages 170–181. Springer, 2020

    work page 2019

  3. [3]

    Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension

    Anna Rogers, Matt Gardner, and Isabelle Augenstein. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension. ACM Computing Surveys, 55:1 – 45, 2021

    work page 2021

  4. [4]

    Ask me: A question answering system via dynamic memory networks

    Gulsum Yigit and Mehmet Fatih Amasyali. Ask me: A question answering system via dynamic memory networks. 2019 Innovations in Intelligent Systems and Applications Conference (ASYU), pages 1–5, 2019

    work page 2019

  5. [5]

    A goal-driven tree-structured neural model for math word problems

    Zhipeng Xie and Shichao Sun. A goal-driven tree-structured neural model for math word problems. InInternational Joint Conference on Artificial Intelligence, 2019

    work page 2019

  6. [6]

    Teacher-student networks with multiple decoders for solving math word problem

    Jipeng Zhang, Roy Ka-Wei Lee, Ee-Peng Lim, Wei Qin, Lei Wang, Jie Shao, and Qianru Sun. Teacher-student networks with multiple decoders for solving math word problem. In International Joint Conference on Artificial Intelligence, 2020

    work page 2020

  7. [7]

    Mwp-bert: Numeracy-augmented pre-training for math word problem solving

    Zhenwen Liang, Jipeng Zhang, Lei Wang, Wei Qin, Yunshi Lan, Jie Shao, and Xiangliang Zhang. Mwp-bert: Numeracy-augmented pre-training for math word problem solving. In NAACL-HLT, 2021

    work page 2021

  8. [8]

    Template- based math word problem solvers with recursive neural networks

    Lei Wang, Dongxiang Zhang, Jipeng Zhang, Xing Xu, Lianli Gao, Bing Tian Dai, and Heng Tao Shen. Template- based math word problem solvers with recursive neural networks. In AAAI Conference on Artificial Intelligence, 2019

    work page 2019

Show all 63 references
  1. [9]

    Graph-to-tree learning for solving math word problems

    Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. Graph-to-tree learning for solving math word problems. In Annual Meeting of the Association for Computational Linguistics, 2020

    2020

  2. [10]

    Solving math word problems with multi-encoders and multi-decoders

    Yibin Shen and Cheqing Jin. Solving math word problems with multi-encoders and multi-decoders. In Interna- tional Conference on Computational Linguistics, 2020

    2020

  3. [11]

    Some challenges and grand challenges for computational intelligence

    Edward A Feigenbaum. Some challenges and grand challenges for computational intelligence. Journal of the ACM (JACM), 50(1):32–40, 2003. 24 A PREPRINT - SEPTEMBER 5, 2025

    2003

  4. [12]

    Mathbot–a deep learning based elementary school math word problem solver

    Anish Kumar Nayak, Rajeev Patwari, and Viswanathan Subramanian. Mathbot–a deep learning based elementary school math word problem solver. https://www. semanticscholar. org/paper/MathBot-% E2, 80:7

  5. [13]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  6. [14]

    Scaling language models: Methods, analysis & insights from training gopher

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  7. [15]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  8. [16]

    List of languages by total number of speakers, 2024

    Wikipedia. List of languages by total number of speakers, 2024. Accessed: 2024-11-22

    2024

  9. [17]

    Gpt-4o: A new multimodal ai model, 2024

    OpenAI. Gpt-4o: A new multimodal ai model, 2024. Accessed: 2024-06-14

    2024

  10. [18]

    Gpt-3.5 turbo model documentation, 2023

    OpenAI. Gpt-3.5 turbo model documentation, 2023. Accessed: 2024-06-14

    2023

  11. [19]

    Llama 3: Open and efficient foundation language models, 2024

    Meta AI. Llama 3: Open and efficient foundation language models, 2024. Accessed: 2024-06-14

    2024

  12. [20]

    Deepseek platform documentation, 2025

    DeepSeek. Deepseek platform documentation, 2025. Accessed: 2025-03-14

    2025

  13. [21]

    Qwen chat interface, 2025

    Qwen. Qwen chat interface, 2025. Accessed: 2025-03-14

    2025

  14. [22]

    Empowering bengali education with ai: Solving bengali math word problems through transformer models

    Jalisha Jashim Era, Bidyarthi Paul, Tahmid Sattar Aothoi, Mirazur Rahman Zim, and Faisal Muhammad Shah. Empowering bengali education with ai: Solving bengali math word problems through transformer models. In 2024 27th International Conference on Computer and Information Techno...

    2024

  15. [23]

    Deep neural solver for math word problems

    Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 845–854, 2017

    2017

  16. [25]

    Variational neural machine translation

    Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. Variational neural machine translation. arXiv preprint arXiv:1605.07869, 2016

    2016 arXiv

  17. [26]

    Neural math word problem solver with reinforcement learning

    Danqing Huang, Jing Liu, Chin-Yew Lin, and Jian Yin. Neural math word problem solver with reinforcement learning. In Proceedings of the 27th International Conference on Computational Linguistics, pages 213–223, 2018

    2018

  18. [27]

    Graph-to-tree learning for solving math word problems

    Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. Graph-to-tree learning for solving math word problems. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3928–3937, 2020

    2020

  19. [28]

    Graph-to-tree neural networks for learning structured input-output translation with applications to semantic parsing and math word problem

    Shucheng Li, Lingfei Wu, Shiwei Feng, Fangli Xu, Fengyuan Xu, and Sheng Zhong. Graph-to-tree neural networks for learning structured input-output translation with applications to semantic parsing and math word problem. arXiv preprint arXiv:2004.13781, 2020

    2004 arXiv

  20. [29]

    Solving math word problems with multi-encoders and multi-decoders

    Yibin Shen and Cheqing Jin. Solving math word problems with multi-encoders and multi-decoders. InProceedings of the 28th International Conference on Computational Linguistics, pages 2924–2934, 2020

    2020

  21. [31]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  22. [32]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  23. [33]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  24. [34]

    Transformers as soft reasoners over language

    Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. arXiv preprint arXiv:2002.05867, 2020

    2002 arXiv

  25. [35]

    Rulebert: Teaching soft rules to pre-trained language models

    Mohammed Saeed, Naser Ahmadi, Preslav Nakov, and Paolo Papotti. Rulebert: Teaching soft rules to pre-trained language models. arXiv preprint arXiv:2109.13006, 2021. 25 A PREPRINT - SEPTEMBER 5, 2025

    2021 arXiv

  26. [36]

    Explainable multi-hop verbal reasoning through internal monologue

    Zhengzhong Liang, Steven Bethard, and Mihai Surdeanu. Explainable multi-hop verbal reasoning through internal monologue. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 122...

    2021

  27. [37]

    Mwp-bert: Numeracy-augmented pre-training for math word problem solving

    Zhenwen Liang, Jipeng Zhang, Lei Wang, Wei Qin, Yunshi Lan, Jie Shao, and Xiangliang Zhang. Mwp-bert: Numeracy-augmented pre-training for math word problem solving. arXiv preprint arXiv:2107.13435, 2021

    2021 arXiv

  28. [38]

    Math word problem solving by generating linguistic variants of problem statements

    Syed Rifat Raiyan, Md Nafis Faiyaz, Shah Md Jawad Kabir, Mohsinul Kabir, Hasan Mahmud, and Md Kamrul Hasan. Math word problem solving by generating linguistic variants of problem statements. arXiv preprint arXiv:2306.13899, 2023

    2023 arXiv

  29. [39]

    Translating a math word problem to an expression tree

    Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. Translating a math word problem to an expression tree. arXiv preprint arXiv:1811.05632, 2018

    2018 arXiv

  30. [40]

    A goal-driven tree-structured neural model for math word problems

    Zhipeng Xie and Shichao Sun. A goal-driven tree-structured neural model for math word problems. In Ijcai, pages 5299–5305, 2019

    2019

  31. [41]

    Warm: A weakly (+ semi) supervised model for solving math word problems

    Oishik Chatterjee, Isha Pandey, Aashish Waikar, Vishwajeet Kumar, and Ganesh Ramakrishnan. Warm: A weakly (+ semi) supervised model for solving math word problems. arXiv preprint arXiv:2104.06722, 2021

    2021 arXiv

  32. [42]

    On the advance of making language models better reasoners, 2022b

    Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners, 2022b. URL https://arxiv.org/abs/2206.02336

    arXiv

  33. [43]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  34. [44]

    Can rationalization improve robustness? arXiv preprint arXiv:2204.11790, 2022

    Howard Chen, Jacqueline He, Karthik Narasimhan, and Danqi Chen. Can rationalization improve robustness? arXiv preprint arXiv:2204.11790, 2022

    2022 arXiv

  35. [45]

    Training classifiers with natural language explanations

    Braden Hancock, Martin Bringmann, Paroma Varma, Percy Liang, Stephanie Wang, and Christopher Ré. Training classifiers with natural language explanations. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2018, page 1884. NIH Public Ac...

    2018

  36. [46]

    Benefits of intermediate annotations in reading comprehension

    Dheeru Dua, Sameer Singh, and Matt Gardner. Benefits of intermediate annotations in reading comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5627–5634, 2020

    2020

  37. [47]

    annotator rationales

    Omar Zaidan, Jason Eisner, and Christine Piatko. Using “annotator rationales” to improve machine learning for text categorization. In Human language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of...

    2007

  38. [48]

    Refining language models with compositional explanations

    Huihan Yao, Ying Chen, Qinyuan Ye, Xisen Jin, and Xiang Ren. Refining language models with compositional explanations. Advances in Neural Information Processing Systems, 34:8954–8967, 2021

    2021

  39. [49]

    When can models learn from explanations? a formal framework for understanding the roles of explanation data

    Peter Hase and Mohit Bansal. When can models learn from explanations? a formal framework for understanding the roles of explanation data. arXiv preprint arXiv:2102.02201, 2021

    2021 arXiv

  40. [50]

    Dream: Uncovering mental models behind language models

    Yuling Gu, Bhavana Dalvi Mishra, and Peter Clark. Dream: Uncovering mental models behind language models. NAACL, 2022

    2022

  41. [51]

    Learning with latent language

    Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. arXiv preprint arXiv:1711.00482, 2017

    2017 arXiv

  42. [52]

    e-snli: Natural language inference with natural language explanations

    Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. Advances in Neural Information Processing Systems, 31, 2018

    2018

  43. [53]

    Explain yourself! leveraging language models for commonsense reasoning

    Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361, 2019

    1906 arXiv

  44. [54]

    Star: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    2022

  45. [55]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  46. [56]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  47. [57]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. 26 A PREPRINT - SEPTEMBER 5, 2025

    2021 arXiv

  48. [58]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021

    arXiv

  49. [59]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  50. [60]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  51. [61]

    Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunk- umar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv p...

    2022 arXiv

  52. [62]

    Mega: Multilingual evaluation of generative ai

    Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, et al. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303.12528, 2023

    2023 arXiv

  53. [63]

    Fine-tuning gpt-3.5 models, 2023

    OpenAI. Fine-tuning gpt-3.5 models, 2023. Accessed: 2024-06-14

    2023

  54. [64]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  55. [65]

    Lora: Low-rank adaptation of large language models, 2024

    Hugging Face. Lora: Low-rank adaptation of large language models, 2024. Accessed: 2024-06-14. 27 A PREPRINT - SEPTEMBER 5, 2025 A Prompting in OpenAI Playground and their Output examples along with Response of Llama 3.3 on Bengali Math Word Problems Figure 15: SOMADHAN dataset...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.