{"id":"a0c23fb4-8850-4b36-bc85-a5eb6ec696c8","arxiv_id":"2501.04425","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Fine-tuning Qwen2.5-7B with translated math datasets plus retrieval and tool-integrated reasoning yields 71/100 on a Bangla math olympiad test set, versus 77/100 for a larger base model.","lead":"This paper tests fine-tuned and prompted large language models on Bangla math olympiad problems, combining retrieval-augmented generation with tool-based reasoning. The best result, 77 out of 100 on the test set, came from a larger non-fine-tuned model, while fine-tuning a 7B model improved its score by about one point.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 70-to-71 improvement from fine-tuning plus RAG is a one-point shift on a 100-problem test set with no error bars or control; the central empirical claim is not yet established.","rationale":"The reader's weakest assumption (translation quality) is a genuine concern, but I see a more immediately load-bearing problem: the headline numerical gain is one point on a 100-problem benchmark, with no error bars, no repeated runs, and no controlled baseline under identical evaluation conditions. For the paper's central empirical claim to be true, the 70-to-71 difference must be distinguishable from sampling noise and attributable to the stated intervention. Table 6 does not demonstrate either. The conclusion's own language ('slightly better result', RAG 'does not improve up to satisfactory level') suggests the authors themselves do not assert a robust effect. The consistent ordering of model sizes (32B > 7B) is some evidence for the parameter-scaling sub-claim, but it is not the paper's novel contribution and is also based on single runs. I am not rejecting the work; the experiments are suggestive and the benchmark is useful. The right outcome is to require a minimal statistical/experimental control before accepting the central claim, which matches the reader's CONDITIONAL verdict. Hence I recommend UNCHANGED rather than a harsher verdict.","tokens_in":6766,"tokens_out":4177,"duration_ms":42702,"concrete_test":"Rerun the three decisive Table 6 configurations—TIR only, TIR+RAG, and CoT+TIR+RAG—on the same 100-problem test set at least 10 times each with different random seeds, under identical TIR agent counts, depth, and RAG retrieval settings. Report the per-run scores, mean, standard deviation, and a paired significance test (e.g., McNemar's test on per-problem correctness between TIR and TIR+RAG). Also run the base, non-fine-tuned Qwen2.5-7B-Instruct with the exact same RAG + 5-agent + 5-depth protocol used for the fine-tuned models. If the 70-vs-71 difference is not consistently reproduced, or if confidence intervals overlap, the claim that RAG and curated datasets improve performance is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the integrated fine-tuning/RAG pipeline lifts Qwen2.5-7B-Instruct to 71/100 and that retrieval-augmented methods and curated datasets improve performance—rests on one-point differences in Table 6. TIR alone scores 70 (whether trained for 1 or 3 epochs), while TIR+RAG and CoT+TIR+RAG both score 71. With only 100 test problems, a one-point difference is a single question, and the paper reports no repeated trials, confidence intervals, or significance tests. There is also no controlled comparison of the base Qwen2.5-7B-Instruct under the same RAG/TIR evaluation protocol: Table 5 reports 70 for the non-fine-tuned model in English and 68 in Bangla, but with different sampling depths (5 vs 9) and no explicit statement that RAG is active, so the contribution of fine-tuning cannot be isolated from prompting, RAG, or self-consistency. The conclusion itself concedes only a 'slightly better result' and admits that RAG 'does not improve up to satisfactory level,' undercutting the abstract's 'crucial discoveries.' A related but secondary concern is the lack of any human evaluation of the gemini-1.5-flash translations (Section 2.2); if translations are noisy, the fine-tuning data are corrupted, but the primary issue is that even with perfect translations the reported effect is not shown to be distinguishable from noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes an end-to-end system for solving Bangla AI Math Olympiad problems. The authors evaluate several LLMs (GPT-4o, DeepSeek-Math, NuminaMath, Qwen2.5 variants), fine-tune Qwen2.5-7B-Instruct on translated Numina TIR and CoT datasets plus synthetic GPT-4o-generated data, and combine the fine-tuned model with retrieval-augmented generation (RAG), tool-integrated reasoning (TIR) agents, and self-consistency voting. The main reported results are 71/100 on the Bangla AI Math test set for the fine-tuned Qwen2.5-7B-Instruct pipeline, 77/100 for Qwen2.5-32B-Instruct-AWQ without fine-tuning, and the conclusion that retrieval-augmented methods and well-curated datasets improve performance in multilingual specialized domains. The paper also reports exploratory observations about tailored prompting, problem categorization, and prompt phrasing.","tokens_in":7131,"tokens_out":2899,"duration_ms":27901,"significance":"If the reported effects were statistically reliable, the work would be a useful case study in adapting open-weight LLMs to a low-resource language for Olympiad-level mathematical reasoning, and the released datasets and fine-tuned models would be of practical value to the Bengali NLP and AI4Math communities. The paper also provides a reproducible recipe (fine-tuning on translated Numina data, synthetic augmentation, RAG-based few-shot retrieval, TIR with self-consistency) that could inform future multilingual reasoning systems. However, the significance is currently limited by the lack of statistical grounding: the central improvements are one-point differences on a 100-problem test set, and the evaluation protocol does not isolate the contribution of fine-tuning from prompting, RAG, or self-consistency. The exploratory observations about prompt phrasing and politeness are interesting but anecdotal.","major_comments":[{"comment":"The central claim that fine-tuning plus RAG improves accuracy rests on a one-point difference (70 to 71) on a 100-problem test set. The table shows TIR alone achieving 70 with both 1 and 3 epochs, while TIR+RAG and CoT+TIR+RAG achieve 71; CoT+TIR without RAG achieves 68. A single test question separates these configurations, and the paper reports no repeated trials, confidence intervals, or significance tests. Under standard binomial sampling, a one-point difference between 70/100 and 71/100 is entirely consistent with noise. The abstract's phrasing 'Crucial discoveries' and the conclusion's statement that 'retrieval-augmented methods and well-curated datasets improve model performance' overstate what this table can support.","section":"Table 6"},{"comment":"There is no controlled comparison that isolates the effect of fine-tuning. Table 5 reports Qwen2.5-7B-Instruct without fine-tuning at 70/100 in English (5 samples, depth 5) and 68/100 in Bangla (50 samples, depth 9). Table 6 reports fine-tuned models at 68-71/100 but does not state the problem language, number of samples, reasoning depth, or whether RAG/TIR agents were used in the base-model rows. Consequently, the difference between the fine-tuned and base models could be due to the evaluation protocol (e.g., more samples or different depth) rather than to the fine-tuning itself. The paper should report all configurations under identical inference settings, including the base model with the same RAG/TIR/self-consistency pipeline.","section":"Sections 2.5 and 3; Tables 5 and 6"},{"comment":"The load-bearing assumption that the translated Numina TIR and CoT datasets preserve mathematical content and step-by-step solution structure is unvalidated. The text says 'these were translated using gemini-1.5-flash model into Bangla for fine-tuning,' but no human evaluation, back-translation check, or sample quality assessment is provided. If the translations contain errors, the fine-tuned model learns from corrupted examples, and the measured scores on the Bangla test set cannot be interpreted as evidence of Bangla mathematical reasoning ability. At minimum, the authors should provide a human-annotated sample of translation quality and a discussion of translation-induced errors.","section":"Section 2.2"},{"comment":"The claim that 'models with larger parameters perform better' is confounded. Qwen2.5-32B-Instruct-AWQ is evaluated with 10 samples and depth 4 in Bangla, while Qwen2.5-7B-Instruct is evaluated with 5 samples/depth 5 (English) and 50 samples/depth 9 (Bangla). The 32B model is also not fine-tuned or run through the same RAG/TIR pipeline, and it uses AWQ quantization. The 77 vs 70 comparison therefore conflates parameter count, sampling budget, reasoning depth, quantization, and language. A fair scaling comparison would hold the inference protocol fixed across model sizes.","section":"Section 2.1 and Table 5"},{"comment":"The conclusion states that 'incorporating RAG which involves keyword search based similarity does not improve up to satisfactory level,' and that fine-tuning with augmented datasets provides 'comparatively a slight better result.' This directly qualifies, and in the case of RAG partially contradicts, the abstract's 'Crucial discoveries' and the Section 3 statement that 'retrieval-augmented methods and well-curated datasets improve model performance.' The manuscript should present a single consistent interpretation of the evidence, ideally with error bars, rather than offering strong claims in the abstract and weak caveats in the conclusion.","section":"Section 5 versus Abstract and Section 3"}],"minor_comments":[{"comment":"There are numerous typographical and formatting issues, including 'T able' in several places, 'Qwen2.5-32B-Instruct-A WQ' instead of 'AWQ', and 'Gemini-1.5-flash' capitalization. A thorough proofread is needed.","section":"Throughout"},{"comment":"The dataset and model availability sections say 'can be downloaded from here' without visible URLs. The paper should include the actual repository or Hugging Face links.","section":"Section 6 and 7"},{"comment":"The architecture figure is referenced but the caption and in-text description do not explain several components (e.g., how keywords are derived, how the RAG similarity search is implemented, and how the voting mechanism aggregates agents). Adding a precise description would improve reproducibility.","section":"Section 2.6 and Figure 1"},{"comment":"The Numina-7B-TIR results vary from 50 to 64 with different numbers of agents and languages, but there is no discussion of variance or why 10 agents in English outperforms all other configurations. A brief explanation of these differences would be helpful.","section":"Table 4"},{"comment":"The fine-tuning procedure reports using 250,000 of the 800,000 CoT examples but does not specify how the subset was sampled or whether this choice was ablated. Similarly, the number of synthetic problems and the augmentation ratio are not quantified.","section":"Section 2.5"}],"recommendation":"major_revision","confidential_remarks":"The paper reads more like a competition write-up or technical report than a full research paper: the empirical claims are plausible and the released artifacts may be useful, but the evidence is not yet at the level expected for a journal publication. The authors should be encouraged to add repeated runs with variance estimates, a controlled comparison with identical inference settings, and a translation-quality evaluation. The citation list is adequate and no attribution problems are apparent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely new contribution is the language domain: a Bangla math olympiad test set with scores for several LLM configurations. That fills a real gap, and the authors deserve credit for testing a public 100-problem benchmark rather than inventing their own. They also report a systematic set of configurations and make the fine-tuned models available, which is the kind of reproducibility-minded behavior we like to see.\n\nThe paper does several things well. The observation that the larger Qwen2.5-32B beats the 7B is confirmatory but useful. The practical notes on problem categorization, tailored prompts, and the value of English reasoning for Bangla problems are plausible and should help others working in low-resource math reasoning. The use of an external test set keeps the circularity risk low.\n\nThe soft spots are substantial. The central claim of improvement is a one-point gain (70 to 71 on 100 problems) with no repeated runs, confidence intervals, or significance tests. A single question makes that difference. The ablations also fail to isolate fine-tuning: the baseline in Table 5 uses different sample depths and no explicit RAG, while the fine-tuned rows mix RAG, self-consistency, and prompt changes, so the contribution of fine-tuning alone is not identifiable. The paper's own conclusion concedes that RAG 'does not improve up to satisfactory level,' which undercuts the abstract's 'crucial discoveries.' There is also no human evaluation of the gemini-1.5-flash translations used for fine-tuning; if those are noisy, the model trains on corrupted examples, though the headline issue remains the missing statistical support.\n\nWho should read this: anyone building Bangla or low-resource math datasets, and those interested in practical prompt engineering for multilingual LLMs. It is not a methodological advance, but it is a legitimate empirical data point with real limitations.\n\nI would send this to peer review: the benchmark and scores are worth scrutinizing. But I would ask the authors for repeated runs or confidence intervals, a cleaner ablation that separates fine-tuning from RAG and self-consistency, and a human check on translation quality. The abstract should be softened to match the conclusion.","headline":"Useful new Bangla math-olympiad scores, but the paper's headline gain is a one-point difference without error bars, so treat the improvement claim as unproven.","tokens_in":7614,"tokens_out":3015,"would_cite":false,"duration_ms":28479,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7B model fine-tuned with retrieval reaches 71/100 on Bangla math olympiad problems.","keywords":["Bangla math olympiad","large language models","fine-tuning","retrieval-augmented generation","tool-integrated reasoning","self-consistency","multilingual mathematical reasoning","data augmentation"],"falsifier":"A direct falsifier is a bilingual audit of a random sample of the translated training solutions: if a substantial fraction contains mathematical or arithmetic errors that change the answer, then the fine-tuned model is learning from corrupted examples and its test score does not measure Bangla reasoning.","tokens_in":6614,"feed_emoji":"🧮","tokens_out":8899,"duration_ms":75107,"temperature":0.7,"pith_summary":"This paper argues that a modestly sized open language model can be made to solve Bangla-language math olympiad problems by combining fine-tuning on translated competition-math datasets, synthetic data augmentation, retrieval-augmented few-shot examples, and tool-integrated reasoning with self-consistency voting. The authors report that this integrated pipeline lifts Qwen2.5-7B-Instruct to 71/100 on the Bangla AI math test set, and that a larger model, Qwen2.5-32B-Instruct-AWQ, reaches 77/100 without fine-tuning. A GPT-4o baseline reaches 130/209 on the 209-problem Bangla olympiad development set only when both problem and reasoning are in English, which underscores the language gap the paper targets. The claimed upshot is that retrieval-augmented methods and well-curated datasets improve model performance in multilingual, specialized domains, which matters for extending math AI beyond English.","feed_headline":"Fine-tuned 7B model scores 71/100 on Bangla math olympiad","feed_subtitle":"Retrieval-augmented fine-tuning lifts Qwen2.5-7B to 71/100; a 32B model hits 77/100 with no fine-tuning.","key_machinery":"The load-bearing mechanism is the pairing of retrieval-augmented generation (RAG) with tool-integrated reasoning (TIR) agents. Keywords are extracted from each Bangla problem, a similarity search over the training problems retrieves analogous solved examples, and those examples are inserted into the prompt as few-shot demonstrations. At inference, one or more TIR agents generate Python code, execute it in a REPL, and iterate when a solution fails; the final answer is chosen by self-consistency majority voting across the agents' outputs. The other pillar is the training-data mix: translated TIR and CoT solution sets plus synthetic paraphrased problems, which together teach the model Bangla mathematical phrasing and stepwise verification.","core_discovery":"The central claim is that a stepwise pipeline—fine-tune on translated tool-integrated reasoning (TIR) solutions, then on translated chain-of-thought (CoT) solutions, then augment with synthetic paraphrased problems, then retrieve similar solved problems at inference time and feed them as few-shot prompts—produces measurable gains for Bangla math olympiad solving. The fine-tuned 7B model scores 71/100 on the 100-problem test set, up from 70 without retrieval augmentation, while the same model family at 32B scores 77/100 without any fine-tuning. GPT-4o, by contrast, reaches 130/209 on the 209-problem development set only when both problem and reasoning are in English, and only with tool-integrated reasoning. The paper also finds that reasoning in English while the problem stays in Bangla generally beats reasoning entirely in Bangla, that tailored prompts per problem category help, and that multiple tool-integrated agents with majority voting beat single-shot answering. The authors conclude that retrieval-augmented generation and carefully curated training data are the effective ingredients for multilingual specialized math performance.","pith_inferences":["Because the training data were machine-translated without human verification, an obvious extension is to measure translation fidelity directly; noisy translations would mean the reported scores understate what a curated Bangla dataset could achieve.","The 6-point gap between the fine-tuned 7B model (71) and the zero-shot 32B model (77) is within typical run-to-run variance on a 100-question test, so the ranking claim should not be treated as settled.","A testable extension is to replace the keyword-based similarity search with a semantic retriever; the paper itself notes the retrieval step did not improve 'up to satisfactory level,' so a better retriever may close the gap.","The reported politeness effect (adding 'please' improves adherence) suggests instruction-following depends on prompt surface form, which could be studied systematically across models and languages."],"forward_implications":["A 7B-parameter open model can stay competitive with much larger proprietary systems on a low-resource-language math benchmark when it is given translated training data and retrieval support.","Retrieval augmentation contributes a small but consistent gain (70 to 71 on the 100-problem test set), and improving the retriever is the obvious next lever.","Parameter count remains decisive: the 32B model outperforms the fine-tuned 7B model with no fine-tuning at all, so scaling the same Bangla fine-tuning to 32B should yield the largest gains.","Category-specific prompts—brute-force hints for number theory, dynamic-programming hints for combinatorics—are a zero-cost accuracy lever that transfers to other languages and benchmarks.","Mixed-language inference, keeping the problem in Bangla but reasoning in English, outperforms fully Bangla reasoning in most reported configurations."],"supporting_citations":[{"why":"Supplies the large competition-math problem–solution corpus that is translated into Bangla and used for fine-tuning.","marker":"[12]"},{"why":"Defines the Qwen2.5 model family that the pipeline fine-tunes and the larger model it compares against.","marker":"[13]"},{"why":"Provides the self-consistency majority-voting method used to aggregate multiple agent solution paths.","marker":"[14]"},{"why":"Introduces the tool-integrated reasoning (TIR) agents that generate and execute Python code to verify solutions.","marker":"[15]"},{"why":"Grounds the retrieval-augmented generation step that retrieves similar solved problems for few-shot prompting.","marker":"[16]"},{"why":"Sets the baseline of a math-specialized 7B model, which underperformed and motivated the chosen architecture.","marker":"[11]"}],"fun_headline_variants":["Bangla math olympiad: fine-tuned 7B with RAG scores 71/100","RAG plus fine-tuning lifts Bangla math solver to 71/100","7B fine-tuned model cracks Bangla math at 71/100","Bangla math: fine-tuning plus RAG lifts 7B to 71/100","How fine-tuning and RAG help LLMs ace Bangla math olympiad"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the machine-translated versions of the competition-math training data preserve both the mathematical content and the step-by-step solution structure, and the paper reports no human evaluation of translation quality.","fun_headline_variants_meta":{"raw":{"variants":["Bangla math olympiad: fine-tuned 7B with RAG scores 71/100","RAG plus fine-tuning lifts Bangla math solver to 71/100","7B fine-tuned model cracks Bangla math at 71/100","Bangla math: fine-tuning plus RAG lifts 7B to 71/100","How fine-tuning and RAG help LLMs ace Bangla math olympiad"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001364,"raw_usage":{"total_tokens":5490,"prompt_tokens":862,"completion_tokens":4628,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":4519}},"tokens_in":478,"tokens_out":4628,"duration_ms":32905,"temperature":1.0,"reasoning_tokens":4519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:32:50.296638+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier is a bilingual audit of a random sample of the translated training solutions: if a substantial fraction contains mathematical or arithmetic errors that change the answer, then the fine-tuned model is learning from corrupted examples and its test score does not measure Bangla reasoning.","supporting_citations":[{"cited_title":"& Polu, S","cited_arxiv_id":null,"evidence_quote":"Supplies the large competition-math problem–solution corpus that is translated into Bangla and used for fine-tuning."},{"cited_title":"& Kiela, D","cited_arxiv_id":null,"evidence_quote":"Grounds the retrieval-augmented generation step that retrieves similar solved problems for few-shot prompting."}],"review_version":1}