{"id":"54431532-a4df-4fe3-b2c6-7468b166b519","arxiv_id":"2509.09101","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"TigerCoder is a fine-tuned Bangla code-generation LLM family that posts 0.82 Pass@1 on the new MBPP-Bangla benchmark, but its gains partly come from model selection on the test set.","lead":"This paper trains and releases TigerCoder, the first dedicated Bangla-language code generation models (1B and 9B), along with a 300K-instruction Bangla code dataset and a translated 974-problem benchmark. The 9B model reaches 0.75-0.82 Pass@1 on Bangla coding benchmarks, beating much larger general models, but the reported gains come with evaluation choices that need scrutiny.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MBPP-Bangla's non-Python reference/test porting is the load-bearing unknown; if the ported tests are unfaithful, the across-5-PL Pass@1 gains are unsupported.","rationale":"The reader's weakest assumption is exactly the load-bearing one. The paper's own contribution list (C2, C3) and abstract make 'across all 5 PLs' part of the headline; the evidence for the non-Python claims is entirely inside MBPP-Bangla's self-ported reference implementations. The porting step is described only in Appendix D and uses an uncited TRANSCODER-ST, with no test conversion protocol. This is not an outside-consensus disagreement; it is an internal gap between the claim of 'expert-validated cross-language consistency' (Table 2) and the reported procedure. If the released code passes an independent audit, the concern disappears. Model selection on the test benchmarks (Figure 2) and missing sampling parameters are real but secondary: they affect the magnitude and reproducibility of the gains, not whether the benchmark measures what it claims. The paper has genuine strengths: the RQ2 negative result is a clean finding, the datasets are a resource, and the models are open-sourced. Those justify conditional acceptance, not rejection, pending benchmark validation.","tokens_in":16955,"tokens_out":7775,"duration_ms":84650,"concrete_test":"Audit a random sample of 100 MBPP-Bangla tasks from the released GitHub repository: (1) independently back-translate the Bangla prompts to English and compare semantic equivalence to the original MBPP prompts; (2) compile and run the released Java, C++, JavaScript, and Ruby reference solutions against a language-native test suite derived from the original MBPP test cases (translating assertion semantics, not just I/O). If >5% of prompts are semantically inequivalent or >5% of non-Python references fail their tests, MBPP-Bangla is unreliable and the cross-5-PL gains should be recomputed on a validated subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—TigerCoder achieving 11–18% Pass@1 gains over baselines across all 5 PLs—rests on the validity of MBPP-Bangla, a benchmark the authors constructed. Appendix D Step 4 states that Python solutions were ported to Java/C++/JS/Ruby via TRANSCODER-ST and then manually hardened 'until all variants satisfy the original tests.' However, the paper never specifies how the original Python-only test cases were converted to each target language, nor whether the 'original tests' are executed through a language-agnostic harness. The entire multi-language evaluation in Appendix E therefore depends on an undocumented, single-verifier porting process. If the ported reference solutions are incorrect, or if the tests are silently weakened (e.g., only checking I/O behavior rather than logic), then the reported 0.82 Pass@1 and the Appendix E deltas are not comparable across languages and the 'across all 5 PLs' contribution is unsupported. The Python-only results might survive, but the abstract's unqualified 11–18% claim would still be overbroad.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses Bangla code generation, a low-resource setting, and makes three main contributions: (1) MBPP-Bangla, a 974-problem benchmark with Bangla prompts and reference solutions in Python, Java, JavaScript, Ruby, and C++; (2) three 100K-pair Bangla code-instruction datasets (self-instruct, synthetic, and machine-translated); and (3) TigerCoder 1B and 9B models obtained by fine-tuning TigerLLM on combinations of those datasets. The authors also investigate two research questions: whether multilingual LLMs lose code-generation quality when prompted in Bangla instead of English (RQ1), and whether a simple Bangla-to-English machine-translation step helps (RQ2). Evaluation on mHumanEval-Bangla and MBPP-Bangla reports TigerCoder-9B reaching 0.82 Pass@1 on MBPP-Bangla and claims 11-18% gains over the strongest prior baseline, with similar cross-language gains in Appendix E. All models, datasets, and benchmarks are promised to be open-sourced.","tokens_in":17276,"tokens_out":5601,"duration_ms":58725,"significance":"If the results hold, this is a useful contribution to low-resource code generation: it provides a dedicated Bangla code benchmark, instruction-tuning datasets, and openly described models, and the comparison against external baselines (Gemma, GPT, Gemini, TigerLLM) is a strength. The paper also includes a sensible two-RQ design separating language comprehension from code-generation ability. However, the headline numerical claims currently rest on two procedures that are not yet adequately established: the conversion of Python test cases to four other programming languages, and the use of the evaluation benchmarks to select the final dataset combination and hyperparameters. Because both procedures are load-bearing for the abstract claim of 11-18% gains across all five programming languages, the reported numbers should be treated as unverified until the authors supply a proper test-conversion protocol and held-out evaluation.","major_comments":[{"comment":"The five-language Pass@K results in Appendix E, and the abstract's 'across all 5 PLs' claim, rest entirely on the undocumented conversion of MBPP's Python test cases. Step 4 says the verifier patches ports 'until all variants satisfy the original tests,' but the paper never specifies how Python-specific tests/assertions are converted to Java, C++, JavaScript, and Ruby, whether the same edge cases are covered, or how equivalence is established. Without a concrete conversion protocol, examples, and a language-agnostic test harness, the non-Python numbers (Tables 11-14) may be measuring weaker or I/O-only behavior, making the reported 0.82 Pass@1 and the cross-PL deltas non-comparable. Please provide the full protocol, release the harness, and ideally include an independent re-run of a sample of ported tests.","section":"Appendix D, Step 4; Section 3"},{"comment":"The model-selection procedure appears to use the evaluation benchmarks as the validation set. Figure 2 chooses the dataset combination based on mHumanEval and MBPP performance, and Table 6 reports 'empirically selected hyperparameters'; Table 7 then reports final scores on the same two benchmarks. This is a form of test-set fitting, so the reported 11-18% gains are optimistic and do not estimate generalization to unseen Bangla coding problems. Please hold out a subset of MBPP-Bangla and mHumanEval-Bangla for final evaluation, or at minimum report all configurations explored and clearly label Figure 2 as validation-only with independent test results.","section":"Section 7, Figure 2, Table 6"},{"comment":"No sampling details accompany any Pass@K estimate. The paper defines the formula but does not report n (number of sampled programs), temperature, maximum tokens, decoding scheme, random seeds, or repetitions. The metric is a point estimate with nontrivial variance at moderate n, and several headline deltas are only 4-5 points (e.g., Table 7: TigerCoder-1B 0.69 vs TigerLLM-1B 0.63/0.65 on mHumanEval; Table 3: TigerLLM-9B 0.61 vs Gemma-3 0.69 on MBPP-Bangla). Without variance estimates or confidence intervals, the word 'significant' in the abstract is unsupported. Please report n, temperature, standard errors or bootstrap confidence intervals, and reproducibility runs.","section":"Section 4, Pass@K metric; Tables 3 and 7"},{"comment":"No decontamination step is described between the 300K instruction pairs and the two evaluation benchmarks. MBPP-Bangla is derived from the public MBPP, and the instruction datasets include GPT-4o/Claude self-instruct generations and translated Evol-Instruct data; such corpora often contain HumanEval/MBPP-style problems. If any training prompt overlaps with an evaluation task, the corresponding Pass@1 is inflated. Please conduct and report n-gram overlap and embedding-similarity analyses between Bangla-Code-Instruct and mHumanEval-Bangla/MBPP-Bangla, and describe any filtering applied.","section":"Section 6; Section 3"}],"minor_comments":[{"comment":"Please fix typos and inconsistent notation: 'comapred' (Section 8), inconsistent use of 'JAVA' vs 'Java', 'Bangla-Instruct-Syn' vs 'Bangla-Code-Instruct-Syn', and 'mHumanEval-ben' vs 'mHumanEval-Bangla'.","section":"Throughout"},{"comment":"The NLLB entry is cited as 'NLLB ()' with no reference. TransCoder-ST (Appendix D, Step 4) is used without a citation. CometKiwi-22 QE is referenced to Rei et al. (2020), but that paper describes COMET, not the Kiwi-22 checkpoint; please update the references.","section":"Appendix C, Appendix D"},{"comment":"The columns 'Paper?' and 'Reprod.' use 'Y/X' without a legend. Clarify what each symbol means, and consider adding a column for code-generation task support.","section":"Table 1"},{"comment":"Hyperparameter selection reports 'Save Steps Varies' and provides no search range or early-stopping details. Include the ranges searched and the criterion used for selecting final hyperparameters so the tuning process is reproducible.","section":"Table 6"},{"comment":"The Limitations paragraph calls MBPP-Bangla 'Python-focused' although the benchmark includes five programming languages and the paper claims cross-language evaluation. Rephrase to avoid contradicting the benchmark description in Section 3.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The core idea is valuable and the open-resource plan is commendable, but the current manuscript overstates reliability. The two load-bearing issues—undocumented non-Python test conversion and model selection on the evaluation benchmarks—are fixable within the scope of a revision. The reader's concern about MBPP-Bangla porting is legitimate and should be addressed head-on, not in a sentence. I would be willing to review a revised version that includes a proper held-out evaluation, full test-conversion protocol, and sampling details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe real thing here is the resource pack: three 100K Bangla code-instruction datasets, an MBPP-Bangla benchmark with 974 problems, and two fine-tuned TigerCoder models, all open-sourced. That is a genuine first for Bangla, and the authors deserve credit for shipping data and code. The evaluation is broad, with a sensible set of baselines, and the RQ2 finding—machine-translating Bangla prompts to English does not help—is clean and useful.\n\nThe soft spots are real but not fatal. Figure 2 and Table 6 show that the final dataset combination and hyperparameters were selected on the exact benchmarks used for reporting, and there is no held-out validation. That makes the 11-18% gains optimistic, though not fake. Pass@K is also reported without variance, temperature, or sample counts, so the numbers are not independently reproducible.\n\nThe bigger concern is the multi-language porting in Appendix D. Step 4 says Python solutions were ported to Java, C++, JS, and Ruby via TRANSCODER-ST and then manually hardened until all variants satisfy the original tests. But the paper never says how the original Python test cases were converted to each target language, or how 'satisfy the original tests' is checked in a language-agnostic way. If those ported tests are weaker or non-equivalent, then the across-5-PL results in Appendix E—and the abstract's unqualified 11-18% claim—are unsupported. The Python-only results would still stand, so the core direction is probably right, but the claim is overbroad as written.\n\nMy take: this deserves a serious referee. The resource contribution is valuable, and the core method is standard fine-tuning—so the questions are about evaluation rigor, not novelty. A referee should ask for a held-out split, full sampling details, and a precise description of the test-porting pipeline. If the porting is done properly, this is a solid paper.\n\nSend it to review.","headline":"Real resource contribution with a plausible core result, but the headline numbers are softer than they look once you notice model selection on the test set and the under-specified multi-language test porting.","tokens_in":17690,"tokens_out":2607,"would_cite":true,"duration_ms":28674,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a compact, Bangla-only code model family—TigerCoder—achieves the best reported Pass@1 scores on Bangla code benchmarks across five programming languages, beating models up to 27 times its size.","keywords":["Bangla code generation","low-resource language models","instruction tuning","Pass@1","MBPP-Bangla","multilingual code evaluation","machine translation for code","TigerCoder"],"falsifier":"Inspect the released MBPP-Bangla records: if the non-Python reference solutions are paired with Python-specific test functions that were never semantically ported to each target language, or if re-running the ported solutions against the original MBPP Python test assertions fails for reasons other than trivial syntax, the multi-language Pass@K results are not measuring what they claim.","tokens_in":16877,"feed_emoji":"🐯","tokens_out":4035,"duration_ms":41666,"temperature":0.7,"pith_summary":"TigerCoder is the first family of code-generation LLMs built specifically for Bangla, in two sizes (1B and 9B). The paper's central claim is that fine-tuning a strong Bangla base model on 300K curated Bangla instruction–code pairs yields large Pass@1 gains—about 11–18 percentage points over prior multilingual and Bangla-specific systems—and that the 1B model already beats models 27 times larger. To support this, the authors construct MBPP-Bangla, a 974-problem benchmark with human-translated prompts and reference solutions in Python, Java, C++, JavaScript, and Ruby. They also show that machine-translating Bangla prompts to English does not recover the performance lost from Bangla prompting. If correct, this is evidence that high-quality targeted data can outweigh model scale for low-resource code generation.","feed_headline":"Bangla code model beats rivals 27x larger—and 9B hits 82% Pass@1","feed_subtitle":"TigerCoder-9B tops Bangla benchmarks in Python, C++, Java, JavaScript, and Ruby, proving small models can win with curated data.","key_machinery":"The operative mechanism is three 100K-pair instruction datasets—self-instructed, synthetic, and machine-translated—each filtered for syntax and execution validity, used to fine-tune the TigerLLM base. The evaluation anchor is MBPP-Bangla, a 974-task benchmark with independently human-translated Bangla prompts and expert-verified reference solutions in five languages, scored with Pass@K.","core_discovery":"On the paper's own terms, the discovery is that a small open model, fine-tuned on a purpose-built Bangla instruction corpus, can outperform both proprietary multilingual systems and all existing Bangla LLMs on Bangla code generation. The strongest result is TigerCoder-9B reaching 0.82 Pass@1 on MBPP-Bangla and 0.75 on mHumanEval-Bangla, with similar leads in C++, Java, JavaScript, and Ruby. The authors interpret this as demonstrating that curated, executable, language-specific instruction data is the decisive factor, challenging the assumption that scale alone drives code-generation quality.","pith_inferences":["A direct test of the data-quality thesis would be to train the same TigerLLM base on the same 300K pairs but with the Bangla instructions replaced by English translations; if Pass@1 drops, the benefit is linguistic, not merely instructional.","If the multi-language reference solutions are truly equivalent, the same benchmark could be used to study cross-lingual transfer and to expose whether models generate idiomatic code or merely token-mimicry.","The benchmark's portability hinges on the unstated test-conversion protocol; an explicit release of language-specific test harnesses would make the 0.82 number independently checkable.","The same curation template—native-speaker translation, multilingual expert verification, execution-validated instruction data—likely transfers to other under-resourced languages, with human verification costs as the main bottleneck."],"forward_implications":["A 1B model that surpasses systems 27 times its size suggests practical, low-cost paths to code assistants for other low-resource languages.","Machine translation of prompts is not a substitute for native-language instruction data in code generation; future MT pipelines must preserve code-specific keywords.","The combined SI+Syn+TE dataset recipe—diverse instruction sources with execution validation—appears to be the configuration yielding the largest gains.","MBPP-Bangla extends code-generation evaluation beyond Python, allowing per-language analysis in five PLs.","The reported 9B leadership makes smaller open checkpoints a viable alternative to proprietary APIs for Bangla coding tasks."],"fun_headline_variants":["TigerCoder: small Bangla code models beat 27x bigger rivals","Curated data helps tiny Bangla LLMs ace code generation","Open-source Bangla code LLMs outperform larger models","1B and 9B TigerCoder top Bangla code benchmarks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole evaluation stands on the assumption that MBPP-Bangla's translated prompts and its ported Java, C++, JavaScript, and Ruby solutions faithfully preserve the original Python tasks and tests; if those ported tests do not truly match the Python semantics, every Pass@K number loses its meaning.","fun_headline_variants_meta":{"raw":{"variants":["TigerCoder: small Bangla code models beat 27x bigger rivals","Curated data helps tiny Bangla LLMs ace code generation","Open-source Bangla code LLMs outperform larger models","1B and 9B TigerCoder top Bangla code benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1292,"prompt_tokens":694,"completion_tokens":598,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":523}},"tokens_in":438,"tokens_out":598,"duration_ms":7197,"temperature":1.0,"reasoning_tokens":523,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:40:18.544602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released MBPP-Bangla records: if the non-Python reference solutions are paired with Python-specific test functions that were never semantically ported to each target language, or if re-running the ported solutions against the original MBPP Python test assertions fails for reasons other than trivial syntax, the multi-language Pass@K results are not measuring what they claim.","supporting_citations":[],"review_version":1}