{"id":"2722ddd3-0974-4d65-a26c-d8af89eac843","arxiv_id":"2501.19085","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"No single technique boosts LLM code generation for low-resource languages; the best choice depends on model size, with in-context learning typically the safest.","lead":"This paper compares five techniques for improving LLM code generation in low-resource languages like R and Racket. It finds that fine-tuning helps small models, while in-context learning is a safer and cheaper choice for larger models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-epoch selection on the MultiPL-E test set inflates fine-tuning results and may distort the size-based technique ranking; this needs a fixed-epoch or validation-based reanalysis.","rationale":"I read the paper as a comparative empirical study whose load-bearing condition is that fine-tuning and in-context learning are compared fairly. The oracle epoch selection over the test benchmark violates that condition, and it is exactly the weakest assumption identified by the reader. The concern is concrete and located in the text (Section IV-C), and it is checkable because the authors state that all per-epoch results are in the replication package. I do not see a more load-bearing internal flaw: the limited number of languages and the use of default hyperparameters are acknowledged as external-validity and internal-validity limitations, respectively, rather than hidden assumptions that would invalidate the reported comparisons. The test-set peeking is real, but it is not automatically fatal: for the 33B model, fine-tuning is worse than baseline even after selecting the best epoch, and for the 1B model on Racket the reported gain is large enough that it may survive a fixed-epoch reanalysis. Therefore the appropriate outcome is the same conditional verdict: the paper's central claim is plausible and well documented, but it should not be accepted as definitive until the epoch-selection bias is removed or shown not to change the rankings. The reader's conditional verdict is unchanged.","tokens_in":18709,"tokens_out":6203,"duration_ms":66083,"concrete_test":"Using the replication package's per-epoch pass@1 values, redo the RQ2 comparison with a fixed epoch (e.g., epoch 1, or an epoch selected on a held-out validation split of MultiPL-T rather than on MultiPL-E). For each model and language, record whether fine-tuning and pre-training plus fine-tuning still beat baseline and the best in-context learning technique. Also tabulate the index of the best epoch: if the best epoch varies unsystematically (e.g., epoch 1 for R, epoch 3 for Racket) and fixed-epoch results erase several fine-tuning wins, the size-based conclusion in Section IV-D should be revised. If the fixed-epoch results preserve the same rankings, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the epoch-selection protocol in Section IV-C: 'For fine-tuned models, we evaluate each epoch on the MultiPL-E benchmark and only report the best model's results.' Because MultiPL-E is also the test set on which all final pass@1 numbers are computed, reporting the maximum over three epochs is a form of test-set peeking. It can only improve the apparent performance of fine-tuning relative to the baseline and to in-context learning, which are reported as single configurations. This matters because the paper's central size-based conclusion—fine-tuning is best for small models and degrades for 33B—depends on comparisons where fine-tuning is close to the baseline or to ICL, e.g., DeepSeek Coder 1B on R: 16.7 vs 13.9, and Code Llama 7B on R: 14.6 vs 13.9. If the best epoch is not stable across models and languages, or if a fixed epoch gives materially lower pass@1, the claimed advantage of fine-tuning for smaller models and the claimed 'no silver bullet' pattern would weaken. The 33B finding is less affected because fine-tuning is worse than baseline even after oracle epoch selection, but the small- and mid-size comparisons are not yet trustworthy. The authors do provide per-epoch results in the replication package, so this is checkable without re-running training.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of techniques for improving LLM code generation in low-resource programming languages. It first measures the performance gap between high-resource languages (Python, Java) and four low-resource languages (Julia, Lua, R, Racket) across six LLMs (RQ1), then compares, on R and Racket, three in-context-learning variants (translation examples, translation rules, few-shot) and two fine-tuning variants (code-generation fine-tuning, and code-translation pre-training followed by fine-tuning) across DeepSeek Coder 1B/7B/33B, Code Llama 7B/13B, and GitHub Copilot (RQ2). The main findings are that fine-tuning helps the 1B model most, in-context learning is a safe and cheap improvement for larger models, and fine-tuning degrades DeepSeek Coder 33B; overall, no single technique is best across all settings.","tokens_in":18953,"tokens_out":7591,"duration_ms":76877,"significance":"The study addresses a timely and practical question, and its design has notable strengths: 50 repetitions per problem, statistical testing with McNemar's test and Benjamini-Hochberg correction, multiple model families and sizes, and a replication package with per-epoch results. If the quantitative conclusions are robust, the paper provides actionable guidance: use fine-tuning for sub-7B models and in-context learning with translation examples for larger models; the negative result for 33B fine-tuning is particularly noteworthy. However, the epoch-selection protocol currently biases the fine-tuning numbers upward, so the headline size-based pattern is not yet established.","major_comments":[{"comment":"Section IV-C states: 'For fine-tuned models, we evaluate each epoch on the MultiPL-E benchmark and only report the best model's results.' Since MultiPL-E is also the test set on which all final pass@1 values are reported, selecting the best of three epochs on the test benchmark is a form of test-set peeking: it can only improve the apparent performance of fine-tuning relative to the baseline and to the in-context-learning techniques, which are each evaluated as a single configuration. This is load-bearing for the paper's central size-based conclusion. For example, in Table III the fine-tuning advantage for DeepSeek Coder 1B on R (16.7 vs 13.9 baseline) and for Code Llama 7B on R (14.6 vs 13.9) is small enough that a non-oracle epoch choice could change the qualitative ranking. I request a reanalysis using a fixed epoch (e.g., the final epoch, or an epoch selected on a held-out validation split of MultiPL-E), and that the per-epoch pass@1 values be reported in the paper or supplementary material so the sensitivity of the conclusions to the selection rule can be assessed. The 33B deterioration is less affected because it appears even under oracle selection, but the small- and mid-size comparisons must be rechecked.","section":"IV-C, Table III"},{"comment":"Beyond epoch selection, the comparison uses unequal configuration protocols: fine-tuning results are the best over three epochs, while baseline and in-context-learning results are single configurations, and hyperparameters (learning rate, batch size, sequence length, number of epochs) are fixed to defaults. Section V acknowledges the lack of hyperparameter tuning, but the abstract and conclusion state the size-based ranking without this caveat. Since low-resource fine-tuning is expensive, I do not require a full hyperparameter search, but the paper should either provide a sensitivity analysis (e.g., a few learning rates or an early-stopping check on a small validation set) or explicitly frame the conclusion as holding under default hyperparameters and with a fair epoch-selection protocol. Without this, the 'no silver bullet' claim is not fully supported.","section":"V (Internal validity), IV-B"}],"minor_comments":[{"comment":"The RQ1 baselines in Table I are computed on the 157 programs common to all languages, while the RQ2 baselines in Table III use the full 161 programs; this difference is only mentioned in Section IV-C and not in the table captions, which can confuse readers comparing the two tables.","section":"Tables I and III"},{"comment":"The 'pre-training' step is performed on already-pretrained models and is actually a second fine-tuning on a translation objective; the authors acknowledge this in the text, but the terminology in the abstract and in Table III ('Pre-training & Fine-tuning') may still mislead readers, so consider renaming it to something like 'translation fine-tuning + generation fine-tuning'.","section":"IV-A5"},{"comment":"The statement that 'all ORs reported in the text are statistically significant' cannot be checked from the paper because all 162 statistical tests are relegated to the replication package; I suggest including a compact table of adjusted p-values for the comparisons cited in the text, or at least for the best-performing technique versus baseline for each model and language in Table III.","section":"IV-D"},{"comment":"The paper initially classifies languages by GitHub repository counts and later states that repository count is not a good discriminator (e.g., Julia vs R); the text should reconcile these statements, for example by noting that repository counts are only a rough proxy and are not the sole determinant of model performance.","section":"III-A and III-E"}],"recommendation":"major_revision","confidential_remarks":"The core issue (test-set epoch selection) is checkable because per-epoch results are in the replication package; if the authors provide a fixed-epoch or validation-based reanalysis and the qualitative conclusions hold, the paper should be publishable. I would not reject on the current evidence, but I would not accept without the reanalysis. The workload for the revision is moderate and within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou can trust the paper's main practical message: if you want to improve code generation for a low-resource language without spending on fine-tuning, in-context learning with translation examples is a safe, cheap bet for models of 7B and up. Translation rules are weaker, and few-shot helps less consistently. That conclusion is supported by the data and is worth knowing.\n\nWhat is actually new here is the systematic comparison. They are the first, as far as I know, to put fine-tuning, few-shot prompting, translation-example prompts, translation-rule prompts, and a code-translation pre-training objective in the same harness across six models spanning 1B to 33B. The evaluation is careful: 50 repetitions, McNemar with Benjamini-Hochberg, and a replication package with per-epoch results. Give them credit for that.\n\nThe soft spot is real and it is exactly where the reader put it. In Section IV-C they say they evaluate each fine-tuning epoch on the MultiPL-E benchmark and report only the best. MultiPL-E is the same benchmark on which all final numbers are computed, so this is test-set peeking. It can only favor fine-tuning. The 33B degradation (fine-tuning worse than baseline) is robust because it appears even after oracle epoch selection, but the claimed advantage for the 1B and 7B models rests on comparisons that could shrink once you use a fixed epoch or a validation split. They shipped per-epoch results, so this is checkable without rerunning training; the revision should just do it.\n\nOther limitations are milder and mostly acknowledged: two low-resource languages, two model families, no hyperparameter tuning, three epochs. The translation pre-training did not help, which is a null result but an honest one. I would not call the scope a load-bearing flaw; it is enough for an empirical study.\n\nFor a reader, this is a useful paper if you work on code generation for niche languages or on practical LLM adaptation. It deserves a serious peer review, but I would make the revision conditional on the epoch-selection fix. If the fine-tuning numbers hold, the size-based pattern is a decent contribution; if they do not, the ICL-safe-bet message survives anyway.","headline":"Useful head-to-head comparison of boosters for low-resource code generation, but the test-set epoch picking means the fine-tuning results need a reanalysis before the size-based pattern is taken as fact.","tokens_in":19479,"tokens_out":3381,"would_cite":true,"duration_ms":32693,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across six LLMs, no single technique fixes low-resource code generation; translation-example prompting is the only reliably safe gain.","keywords":["code generation","low-resource programming languages","in-context learning","fine-tuning","large language models","pass@1","R","Racket"],"falsifier":"Re-run the boosting comparison with the fine-tuning training stop point selected on a validation subset split from the training data, and evaluate only that stop point on the 161 test problems. If fine-tuning no longer beats in-context learning for the 1B model, or if the 33B model's fine-tuning degradation disappears, the size-based conclusion is an artifact of test-set peeking.","tokens_in":18484,"feed_emoji":"💻","tokens_out":10840,"duration_ms":89677,"temperature":0.7,"pith_summary":"The paper asks whether any single technique can close the gap between LLM code generation in high-resource languages (Python, Java) and low-resource ones (R, Racket). Across six models—five openly released models from 1B to 33B parameters plus Copilot—it compares three in-context-learning prompts with two fine-tuning routes, measuring pass@1 (the fraction of single generations that pass unit tests) on the same 161 translated programming problems. Its central finding is that no technique dominates: fine-tuning gives the smallest model its largest gains, in-context learning becomes the stronger family as models grow, and fine-tuning the largest model drops performance below baseline. The practical conclusion is that prompting with Python-to-target translation examples is a cheap, broadly reliable default, while fine-tuning should be reserved for small models with enough data to update their weights.","feed_headline":"Fine-tuning helps small code models; prompting wins at scale","feed_subtitle":"Translation-example prompts lift R and Racket performance for every model size that can use them, at near-zero cost.","key_machinery":"The comparison rests on five treatments applied to the same underlying problem prompts. Three are in-context learning: a prompt with two Python-to-target translation examples, a prompt with hand-written mapping rules from Python syntax to the target, and a standard two-shot prompt with target-language implementation examples. Two are training-based: fine-tuning on the released R and Racket function datasets, and the same fine-tuning preceded by a pre-training phase that asks the model to translate Python functions into the target language. All are evaluated with pass@1 (one generation per problem, 50 repetitions, temperature 0.2) on the 161 translated HumanEval programs, and the central comparison is how the gap between the two families shifts with parameter count.","core_discovery":"The paper claims that model size, not the technique itself, determines which boosting strategy works for low-resource code generation. For the 1B model, fine-tuning (with or without a preceding translation pre-training phase) is the only family that substantially improves Racket generation, from 7.0% to 18.4% pass@1, and the same model barely moves under any in-context prompt. For 7B and 13B models the picture splits by language: in-context learning is ahead on R, fine-tuning is ahead on Racket. At 33B, fine-tuning degrades both languages relative to the unmodified baseline, while translation-example and few-shot prompts help; Copilot, which cannot be fine-tuned, also gains from these prompts. The paper's summary claim is that translation-example in-context learning is a safe bet across all models that can interpret complex prompts.","pith_inferences":["Editorial inference: the size-based ordering suggests a practical scaling rule — the amount of fine-tuning data needed to update a model usefully grows with its parameter count, so for a new low-resource language the first question is model size.","Editorial inference: because the paper chooses the best fine-tuning training stop point by looking at the test benchmark, the reported fine-tuning gains, especially for the 1B model, may be optimistic; a held-out validation split could change the ranking.","Editorial inference: translation-example prompting borrows knowledge from Python, so its success may depend on the target language resembling languages the model already knows; whether it helps a truly unseen language with very different syntax remains open.","Editorial inference: for closed commercial models that cannot be fine-tuned, translation-example prompting is the only tested boosting strategy, making it the obvious candidate for API-based low-resource code generation tools."],"forward_implications":["For a 1B model, fine-tuning on a low-resource language is the only tested route that produces large gains; users of such models should not expect prompting alone to move the metric much.","For 7B–13B models, the best choice depends on the language: in-context learning is more reliable for R, fine-tuning for Racket, so selecting a technique requires a language-specific check.","For models around 33B and above, fine-tuning on small low-resource datasets is risky; it can degrade code generation below the unmodified model, whereas translation-example prompting reliably improves it.","Translation-example prompting is the most reliable cheap option: for models above 1B, it improves over the baseline in every tested language, while the three in-context prompts are not all consistently helpful.","Adding a code-translation pre-training stage before fine-tuning does not consistently beat fine-tuning alone, so the extra training cost is not justified by these results."],"supporting_citations":[{"why":"supplies the fine-tuning datasets for R and Racket and the translation-based pipeline used to build them.","marker":"[2]"},{"why":"provides the benchmark of translated HumanEval programs and the pass@1 evaluation setup (temperature 0.2, 50 repetitions).","marker":"[19]"},{"why":"motivates few-shot in-context learning for low-resource and unseen languages, one of the three prompting techniques.","marker":"[5]"},{"why":"supplies the DeepSeek Coder model family, contributing the 1B, 7B, and 33B sizes that anchor the size-based comparison.","marker":"[6]"},{"why":"supplies the Code Llama model family, contributing the 7B and 13B sizes.","marker":"[7]"},{"why":"is the original set of hand-written problems whose translated subset forms the test benchmark.","marker":"[16]"},{"why":"supplies the rationale that a pre-training objective close to the downstream task should improve fine-tuning, the basis for the translation pre-training variant.","marker":"[29]"}],"fun_headline_variants":["No one-size-fits-all fix for low-resource code generation","Model size dictates best boost for niche programming languages","Fine-tune small, prompt large: low-resource code","Small models fine-tune, big models prompt for low-resource code","Translation prompts: safe bet for any model that can use them"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that scoring every fine-tuning training stop point on the test benchmark and keeping only the best score gives a fair comparison with prompting; if the stop point had to be chosen before seeing the test set, the apparent benefit of fine-tuning could shrink.","fun_headline_variants_meta":{"raw":{"variants":["No one-size-fits-all fix for low-resource code generation","Model size dictates best boost for niche programming languages","Fine-tune small, prompt large: low-resource code","Small models fine-tune, big models prompt for low-resource code","Translation prompts: safe bet for any model that can use them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000929,"raw_usage":{"total_tokens":4035,"prompt_tokens":1058,"completion_tokens":2977,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":2891}},"tokens_in":674,"tokens_out":2977,"duration_ms":18244,"temperature":1.0,"reasoning_tokens":2891,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T21:21:38.990197+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the boosting comparison with the fine-tuning training stop point selected on a validation subset split from the training data, and evaluate only that stop point on the 161 test problems. If fine-tuning no longer beats in-context learning for the 1B model, or if the 33B model's fine-tuning degradation disappears, the size-based conclusion is an artifact of test-set peeking.","supporting_citations":[{"cited_title":"Knowledge transfer from high-resource to low-resource pr ogramming languages for code llms,","cited_arxiv_id":null,"evidence_quote":"supplies the fine-tuning datasets for R and Racket and the translation-based pipeline used to build them."},{"cited_title":"Multipl-e: a scalable and polyglot approach to benchmarki ng neural code generation,","cited_arxiv_id":null,"evidence_quote":"provides the benchmark of translated HumanEval programs and the pass@1 evaluation setup (temperature 0.2, 50 repetitions)."},{"cited_title":"Multi-lingual evaluat ion of code generation models,","cited_arxiv_id":null,"evidence_quote":"motivates few-shot in-context learning for low-resource and unseen languages, one of the three prompting techniques."},{"cited_title":"PEGASUS: pre- training with extracted gap-sentences for abstractive summarizati on,","cited_arxiv_id":null,"evidence_quote":"supplies the rationale that a pre-training objective close to the downstream task should improve fine-tuning, the basis for the translation pre-training variant."}],"review_version":1}