{"id":"290ece26-5338-49e6-b711-d72778a8e335","arxiv_id":"2505.06004","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Gemma 9B beats 16 other small open LLMs on multilingual grammar correction across English, German, Italian, and Swedish under automatic referenceless evaluation.","lead":"The authors tested 17 small open-source language models (up to 9B parameters) on correcting grammar mistakes in English, German, Italian, and Swedish using one model per language, and ranked them with automatic scoring. They report that Google's Gemma 9B is the best overall choice, with six models capable of improving correctness in all four languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gemma 9B's 'best overall' ranking collapses if LanguageTool's error counts are not a language-fair correctness measure; the rank gap over Qwen 2.5 is carried almost entirely by LT.","rationale":"The reader's verdict was CONDITIONAL, with the weakest assumption being that LanguageTool error counts are a valid, language-fair proxy for grammatical correctness. My stress-test confirms and sharpens this concern: the top ranking is not merely 'supported by' LT, it is numerically dependent on LT. Removing or down-weighting LT flips the winner from Gemma 9B to Qwen 2.5. This makes the central claim conditional on exactly the metric whose validity the paper itself flags as limited. I did not find a stronger internal inconsistency: the experiments are transparent, the outputs are released, the stability check is reasonable, and the Borda aggregation is clearly described. The main unresolved risk is external validity of the evaluation framework, not internal logic. Therefore the reader's CONDITIONAL verdict is appropriate, and my read does not change it. A human evaluation on a sample of outputs, or even a sensitivity analysis dropping LT, would settle whether the condition is met. I agree with the reader's identification of the load-bearing concern.","tokens_in":21638,"tokens_out":4823,"duration_ms":45381,"concrete_test":"Conduct a human evaluation on a stratified random sample of outputs from the two top-ranked models (e.g., 100 sentences per language for EN, DE, IT, SV, from Gemma 9B and Qwen 2.5, using prompt P3). Have at least two native speakers per language rate each output for grammatical correctness, meaning preservation, and minimal-edit quality. If Qwen 2.5's human-rated grammatical correctness is not significantly worse than Gemma 9B's, or if Gemma's advantage appears only for LanguageTool-detectable error types, then the 'best overall' claim should be revised. A cheap complementary check: recompute the Table 5 Borda aggregation with the LT column excluded; if Qwen 2.5 moves to rank 1, the headline claim is contingent on LT's validity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Gemma 9B is 'the best model overall for the GEC task' (Section 6) rests on the Borda aggregation in Table 5, which gives equal weight to four perspectives: LanguageTool correctness, Correct-F1, semantic similarity, and syntactic similarity. From Table 6, Gemma 9B ranks 1st on LT and Correct-F1, tied-1st on semantic similarity, but only 9th on Levenshtein; Qwen 2.5 ranks 8th on LT but 1st on syntactic similarity, tied-1st on semantic similarity, and 2nd on Correct-F1. Recomputing the second-level Borda sums: Gemma 9B scores 7.67 vs. Qwen 2.5's 13.0, so Gemma wins. But if the LT column is removed, the three-metric Borda sum becomes Gemma 6.67 vs. Qwen 5.0, making Qwen the winner. Thus the headline ranking is almost entirely decided by the LanguageTool correctness metric, specifically the 1-vs-8 LT rank difference. The paper's own Limitations admit that 'selecting LanguageTool as the source of error information may introduce some bias towards errors detected by LanguageTool,' and there is no human evaluation, no gold-reference comparison, and no per-language precision/recall analysis for LT. Because Qwen 2.5 actually decreases LT on Swedish while Gemma improves it, a language-dependent bias in LT (e.g., different error coverage or false-positive rates in Swedish, German, Italian, or English) could invert the overall conclusion. This is not an external-consensus objection; it is a measurement-validity risk at the core of the argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a systematic comparison of 17 open-weight LLMs with up to 9B parameters for multilingual grammatical error correction (GEC) on English, German, Italian, and Swedish. Using the MultiGED dev set, it computes reference-less metrics for five requirements: LanguageTool-based grammatical correctness, semantic similarity (BERTScore, BLEURT, SentenceBERT), syntactic minimality (Levenshtein, GLEU, length difference), preservation of correct sentences (F1), and language drift. The authors select the longest and most concrete prompt (P3) by comparing three prompts on the same dev data, then aggregate per-metric rankings with a two-level Borda procedure. They conclude that six models improve LanguageTool-scored correctness in all four languages and that Gemma 9B is currently the best overall model; they also discuss model-size effects, recurring failure modes, and release the generated outputs publicly.","tokens_in":21952,"tokens_out":10073,"duration_ms":97342,"significance":"If the claims hold, this is a useful practical benchmark: it covers a wide and current model set, uses a coherent multi-criteria reference-less protocol, performs a stability check for two models, and publicly releases the generated outputs, which supports reproducibility. The internally consistent ranking and the authors' explicit Limitations section are strengths. However, the significance is bounded by the fact that the headline 'best model' conclusion is an empirical measurement claim whose validity depends on LanguageTool being a language-fair correctness measure and on the absence of selection effects in the prompt choice; both of these are load-bearing concerns rather than presentation issues. The strongest assets are the public data release and the explicit decomposition of quality into semantic, syntactic, correctness, and preservation perspectives.","major_comments":[{"comment":"The conclusion that Gemma 9B is 'the best model overall for the GEC task' (Section 6) rests on the LanguageTool error count being a language-fair measure of grammatical correctness. Table 6 shows that Gemma 9B is 1st on LT and tied-1st on the semantic perspective, but only 9th on Levenshtein, while Qwen 2.5 is 8th on LT, tied-1st on the semantic perspective, and 1st on several syntactic metrics; the LT rank gap is the main separator in the Borda aggregation. A sensitivity check using Table 6 shows that removing the LT perspective leaves the two models roughly tied on the remaining perspectives, or puts Qwen ahead if the F1 preservation perspective is also excluded. The Limitations concede that LT 'may introduce some bias towards errors detected by LanguageTool,' and no per-language precision/recall analysis of LT, no human evaluation, and no gold-reference comparison are reported. I request per-language LT validation (for example, against MultiGED token-level labels) or a human-annotated sample, together with a sensitivity analysis of the Borda ranking when the LT perspective is removed or reweighted.","section":"Section 4.3 (Req. 1) and Tables 5-6"},{"comment":"The best prompt (P3) is selected by comparing P1-P3 on the same MultiGED dev split that is then used for all subsequent model ranking and for answering RQ3. This is a selection-on-the-evaluation-set design: the reported advantage of P3, and potentially the final model ranking, are optimistically biased because the evaluation data were used to choose the prompt. The paper should hold out a test portion of MultiGED for the final evaluation, or at minimum report the model ranking under P1 and P2 and state explicitly whether the conclusions are invariant to the prompt choice.","section":"Section 5 (Prompt selection) and Tables 3-5"},{"comment":"The stability analysis is too thin to support the precision of the reported ranking. It checks only two of the seventeen models, reports only the LanguageTool score for run-to-run variation, and provides no confidence intervals or significance tests on the Borda totals. Table 5 has several adjacent models with close aggregate ranks, so the final ordering may not be distinguishable from noise. The paper should provide bootstrap intervals on the Borda scores, or at least per-model run-to-run variance for all constituent metrics, and state explicitly which adjacent rank differences are larger than the measurement noise.","section":"Section 6 (Result stability) and Tables 4-5"}],"minor_comments":[{"comment":"There are typos in the prose: 'ranked foruth overall' should be 'ranked fourth overall', and 'an par with' should be 'on a par with'.","section":"Section 6 and Section 8"},{"comment":"The caption of Table 8 says the prompt-quality analysis is 'averaged over all 7 models supporting all four languages considered,' but the text lists nine supporting models (Aya, EuroLLM 1.7B, EuroLLM 9B, Gemma 2B, Gemma 9B, Llama 3.1, OpenChat 3.5, Qwen 2.5, and Yi). This inconsistency should be reconciled.","section":"Section 5 and Table 8"},{"comment":"The sentence 'The parameter selection is inspired by Karen 2, which is the only model (considered) fine-tuned for GEC' is inaccurate because both Karen-strict and Karen-creative are fine-tuned for GEC; please rephrase to refer to the Karen models.","section":"Section 4.2"},{"comment":"The specific LanguageTool wrapper version and configuration used are not reported; specifying them would improve reproducibility.","section":"Section 4.3 (Req. 1)"},{"comment":"The phrase 'selected the best performing prompt, and used it to verify which LLMs support all the languages' should say 'evaluate' rather than 'verify', since the prompt is part of the evaluation protocol rather than an external ground truth.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical benchmark whose main value is the reproducible comparison and the public release of generated outputs. The methodological novelty is modest, but the paper would be acceptable for a venue that welcomes system-oriented empirical studies. The key risk is measurement validity: the authors should treat the LanguageTool dependence and the prompt-selection-on-dev issue as substantive, not cosmetic, and address them before the 'best model' claim can be endorsed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — this is a useful, honest empirical comparison, but read the headline as conditional. The ranking is built on referenceless metrics, and the top spot flips if you drop the LanguageTool column. The paper knows it, but the abstract doesn't. What's new: nobody has systematically compared these 17 small open LLMs on EN/DE/IT/SV GEC with a single model. The prompt study is sensible — longer, more specific prompt wins almost everywhere. The finding that six models improve LT in all four languages is practical, as is the observation that Gemma 2B punches above its weight. They release outputs, describe generation parameters, and report a stability check for two models (run-to-run variance is tiny). The qualitative error tables are genuinely informative. Citations are appropriate; the related work covers the relevant English-only and GPT-3.5 multilingual studies. Where it's soft: the central claim 'best model overall' rests almost entirely on LT. Recompute the Borda aggregation without LT and Qwen 2.5 overtakes Gemma 9B, because Qwen is worse on LT but better on syntax and F1. LT is a rule-based detector with different coverage per language; the paper's own limitation admits bias 'towards errors detected by LanguageTool.' That's not a footnote issue — it is the difference between first and second. Second, there is no human evaluation or gold-reference comparison, so we don't know if LT improvements correspond to actually better texts in Swedish, German, or Italian. The stability check only addresses sampling variance, not metric validity. Third, prompt P3 was chosen on the same dev set used for the final ranking; that's mild selection-on-test, though the superiority of P3 is consistent and intuitive. None of this makes the paper a waste. It's a well-executed benchmark with transparent methodology. But the conclusion should be 'under this metric framework, Gemma 9B ranks first,' not a blanket recommendation. A referee should ask for a sensitivity analysis (e.g., alternate Borda weights, per-language LT precision), and ideally a small human evaluation on Swedish/Italian where LT's reliability is most questionable. Verdict: deserves serious peer review. I'd cite it for the practical ranking and would bring it to reading group to discuss referenceless evaluation pitfalls.","headline":"A transparent, useful multilingual GEC benchmark, but the 'Gemma 9B is best' claim flips without the LanguageTool column; solid work that needs a softer conclusion.","tokens_in":22500,"tokens_out":3052,"would_cite":true,"duration_ms":30868,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single open 9B-parameter language model, Gemma 9B, outperforms 16 other models up to 9B parameters at correcting grammar across English, German, Italian, and Swedish in a reference-free evaluation.","keywords":["multilingual grammatical error correction","large language models","reference-free evaluation","LanguageTool","zero-shot prompting","Borda aggregation","language drift","Gemma 9B"],"falsifier":"Re-evaluate a random sample of the same four-language outputs with human annotators counting grammar errors directly (or using MultiGED's token-level gold annotations instead of LanguageTool); if the corrected-error rates put a different model on top, or if fewer than six models improve in all four languages, the central claim collapses.","tokens_in":21421,"feed_emoji":"✏️","tokens_out":8480,"duration_ms":73920,"temperature":0.7,"pith_summary":"This paper asks whether one moderately sized open language model can correct grammar in several languages at once, and which of 17 models up to 9B parameters does that best for English, German, Italian, and Swedish. Using a reference-free protocol that measures grammatical correctness, meaning preservation, edit minimality, preservation of already-correct sentences, and language fidelity, the authors find that Gemma 9B comes out on top overall, with Gemma 2B, EuroLLM 9B, OpenChat 3.5, EuroLLM 1.7B, and Llama 3.1 also improving grammatical correctness in all four languages. They also show that a longer, more specific prompt that asks for minimal changes and explicitly tells the model to return correct texts unchanged wins in 32 of 36 metric-language scenarios. The results matter because they suggest a single consumer-GPU-sized model can replace per-language GEC systems, and they expose language drift as the main failure mode of smaller multilingual LLMs.","feed_headline":"Gemma 9B is the best single multilingual grammar fixer","feed_subtitle":"Seventeen small open LLMs tested on English, German, Italian, and Swedish; six improve all four languages.","key_machinery":"The load-bearing mechanism is a reference-free evaluation scheme built from eight metrics organized into five requirements. Grammatical correctness is scored as $1/(1+\\text{errors})$ with LanguageTool error counts; semantic preservation uses BERTScore, BLEURT, and SentenceBERT; minimal edits use Levenshtein distance, GLEU, and token-length difference; correct-sentence preservation uses an F1 score over sentences originally marked correct; and language fidelity uses a FastText-based language-identification drift score. Per-metric rankings are fused with a two-stage Borda aggregation, a voting-style method that assigns points by rank position, giving equal weight to correctness, semantics, syntax, and text preservation. The scheme converts raw model outputs into a defensible ordering and is what the ranking claims stand on.","core_discovery":"The central claim is that among 17 open LLMs of up to 9B parameters, Gemma 9B is currently the best overall multilingual grammatical error corrector for English, German, Italian, and Swedish. It ranks first on the language-averaged grammatical-correctness score (LanguageTool 0.948), first on two of the three semantic preservation metrics (BERTScore 0.937, SentenceBERT 0.954), first on length difference (0.942) and on keeping already-correct sentences unchanged (F1 0.560), while avoiding language drift in all four languages. Six models -- Gemma 9B, Gemma 2B, EuroLLM 9B, EuroLLM 1.7B, OpenChat 3.5, and Llama 3.1 -- improve grammatical correctness in every one of the four languages, whereas eight models fail to support all four languages at all and frequently fall back to English. The paper also claims that the longest, most concrete prompt (ask for minimal changes, return only corrected text, do not touch correct inputs) is the best in 32 of 36 metric-language scenarios, primarily because it substantially raises the score for preserving correct sentences.","pith_inferences":["The equal-weight Borda aggregation is a modeling choice; if a user weights raw error reduction above all else, Qwen 2.5 and Aya become competitive or better per language, so 'best' is partly policy-dependent.","The same protocol could be ported to other language triples or quadruples (e.g., Spanish, French, Portuguese) to test whether Gemma 9B's lead is a Germanic/Romance artifact or a general property of the model.","Because LanguageTool misses certain error types, the six-model list that 'improves grammatical correctness' is a lower bound; a checker with wider coverage might certify additional models such as Aya or Qwen on Swedish.","The P3 prompt's explicit 'return correct text unchanged' instruction appears to be the main driver of the F1 gains; varying only that clause while keeping prompt length fixed would isolate whether it is the instruction or the length that matters."],"forward_implications":["A single open ~9B model can serve as the GEC backend for English, German, Italian, and Swedish, removing the need for per-language models or paid APIs.","Prompt design can substitute for fine-tuning: the longest, most concrete prompt (P3) lifted the average F1 for keeping correct sentences from roughly 0.12 to 0.34 on English and by similar margins on other languages.","Task-specific fine-tuning works: the Karen models, fine-tuned from Mistral, beat their base model on English while making the smallest edits.","Training data and distillation matter more than raw parameter count: Gemma 2B and EuroLLM 1.7B support all four languages while larger models such as BLOOM and SmolLM do not."],"supporting_citations":[{"why":"Introduces LanguageTool, the grammar checker whose error counts define the grammatical-correctness metric and the 'improves LT' criterion.","marker":"(Miłkowski, 2010)"},{"why":"Provides the MultiGED dataset, the source of all 6,363 test sentences across the four languages and their correct/incorrect labels.","marker":"(Volodina et al., 2023)"},{"why":"Presents the Gemma models that become the top-ranked and fourth-ranked systems in the comparison.","marker":"(Mesnard et al., 2024)"},{"why":"Supplies the FastText-based language identifier used to compute the language-drift metric that filters out non-multilingual models.","marker":"(Bojanowski et al., 2016)"},{"why":"Defines SentenceBERT, one of the three semantic-similarity metrics used to verify meaning preservation.","marker":"(Reimers and Gurevych, 2019)"},{"why":"Defines BLEURT, the second semantic-similarity metric in the evaluation suite.","marker":"(Sellam et al., 2020)"},{"why":"Defines BERTScore, the third semantic-similarity metric used to compare original and corrected sentences.","marker":"(Zhang et al., 2020)"},{"why":"Defines GLEU, one of the three syntactic-similarity metrics measuring how minimally the text was edited.","marker":"(Mutton et al., 2007)"},{"why":"Describes the Borda rank-aggregation method that combines the per-metric rankings into the final model ordering.","marker":"(McLean, 1990)"}],"fun_headline_variants":["Gemma 9B tops 17 small LLMs in multilingual grammar fixing","Single LLM up to 9B: Gemma 9B beats 16 rivals on 4 languages","Best multilingual grammar corrector? Gemma 9B, says 17-model study","Six small LLMs fix all four languages; Gemma 9B leads","Gemma 9B outperforms 17 open models in 4-language grammar correction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking rests on LanguageTool's error counts being an equally reliable measure of grammatical correctness across English, German, Italian, and Swedish; if it systematically misses, invents, or weights error types differently in any of these languages, the ranking measures the wrong target.","fun_headline_variants_meta":{"raw":{"variants":["Gemma 9B tops 17 small LLMs in multilingual grammar fixing","Single LLM up to 9B: Gemma 9B beats 16 rivals on 4 languages","Best multilingual grammar corrector? Gemma 9B, says 17-model study","Six small LLMs fix all four languages; Gemma 9B leads","Gemma 9B outperforms 17 open models in 4-language grammar correction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1289,"prompt_tokens":926,"completion_tokens":363,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":251}},"tokens_in":542,"tokens_out":363,"duration_ms":3256,"temperature":1.0,"reasoning_tokens":251,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:50:18.395040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-evaluate a random sample of the same four-language outputs with human annotators counting grammar errors directly (or using MultiGED's token-level gold annotations instead of LanguageTool); if the corrected-error rates put a different model on top, or if fewer than six models improve in all four languages, the central claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MultiGED dataset, the source of all 6,363 test sentences across the four languages and their correct/incorrect labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines GLEU, one of the three syntactic-similarity metrics measuring how minimally the text was edited."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the Borda rank-aggregation method that combines the per-metric rankings into the final model ordering."}],"review_version":1}