{"id":"fb3522f1-3864-4e6e-986d-2963bc241300","arxiv_id":"2502.08507","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Retrieving in-context demonstrations by matching natural-language grammatical error explanations beats input-text similarity for few-shot multilingual GEC.","lead":"This paper proposes choosing few-shot examples for LLM grammar correction by matching written explanations of the errors rather than matching the sentences themselves. The approach improves multilingual grammar correction results slightly without extra training, and suggests error patterns, not wording, are what matter for picking examples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption that the label-free detection explanation dT is a reliable retrieval query is unverified; Section 5.2 shows the same dT is harmful as CoT, and no retrieval-quality analysis exists.","rationale":"The central claim is that retrieving demonstrations by matching grammatical error explanations is what drives the F0.5 gains. That claim depends on the test-time explanation dT, produced without a gold correction, being an informative proxy for the error pattern. Section 5.2 directly shows these label-free explanations are poor enough to hurt the model when used as reasoning, so the paper's own evidence raises the possibility that dT is too noisy to retrieve reliably. The authors' response that dT is 'only useful as a retrieval tool' is plausible but untested: retrieval can tolerate noisy queries, and the case studies are only two favorable examples. An oracle-query comparison would settle whether the label-free bottleneck is real, and editing-overlap measurements would directly test the 'matching error patterns' mechanism. I considered the Estonian correct-sample database inconsistency (Table 6 shows only 2 correct samples with kC=4) and the absence of significance tests; both are real but secondary, because the main comparison remains fair across methods and the point estimates are mostly consistent. Keeping the reader's conditional verdict is appropriate: the method is promising, but the mechanism should be validated with the proposed oracle check before the claim 'matching error patterns is key' is accepted at face value.","tokens_in":20618,"tokens_out":11053,"duration_ms":116966,"concrete_test":"On a held-out subset (e.g., 200 samples per language) with gold corrections, run the full pipeline twice per test input: once with the proposed dT query from Eq. (4) and once with an oracle explanation eT generated by the same teacher prompt from (xT, yT). Compare F0.5 and compute edit/error-type overlap between the top-4 retrieved demonstrations and the test errors. If oracle queries substantially improve F0.5 or overlap, the label-free dT is a real bottleneck and the stated mechanism is only partially realized; if oracle and dT are comparable, the method is robust to imperfect detection and the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (4) defines the query for the erroneous-sample database as dT = LLMp(prompt_d(xT)), an explanation generated from the test input alone. The mechanism requires dT's xlm-roberta-large embedding to be close to teacher-generated explanations e in the database for sentences with the same error pattern. Table 3 shows that for Llama3.1, using dT directly as a chain of thought is strongly harmful (English F0.5 drops from 51.70 to 42.24 in zero-shot; every language drops by roughly 7-10 points), which the authors attribute to poor label-free explanation quality. A poor explanation can still serve as a retrieval key, but the paper provides no direct evidence of retrieval quality: the only qualitative support is two selected case studies, and there is no measurement of error-type or edit overlap between test sentences and retrieved demonstrations. Because database keys are generated with gold labels while dT is generated without them, there is a query/key distribution shift at the heart of the method. If dT is noisy in embedding space, the observed F0.5 gains could plausibly come from the extra detection call, prompt wording, or correct-sample retrieval, rather than from 'matching error patterns,' which is the stated mechanism and the basis for the method's claimed generality. This unvalidated assumption is the load-bearing step separating the empirical result from the stated mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a retrieval method for few-shot in-context demonstrations in multilingual grammatical error correction (GEC). It constructs a database of erroneous samples whose keys are LLM-generated grammatical error explanations (GEE), and a database of correct samples keyed by input text. At inference, a detection prompt produces an initial explanation dT from the test input; kNN retrieval matches dT against the erroneous-sample keys and the test text against the correct-sample keys, and the retrieved (x,y) pairs are used as few-shot demonstrations for the final prediction. Experiments across five languages and four LLMs compare against Random, Semantic embedding, and BM25 retrieval using F0.5. The central claim is that explanation-based retrieval outperforms semantic and BM25 baselines without additional training or language adaptation, suggesting that matching error patterns is the key to demonstration selection.","tokens_in":20912,"tokens_out":6412,"duration_ms":59727,"significance":"Demo selection for GEC is a real problem, and replacing input-to-input similarity with error-pattern similarity is a plausible and useful idea. The paper's strengths are its training-free pipeline, its cross-lingual scope, the extension to BEA-19 in Section 5.1, the analysis of the correct/erroneous ratio in Section 5.3, and the release of code. If the mechanism is validated, the method would be a practical contribution to multilingual GEC and to interpretable in-context learning. However, the current evidence does not yet establish the mechanism: effect sizes over the Semantic baseline are often below one F0.5 point, several per-language/per-model cells show the proposed method underperforming Semantic, no statistical significance is reported, and the claim that retrieval operates on error-pattern similarity rests on only two case studies. The paper would be substantially strengthened by retrieval-quality measurements (e.g., error-type agreement between test inputs and retrieved demonstrations) and by a broader sensitivity analysis of the free parameters.","major_comments":[{"comment":"The load-bearing assumption of the method is that the label-free detection explanation dT (Eq. 4) is a reliable query in the xlm-roberta-large embedding space for retrieving samples with similar error patterns. Section 5.2 and Table 3 show that the same dT, when inserted as chain-of-thought, drastically reduces F0.5 (e.g., Llama3.1 English drops from 51.70 to 42.24 in zero-shot), and the authors attribute this to poor label-free explanation quality. That finding does not by itself invalidate dT as a retrieval key, but the paper currently provides no direct evidence that dT-based retrieval actually retrieves demonstrations with matching errors: the only qualitative support is the two selected case studies in Appendix C, and there is no measurement of error-type or edit overlap between test sentences and retrieved demonstrations. Because database keys are generated with gold labels while dT is generated without them, a query/key distribution shift is at the heart of the method. Please add quantitative retrieval-quality analysis (e.g., error-type precision@k, or a comparison of retrieval using dT versus using gold-label explanations) to substantiate the stated mechanism and the abstract's claim that matching error patterns is key.","section":"Section 3.2 and Section 5.2"},{"comment":"The claimed consistent superiority over semantic retrieval is not supported by the reported effect sizes. For instance, Qwen2.5 on German obtains Explanation F0.5 55.08 versus Semantic 55.76; Deepseek2.5 on Chinese obtains 37.62 versus Semantic 38.44; and on Llama3.1 CoNLL-14 the gain over Semantic is only 0.16 (54.60 vs 54.44). No error bars, confidence intervals, or significance tests are reported; the Random baseline averages three seeds while the other methods are reported from single runs. Because the central claim is that the proposed method 'consistently outperforms' semantic and BM25 retrieval, please report variance (e.g., bootstrap over test samples or multiple seeds for API models) and a paired significance test, and discuss the fraction of language/model settings in which the improvement is statistically reliable.","section":"Section 4.3, Table 1"},{"comment":"The number of correct demonstrations kC=4 is selected from a sweep on the evaluation datasets (Figure 3) rather than fixed a priori or chosen on a held-out set, so the comparison may be favorable to the proposed method relative to baselines evaluated only at the default setting. Additionally, the Estonian database contains only 2 correct samples after filtering (Table 6), while Section 4.2 states kC=4 for all methods; it is unclear how four correct demonstrations are obtained for Estonian. Please clarify the Estonian procedure (e.g., retrieval with replacement, duplication, or fewer correct demonstrations) and provide a validation strategy or sensitivity analysis showing that the main conclusions are robust to the choice of kC.","section":"Sections 4.2 and 5.3, Table 6"}],"minor_comments":[{"comment":"The Limitations section states that experiments were conducted on '2 open-source LLMs and 1 closed-source LLM,' but Table 1 reports results for two closed-source models (Deepseek2.5 and GPT4o-mini) in addition to Llama3.1 and Qwen2.5; please correct this count.","section":"Limitations"},{"comment":"BM25 results are missing for Estonian in Table 1, and the text notes in the Limitations that LlamaIndex encountered issues with the Estonian dataset. Please state this explicitly in the table or caption (e.g., 'not available') so readers do not infer that BM25 is inapplicable by design.","section":"Section 4.2 and Table 1"},{"comment":"The comparison in Table 3 is run only with Llama3.1, and the 'Post' condition places the explanation after the corrected text, which is not a chain-of-thought condition; the statement that using the initial explanation as an intermediate reasoning step is harmful should be restricted to the single model tested and to the 'Pre' condition.","section":"Section 5.2, Table 3"},{"comment":"Equation (7) defines D as DE ⊕ DC without specifying the concatenation order; please define whether erroneous demonstrations always precede correct demonstrations and whether the order is shuffled, since demonstration ordering is known to affect ICL performance.","section":"Section 3.3, Eq. (7)"},{"comment":"The concluding sentence 'This also suggests that matching error patterns is key to selecting examples' goes beyond the correlational evidence; the paper demonstrates that explanation-based retrieval helps in some settings, but does not isolate error-pattern matching from other differences (e.g., the extra detection call, the correct-sample retrieval, or prompt wording). Please soften this claim or add the proposed retrieval-quality analysis.","section":"Abstract"},{"comment":"Qwen2.5 uses a different, shorter detection prompt than the other three models (Table 7). This is a potential confound in cross-model comparisons of dT quality and final F0.5; please either report results with a common detection prompt or discuss the implications.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a worthwhile problem and the proposed method is simple and, in principle, reproducible. The main gap is not novelty but validation of the mechanism: the paper currently rests on aggregate F0.5 differences of under one point without significance testing, and the central retrieval hypothesis is supported only by case studies. I recommend major revision rather than reject because the issues are addressable within the manuscript's scope: add retrieval-quality analysis, significance testing, and clarify the kC/Estonian handling. The contribution statement should also be checked against prior work for the claim of 'first aligned evaluation' of few-shot GEC."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. The idea is simple and new: instead of retrieving GEC demonstrations by input similarity, they generate a grammatical error explanation for the test sentence and match it against a database of explanations for erroneous samples, with correct samples retrieved by input. That combination is not in the prior work they cite, and it makes sense. The evaluation is appropriately broad—five languages, four LLMs, including Estonian—and they ship code.\n\nThe paper does a few things well. The section showing that putting the initial explanation into the context as CoT hurts performance is a useful negative result; it strengthens the case that retrieval rather than reasoning is the right use. The kC sweep is a nice sanity check, and the transfer experiment to BEA-19 helps.\n\nNow the soft spots. First, the gains are small and not always consistent. On German with Qwen2.5, explanation retrieval is actually worse than semantic retrieval. There are no error bars or significance tests; for the API models you have single runs with default sampling. Second, the choice kC=4 comes from a sweep on the test sets; the paper should use a validation split. Also, the Estonian database has only two correct samples while the paper says kC=4, so that setup is inconsistent as described. Third, they omit the most directly relevant baseline: Tang et al. (2024) on ungrammatical-syntax-based demonstration selection. They cite it but never compare. Fourth, the 'approaching SOTA' claim in the contributions is overstated—their own Table 2 shows a 6-10 point gap to fine-tuned models.\n\nThe deeper issue the stress-test raises is the mechanism. The claim that matching error patterns is key is supported only by two case studies. The retrieval query dT is generated without labels, while the database keys are teacher-generated with labels; that shift is real. The paper shows dT is poor as CoT, but that doesn't tell us whether it's good as a retrieval key. An ablation using a random or input-based query, or measuring error-type overlap between retrieved demonstrations and test sentences, would directly address this. Given that the gains over semantic retrieval are often below one F0.5 point, it's worth ruling out the possibility that the extra detection call or the correct-sample component is doing the work.\n\nNet: the paper is a solid empirical starting point, not a finished proof of mechanism. It deserves review; the reviewer should ask for significance testing, a validation-based hyperparameter choice, the Tang et al. baseline, and at least one direct retrieval-quality measure. I'd accept it with major revisions.","headline":"Genuinely new retrieval idea for few-shot GEC, with broad but noisy gains; the mechanism is plausible but not directly verified.","tokens_in":21441,"tokens_out":4401,"would_cite":true,"duration_ms":42703,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that selecting few-shot correction examples by matching LLM-generated grammatical error explanations, rather than by input-text similarity, improves multilingual grammatical error correction across five languages and four…","keywords":["grammatical error correction","in-context learning","demonstration retrieval","grammatical error explanations","multilingual GEC","few-shot learning","large language models","kNN retrieval"],"falsifier":"Take a test set with known error types, such as those extracted by ERRANT, and compute whether the top-$k$ demonstrations retrieved by explanation matching share at least one error type with the test input. If explanation-based retrieval does not achieve higher error-type overlap than input-text retrieval, or if the overlap is no better than random, the central claim that error patterns drive the gains would be falsified.","tokens_in":20426,"feed_emoji":"✏️","tokens_out":6784,"duration_ms":55632,"temperature":0.7,"pith_summary":"The paper sets out to show that for few-shot grammatical error correction (GEC), the right demonstrations are ones that share the same grammatical error pattern with the test input, not ones whose text is semantically similar. It builds a database of erroneous sentences, each indexed by an LLM-generated natural-language grammatical error explanation (GEE), and at test time it first runs a detection prompt on the input to produce an initial explanation that serves as the retrieval query. Across five languages and four LLMs, the authors report that this explanation-based retrieval yields higher F0.5 than input-text semantic retrieval and BM25, without any additional training or per-language adaptation. The paper's central conclusion is that matching error patterns, not surface texts, is the key to selecting in-context examples for GEC.","feed_headline":"Error explanations beat text similarity for grammar AI demos","feed_subtitle":"Across five languages and four LLMs, matching error patterns lifts few-shot grammatical error correction without retraining.","key_machinery":"The load-bearing mechanism is the grammatical error explanation (GEE) used as the retrieval key: a short natural-language statement of what is wrong in a sentence and the grammar rule behind the correction. The paper has the teacher model Llama-3.1-8B-Instruct generate these explanations for each erroneous labeled sample, stores them in a database, embeds both the test input's detection-prompt explanation and the database explanations with xlm-roberta-large, and retrieves the nearest neighbors by cosine distance. Because the explanation describes the error rather than the topic of the sentence, two sentences about completely different subjects can be paired if they contain the same mistake. This is what allows the method to decouple demonstration selection from input-text similarity and to transfer across languages and test sets.","core_discovery":"The central discovery claimed is that natural-language grammatical error explanations make a better retrieval key than the input sentence itself when choosing in-context demonstrations for GEC. The paper creates two databases from labeled data: erroneous samples whose keys are teacher-generated explanations of the error, and correct samples whose keys are the raw input texts because error-free sentences have no explanation. At inference, a detection prompt converts the test input into an initial explanation $d_T$, which is embedded and matched against the explanation keys with kNN ($k_E=4$); the correct-sample query is the raw input ($k_C=4$). The retrieved text pairs are concatenated into a fixed few-shot prompt without including the explanations. The authors find that this method generally outperforms semantic and BM25 retrieval in F0.5 across English, Chinese, German, Russian, and Estonian, and that the same database carries over to a second English test set, leading them to conclude that matching error patterns is key to selecting examples.","pith_inferences":["The method uses the detection-prompt explanation only as a retrieval key, and the paper's own experiments show the same explanation is harmful when inserted as a chain of thought; a likely general lesson is that noisy intermediate representations can still be useful for selecting examples even when they are not useful for reasoning.","With better explanation generators, retrieval quality should improve without changing the pipeline, so the approach's ceiling is partly set by the teacher model rather than by the retriever.","The Estonian correct-sample database contains only two examples while the method asks for four, so the reported gains there are probably driven almost entirely by the erroneous-sample retrieval; this suggests the correct-sample pool could be omitted in very low-resource settings or supplemented with synthetic error-free sentences."],"forward_implications":["Few-shot multilingual GEC can be improved for a new language without fine-tuning, provided a small labeled database with LLM-generated explanations exists for that language.","A single explanation-indexed database transfers to multiple test sets in the same language, as shown on BEA-19 with a database built from W&I+LOCNESS.","The finding that matching error patterns matters more than text similarity should carry over to other editing tasks, such as text simplification or style transfer, where the input-output relation is local.","Balancing erroneous and correct demonstrations (here 4 and 4) sits near the optimum of the precision-recall tradeoff measured by F0.5."],"supporting_citations":[{"why":"Defines the few-shot in-context learning setup that the paper uses for GEC prediction.","marker":"Brown, 2020"},{"why":"Supplies the kNN retrieval mechanism used both for demonstration selection and as the Semantic baseline.","marker":"Khandelwal et al., 2020"},{"why":"Defines BM25, the term-based retrieval baseline the proposed method is compared against.","marker":"Robertson et al., 2009"},{"why":"Provides xlm-roberta-large, the multilingual embedding model that encodes explanation queries and keys.","marker":"Conneau et al., 2019"},{"why":"Shows LLMs can generate grammatical error explanations, the representation this method uses as retrieval keys.","marker":"Song et al., 2024"},{"why":"Provides the English W&I+LOCNESS database and the BEA-19 test set used in evaluation.","marker":"Bryant et al., 2019"},{"why":"Provides the CoNLL-14 English test set used for evaluation.","marker":"Ng et al., 2014"},{"why":"Informs the few-shot prompt template and the in-context example selection line of work.","marker":"Tang et al., 2024"}],"fun_headline_variants":["Error explanations outperform text similarity for grammar AI demos","Matching error patterns lifts multilingual grammar correction","Explanations, not text, pick better examples for grammar AI","Error-pattern retrieval boosts few-shot grammar correction","For grammar correction, explain errors to find better examples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the detection prompt's initial explanation of the test input lands near the teacher-generated explanations of genuinely similar errors in the shared embedding space, even though the same explanation is too unreliable to serve as a chain-of-thought for the final correction.","fun_headline_variants_meta":{"raw":{"variants":["Error explanations outperform text similarity for grammar AI demos","Matching error patterns lifts multilingual grammar correction","Explanations, not text, pick better examples for grammar AI","Error-pattern retrieval boosts few-shot grammar correction","For grammar correction, explain errors to find better examples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1456,"prompt_tokens":921,"completion_tokens":535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":459}},"tokens_in":537,"tokens_out":535,"duration_ms":4647,"temperature":1.0,"reasoning_tokens":459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:48:33.009968+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a test set with known error types, such as those extracted by ERRANT, and compute whether the top-$k$ demonstrations retrieved by explanation matching share at least one error type with the test input. If explanation-based retrieval does not achieve higher error-type overlap than input-text retrieval, or if the overlap is no better than random, the central claim that error patterns drive the gains would be falsified.","supporting_citations":[],"review_version":1}