{"id":"fd2a2491-5062-43e3-b36e-681ccd5fd7be","arxiv_id":"2411.15523","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning BERT-base-uncased on a hand-cleaned Lang-8 subset yields F1 0.91 on that same distribution, but the result is not benchmarked against standard GED tests.","lead":"The authors fine-tuned BERT and RoBERTa models on a cleaned subset of the Lang-8 learner corpus for sentence-level grammatical error detection, reporting an F1 of 0.91 for BERT-base. The evaluation uses only a self-constructed test set, so the result is best read as a data-cleaning case study, not a benchmark claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 2.1.8's label rule treats any Lang-8 edit as a grammatical error and never filters to English; the 0.91 F1 may reflect language or position artifacts rather than grammar.","rationale":"The paper makes an empirical claim with no standard benchmark evaluation, so internal validity is decisive. I considered the absence of CoNLL/BEA benchmarks and the non-random split; both matter, but the most fundamental issue is the semantic validity of the labels. If the labels do not denote English grammaticality, no metric computed on them supports the abstract. The language-filter omission is the sharpest, checkable version of this: it is a single missing preprocessing step that would invalidate the BERT-base result regardless of benchmark. The reader identified label validity as the weakest assumption; I agree and add that the English-only restriction is the concrete first test. The self-reported chance performance on discarded Lang-8 (Table 3) corroborates that the model's skill is tied to the cleaning filter. I recommend keeping the reader's REJECT; the paper would need at least a language-filtered, randomly split, standard-benchmark evaluation to support the claim.","tokens_in":12044,"tokens_out":12784,"duration_ms":123483,"concrete_test":"Download the released Cleaned Lang-8 dataset (Nihalani and Shah 2024), run langdetect on column 0, then fine-tune and evaluate BERT-base-uncased on the English-only rows using a random stratified 90/10 split with the same hyperparameters. If any non-English rows are present, or if the English-only random-split F1 is materially below 0.91, the reported F1 is not a clean English GED result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the label construction in §2.1.8: every retained Lang-8 pair is treated as 'column 0 = grammatically incorrect, column 1 = corrected.' This conflates any learner-corrector edit with a grammaticality label; stylistic rewrites and annotator disagreements are labeled as errors, and corrected versions may themselves be non-native. The same section never restricts the corpus to English, although BERT-base-uncased is English-only and the raw Lang-8 corpus is multilingual. Because the train/validation split is taken 'from the top' and 'from the bottom' (§2.1) rather than randomly stratified, any ordering by language, author, or document can make the label predictable from position or script. The paper's own Table 3 shows the model is near chance (F1 ≈ 0.41–0.50) on discarded Lang-8 sentences, which is exactly what one would expect if the model exploits the cleaning filter rather than detecting grammar. If the 0.91 F1 reflects these cues, the central claim about GED is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes a data-cleaning pipeline for the Lang-8 learner corpus and fine-tunes BERT-family models for sentence-level grammatical error detection (GED). The cleaning pipeline removes identical pairs and filters by Levenshtein distance and length, producing a 200,000-sentence dataset. The authors report that BERT-base-uncased fine-tuned on 180k cleaned sentences achieves F1 0.91 and 90.53% test accuracy on a self-constructed validation split, and that larger models (BERT-large, RoBERTa-large) do not improve over BERT-base. The paper also reports zero-shot comparisons with GPT-4 and Llama-3-70B-instruct, on which the fine-tuned models perform better.","tokens_in":12293,"tokens_out":4372,"duration_ms":37851,"significance":"If the findings were transferable to standard GED benchmarks, the paper would provide a useful demonstration that rigorous cleaning of learner data and a moderately sized model can outperform larger models, and the released cleaned dataset could be a practical resource. The paper deserves credit for making its dataset publicly available on GitHub and for reporting training times. However, the significance is currently conditional: the headline F1 is measured only on the paper's own cleaned Lang-8 split, the label construction conflates any learner-corrector edit with a grammatical error, and no independent benchmark is used. These issues make the results difficult to interpret as progress on GED itself.","major_comments":[{"comment":"The label rule in §2.1.8 states that every Lang-8 pair surviving the cleaning filters is treated as 'column 0 grammatically incorrect, column 1 corrected.' This is the load-bearing premise of the entire evaluation. Lang-8 corrections can reflect stylistic rewrites, content additions, spelling changes, or annotator disagreement, not only grammatical errors, and the corrected sentence itself may be non-native. Additionally, the cleaning procedure never restricts the corpus to English despite using English-only BERT-base-uncased. Please validate the labels by expert or crowd-sourced annotation on a random sample (with inter-annotator agreement), and report the language distribution of the 200,000 retained sentences. Without such validation, the F1 0.91 could be measuring 'editedness' or script-based artifacts rather than grammaticality.","section":"§2.1.8"},{"comment":"Table 3 shows that when the same BERT-base-uncased models are tested on discarded Lang-8 sentences (sentences removed by the cleaning filters), test accuracy is consistently near 50% and F1 is between 0.41 and 0.50 across all training compositions. This is precisely what would be expected if the model has learned the distributional signature of the cleaning filter rather than a general grammaticality signal. The paper interprets Table 3 as underscoring the importance of cleaning, but it also demonstrates that the model's knowledge does not transfer to sentences that do not satisfy the filter. Please provide a concrete explanation of why near-chance accuracy on discarded sentences is consistent with the claim that the model performs grammatical error detection in general, and ideally evaluate on an independently annotated corpus.","section":"Table 3"},{"comment":"The train/validation split is non-random: 90k sentences 'from the top' of column 0, 90k 'from the bottom' of column 1, and the remaining 20,000 'mid' sentences for validation. If the Lang-8 rows are ordered by language, author, or document, this split can create positional or script-based cues, and the label may be predictable without any linguistic understanding. The split should be randomized and stratified (e.g., by language and author), and results should be reported with variance across multiple seeds. The duplicate '10k + 10k' rows in Table 2 (F1 0.81 and 0.76) indicate run-to-run variability that is undocumented, so single-point results are not sufficient.","section":"§2.1 (last paragraph)"},{"comment":"No external GED benchmark is used. The only test set is the self-constructed cleaned Lang-8 split, and the dataset citation (Nihalani and Shah 2024) is the authors' own GitHub artifact. Consequently, the reported F1 0.91 is not comparable to the prior GED numbers cited in the introduction (e.g., Bi-LSTM F1 0.80–0.90). Please evaluate on a standard benchmark such as CoNLL-2014 or BEA-2019, and report precision, recall, and F1 with error bars. Without an independent test set, the claims that the proposed method 'improves GED' and that 'larger models are not always better' are not supported.","section":"Evaluation protocol (Tables 2 and 5)"}],"minor_comments":[{"comment":"Text says BERT-base-uncased 20k has a recall of 0.97, but Table 5 reports recall 0.87; correct the inconsistency.","section":"§3.2.1 / Table 5"},{"comment":"The caption defines 'UT - Undertrained layers, OT - Over-trained layers' but the table does not identify which specific layers are under- or over-trained; state how these labels are assigned.","section":"Table 4"},{"comment":"The caption explains 'Batch Sizes mentions how many sentences were taken from Cleaned and Discarded Lang8 set respectively,' but the table lists identical batch compositions (e.g., 10k + 10k, 8k + 12k) with different metrics; explain the difference (e.g., different random seeds or checkpoint epochs).","section":"Table 2 caption"},{"comment":"'Only sentences with a Levenshtein distance between 7 and 42 and lengths less than 101 characters were kept' does not specify which column's length is used; state whether the length threshold applies to column 0, column 1, or both.","section":"§2.1.7"},{"comment":"The caption says TP is 'Correct sentences correctly identified as Correct' and TN is 'Incorrect sentences correctly identified as Incorrect,' which inverts the usual TP/TN convention when the positive class denotes incorrect sentences; clarify the label convention in the text.","section":"Table 6 caption"}],"recommendation":"reject","confidential_remarks":"The central evaluation is circular in the sense that the model is trained and tested on splits of the same hand-filtered dataset, with no independent validation of the labels or comparison to standard GED benchmarks. The near-chance results on discarded sentences in Table 3 further weaken the claim that the model learns generalizable grammatical error detection. These issues cannot be addressed by local revisions because they require re-annotating or re-sourcing the evaluation data. The cleaned dataset may still be useful to the community, but the paper's claims as stated are not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a dataset-and-ablation paper dressed as a GED results paper. The genuinely new bits are a cleaned Lang-8 subset of ~200k pairs and a step-by-step cleaning pipeline with row counts for each filter. The authors also show that on 20k training examples BERT-base matches RoBERTa-large and that adding discarded sentences to training hurts test performance on the cleaned distribution. Those results are internally consistent, and the paper earns credit for openly reporting near-chance performance on discarded sentences.\n\nThe load-bearing weakness is Section 2.1.8. Every pair that survives the Levenshtein filters is treated as 'column 0 = incorrect, column 1 = correct.' That conflates any edit—stylistic rewrite, annotator disagreement, or correction by another learner—with a grammaticality judgment. The section never restricts the corpus to English, despite BERT-base-uncased being English-only. The train/validation split is non-random (top of column 0, bottom of column 1, middle as validation), so positional or document order artifacts are plausible. Table 3's near-chance accuracy on discarded sentences is exactly what you'd expect from a model that learned the cleaning filter rather than grammar. There are also no standard benchmarks (CoNLL-2014, FCE), no error bars, and no significance tests, so the headline 0.91 F1 is not comparable to prior GED numbers.\n\nThe cleaning ablation itself is sound on its own terms: within the cleaned distribution, cleaning clearly matters. So the paper is a plausible empirical report about BERT on a particular filtered slice of Lang-8, not a demonstration of improved general GED. The novelty over Bell et al. (2019) is the cleaned dataset artifact and the cleaning-ablation, which is legitimate but small.\n\nWho gets value: practitioners who want a quick BERT baseline for language-learning tools on Lang-8, and researchers studying the effect of aggressive cleaning. It is not a generalizable GED result as written.\n\nMy recommendation for an editor: send it to peer review with a strong request for major revision—random stratified split, at least one standard GED benchmark, and a modest manual validation of the labels. If the venue does not allow that kind of revision path, desk reject with clear instructions. The authors report negative results and release their data, which suggests they would respond well to referee guidance, so I would give a serious referee a shot at it.","headline":"A cleaned Lang-8 dataset and an internally consistent cleaning ablation that cannot support the paper's general GED claim because the labels are filter-defined.","tokens_in":12800,"tokens_out":4624,"would_cite":false,"duration_ms":42169,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning BERT-base on a cleaned Lang-8 dataset yields 0.91 F1 for grammatical error detection, outperforming larger models and zero-shot GPT-4.","keywords":["Grammatical Error Detection","BERT","Lang-8 corpus","data cleaning","Levenshtein distance","transfer learning","transformer fine-tuning","sentence classification"],"falsifier":"Evaluate the same fine-tuned model on a held-out set of learner sentences whose grammaticality labels were assigned by human annotators rather than inferred from the difference between original and corrected text; if the F1 drops far below 0.91, the model is detecting edit propensity rather than grammar.","tokens_in":11829,"feed_emoji":"✍️","tokens_out":11375,"duration_ms":80777,"temperature":0.7,"pith_summary":"This paper tries to establish that rigorous data cleaning, not model scale, is the main lever for grammatical error detection with modern transformers. The authors build a cleaned Lang-8 dataset of 180,000 learner sentences, filtered through eight steps that delete identical pairs, normalize casing and punctuation, expand contractions, and keep only pairs whose Levenshtein distance falls in a chosen window. Fine-tuning BERT-base-uncased on that set achieves an F1 score of 0.91 and 90.53% test accuracy on held-out cleaned sentences, outperforming BERT-large and RoBERTa-large trained on the same 20,000-sentence budget. The paper also reports that zero-shot generative models (GPT-4, Llama-3-70B-instruct) score far lower, and that training on discarded, noisier Lang-8 pairs degrades F1, supporting the cleaning-first argument.","feed_headline":"BERT-base fine-tuned on cleaned data hits 0.91 F1 for grammar errors","feed_subtitle":"Fine-tuning a small BERT on 180k cleaned learner sentences beats larger models and zero-shot GPT-4 and Llama-3","key_machinery":"The load-bearing object is the cleaned Lang-8 dataset itself: an eight-filter pipeline that converts 2,350,982 raw learner sentences into 200,000 pairs whose original (column 0) is treated as ungrammatical and whose corrected version (column 1) is grammatical. The decisive filters are the Levenshtein-distance windows — raw distance 7–42, normalized distance 0.08–0.5 — and the 101-character length cap, which keep only pairs that differ by a substantial but not extreme amount of editing. On top of this, the second mechanism is a binary classification head on BERT-base-uncased, fine-tuned with AdamW, a 0.65 dropout on the final layer, and four epochs.","core_discovery":"The central discovery is that BERT-base-uncased, fine-tuned on a carefully filtered subset of Lang-8, reaches 0.91 F1 and 90.53% accuracy for binary sentence-level grammaticality classification on cleaned Lang-8 test data, while larger models (BERT-large-uncased, RoBERTa-large) trained on a 20,000-sentence sample do not meaningfully improve over the base model, and zero-shot GPT-4 and Llama-3-70B-instruct perform substantially worse. The paper attributes the gain primarily to an eight-step cleaning pipeline that removes near-identical pairs, normalizes casing and punctuation, expands contractions, and restricts pairs to a normalized Levenshtein-distance window between 0.08 and 0.5, so that every training pair is a clear original/correction contrast. It also shows that fine-tuning on sentences discarded by the cleaning process yields test accuracy around 50% on the cleaned test set, which the authors interpret as evidence that data quality, not model capacity, drives performance.","pith_inferences":["Because the labels are defined by the cleaning filters, the 0.91 F1 measures agreement with an edit-distance operationalization of error, not with human annotation; an independent human-labeled test set would likely show smaller gaps between models but lower absolute F1.","A testable extension would be to feed the discarded Lang-8 pairs back into training with soft labels derived from their Levenshtein distance to the corrected version, which might improve robustness on unfiltered learner text without sacrificing performance on the cleaned distribution.","The normalized Levenshtein window suggests an interpretable trade-off curve: tightening the window makes training pairs easier to separate (higher F1 on filtered test sets) while loosening it should improve generalization to noisier text; plotting F1 against the window bounds would quantify how much of the reported gain is an artifact of filtering."],"forward_implications":["A cleaned, filtered version of Lang-8 — 180,000 pairs with Levenshtein distance between 7 and 42 and normalized distance between 0.08 and 0.5 — is enough to train a 110M-parameter BERT-base to 0.91 F1 on the same cleaned distribution, so data curation can substitute for model scale in GED.","Models trained on this cleaned set do not transfer to the 1.3 million discarded Lang-8 pairs: test accuracy there stays near 50%, meaning the claimed performance is distribution-specific.","Fine-tuned BERT-base (180k) outperforms zero-shot GPT-4 and Llama-3-70B-instruct by a wide margin on a 500-sentence sample (F1 0.99 vs 0.63 and 0.60), implying that task-specific fine-tuning on cleaned data remains necessary for sentence-level grammaticality classification.","Freezing most BERT layers individually leaves validation F1 at 0.84, so the grammaticality signal appears to be distributed across the network rather than concentrated in particular layers."],"supporting_citations":[{"why":"Supplies the early neural GED method using error- and grammaticality-specific word embeddings that the paper cites as the pre-transformer deep-learning baseline.","marker":"Kaneko et al. [2017]"},{"why":"Provides the Bi-LSTM joint sentence/token labeling approach that sets the deep-learning F1 range (0.80–0.90) the paper aims to beat with BERT.","marker":"Rei and Søgaard [2018]"},{"why":"Establishes that contextual word representations such as ELMo and BERT improve GED, motivating the choice of BERT as the backbone.","marker":"Bell et al. [2019]"},{"why":"Defines the standard evaluation methodology for GED/GEC that the paper's accuracy and F1 reporting follows.","marker":"Felice and Briscoe [2015]"},{"why":"Supplies the WeightWatcher tool used to analyze per-layer overfitting and to decide which BERT layers to freeze.","marker":"Martin et al. [2021]"},{"why":"Documents the GPT-4 model used as the zero-shot generative baseline in the inference comparison.","marker":"OpenAI et al. [2024]"},{"why":"Documents the Llama-3-70B-instruct model used as the second zero-shot generative baseline.","marker":"Meta AI [2024]"},{"why":"The cleaned Lang-8 GED dataset that the paper trains and evaluates on.","marker":"Nihalani and Shah [2024]"}],"fun_headline_variants":["Data cleaning beats model size in grammar error detection","Clean data, not big models, boosts grammar detection to 0.91 F1","BERT-base with cleaned data outperforms larger models for grammar errors","Data cleaning lifts BERT to 0.91 F1 for grammar error detection","Rigorous cleaning gives BERT-base 0.91 F1, beating larger models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that every sentence pair surviving its cleaning filters is a genuine example of a grammatical error, meaning the difference between the learner's original text and the corrected version is exactly what a GED model should learn to flag.","fun_headline_variants_meta":{"raw":{"variants":["Data cleaning beats model size in grammar error detection","Clean data, not big models, boosts grammar detection to 0.91 F1","BERT-base with cleaned data outperforms larger models for grammar errors","Data cleaning lifts BERT to 0.91 F1 for grammar error detection","Rigorous cleaning gives BERT-base 0.91 F1, beating larger models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000609,"raw_usage":{"total_tokens":2875,"prompt_tokens":1025,"completion_tokens":1850,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":1752}},"tokens_in":641,"tokens_out":1850,"duration_ms":11415,"temperature":1.0,"reasoning_tokens":1752,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:11:29.598960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the same fine-tuned model on a held-out set of learner sentences whose grammaticality labels were assigned by human annotators rather than inferred from the difference between original and corrected text; if the F1 drops far below 0.91, the model is detecting edit propensity rather than grammar.","supporting_citations":[],"review_version":1}