Pith. sign in

REVIEW 4 major objections 5 minor 16 references

Enhancing Grammatical Error Detection using BERT with Cleaned Lang-8 Dataset

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning BERT-base on a cleaned Lang-8 dataset yields 0.91 F1 for grammatical error detection, outperforming larger models and zero-shot GPT-4.

desk verdict A cleaned Lang-8 dataset and an internally consistent cleaning ablation that cannot support the paper's general GED claim because the labels are filter-defined. read the letter →

arxiv 2411.15523 v1 pith:J3MTISB7 submitted 2024-11-23 cs.CL cs.AI

Rahul Nihalani , Kushal Shah This is my paper · ORCID
classification cs.CLcs.AI
keywords GrammaticalErrorDetectionBERTLang-8corpusdatacleaningLevenshteindistancetransferlearningtransformerfine-tuningsentenceclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that rigorous data cleaning, not model scale, is the main lever for grammatical error detection with modern transformers. The authors build a cleaned Lang-8 dataset of 180,000 learner sentences, filtered through eight steps that delete identical pairs, normalize casing and punctuation, expand contractions, and keep only pairs whose Levenshtein distance falls in a chosen window. Fine-tuning BERT-base-uncased on that set achieves an F1 score of 0.91 and 90.53% test accuracy on held-out cleaned sentences, outperforming BERT-large and RoBERTa-large trained on the same 20,000-sentence budget. The paper also reports that zero-shot generative models (GPT-4, Llama-3-70B-instruct) score far lower, and that training on discarded, noisier Lang-8 pairs degrades F1, supporting the cleaning-first argument.

What carries the argument

The load-bearing object is the cleaned Lang-8 dataset itself: an eight-filter pipeline that converts 2,350,982 raw learner sentences into 200,000 pairs whose original (column 0) is treated as ungrammatical and whose corrected version (column 1) is grammatical. The decisive filters are the Levenshtein-distance windows — raw distance 7–42, normalized distance 0.08–0.5 — and the 101-character length cap, which keep only pairs that differ by a substantial but not extreme amount of editing. On top of this, the second mechanism is a binary classification head on BERT-base-uncased, fine-tuned with AdamW, a 0.65 dropout on the final layer, and four epochs.

What would settle it

Evaluate the same fine-tuned model on a held-out set of learner sentences whose grammaticality labels were assigned by human annotators rather than inferred from the difference between original and corrected text; if the F1 drops far below 0.91, the model is detecting edit propensity rather than grammar.

Watch

Extended reading notes

Core claim

The central discovery is that BERT-base-uncased, fine-tuned on a carefully filtered subset of Lang-8, reaches 0.91 F1 and 90.53% accuracy for binary sentence-level grammaticality classification on cleaned Lang-8 test data, while larger models (BERT-large-uncased, RoBERTa-large) trained on a 20,000-sentence sample do not meaningfully improve over the base model, and zero-shot GPT-4 and Llama-3-70B-instruct perform substantially worse. The paper attributes the gain primarily to an eight-step cleaning pipeline that removes near-identical pairs, normalizes casing and punctuation, expands contractions, and restricts pairs to a normalized Levenshtein-distance window between 0.08 and 0.5, so that every training pair is a clear original/correction contrast. It also shows that fine-tuning on sentences discarded by the cleaning process yields test accuracy around 50% on the cleaned test set, which the authors interpret as evidence that data quality, not model capacity, drives performance.

Load-bearing premise

The paper assumes that every sentence pair surviving its cleaning filters is a genuine example of a grammatical error, meaning the difference between the learner's original text and the corrected version is exactly what a GED model should learn to flag.

Editorial extensions

If this is right

  • A cleaned, filtered version of Lang-8 — 180,000 pairs with Levenshtein distance between 7 and 42 and normalized distance between 0.08 and 0.5 — is enough to train a 110M-parameter BERT-base to 0.91 F1 on the same cleaned distribution, so data curation can substitute for model scale in GED.
  • Models trained on this cleaned set do not transfer to the 1.3 million discarded Lang-8 pairs: test accuracy there stays near 50%, meaning the claimed performance is distribution-specific.
  • Fine-tuned BERT-base (180k) outperforms zero-shot GPT-4 and Llama-3-70B-instruct by a wide margin on a 500-sentence sample (F1 0.99 vs 0.63 and 0.60), implying that task-specific fine-tuning on cleaned data remains necessary for sentence-level grammaticality classification.
  • Freezing most BERT layers individually leaves validation F1 at 0.84, so the grammaticality signal appears to be distributed across the network rather than concentrated in particular layers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the labels are defined by the cleaning filters, the 0.91 F1 measures agreement with an edit-distance operationalization of error, not with human annotation; an independent human-labeled test set would likely show smaller gaps between models but lower absolute F1.
  • A testable extension would be to feed the discarded Lang-8 pairs back into training with soft labels derived from their Levenshtein distance to the corrected version, which might improve robustness on unfiltered learner text without sacrificing performance on the cleaned distribution.
  • The normalized Levenshtein window suggests an interpretable trade-off curve: tightening the window makes training pairs easier to separate (higher F1 on filtered test sets) while loosening it should improve generalization to noisier text; plotting F1 against the window bounds would quantify how much of the reported gain is an artifact of filtering.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript describes a data-cleaning pipeline for the Lang-8 learner corpus and fine-tunes BERT-family models for sentence-level grammatical error detection (GED). The cleaning pipeline removes identical pairs and filters by Levenshtein distance and length, producing a 200,000-sentence dataset. The authors report that BERT-base-uncased fine-tuned on 180k cleaned sentences achieves F1 0.91 and 90.53% test accuracy on a self-constructed validation split, and that larger models (BERT-large, RoBERTa-large) do not improve over BERT-base. The paper also reports zero-shot comparisons with GPT-4 and Llama-3-70B-instruct, on which the fine-tuned models perform better.

Significance. If the findings were transferable to standard GED benchmarks, the paper would provide a useful demonstration that rigorous cleaning of learner data and a moderately sized model can outperform larger models, and the released cleaned dataset could be a practical resource. The paper deserves credit for making its dataset publicly available on GitHub and for reporting training times. However, the significance is currently conditional: the headline F1 is measured only on the paper's own cleaned Lang-8 split, the label construction conflates any learner-corrector edit with a grammatical error, and no independent benchmark is used. These issues make the results difficult to interpret as progress on GED itself.

major comments (4)
  1. [§2.1.8] The label rule in §2.1.8 states that every Lang-8 pair surviving the cleaning filters is treated as 'column 0 grammatically incorrect, column 1 corrected.' This is the load-bearing premise of the entire evaluation. Lang-8 corrections can reflect stylistic rewrites, content additions, spelling changes, or annotator disagreement, not only grammatical errors, and the corrected sentence itself may be non-native. Additionally, the cleaning procedure never restricts the corpus to English despite using English-only BERT-base-uncased. Please validate the labels by expert or crowd-sourced annotation on a random sample (with inter-annotator agreement), and report the language distribution of the 200,000 retained sentences. Without such validation, the F1 0.91 could be measuring 'editedness' or script-based artifacts rather than grammaticality.
  2. [Table 3] Table 3 shows that when the same BERT-base-uncased models are tested on discarded Lang-8 sentences (sentences removed by the cleaning filters), test accuracy is consistently near 50% and F1 is between 0.41 and 0.50 across all training compositions. This is precisely what would be expected if the model has learned the distributional signature of the cleaning filter rather than a general grammaticality signal. The paper interprets Table 3 as underscoring the importance of cleaning, but it also demonstrates that the model's knowledge does not transfer to sentences that do not satisfy the filter. Please provide a concrete explanation of why near-chance accuracy on discarded sentences is consistent with the claim that the model performs grammatical error detection in general, and ideally evaluate on an independently annotated corpus.
  3. [§2.1 (last paragraph)] The train/validation split is non-random: 90k sentences 'from the top' of column 0, 90k 'from the bottom' of column 1, and the remaining 20,000 'mid' sentences for validation. If the Lang-8 rows are ordered by language, author, or document, this split can create positional or script-based cues, and the label may be predictable without any linguistic understanding. The split should be randomized and stratified (e.g., by language and author), and results should be reported with variance across multiple seeds. The duplicate '10k + 10k' rows in Table 2 (F1 0.81 and 0.76) indicate run-to-run variability that is undocumented, so single-point results are not sufficient.
  4. [Evaluation protocol (Tables 2 and 5)] No external GED benchmark is used. The only test set is the self-constructed cleaned Lang-8 split, and the dataset citation (Nihalani and Shah 2024) is the authors' own GitHub artifact. Consequently, the reported F1 0.91 is not comparable to the prior GED numbers cited in the introduction (e.g., Bi-LSTM F1 0.80–0.90). Please evaluate on a standard benchmark such as CoNLL-2014 or BEA-2019, and report precision, recall, and F1 with error bars. Without an independent test set, the claims that the proposed method 'improves GED' and that 'larger models are not always better' are not supported.
minor comments (5)
  1. [§3.2.1 / Table 5] Text says BERT-base-uncased 20k has a recall of 0.97, but Table 5 reports recall 0.87; correct the inconsistency.
  2. [Table 4] The caption defines 'UT - Undertrained layers, OT - Over-trained layers' but the table does not identify which specific layers are under- or over-trained; state how these labels are assigned.
  3. [Table 2 caption] The caption explains 'Batch Sizes mentions how many sentences were taken from Cleaned and Discarded Lang8 set respectively,' but the table lists identical batch compositions (e.g., 10k + 10k, 8k + 12k) with different metrics; explain the difference (e.g., different random seeds or checkpoint epochs).
  4. [§2.1.7] 'Only sentences with a Levenshtein distance between 7 and 42 and lengths less than 101 characters were kept' does not specify which column's length is used; state whether the length threshold applies to column 0, column 1, or both.
  5. [Table 6 caption] The caption says TP is 'Correct sentences correctly identified as Correct' and TN is 'Incorrect sentences correctly identified as Incorrect,' which inverts the usual TP/TN convention when the positive class denotes incorrect sentences; clarify the label convention in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the 0.91 F1 is an empirical evaluation on held-out Lang-8 labels, not a reconstruction of the paper's own inputs.

full rationale

The paper's chain is: (1) take Lang-8 learner/correction pairs; (2) remove pairs that are identical or fail length/Levenshtein filters; (3) label column-0 sentences as incorrect and column-1 sentences as correct; (4) fine-tune BERT-base on 180k of these sentences; (5) evaluate on the remaining 20k. The labels come from human corrections in a pre-existing corpus, not from the model's predictions or from a fitted parameter. The cleaning thresholds (Levenshtein distance 7-42, normalized 0.08-0.5, length <101) define the dataset, but they are not functions of the test labels, and no quantity fitted on the test set is later reported as a prediction. The only self-citation is the GitHub pointer to the author-created cleaned dataset; since the cleaning pipeline is fully specified in Section 2.1, the citation is not load-bearing. The lack of an independent benchmark and the fact that the test set comes from the same cleaning procedure are legitimate generalization concerns, but they are not circularity: the model is not guaranteed to succeed on the held-out set by construction, and the paper's Table 3 even shows near-chance performance on discarded sentences. This is an empirical result, not a definitional tautology.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim rests on hand-picked cleaning thresholds, a self-built label rule, and a test set drawn from the same filtered distribution. No external benchmark or independent ground truth anchors the reported F1.

free parameters (5)
  • Levenshtein distance lower and upper bounds = 7 and 42
    Hand-chosen in Section 2.1.7 to keep sentences 'neither too close nor too far' from corrections; defines the dataset and therefore the reported performance.
  • Normalized Levenshtein distance bounds = 0.08 to 0.5
    Hand-chosen in Section 2.1.8; the second filtering stage further shapes the evaluation distribution.
  • Maximum sentence length = 101 characters
    Hand-chosen cap in Section 2.1.7; excludes long sentences from training and test.
  • Final dataset size = 200,000 sentences
    Created from the 217,018 retained sentences in Section 2.1.8; the 180k training and 20k validation split is based on this.
  • Training hyperparameters = lr 2e-5, weight decay 0.2, epochs 4, dropout 0.65
    Chosen by hand in Section 2.2 with no sensitivity analysis, though they likely do not drive the main claim as much as the cleaning thresholds.
assumptions (4)
  • domain assumption Lang-8 corrected sentences are accurate grammatical ground truth.
    Section 2.1 uses column 1 as the corrected version and labels column 0 as incorrect; learner edits can be stylistic, orthographic, or content changes, not grammar errors.
  • domain assumption Binary sentence-level classification is a valid operationalization of GED.
    The paper evaluates sentence correctness, while the cited GED literature includes token-level detection and standard benchmarks; no comparison is made to those tasks.
  • domain assumption The cleaned Lang-8 subset is representative of English learner text in general.
    Filtering by Levenshtein distance bounds removes many sentence types; Section 3.1.2 shows models are at chance on discarded sentences, so representativeness is doubtful.
  • ad hoc to paper The top/bottom/mid split into training and validation has no leakage.
    Section 2.1.8 splits by row position rather than random assignment, and no duplicate removal across splits is described.
invented entities (1)
  • Cleaned Lang8 GED dataset independent evidence
    purpose: Training and evaluation corpus for binary sentence-level grammatical error detection.
    Publicly linked on GitHub, but no commit hash or checksum is given, so the exact artifact cannot be independently verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Grammatical Error Detection using BERT with Cleaned Lang-8 Dataset." pith.science (2026). https://pith.science/paper/J3MTISB7

@misc{pith2026241115523,
  author       = {Pith},
  title        = {Pith review of: Enhancing Grammatical Error Detection using BERT with Cleaned Lang-8 Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J3MTISB7}},
  note         = {Machine review of arXiv:2411.15523}
}
read the original abstract

This paper presents an improved LLM based model for Grammatical Error Detection (GED), which is a very challenging and equally important problem for many applications. The traditional approach to GED involved hand-designed features, but recently, Neural Networks (NN) have automated the discovery of these features, improving performance in GED. Traditional rule-based systems have an F1 score of 0.50-0.60 and earlier machine learning models give an F1 score of 0.65-0.75, including decision trees and simple neural networks. Previous deep learning models, for example, Bi-LSTM, have reported F1 scores within the range from 0.80 to 0.90. In our study, we have fine-tuned various transformer models using the Lang8 dataset rigorously cleaned by us. In our experiments, the BERT-base-uncased model gave an impressive performance with an F1 score of 0.91 and accuracy of 98.49% on training data and 90.53% on testing data, also showcasing the importance of data cleaning. Increasing model size using BERT-large-uncased or RoBERTa-large did not give any noticeable improvements in performance or advantage for this task, underscoring that larger models are not always better. Our results clearly show how far rigorous data cleaning and simple transformer-based models can go toward significantly improving the quality of GED.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 11 canonical work pages

  1. [6]

    Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation

    URL https://arxiv.org/abs/2305.14734. Lvxiaowei Xu, Jianwang Wu, Jiawei Peng, Jiayu Fu, and Ming Ca i. Fcgec: Fine-grained corpus for chi- nese grammatical error correction. In Findings of the Association for Computational Linguistics : EMNLP

    work page Pith review arXiv

  2. [8]

    Marek Rei

    doi:10.1109/ACCESS.2023.3280549 . Marek Rei. Semi-supervised multitask learning for sequenc e labeling,

    arXiv 2023

  3. [10]

    Wronging a Right: Generating Better Errors to Improve Grammatical Error Detection

    URL https://arxiv.org/abs/1810.00668. Christopher Davis, Christopher Bryant, Andrew Caines, Mar ek Rei, and Paula Buttery. Probing for targeted syntactic knowledge through grammatical error detection,

    work page Pith review arXiv

  4. [11]

    Probing for targeted syntactic knowledge through grammatical error detection

    URL https://arxiv.org/abs/2210.16228. Nora Madi and Hend S. Al-Khalifa. A proposed arabic grammati cal error detection tool based on deep learning. Pro- cedia Computer Science , 142:352–355,

    work page Pith review arXiv

  5. [12]

    doi:https://doi.org /10.1016/j.procs.2018.10.482

    ISSN 1877-0509. doi:https://doi.org /10.1016/j.procs.2018.10.482. URL https://www.sciencedirect.com/science/article/pii/S1877050918321860. Arabic Computa- tional Linguistics. Romila Aziz, Muhammad Waqas Anwar, Muhammad Hasan Jamal, Usama Ijaz Bajwa, Ángel Kuc Castilla, Carlos Uc Rios, Ernesto Bautista Thompson, and Imran Ashraf. Real wor d spelling error ...

    work page doi:10.1016/j.procs.2018.10.482 2018

  6. [13]

    doi:10.1109/ACCESS.2023.3312

    work page arXiv 2023

  7. [14]

    URL https://arxiv.org/abs/2308.08982. R. Nihalani and K. Shah. Cleaned lang8 ged,

    work page Pith review arXiv

  8. [15]

    URL https://doi.org/10.1038/s41467-021-24025-8

    doi:10.1038/s41467-021-24025-8. URL https://doi.org/10.1038/s41467-021-24025-8 . OpenAI, Josh Achiam, and Steven Adler et al. Gpt-4 technical report,

    doi:10.1038/s41467-021-24025-8

Show all 16 references
  1. [2015]

    doi:10.3115/v1/N15-1060

    Association for Computat ional Linguistics. doi:10.3115/v1/N15-1060. URL https://aclanthology.org/N15-1060. Martin Chodorow, Markus Dickinson, Ross Israel, and Joel Te treault. Problems in evaluating grammat- ical error detection systems. In Martin Kay and Christian Bo itet, e...

    doi

  2. [2017]

    Sudhanshu Kasewa, Pontus Stenetorp, and Sebastian Riedel

    URL https://arxiv.org/abs/1704.07156. Sudhanshu Kasewa, Pontus Stenetorp, and Sebastian Riedel. Wronging a right: Generating better errors to improve grammatical error detection,

    arXiv

  3. [2018]

    Mariano Felice and Ted Briscoe

    URL https://arxiv.org/abs/1811.05949. Mariano Felice and Ted Briscoe. Towards a standard evaluation method for grammatical error detection and correction. In Rada Mihalcea, Joyce Chai, and Anoop Sarkar, editors, Proceedings of the 2015 Conference of the North Amer- ican Chapte...

    2015 arXiv

  4. [2019]

    URL http://dx.doi.org/10.18653/v1/W19-4410

    doi:10 .18653/v1/w19-4410. URL http://dx.doi.org/10.18653/v1/W19-4410. H. A. Z. Sameen Shahgir and Khondker Salman Sayeed. Bangla gr ammatical error detection using t5 transformer model,

    doi

  5. [2021]

    Samuel Bell, Helen Y annakoudakis, and Marek Rei

    URL https://arxiv.org/abs/2105.04443. Samuel Bell, Helen Y annakoudakis, and Marek Rei. Context is key: Grammatical error detection with contex- tual word representations. In Proceedings of the F ourteenth W orkshop on Innovative Use of NLP for Building Educational Application...

    arXiv

  6. [2022]

    URL http://dx.doi.org/10.18653/v1/2022.findings-emnlp.1

    doi:10 .18653/v1/2022.findings-emnlp.137. URL http://dx.doi.org/10.18653/v1/2022.findings-emnlp.1

    2022 doi

  7. [2023]

    9 Enhancing Grammatical Error Detection using BERT with Clea ned Lang-8 Dataset A P REPRINT Bashar Alhafni, Go Inoue, Christian Khairallah, and Nizar H abash

    URL https://arxiv.org/abs/2303.10612. 9 Enhancing Grammatical Error Detection using BERT with Clea ned Lang-8 Dataset A P REPRINT Bashar Alhafni, Go Inoue, Christian Khairallah, and Nizar H abash. Advancements in arabic grammatical error detec- tion and correction: An empirica...

    arXiv

  8. [2024]

    URL https://arxiv.org/abs/2303.08774. Meta AI. Meta llama 3,

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.