{"id":"dfe182a9-95e7-4582-93cb-db83bdba9f4d","arxiv_id":"2501.07740","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning Llama-2 and Mistral models on a new GPT-generated essay-syntax-feedback dataset improves the quality of automated syntax corrections, according to human ratings.","lead":"This paper creates a dataset of student essays paired with automated feedback on sentence structure, generated by GPT-3.5, and uses it to fine-tune smaller language models. The fine-tuned models produce better syntax feedback than their base versions, which is a step toward low-cost automated writing support for classrooms.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning's claimed benefit rests on an unblinded human evaluation that may reward format mimicry; a gold-standard error-level test is needed.","rationale":"The reader's weakest assumption concerned the quality of GPT-3.5-generated feedback as both training target and evaluation reference. My analysis agrees that the automatic evaluation is circular and that label errors can propagate, but I identify the more directly load-bearing weakness as the human evaluation: it is the only non-circular evidence for the central claim, and as reported it lacks blinding, inter-annotator agreement, statistical testing, and a comparison against the GPT-3.5 teacher. The rubric's emphasis on structured output makes it plausible that fine-tuned models are rewarded for format imitation rather than for better syntax error detection. This is not an accusation of dishonesty; it is a description of what the manuscript does and does not establish. I give credit for the public dataset and checkpoints, the reproducible LoRA training details, and the fact that the human ratings, even if imperfect, directionally favor fine-tuning. The verdict should remain CONDITIONAL rather than being upgraded to ACCEPT, because the main empirical claim needs the proposed gold-standard and blinded verification before it can be taken as established.","tokens_in":11474,"tokens_out":3492,"duration_ms":38739,"concrete_test":"Construct a gold-standard test by having two expert linguists independently annotate syntax errors, with error spans and corrections, in the seven target categories on a held-out sample of 100 student essays, and report inter-annotator agreement. Then compute per-error recall, precision, and F1 for each base and fine-tuned model against this gold standard, and also obtain blinded pairwise human preference ratings in which raters do not know which output is base versus fine-tuned. If fine-tuned models do not show a statistically significant advantage in error-level F1 or in blinded preferences over the base models, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that fine-tuning significantly enhances syntax feedback quality, cannot rest on the ROUGE scores in Table 1 because those scores compare against GPT-3.5 outputs that are also the training targets; any fine-tuned model trained to imitate GPT-3.5 will trivially raise ROUGE against that same model. The claim therefore depends entirely on the human ratings in Table 2 and Figure 7. Those ratings are not reported with inter-annotator agreement, significance tests, or evidence of blinding, and the rubric explicitly rewards a 'structured approach'. A base model that does not follow the requested bullet format will score D or E regardless of whether it detects real errors, while a fine-tuned model that imitates GPT-3.5's layout will receive B or C even if it identifies no more actual errors. Consistent with this, Table 2 shows the large gains after fine-tuning are in rating B (rising to 44-65%), while rating A, which requires systematically identifying every error, barely moves (2.7-4.7%). Figure 7 additionally labels the evaluated models as gemma2 and llama3.1, model families never mentioned in the text, which further undermines confidence in the reported human evaluation. Thus the load-bearing condition is that the human evaluation isolates genuine syntax error detection and correction quality from format adherence; the paper does not currently establish this.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Essay-Syntax-Instruct, a dataset of student essays from ASAP with syntax feedback generated by GPT-3.5-Turbo under a seven-category prompt, after placeholder replacement. It reports a human quality check of 300 generated feedback samples (29% A, 63.3% B) and uses the dataset to fine-tune Llama-2-7B, Llama-2-13B, and Mistral-7B with LoRA. Evaluation is based on ROUGE scores against GPT-3.5 feedback and on human ratings of base versus fine-tuned outputs. The paper's central claim is that fine-tuning with this dataset significantly improves the quality of automatic syntax feedback for student essays.","tokens_in":11743,"tokens_out":6517,"duration_ms":63369,"significance":"The dataset and checkpoints are released publicly, which is a useful contribution for the AI-in-education community, and the idea of using LLM-generated structured feedback on authentic student essays is timely. If the reported improvements are genuine, the paper provides a low-cost recipe for building syntax feedback systems from open models. However, the current evidence is conditional: the automatic metric is partly circular, and the human evaluation is not yet shown to measure error-identification quality independently of format adherence. The significance of the work therefore depends on whether the authors can strengthen the evaluation with agreement metrics, significance tests, and an error-level analysis.","major_comments":[{"comment":"The ROUGE evaluation compares model outputs against GPT-3.5-Turbo feedback that also served as the fine-tuning target. Since the fine-tuned models are trained to imitate that teacher, higher ROUGE-1/2/L after fine-tuning is expected and does not by itself demonstrate improved syntax feedback quality. The text acknowledges that ROUGE is unreliable, but the paper still presents the gains as supportive; please reframe Table 1 as a fidelity-to-teacher measure or replace it with an evaluation on a human-authored gold standard.","section":"Section 4.2, Table 1"},{"comment":"The human evaluation has no inter-annotator agreement, no significance testing, and no evidence of blinding, and the rating rubric explicitly rewards a structured approach. The post-fine-tuning gains are concentrated in Rating B (e.g., Mistral from 10.33% to 65.67%) while Rating A stays nearly flat (4.00% to 4.67%), which is consistent with the hypothesis that fine-tuning improves format adherence rather than error identification. To support the central claim, report per-rater distributions and agreement (e.g., Cohen's kappa), perform significance tests, and ideally evaluate at the level of individual errors (precision/recall of flagged errors against a gold standard).","section":"Section 4.2, Table 2 and Figure 7"},{"comment":"The figure legend names 'gemma2 7b' and 'llama3.1 8b', models that are never mentioned in the methodology or results text, while Table 2 reports Llama2 and Mistral. This prevents the reader from knowing which bars correspond to which fine-tuned model. Please correct the figure or align its labels with the model names used throughout the paper.","section":"Section 4.2, Figure 7"},{"comment":"The paper says the 300 annotated instances were used as the test set, and later describes human evaluation on a 'subset of 300 essays from the test dataset.' It is not stated explicitly whether these are the same 300 essays. Please clarify the split and state, for the human evaluation, whether the raters saw the same essays as those used in the dataset quality check; this is needed to interpret the comparison.","section":"Sections 3.1 and 4.1"},{"comment":"The quality check of the training data shows only 29% Rating A and 63.3% Rating B for GPT-3.5 feedback. Because the fine-tuning target contains a substantial amount of imperfect feedback, the paper should analyze whether systematic teacher errors are propagated by the fine-tuned models. A minimal addition would be a breakdown of the low-rated feedback by syntax category and a qualitative check of whether such errors reappear in the fine-tuned outputs.","section":"Section 3.1"}],"minor_comments":[{"comment":"The paper repeatedly writes 'ROGUE scores' where the metric is ROUGE; please correct this throughout, starting in Section 3.1.","section":"Throughout"},{"comment":"There are typographical errors in model names, including 'LLlama-2-13b-chat-hf' and 'LLlama-2-chat-hf'; please proofread all model names.","section":"Introduction and Related Work"},{"comment":"The legend contains the typo 'mistal 7b FT'; it should be 'Mistral 7b FT'.","section":"Figure 7"},{"comment":"The phrase 'we randomly selected a sample of clues' should be 'a sample of essays' or 'a sample of feedback instances'.","section":"Section 3.1"},{"comment":"The sentence 'model distribution sampling facilitated clue generation' is unclear; it should be rewritten as, for example, 'output generation'.","section":"Section 4.1"},{"comment":"The final word 'Students' is capitalized; it should be lowercase.","section":"Abstract"},{"comment":"The related work does not discuss grammatical error correction (GEC) systems or widely used grammar feedback tools; adding a brief positioning paragraph would help readers understand the contribution relative to existing writing-support technology.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an applied NLP/AIED venue, but the evaluation section is not yet at the standard required for the central claim. I do not suspect intentional misreporting, but the Figure 7 labels and the missing experimental details must be corrected. The main risk is that the reported gains are largely format effects; if the authors add agreement metrics, significance tests, and error-level analysis in revision, the contribution could become suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this paper's real asset is the Essay-Syntax-Instruct dataset — 8,320 student essays with GPT-3.5-generated syntax feedback in seven categories, plus fine-tuned Llama-2 and Mistral checkpoints. That is a genuinely new resource and they put it on Hugging Face. If you work in educational NLP, it is worth a look.\n\nWhat the paper does well: it is transparent about the generation pipeline, releases the models and data, and it explicitly admits ROUGE is not reliable for this task. The human evaluation, while flawed, does show a consistent directional shift toward higher ratings after fine-tuning. The authors also flag that GPT-3.5's own feedback was only 29% A and 63.3% B, which is honest.\n\nThe soft spots are real and load-bearing. The central claim — that fine-tuning significantly improves syntax feedback — rests almost entirely on the human ratings in Table 2 and Figure 7. But those ratings have no inter-annotator agreement, no significance testing, no evidence of blinding, and the rubric explicitly rewards a structured format. The pattern in Table 2 is telling: the big gains are in rating B, while rating A, which requires catching every error, barely moves (2.7–4.7%). That is exactly what you would expect if fine-tuning mostly teaches the model to imitate GPT-3.5's bullet layout, not to detect more real errors. The stress-test note also caught something important: Figure 7 labels the models as gemma2 7b and llama3.1 8b, which are never mentioned in the text or tables. That discrepancy makes me wonder whether the reported human evaluation was actually run on the models the paper describes.\n\nThe automatic ROUGE comparison is circular — GPT-3.5 outputs are the training targets and the references — but the authors acknowledge its limits. The missing piece is a serious external evaluation: an error-level gold standard, two independent blinded raters, a comparison against the teacher model GPT-3.5, and significance tests. Without that, the paper's headline claim overreaches.\n\nWho this is for: researchers in educational NLP who want a reusable dataset and cheap open models for syntax feedback. It deserves a serious referee because the resource is valuable and the question is important, but it needs major revision on evaluation before the claims can stand.\n\nMy recommendation: send it to review, but make clear the evaluation section needs substantial work — a blinded, error-level study with agreement metrics, not another format-mimicry comparison.","headline":"The dataset is a useful, public resource; the performance claim needs a blinded, error-level evaluation before I'd trust it.","tokens_in":12294,"tokens_out":2490,"would_cite":true,"duration_ms":24175,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning turns open LLMs into useful essay syntax tutors","keywords":["syntax feedback","automated essay feedback","large language models","fine-tuning","Essay-Syntax-Instruct dataset","student writing","Llama-2","Mistral-7B"],"falsifier":"Take a held-out set of student essays, have expert writing instructors independently identify every syntax error and correction in the seven categories, and compare the fine-tuned models' feedback against those expert annotations. If the fine-tuned models recover the expert corrections no more often than their base versions do—or if human raters who are blind to which model produced the feedback see no quality difference—then the paper's central claim of improvement would be refuted.","tokens_in":11287,"feed_emoji":"✍️","tokens_out":4937,"duration_ms":42246,"temperature":0.7,"pith_summary":"This paper argues that fine-tuning open-weight large language models on a purpose-built dataset of essay feedback turns them into usable automatic syntax tutors. The authors built Essay-Syntax-Instruct, 8,320 student essays paired with structured syntax feedback generated by GPT-3.5-Turbo across seven error categories. Fine-tuning Llama-2-7B, Llama-2-13B, and Mistral-7B on this dataset raised their ROUGE overlap with the reference feedback and, in human evaluation, shifted outputs from poor ratings toward good ones. If the result holds, schools and researchers get a low-cost, reusable way to generate constructive syntax feedback on student writing without relying on a paid API.","feed_headline":"Fine-tuning turns open LLMs into useful essay syntax tutors","feed_subtitle":"A new 8,320-pair dataset lifts Llama-2 and Mistral feedback from D-level to mostly B-level, no API required.","key_machinery":"The load-bearing object is the Essay-Syntax-Instruct dataset: 8,320 essay–feedback pairs built from ASAP essays after GPT-3.5 replaced anonymized placeholders, with feedback generated by a GPT-3.5-Turbo prompt that checks seven syntax categories (misspelled words, conjunctions and linking phrases, modifiers, prepositions, modal verbs, punctuation, articles) and reports each error with a corrected version or writes N/A. Fine-tuning uses LoRA (r=32, alpha=64) for three epochs at a 3e-4 learning rate, and evaluation combines ROUGE scores against GPT-3.5's feedback with human ratings from a five-tier A–E scale. The dataset's structured format is what teaches the models to output feedback that is both formatted and substantively aligned.","core_discovery":"The central claim is that base chat models produce noisy, often misformatted syntax feedback, and that a modest amount of instruction fine-tuning on this dataset substantially corrects that behavior. After fine-tuning, the models' generated feedback follows the requested seven-category structure more faithfully, repeats fewer errors across categories, and earns higher human ratings; for example, Mistral-7B's B ratings rose from 10.33% to 65.67% and its D ratings fell from 43.33% to 4.00%. The paper treats the Essay-Syntax-Instruct dataset, with its structured error-correction format, as the mechanism that induces this improvement.","pith_inferences":["Because the same GPT-3.5 model generated the training labels and serves as the ROUGE reference, part of the measured gain may reflect imitation of the teacher's style rather than objectively better corrections; an independent expert-labeled gold set would separate the two.","The human evaluation suggests a ceiling set by the teacher: even the best fine-tuned model reached only 4.67% A ratings, roughly the rate at which the teacher itself was rated A on the same scale, so improving the teacher (or adding expert corrections) may be the next bottleneck.","A natural testable extension would be a classroom study measuring whether students who receive this feedback show measurable syntax gains over a semester; the paper lists this as future work, but it is the decisive practical test."],"forward_implications":["Schools and districts can deploy the released Llama-2 and Mistral checkpoints locally to give students immediate, structured syntax feedback without sending essays to a commercial API.","The dataset provides a new benchmark for syntax feedback generation, letting future models be compared on the same essay set and rating rubric.","The observed shift from D-level to B-level outputs suggests that instruction tuning on structured error-correction data is an effective general recipe for feedback tasks, not just for syntax.","Researchers can reuse the pipeline—ASAP essays, placeholder replacement, GPT-generated feedback, human vetting—to build similar datasets for other writing dimensions such as coherence or argument strength.","The fine-tuned models' higher ROUGE and human ratings imply that even small 7B-parameter models can serve as practical feedback generators when the training data is well structured."],"supporting_citations":[{"why":"Supplies the five-tier human rating scale used to evaluate both the GPT-3.5-generated dataset and the fine-tuned models' feedback.","marker":"[24]"},{"why":"Provides the Mistral-7B-Instruct-v0.2 base model that the paper fine-tunes.","marker":"[25]"},{"why":"Gives the low-rank adaptation method used for all fine-tuning experiments.","marker":"[28]"},{"why":"Provides the Llama-2-7b-chat and Llama-2-13b-chat base models that the paper fine-tunes.","marker":"[30]"}],"fun_headline_variants":["Fine-tuned LLMs give sharper essay syntax tips","8K examples lift open LLM grammar feedback to B-level","From D to B: Fine-tuning boosts syntax tutoring","Mistral's grammar advice improves with new dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training labels and the evaluation reference both come from GPT-3.5, whose own feedback earned only 29% A and 63.3% B ratings in the paper's human check; if that teacher makes systematic syntax mistakes, fine-tuning will bake them in and ROUGE will reward imitation rather than correction.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned LLMs give sharper essay syntax tips","8K examples lift open LLM grammar feedback to B-level","From D to B: Fine-tuning boosts syntax tutoring","Mistral's grammar advice improves with new dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1318,"prompt_tokens":881,"completion_tokens":437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":372}},"tokens_in":497,"tokens_out":437,"duration_ms":5426,"temperature":1.0,"reasoning_tokens":372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:36:20.626864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of student essays, have expert writing instructors independently identify every syntax error and correction in the seven categories, and compare the fine-tuned models' feedback against those expert annotations. If the fine-tuned models recover the expert corrections no more often than their base versions do—or if human raters who are blind to which model produced the feedback see no quality difference—then the paper's central claim of improvement would be refuted.","supporting_citations":[],"review_version":1}