{"id":"8399b5d8-b3d2-4730-b8e8-4797916b42bf","arxiv_id":"2502.09497","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Adding handcrafted linguistic features to zero-shot LLM prompts modestly improves automatic essay scoring for Mistral-7B on ASAP and ELLIPSE, but GPT-4 shows no benefit on ASAP and no significance tests are provided.","lead":"This paper tests whether giving a large language model a short list of essay statistics, like word count and unique words, improves its essay grading accuracy. The method helps a 7-billion-parameter model score essays slightly better on both familiar and unfamiliar prompts, but the gains are small and not statistically verified.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-fold QWK gains (Mistral: .454→.492 ASAP, .454→.481 ELLIPSE) have no confidence intervals or significance tests, and the prompt both adds text and tells the model higher features mean higher scores; the improvement is not yet distinguished from noise or prompt-length effects.","rationale":"The reader's weakest assumption is exactly the one I find most load-bearing: the observed QWK gains are asserted as a causal effect of linguistic features, but the paper provides no variance estimates, significance tests, or a non-informative-prompt control. My reading of Table 1 strengthens this concern: the average ASAP improvement is carried by a minority of subsets, GPT-4 results do not replicate the benefit on ASAP, and the ELLIPSE 'best' configuration was chosen after inspecting test-set numbers. These facts do not prove the method is useless; they mean the central empirical claim is not yet statistically substantiated. The paper does release code and reports parsing error rates, which is helpful, but reproducibility of the pipeline does not fix the missing inferential support. Because the reader already recommended conditional acceptance, this concern does not move the verdict; it reinforces the conditions: confidence intervals, a length-matched control, and a corrected abstract. I therefore mark agreement as 'agree' and leave the verdict unchanged.","tokens_in":14082,"tokens_out":5343,"duration_ms":49571,"concrete_test":"Run a 5-fold cross-validation of the exact Mistral pipeline with the None, Unique-Word, Top-3, Top-10, and a matched-length non-informative control prompt, then bootstrap the per-fold QWK differences (e.g., 10,000 resamples of essays within folds). Report the 95% confidence interval for the mean difference between each feature prompt and None on ASAP and ELLIPSE. If any CI includes 0, or the non-informative control produces the same gain, the 'notably improve' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Table 1: Mistral Top-10 vs None = .492 vs .454 (ASAP) and Top-3 vs None = .481 vs .454 (ELLIPSE). These are single-run comparisons: ASAP uses one held-out fold (§4.1) and ELLIPSE is scored once in full; no confidence intervals, bootstrap, or significance test is reported. The subset-level pattern is mixed: Top-10 improves only 5 of 8 ASAP sets and lowers QWK on sets 3 and 6, so the average gain is driven mainly by sets 1 and 7. The prompt also explicitly instructs the model that 'higher features typically mean higher end score' and adds ~10 lines of text, with no control prompt of matched length containing non-informative statistics; either the instruction or the extra length could produce part of the gain. Feature choice and prompt phrasing were tuned on the ASAP dev set (Appendix C), and the ELLIPSE Top-3 result was selected after seeing the test outputs, so multiple comparisons inflate the apparent effect. With GPT-4 the Top-10 condition actually lowers ASAP QWK (.499 to .488), which undercuts the abstract's 'both in-domain and out-of-domain' claim. Thus the improvement is not yet shown to be real beyond noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a zero-shot prompting method for automatic essay scoring (AES) that augments the prompt with handcrafted linguistic features of the essay, such as unique word count, lemma count, and Dale-Chall word count. The method is evaluated on two datasets, ASAP and ELLIPSE, with Mistral-7B-Instruct and GPT-4, using Quadratic Weighted Kappa (QWK) as the metric. The central claim is that incorporating linguistic features into existing zero-shot prompts notably improves score prediction, both in-domain (ASAP) and out-of-domain (ELLIPSE), without fine-tuning. The paper also compares against a supervised BERT baseline on ASAP and discusses the headroom for open-source LLMs. The main numerical support is Table 1, where Mistral-7B with Top-10 features reaches 0.492 on ASAP versus 0.454 without features, and with Top-3 features reaches 0.481 on ELLIPSE versus 0.454.","tokens_in":14458,"tokens_out":4647,"duration_ms":39671,"significance":"If the reported effect is real, the contribution is practically valuable: a simple, training-free prompt modification that improves LLM-based AES agreement with human scores, with some evidence of cross-prompt transfer. The paper is clearly written and releases code, uses two established datasets, and compares open and closed LLMs, which are strengths. However, the empirical evidence is currently too thin to support the central claim. The results are single-point QWK estimates without confidence intervals or significance tests, the feature condition is confounded with extra prompt text and an explicit correlation instruction, and feature/prompt variants were selected on the development or test data. These issues are fixable within the manuscript's scope, so the contribution is plausible but not yet demonstrated.","major_comments":[{"comment":"The central claim rests on single-point QWK estimates: Mistral Top-10 vs None on ASAP (.492 vs .454) and Top-3 vs None on ELLIPSE (.481 vs .454). No confidence intervals, bootstrap, or significance test is provided, and ASAP is evaluated on a single fold. These differences could be within sampling noise; the contribution statement that linguistic features 'notably improve' score prediction is not supported by the reported statistics without an uncertainty estimate.","section":"Table 1, §4.1, §5.1"},{"comment":"The 'additional information' condition adds both several lines of feature text and the explicit instruction that 'higher features typically mean higher end score.' There is no control prompt of matched length with non-informative statistics, so the observed gain could be due to extra token length, instruction-following, or the specific feature values rather than the features themselves. This confound is load-bearing because the paper's contribution is specifically the addition of linguistic features, not generic prompt extension.","section":"§3.1, Appendix C.0.2"},{"comment":"The feature variants and prompt phrasing were tuned on the ASAP dev set, and the ELLIPSE 'best' Top-3 result is identified after seeing test outputs. Since the paper reports three Mistral feature conditions and highlights the best-performing one on ELLIPSE to support the out-of-domain claim, the effect is inflated by multiple comparisons. The paper should report uncertainty for all conditions and, at a minimum, acknowledge this selection effect; ideally, the feature choice would be pre-registered or corrected.","section":"§4.3, Appendix C"},{"comment":"The abstract claims the hybrid method 'outperforms baseline models for both in-domain and out-of-domain writing prompts,' but Table 1 shows GPT-4 with Top-10 features scores lower on ASAP than GPT-4 with no features (.488 vs .499), and §5.2 concedes 'with the exception of GPT-4.' The abstract and the contributions section should be qualified to match the actual results and avoid overstating the improvement.","section":"Abstract, §5.2, Table 1"}],"minor_comments":[{"comment":"The Top-10 prompt lists 'total number of characters: 279' and later 'total number of characters: 1229' with no distinguishing label; one is the essay character length and the other the total character count. This duplicate label could confuse the model and should be fixed with separate names.","section":"Appendix C.0.2"},{"comment":"The sentence beginning 'Another research Chen et al. (2024) that conducted a zero-shot prompting experiment...' is grammatically incomplete and the citation should be formatted properly.","section":"§5.1"},{"comment":"The Limitations section says 'there is only one open-source and one close-source one'; this should read 'only one open-source and one closed-source model.'","section":"§7"},{"comment":"The statement 'We use default sampling parameters in vllm framework during decoding' should specify the exact decoding parameters (e.g., temperature, top_p, max_tokens) for reproducibility, since temperature 0 is already mentioned but other defaults are not.","section":"§4.3"},{"comment":"The bibliography entry for Devlin et al. is missing a publication year; the year should be added.","section":"References"},{"comment":"Appendix E reports 'Overall parsing error is less than 7%' without explaining how the error was measured or on which essays; a brief description would improve reproducibility.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of a computational linguistics journal, and the availability of code and use of two standard datasets are positive features. The main weakness is statistical: the headline comparisons are single-fold point estimates with no confidence intervals, and the feature condition is confounded with instruction-following. I believe these issues can be addressed with additional experiments and revised framing, so major revision is appropriate. The abstract currently overstates the results and should be corrected in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a sensible, clearly-written empirical study of a simple idea—inject handcrafted linguistic feature counts into zero-shot LLM prompts for essay scoring—and the Mistral results are consistently in the right direction. But the central claim is not yet supported: single-fold QWK comparisons with no confidence intervals or significance tests, no control for prompt-length or instruction-induced bias, and feature/prompt selection on the dev set. The abstract overstates the result for GPT-4 on ASAP. Still, the paper is worth a serious referee because the question is real and the method is transparent.\n\nWhat's new: the specific comparison of zero-shot LLM AES with versus without numeric linguistic feature values in the prompt does not appear in the cited prior work. The paper builds on Stahl et al. and Ridley et al. in an honest way, reuses their templates and features, and releases code. The Mistral improvements—Top-10 on ASAP .454→.492, Top-3 on ELLIPSE .454→.481—are plausible and consistent across most subsets. The subset-level analysis is honest: it shows some sets don't improve.\n\nWhere it's soft: the evaluation is a single fold for ASAP and a single full-dataset pass for ELLIPSE. Table 1 has no error bars or significance tests. The gain is driven by a couple of sets. The prompt tells the model that higher features mean higher scores, and adds substantial text; without a control prompt of matched length with non-informative features, we can't separate the effect of information from instruction-following or length. Feature selection and prompt phrasing were tuned on the ASAP dev set, and the ELLIPSE Top-3 was chosen after seeing test outputs, so multiple comparisons are unaccounted for. GPT-4's drop on ASAP with Top-10 (.499→.488) directly contradicts the abstract's 'both in-domain and out-of-domain' claim; the paper even notes the exception in Section 5.2 but the abstract doesn't qualify it.\n\nThe paper would be much stronger with bootstrap CIs, a significance test (even a permutation test on QWK), and a non-informative-feature control. The code and prompt templates are already available, so these are feasible additions.\n\nWho it's for: researchers working on AES, LLM-as-evaluator, or prompt engineering for automated scoring. It's not a landmark, but it's a legitimate data point if the effect survives re-analysis. I'd send it to review, with the expectation of major revision. A careful referee should ask for the controls before accepting the causal claim.","headline":"Potentially useful prompt-level finding, but the evidence is not yet there: single-fold QWK without error bars, no control prompt, and an abstract that overstates the GPT-4 result.","tokens_in":14852,"tokens_out":1903,"would_cite":false,"duration_ms":17623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that appending ten hand-calculated linguistic features to a zero-shot LLM prompt raises agreement with human essay scores on both in-domain and out-of-distribution essays.","keywords":["automatic essay scoring","linguistic features","zero-shot prompting","LLM as evaluator","cross-prompt AES","quadratic weighted kappa","prompt engineering"],"falsifier":"Repeat the zero-shot evaluation on the same folds but replace the feature list with an equal-length list of random numbers, or with the same numbers presented without the correlation instruction; if QWK stays at the same level, the features are not the driver. A second, statistical test would compute bootstrap confidence intervals for the QWK differences; the claim is unsupported if the intervals for the with- and without-feature conditions overlap.","tokens_in":13872,"feed_emoji":"✍️","tokens_out":7096,"duration_ms":54830,"temperature":0.7,"pith_summary":"This paper tries to show that a zero-shot large language model scores student essays more accurately when the prompt is supplemented with a short list of hand-calculated linguistic features of the essay. On the ASAP dataset, Mistral-7B's quadratic weighted kappa with human graders rises from 0.454 with a plain prompt to 0.492 when ten features are included, without any fine-tuning. The improvement also transfers to the held-out ELLIPSE dataset, where the best feature set raises the score from 0.454 to 0.481. The result matters because it suggests a cheap, training-free route to better open-source essay grading, and because the gain is uneven across models, hinting at how LLM scores depend on prompt content.","feed_headline":"Linguistic features in LLM prompts improve essay scoring","feed_subtitle":"Adding ten handcrafted features to a zero-shot prompt raises Mistral's agreement with human graders on ASAP and ELLIPSE.","key_machinery":"The mechanism is the prompt construction: the model is given a persona, the essay prompt, an analysis instruction, the essay text, and then an 'Additional Information' section listing precomputed linguistic features with the framing that higher feature values typically mean higher end scores. The features themselves come from a published set of essay-grading statistics, limited to ten to avoid excessively long prompts. A separate few-shot LLM parser converts the model's text output into structured scores. The argument rests on comparing quadratic weighted kappa across four feature settings—none, the single most correlated feature, the top three, and all ten—on two datasets.","core_discovery":"The central claim is that appending an 'Additional Information' block listing ten precomputed essay statistics to an existing zero-shot prompt improves agreement with human holistic scores. The features—unique word count, word count, sentence count, lemma count, noun count, stopword count, Dale-Chall complex-word count, and two character counts—are presented to the model with the statement that higher values typically correspond to higher grades. Using the prompt template of a prior zero-shot baseline, the authors report that Mistral-7B improves from 0.454 to 0.492 average QWK on ASAP when all ten features are added, and from 0.454 to 0.481 on ELLIPSE with the top three features. They conclude that linguistic features can benefit LLM-based zero-shot AES, and that the benefit persists on out-of-distribution essays.","pith_inferences":["A plausible alternative reading, which the paper does not rule out, is that Mistral is simply following the prompt's stated rule that higher feature values mean higher scores; a control condition with random numbers, or with features presented without the correlation claim, would separate feature-driven improvement from instruction-following.","The single evaluation fold and absence of significance tests mean the reported QWK gaps could lie within noise; a bootstrap over the test essays would test whether 0.454 to 0.492 is a real difference.","The feature list is built from counts and readability statistics only; the same prompt scaffold could be tested with content-based features (e.g., rubric-aligned topic coverage) to see whether the mechanism generalizes beyond surface statistics."],"forward_implications":["If the central claim holds, educators can improve open-source LLM grading by computing a handful of text statistics and stating their positive correlation with scores, at no training cost.","The effect transfers across datasets, so the method supports the cross-prompt AES goal of scoring essays from prompts not seen during development.","The benefit is concentrated in the open-source model; the closed model shows mixed results, meaning the method's usefulness depends on the base LLM.","Even with the boost, a fine-tuned supervised model scores higher on ASAP, so prompt-only LLM grading still trails supervised methods.","The improved alignment between LLM and human scores suggests that additional linguistic context can partially substitute for the statistical prior that fine-tuning provides."],"supporting_citations":[{"why":"provides the zero-shot prompt template, persona, and instructions that the method builds on","marker":"Stahl et al. (2024)"},{"why":"source of the linguistic features and of the claim that they correlate with essay scores","marker":"Ridley et al. (2020b)"},{"why":"supplies the ASAP dataset and the QWK evaluation metric used in the experiments","marker":"Hamner et al. (2012)"},{"why":"provides the ELLIPSE dataset used as the out-of-distribution test set","marker":"Crossley (2024)"},{"why":"defines the supervised BERT baseline that upper-bounds performance on ASAP","marker":"Wang et al. (2022)"},{"why":"uses the same feature set in a cross-prompt AES context, supporting the choice of features","marker":"Li and Ng (2024)"}],"fun_headline_variants":["LLM essay scoring boosted by adding linguistic stats","Hybrid AES: zero-shot LLM plus handcrafted features","Ten handcrafted features lift LLM agreement on essays","Zero-shot LLM essay grading improves with feature block","Mistral essay scores rise when fed linguistic counts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported QWK improvements are caused by the linguistic features themselves, rather than by the longer prompt, the instruction telling the model that higher features mean higher grades, or chance variation across a single evaluation fold.","fun_headline_variants_meta":{"raw":{"variants":["LLM essay scoring boosted by adding linguistic stats","Hybrid AES: zero-shot LLM plus handcrafted features","Ten handcrafted features lift LLM agreement on essays","Zero-shot LLM essay grading improves with feature block","Mistral essay scores rise when fed linguistic counts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1223,"prompt_tokens":828,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":316}},"tokens_in":444,"tokens_out":395,"duration_ms":4094,"temperature":1.0,"reasoning_tokens":316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:15:41.139808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the zero-shot evaluation on the same folds but replace the feature list with an equal-length list of random numbers, or with the same numbers presented without the correlation instruction; if QWK stays at the same level, the features are not the driver. A second, statistical test would compute bootstrap confidence intervals for the QWK differences; the claim is unsupported if the intervals for the with- and without-feature conditions overlap.","supporting_citations":[{"cited_title":"Exploring LLM prompting strategies for joint essay scoring and feedback generation","cited_arxiv_id":null,"evidence_quote":"provides the zero-shot prompt template, persona, and instructions that the method builds on"},{"cited_title":"The hewlett foundation: Automated essay scoring","cited_arxiv_id":null,"evidence_quote":"supplies the ASAP dataset and the QWK evaluation metric used in the experiments"}],"review_version":1}