Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper aims to establish that zero-shot LLM essay scorers systematically under-score lower-order writing traits such as grammar and conventions, that this harshness is stable rather than random, and that small human-scored validation set

desk verdict Solid, reproducible empirical core on trait-level LLM scoring bias, but the small-sample detection claims outrun the bootstrap analysis. read the letter →

arxiv 2604.00259 v1 pith:OV4ZCCD2 submitted 2026-03-31 cs.CL cs.AI

Filip J. Kucia , Anirban Chakraborty , Anna Wróblewska This is my paper · ORCID
classification cs.CLcs.AI
keywords automatedessayscoringLLM-as-a-judgeholisticvsanalyticrubricsbiaslower-orderconcernsbootstrapminimumsamplesizepromptstrategyzero-shotevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-weight, instruction-tuned LLMs, when asked to score essays with no fine-tuning, do not miss evenly across rubric dimensions. The paper finds that they agree reasonably with human raters on overall essay quality (quadratic weighted kappa around 0.6) but consistently under-grade lower-order concern traits such as grammar, conventions, and syntax, often by about one point on a five-point scale. This negative bias is stable: bootstrap analysis detects it with validation sets as small as five essays on the six-trait analytic corpus, while higher-order traits such as content and organization need much larger samples or are not detectable at all. The paper also shows that concise keyword-style prompts beat detailed rubric text for multi-trait analytic scoring, while the reverse holds for holistic scoring. If correct, the practical conclusion is bias-correction-first deployment: estimate the systematic offset from a tiny set of human scores and subtract it, rather than fine-tune.

What carries the argument

The paper's analytical engine is the HOC/LOC distinction (higher-order concerns = global discourse traits; lower-order concerns = local linguistic form) used to organize trait-level results, paired with a bootstrap minimum-sample-size (Nmin) procedure: at each sample size N, draw 10,000 resamples, build a 95% percentile confidence interval for mean signed bias, and record the smallest N at which the interval excludes zero. This tool turns "is the model biased?" into "how many human-scored essays prove the bias?" The prompt contrast (keyword labels vs full rubric guidelines) is the other manipulated factor, used to show that instruction detail interacts with scoring granularity.

What would settle it

Take an analytic-scoring corpus where every essay has multiple independent human raters, and recompute the mean signed bias for grammar and conventions against the average of all raters (and against each rater separately). If the negative bias disappears or becomes inconsistent across raters, the claim of stable model harshness fails; if it persists against every rater, the claim is strengthened.

Watch

Extended reading notes

Core claim

The central discovery is trait-dependent directional bias. In zero-shot scoring against dataset-provided human consensus scores, models assigned numerically lower scores than humans on Lower-Order Concern traits—grammar, conventions, syntax, vocabulary, phraseology—while Higher-Order Concern traits (content, organization, cohesion) often showed smaller or less consistent deviation. On the six-trait corpus, the median minimum sample size for detecting non-zero bias was 5 essays; on the holistic corpus, some configurations required 515–1,460 essays, and on one content trait the bias was not detectable. The authors interpret this as evidence that the LOC harshness reflects stable model behavior

Load-bearing premise

The paper treats each dataset's single human consensus score as the true gold standard, so if that reference is noisy or reflects one rater's leniency, the measured model harshness is partly an artifact of the reference rather than a stable property of the model.

Editorial extensions

If this is right

  • Systematic LOC bias can be corrected with a small bias-estimation set, so a low-cost deployment path exists: measure the offset on 5–20 human-scored essays and apply it to raw zero-shot scores.
  • Analytic trait scores should not be used as exact diagnostics: even the best model reached only moderate agreement (QWK around 0.32–0.46) on multi-trait rubrics, and near-zero mean bias can coexist with low agreement because of score compression.
  • Prompt design is not one-size-fits-all: keyword prompts are preferable for multi-trait analytic scoring, while full rubric descriptions help holistic scoring.
  • Bias-detection sample sizes should be planned per trait, not per model: lower-order traits need very small sets, higher-order traits need hundreds or are undetectable within available data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper links the LOC harshness to the absence of learner-specific context; an immediate testable extension is to add writer metadata (e.g., L1 or grade level) to prompts and see whether the negative bias shrinks.
  • The keywords-over-guidelines reversal suggests that long rubric text may overload the model when several traits are judged at once; a prompt-engineering experiment that presents traits one at a time or in a fixed order could isolate the mechanism.
  • The bootstrap Nmin is, in effect, a ready-made audit rule: score a handful of essays, resample the bias, and decide whether to deploy or correct a scorer. The authors stop short of stating a formal decision threshold, so a practical next step is to derive one with explicit trade-offs.
  • The results imply that any future claim of "LLMs match humans on essay scoring" should be qualified by rubric granularity and trait type, since aggregate agreement can hide large trait-level offsets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates five open-weight instruction-tuned LLMs as zero-shot essay scorers on three open datasets (ASAP 2.0, ELLIPSE, DREsS), comparing holistic and analytic scoring under two prompt strategies (Keywords vs. Guidelines). It reports agreement (QWK, EA), directional bias (mean signed error), score compression, length sensitivity, and a bootstrap-based minimum sample size (Nmin) for detecting non-zero mean bias. The central claims are that (i) strong open-weight models achieve moderate holistic agreement (QWK≈0.6) but this does not transfer to analytic scoring; (ii) there is a large and stable negative bias on Lower-Order Concern (LOC) traits such as Grammar and Conventions; (iii) keyword-based prompts generally outperform rubric-style guidelines for analytic multi-trait scoring; and (iv) the LOC bias is often detectable with very small validation sets (median Nmin=5 on ELLIPSE), supporting a bias-correction-first deployment strategy.

Significance. If the central claims hold, the paper provides a reproducible, multi-dataset characterization of a trait-dependent harshness in LLM essay scoring, with practical implications for using small bias-estimation sets rather than full fine-tuning. Strengths include the use of open datasets and open-weight models, deterministic greedy decoding, a clear separation of agreement and bias metrics, and an explicit acknowledgment of the single-reference-score limitation. However, the most actionable claim — that LOC bias is detectable with N≤10 essays — rests on a descriptive bootstrap statistic that is not a prospective power analysis, and several secondary claims lack uncertainty quantification. The paper is a useful contribution to the LLM-as-judge and AES literature, but the statistical support for the small-sample detectability claim needs substantial additional analysis.

major comments (4)
  1. [§4.4, Tables 4–5, Abstract, §6] The abstract and §6 claim that LOC bias is 'detectable with very small validation sets' (often N≤10) and that the bootstrap analysis shows biases are 'not artifacts of sampling variability.' The procedure in §4.4, however, computes, for each model–trait–split–strategy combination, the smallest N at which a 95% bootstrap percentile interval for the mean bias excludes zero, using 10,000 bootstrap resamples. As described, this is a descriptive statement about the full dataset's bias relative to its own variability; it is not a prospective sample-size or power calculation. The event that the full-data bootstrap distribution excludes zero at N=5 does not imply that an independent validation set of 5 essays would yield a confidence interval excluding zero with any stated probability. Moreover, the description is ambiguous about the sampling universe: are resamples drawn from the full dataset a
  2. [Table 2 footnote] The footnote states: 'All bias estimates are statistically different from zero (p<0.001) according to a Wilcoxon Signed-Rank test.' No test statistics, exact p-values, sample sizes, or multiple-comparison corrections are provided. Given the large number of model–trait–split–strategy combinations (e.g., 120 on ELLIPSE in Table 4), unadjusted p<0.001 across all comparisons is not credible as stated and is not verifiable from the manuscript. If the Wilcoxon test is used, the report should provide the distribution of p-values or a proper correction (e.g., FDR) and specify which comparisons are being tested. This is relevant to the paper's characterization of bias as 'systematic' rather than noise.
  3. [§3.4, Table 3, RQ3] The claim that 'concise keyword-based prompts generally outperform longer rubric-style prompts in multi-trait analytic scoring' is based on point estimates of QWK and EA without confidence intervals or significance tests. For example, on ELLIPSE Cohesion under Keywords vs. Guidelines, QWK changes from 0.566 to 0.412 with a large shift in bias (−0.12 vs. −0.55), but the paper does not quantify uncertainty around these values. Given sample sizes of thousands of essays, some differences may be statistically meaningful, but the reader cannot assess which prompt effects are robust. A bootstrap or other resampling-based CI for QWK and EA, or at least for the QWK difference, should be reported for the key comparisons underlying RQ3.
  4. [§3.4, §4.2, §5, Limitations] The definition of 'bias' is entirely relative to the dataset-provided Human Consensus Score (HCS), which is a single reference score per essay. The paper acknowledges in the Limitations that 'Without modeling inter-rater variability, it is difficult to disentangle true model error from legitimate disagreement.' But the abstract and §5 make stronger statements, e.g., 'models often score these traits more harshly than human raters' and 'large and stable negative directional bias.' If HCS is noisy or reflects a single rater's leniency, the observed negative bias on LOC traits could be partly an artifact of the reference rather than a stable property of the models. Because this is the paper's central empirical finding, the interpretation should be tempered to 'bias relative to the dataset-provided HCS,' and the potential impact of HCS noise on the LOC-vs-HOC contrast should be discussed more
minor comments (5)
  1. [§4.4, Table 4] The table header contains a typo: '#Combinations of model–trait–split–strategy' should be '#Combinations' or '#Combinations (model–trait–split–strategy)'. Also, the meaning of 'Bias Not Reached' is clear from the text but could be defined in the caption.
  2. [§3.4] The notation for bias is introduced twice: as 'Bias (Mean Signed Error)' and then with the formula. The formula itself is correct, but the symbol μ̂_bias is not used consistently in Tables 2–5; the tables just say 'Bias.' Consider defining the abbreviation in a single place and using it throughout.
  3. [§4.3] The correlation comparison (r_Human vs. r_LLM) is reported without confidence intervals or a test of the difference. Given the large sample sizes, the difference between 0.71 and 0.47 on ASAP is likely significant, but the paper does not quantify this. Also, Pearson correlation with word counts may be influenced by outliers; a robust alternative (e.g., Spearman) would strengthen the verbosity discussion.
  4. [§3.1, Table 1] The table header has 'T raits' with a stray space. Also, the table reports average word counts and standard deviations but not the standard deviations for the full dataset; consider adding the split sizes consistently.
  5. [References] The reference list contains several entries with incomplete metadata (e.g., items 1, 2, 3, 5, 8, 11, 13, 14, 15, 19, 21, 23 lack page numbers or full proceedings details in some cases). Please ensure all references are complete and consistently formatted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark against fixed external human scores; no fitted parameter is reused as a prediction.

full rationale

The paper's claimed derivation chain is empirical and self-contained: LLM scores are compared to the dataset-provided Human Consensus Score (HCS), bias is defined as mean signed error relative to HCS, and the Nmin statistic is a descriptive bootstrap interval computed on the full dataset. There is no equation or fitted parameter that is reused as a prediction; the bootstrap procedure is a statistical description of the realized sample, not a prospective power analysis. The acknowledged limitation that HCS is a single reference score and that inter-rater variability is not modeled is a caveat about the reference standard, not evidence of circularity. The paper does not cite its own authors for load-bearing claims, and the HOC/LOC distinction and Keywords/Guidelines prompt strategies are adopted from external prior work with citations, not introduced as renamed versions of the paper's own results. The Nmin overclaim (if any) that small full-data bootstrap intervals imply detectability in fresh validation sets is a statistical inference concern, not a circularity. Therefore, no circular step can be exhibited, and the correct finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims are empirical and rest on the assumptions that HCS is a valid ground truth, that the HOC/LOC taxonomy is meaningful for these rubrics, and that bootstrap CI methodology is appropriate. No new entities are introduced; the only methodological choice is the bootstrap configuration. The paper's conclusions depend on these assumptions, which are partially acknowledged in the Limitations.

free parameters (1)
  • Bootstrap settings (N_start, step, B, CI level) = N_start=5, step=5, B=10,000, 95% CI
    Chosen by the authors to define Nmin. Changing these settings would shift the exact reported sample sizes, though the qualitative LOC-vs-HOC pattern is likely robust.
assumptions (4)
  • domain assumption HCS (dataset-provided human score) is the reference ground truth.
    Bias and agreement are defined relative to HCS. If HCS is noisy (single rater), the measured model bias conflates model error with human disagreement. Acknowledged in Section 5 Limitations.
  • domain assumption HOC/LOC distinction from writing-center pedagogy maps onto rubric traits.
    The authors assign Content/Organization/Cohesion as HOC and Language/Syntax/Vocabulary/Phraseology/Grammar/Conventions as LOC. This classification drives the central bias-contrast claim.
  • standard math Bootstrap percentile CI at 95% is a valid method for estimating sample size needed to detect mean bias.
    Borrowed from references [14,3]; the paper uses this as the definition of Nmin.
  • domain assumption Greedy decoding (temperature=0) produces representative zero-shot model scores.
    All evaluations use deterministic decoding; randomness in model outputs is ignored, so sampling variability in scores is not part of the analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias." pith.science (2026). https://pith.science/paper/OV4ZCCD2

@misc{pith2026260400259,
  author       = {Pith},
  title        = {Pith review of: LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OV4ZCCD2}},
  note         = {Machine review of arXiv:2604.00259}
}
read the original abstract

Despite growing interest in using Large Language Models (LLMs) for educational assessment, it remains unclear how closely they align with human scoring. We present a systematic evaluation of instruction-tuned LLMs across three open essay-scoring datasets (ASAP 2.0, ELLIPSE, and DREsS) that cover both holistic and analytic scoring. We analyze agreement with human consensus scores, directional bias, and the stability of bias estimates. Our results show that strong open-weight models achieve moderate to high agreement with humans on holistic scoring (Quadratic Weighted Kappa about 0.6), but this does not transfer uniformly to analytic scoring. In particular, we observe large and stable negative directional bias on Lower-Order Concern (LOC) traits, such as Grammar and Conventions, meaning that models often score these traits more harshly than human raters. We also find that concise keyword-based prompts generally outperform longer rubric-style prompts in multi-trait analytic scoring. To quantify the amount of data needed to detect these systematic deviations, we compute the minimum sample size at which a 95% bootstrap confidence interval for the mean bias excludes zero. This analysis shows that LOC bias is often detectable with very small validation sets, whereas Higher-Order Concern (HOC) traits typically require much larger samples. These findings support a bias-correction-first deployment strategy: instead of relying on raw zero-shot scores, systematic score offsets can be estimated and corrected using small human-labeled bias-estimation sets, without requiring large-scale fine-tuning.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Cost-efficient generative AI summarization for scalable automated essay scoring in educational assessment

    cs.CL 2026-07 conditional novelty 4.0 of 10

    In a fixed hybrid AES pipeline on ASAP 2.0, GPT-5 mini summarization gives the best human-agreement (QWK 0.8435), ahead of GPT-5 and GPT-5 nano, though the differences are small and no significance tests or baselines ...

Reference graph

Works this paper leans on

24 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    Beigman Klebanov, B., Madnani, N.: Assessing the reliability of large lan- guage models for automated essay scoring. In: BEA. ACL (2023),https:// aclanthology.org/2023.bea-1.1

    2023

  2. [2]

    Transactions ACL (2023) LLM Essay Scoring 15

    Blatz, M., et al.: Calibration and bias in language model evaluation. Transactions ACL (2023) LLM Essay Scoring 15

    2023

  3. [3]

    In: Proceedings of the 19th Workshop on Inno- vative Use of NLP for Building Educational Applications (BEA)

    Chang, Y., et al.: Evaluating sensitivity and robustness of automated essay scoring models via bootstrap resampling. In: Proceedings of the 19th Workshop on Inno- vative Use of NLP for Building Educational Applications (BEA). Association for Computational Linguistics (2024)

    2024

  4. [4]

    Chiang, C.H., Lee, H.y.: Can large language models be an alternative to human evaluations? ACL (2023).https://doi.org/10.18653/v1/2023.acl-long.870

    doi:10.18653/v1/2023.acl-long.870 2023

  5. [5]

    Int Journal of Learner Corpus Research (2023)

    Crossley, S., et al.: The English language learner insight, proficiency and skills evaluation (ellipse) corpus. Int Journal of Learner Corpus Research (2023)

    2023

  6. [6]

    In: NAACL: Human Language Technologies

    Devlin, J., et al.: BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In: NAACL: Human Language Technologies. ACL (2019). https://doi.org/10.18653/v1/N19-1423

    doi:10.18653/v1/n19-1423 2019

  7. [7]

    In: ICTIR

    Dietz, L., et al.: Principles and guidelines for the use of llm judges. In: ICTIR. ACM (2025),https://doi.org/10.1145/3731120.3744588

    arXiv 2025

  8. [8]

    In: Findings of ACL (2025).https://doi.org/10.18653/v1/2025

    Eltanbouly, S., et al.: TRATES: Trait-specific rubric-assisted cross-prompt es- say scoring. In: Findings of ACL (2025).https://doi.org/10.18653/v1/2025. findings-acl.1054

    doi:10.18653/v1/2025 2025

Show all 24 references
  1. [9]

    Transac- tions ACL (2021).https://doi.org/10.1162/tacl_a_00373

    Fabbri, A.R., et al.: Summeval: Re-evaluating summarization evaluation. Transac- tions ACL (2021).https://doi.org/10.1162/tacl_a_00373

    2021 doi

  2. [10]

    Kaggle competition (2012),https://www.kaggle.com/c/asap-aes, last accessed 2025/12/28

    Hewlett Foundation: Automated student assessment prize (asap): Automated essay scoring. Kaggle competition (2012),https://www.kaggle.com/c/asap-aes, last accessed 2025/12/28

    2012

  3. [11]

    In: Findings of ACL (2024)

    Kochnavi, I., et al.: On the sensitivity of llm-based evaluators to prompt design. In: Findings of ACL (2024)

    2024

  4. [12]

    In: IJCAI (2024).https://doi.org/10.24963/ijcai.2024/897

    Li, S., Ng, V.: Automated essay scoring: Recent successes and future directions. In: IJCAI (2024).https://doi.org/10.24963/ijcai.2024/897

    2024 doi

  5. [13]

    Naismith, B., Yao, J.: Automated essay scoring and the search for valid interpre- tations. In: BEA. ACL (2023),https://aclanthology.org/2023.bea-1.38

    2023

  6. [14]

    Journal of Modern Applied Statistical Methods (2013)

    Qumsiyeh, M.: Using the bootstrap for estimating the sample size in statistical experiments. Journal of Modern Applied Statistical Methods (2013)

    2013

  7. [15]

    Artif Intell Rev (2022),https://doi.org/10.1007/ s10462-021-10068-2

    Ramesh, D., Sanampudi, S.: An automated essay scoring systems: a sys- tematic literature review. Artif Intell Rev (2022),https://doi.org/10.1007/ s10462-021-10068-2

    2022

  8. [16]

    Reigstad, T.J., McAndrew, D.A.: Training Tutors for Writing Conferences. No. ED240589, ERIC Clearinghouse on Reading and Communication Skills (1984)

    1984

  9. [17]

    (eds.): Automated Essay Scoring: A Cross-Disciplinary Perspective

    Shermis, M.D., Burstein, J. (eds.): Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates (2003)

    2003

  10. [18]

    In: EMNLP (2016),https://aclanthology.org/D16-1193

    Taghipour, K., Ng, H.T.: A neural approach to automated essay scoring. In: EMNLP (2016),https://aclanthology.org/D16-1193

    2016

  11. [19]

    In: Workshop on Generation, Evaluation and Metrics (GEM²)

    Thakur, A., et al.: Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-judges. In: Workshop on Generation, Evaluation and Metrics (GEM²). ACL (2025),https://aclanthology.org/2025.gem-1.33/

    2025

  12. [20]

    Behaviormetrika (2021).https://doi.org/10.1007/s41237-021-00138-y

    Uto, M.: Item response theory for automated essay scoring. Behaviormetrika (2021).https://doi.org/10.1007/s41237-021-00138-y

    2021 doi

  13. [21]

    In: EMNLP (2023)

    Wang, X., et al.: Large language models are not fair evaluators. In: EMNLP (2023)

    2023

  14. [22]

    Applied Measurement in Education (2012)

    Williamson, D.M., et al.: Automated essay scoring: Psychometric considerations. Applied Measurement in Education (2012)

    2012

  15. [23]

    Yoo, H., Han, J., Ahn, S.Y., Oh, A.: DREsS: Dataset for rubric-based essay scoring onEFLwriting.ACL(2025).https://doi.org/10.18653/v1/2025.acl-long.659

    2025 doi

  16. [24]

    In: NIPS (2023)

    Zheng, L., et al.: Judging LLM-as-a-judge with MT-bench and chatbot arena. In: NIPS (2023)

    2023

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.