Pith. sign in

REVIEW 4 major objections 7 minor 49 references

Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards

T0 review · 4 major / 7 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Exploiting a 166-item rubric as a reward, the paper shows that reinforcement learning can make one LLM both score essays at the top of the LLM baseline range and generate feedback that matches a much larger commercial model.

desk verdict Useful RL recipe for joint AES/AFG, but the feedback-quality comparison is largely the model being optimized to the judge's own rubric; the scoring claim is plausible, the feedback claim needs independent human evaluation. read the letter →

arxiv 2607.19219 v1 pith:5Z5FEQEI submitted 2026-07-21 cs.CL cs.AI

Xuefeng Jin , Jiashuo Zhang , Teng Cao , Bin Yang This is my paper · ORCID
classification cs.CLcs.AI
keywords automatedessayscoringfeedbackgenerationreinforcementlearningrubricrewardsLLM-as-judgequadraticweightedkappaGRPOordinalclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that automated essay scoring and feedback generation should be optimized together with reinforcement learning, using feedback rubrics as the reward signal, rather than relying on prompting or supervised imitation alone. It builds a 166-item binary rubric, evaluated by an LLM judge against each essay, that turns feedback quality into a scalar reward; with this reward, a single 9-billion-parameter model reaches 0.803 QWK on the standard essay-scoring benchmark, the highest among LLM-based methods, while its feedback scores comparably to a much larger commercial model. A key supporting result is that training with score rewards only degrades feedback, while the proposed adaptive gating prevents that degradation at roughly one-sixth the cost of full feedback optimization. The paper's reason to care: RL post-training with rubric rewards gives educators a path to systems that both grade and explain, rather than just predicting a number.

What carries the argument

The load-bearing machinery is RFE (Rubric-based Feedback Evaluation), a rubric of 166 binary items in four dimensions — Coverage, Evidence, Faithfulness, Safety — checked by an LLM-as-judge against the specific essay, giving a scalar feedback reward for GRPO. AGFO (Adaptive Gated Feedback Optimization) is the gate that invokes the judge only when the recent average feedback reward is at or below a threshold, so most RL steps use only the cheap score reward. ACR (Adjacent Contrastive Reasoning) is an output-format prompt strategy that asks the model to state why the essay is not the adjacent lower and higher score, which sharpens ordinal score boundaries. Together, RFE supplies the measurable

What would settle it

Have a fresh panel of multiple expert human raters compare RLAES-AGFO feedback against output from the large commercial model on a hold-out essay set and check whether their preferences match RFE rankings; if humans prefer the commercial model as often as or more often than RFE does, or if RFE ranks a mismatched feedback above gold feedback, the central feedback-quality claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a unified LLM, post-trained with GRPO under a reward that combines score accuracy and an essay-grounded feedback rubric, outperforms existing LLM-based AES methods on scoring (QWK 0.803) while keeping feedback quality at the level of a much larger proprietary model (RFE 0.8399 vs 0.8334). It also claims that score-only RL degrades feedback quality from 0.6978 to 0.5608 on the same measure, and that activating the feedback reward only when quality falls below a threshold both avoids the degradation and cuts judge-query cost by about 83%. The paper further claims that RFE rankings agree with a human expert on 40 of 43 challenging pairs, better than an existing rubric-

Load-bearing premise

The load-bearing premise, acknowledged in the paper's own conclusion, is that the 166-item LLM judge measures feedback quality the way expert human readers do; because the human check is one evaluator on 43 pairs and the final feedback evaluation covers a single fold, a judge-vs-judge agreement could be inflating the reported feedback parity.

Editorial extensions

If this is right

  • Feedback quality becomes a first-class, optimizable objective for automated essay systems, not just something evaluated after the fact.
  • Score-only RL is a real failure mode; joint optimization avoids it while preserving state-of-the-art LLM scoring accuracy.
  • RFE gives the field a reusable, essay-grounded automated feedback metric that can rank gold vs perturbed feedback perfectly in the paper's 88-pair test.
  • ACR's 'why not the adjacent score' prompt can transfer to any ordinal classification setting where neighbors are the natural confusions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If RFE is accepted as a valid feedback metric, the paper's recipe — judge-based rubric reward plus thresholded gating — transfers to other open-ended generation tasks where quality is multidimensional and expensive to evaluate.
  • Because the same kind of LLM judge supplies both the training reward and the final evaluation, part of the reported feedback parity may be a 'judge agrees with judge' effect; a multi-rater human replication study is the natural next test.
  • The gating schedule introduces a general efficiency principle for RL with expensive rewards: periodically probe reward quality and pay for dense evaluation only in the low-performing regime.
  • ACR could enrich rubric-based essay feedback itself, since the contrastive reasoning text is itself a form of diagnostic feedback.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes RLAES, an RL post-training framework for joint automated essay scoring (AES) and feedback generation (AFG). The contribution has three parts: (1) Rubric-based Feedback Evaluation (RFE), an essay-grounded 166-item binary rubric scored by an LLM judge; (2) Adaptive Gated Feedback Optimization (AGFO), which intermittently activates the RFE reward during GRPO training; and (3) Adjacent Contrastive Reasoning (ACR), a prompting strategy for ordinal score calibration. On ASAP, RLAES-AGFO reports QWK = 0.803, the highest among LLM-based methods, and an RFE score of 0.8399, claimed comparable to GPT-5.5 (0.8334), while score-only RL degrades feedback quality. The paper also validates RFE through pairwise discrimination and a 43-pair expert-preference study.

Significance. If the results hold, the paper makes a useful step toward moving LLM-based AES beyond prompt engineering and SFT to RL post-training, and it offers a concrete, interpretable reward for feedback quality. The strengths include a publicly available code/data repository, use of standardized ASAP folds, and a candid discussion of LLM-judge limitations. However, the feedback-quality claim is largely circular: the RFE score used as the reported metric is also the reward optimized during training. The scoring claim, while externally grounded in ASAP labels, is supported by QWK differences of only about 0.01 with no uncertainty estimates and with a joint-training protocol that differs from per-prompt baselines. The AGFO method itself lacks ablations against its own special cases. These issues are significant but addressable; the central scoring result is defensible, while the feedback result needs additional non-circular evidence.

major comments (4)
  1. [§3.1, Eq. (2), §5.2, Table 3] The feedback-quality result is circular. Eq. (2) uses r_feedback from RFE as the RL reward, and Table 3 uses the same RFE score as the evaluation metric. RLAES-AGFO is explicitly optimized to satisfy the 166-item checklist, while GPT-5.5 is not, so the 0.8399 vs 0.8334 comparison largely reflects fit to the judge's checklist. The only human anchor (Table 6) uses 43 synthetic pairs judged by a single evaluator, not the actual outputs of RLAES-AGFO vs GPT-5.5. The paper's own conclusion acknowledges LLM-judge dependence. This does not undermine the scoring claim, but the 'comparable to GPT-5.5' feedback claim requires either a human evaluation of the real system outputs or an independent judge not used in training.
  2. [Table 2; Appendix A] The claim of best LLM-based scoring performance rests on QWK differences of 0.008-0.009 (RLAES-AGFO 0.803, RLAES-Score 0.802, RTS 0.794), with no confidence intervals, fold-level variances, or significance tests. Additionally, RLAES models use the joint-training protocol described in Appendix A, while RTS and other baselines appear to be per-prompt trained. Appendix A shows only that joint training costs at most 0.005 QWK for the authors' own models (NPCR, SFTAES, RLAES), not that the same holds for RTS or Llama-3. A fair comparison requires either per-prompt RLAES results or joint-training results for the baselines, plus uncertainty estimates.
  3. [§4.2, Table 3] Feedback quality is evaluated only on fold 0 due to budget constraints. The Table 3 QWK column (0.8043/0.8082) does not match the five-fold averages in Table 2 (0.802/0.803). Because folds use different test splits and RFE scores vary considerably across prompts, a single fold cannot support the 'comparable to GPT-5.5' claim without variance information. Please report all folds or justify why fold 0 alone is representative, and clarify which QWK is reported in Table 3.
  4. [§3.2, §5.2] AGFO is a core contribution, but the experiments do not isolate its mechanism. The paper claims the gating strategy reduces training cost from about 12x to 2x and avoids feedback degradation, yet no ablation compares RLAES-AGFO against full feedback optimization (tau=1) or periodic-only feedback optimization (tau=0) in terms of both RFE score and QWK. Without these controls, the observed improvement over RLAES-Score could be due to any feedback-reward variant, not specifically to the adaptive gating mechanism.
minor comments (7)
  1. [§3.1] The 166 rubric items are not included in the paper or an appendix. Releasing the full rubric with the code is essential for reproducibility, since RFE is a central contribution.
  2. [§3.3, Eq. (3)] The score reward r_score depends on alpha in Eq. (3), but alpha=0.1 is stated without sensitivity analysis. Please clarify the role of alpha and report whether the main results are robust to it.
  3. [Table 3] 'RLAES-Step 0' has QWK 0.3343, which is presumably before any scoring training. State this explicitly in the table caption or text.
  4. [§4.2] The sentence 'Each fold covers the complete ASAP dataset' is misleading; each fold is a split of the full dataset, and the evaluation is on one test fold. Rephrase to avoid ambiguity.
  5. [Abstract / §5.2] The abstract and results assert 'feedback quality comparable to GPT-5.5' without consistently noting that this is under the proposed RFE metric. Please qualify the claim as RFE-based, especially given the circularity noted above.
  6. [Appendix A] NPCR is introduced in Appendix A but not defined in the main text. Add a reference or brief description.
  7. [Figure 4] The right panel tracks 'rubric reward'; for consistency with the rest of the paper, label it as RFE score and clarify that it is computed on Prompt 1 only.

Circularity Check

1 steps flagged · score 6.0 of 10

Feedback-quality result is an in-distribution fit: the same RFE rubric checklist is both the RL reward (Eq. 2) and the reported outcome metric (Table 3), so the 'comparable to GPT-5.5' claim partially measures fit to the metric rather than independently validated quality.

  1. fitted input called prediction [§3.2 Eq. (2); §4.2; §5.2 and Table 3]
    "Given a student essay and its context e, the gold scores, the model-predicted score ˆs, the model-generated feedback ˆf, and the feedback rubrics Rfb, the total reward rtotal is defined as rtotal = rscore(s,ˆs) + λf · rfeedback(e,Rfb,ˆf) ... Figure 4 tracks QWK and feedback quality (rubric reward, i.e., RFE score) ... For AFG evaluation, we employ our proposed RFE framework with GPT-5.5 as the LLM judge."

    The RFE score is simultaneously the quantity the policy is trained to maximize (r_feedback in Eq. 2, explicitly identified as 'rubric reward, i.e., RFE score') and the metric used to report feedback quality in Table 3. RLAES-AGFO is therefore evaluated on the same rubric checklist R_fb it was optimized against, while the GPT-5.5 baseline was not. The headline comparison (RLAES-AGFO 0.8399 vs. GPT-5.5 0.8334) and the claimed improvement over RLAES-Score (0.6978 → 0.8399) partly reduce to fit-to-metric. The 43-pair expert study (§5.3) validates RFE on specially constructed pairs, not on the actual Table 3 outputs. Using DeepSeek-V4-Pro as training judge and GPT-5.5 as evaluation judge mitigates, but does not break, the identity of the rubric checklist.

full rationale

The scoring claims are externally grounded: QWK is computed against ASAP human-assigned scores, and the ACR results are also judged by QWK. No load-bearing self-citations or author-imported uniqueness constraints appear. The circularity is concentrated in the feedback-quality evaluation. The paper's own equation 2 makes RFE the feedback reward, and Section 5.2 explicitly equates the rubric reward with the RFE score; Section 4.2 then uses the RFE framework as the final feedback-evaluation metric. Thus the central AFG result—that RLAES-AGFO matches GPT-5.5 and avoids the feedback degradation of score-only RL—is substantially an in-distribution comparison against the training objective. The different judge model at evaluation and the small human agreement study provide partial external anchoring, so this is a partial circularity rather than a complete definitional collapse. Score 6 reflects that one or more reported quality results reduce, by construction, to the metric being optimized.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central scoring claim rests on standard ASAP labels; the feedback claim rests on LLM-constructed rubrics, LLM judges, and several hand-set reward hyperparameters. No new physical or theoretical entities are introduced.

free parameters (6)
  • feedback reward weight λ_f = 1
    Eq. (2) combines score and feedback rewards linearly; λ_f=1 is set without sensitivity analysis (§4.4).
  • score-interval scaling factor α = 0.1
    Eq. (3) scales the absolute score difference; chosen in §4.4, no ablation.
  • AGFO gating period n = 30
    The judge is queried every n steps (Algorithm 1); no sensitivity analysis.
  • AGFO feedback reward threshold τ = 0.8
    Gate closes when the running-average feedback reward exceeds τ; 'based on an offline estimate' (§4.4), not a principled derivation.
  • rubric item weights w_j = 1 (all 166)
    RFE score uses equal weights in Eq. (1); alternative weighting is not explored.
  • GRPO KL coefficient = 0
    KL regularization coefficient set to 0 (§4.4), a design choice that affects training dynamics.
assumptions (6)
  • domain assumption ASAP gold scores are treated as ground truth for scoring quality.
    All QWK claims assume the human gold labels on ASAP are reliable; the paper cites the single human rater ceiling of 0.805 but does not model label noise beyond that point.
  • domain assumption LLM judge answers on 166 binary rubric items are a valid measure of feedback quality.
    RFE is built with LLM agents and judged by LLMs (§3.1, §5.2); validity rests on one 43-pair, single-evaluator study.
  • ad hoc to paper The 50 gold feedback samples used to construct rubrics are representative of high-quality feedback.
    §3.1: only 92 pairs are manually verified, 50 are retained as gold; these samples define the target distribution for RFE.
  • domain assumption Feedback can serve as a CoT-like intermediate for scoring (output order feedback-then-score).
    §2.1 defines feedback as an intermediate rationale. Ablation Table 8 shows removing feedback changes QWK by only -0.0007, so this is not load-bearing for scoring but is load-bearing for the joint-training framing.
  • ad hoc to paper Independent per-item rubric judgments aggregated with equal weights (Eq. 1) form a valid scalar reward.
    No evidence of conditional independence or equal importance among 166 items; this aggregation is assumed throughout.
  • domain assumption ACR's anchoring-and-adjustment analogy transfers to LLM ordinal classification.
    §3.3 treats adjacent score levels as hard negatives; the prompt-level QWK gains are small and not statistically tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards." pith.science (2026). https://pith.science/paper/5Z5FEQEI

@misc{pith2026260719219,
  author       = {Pith},
  title        = {Pith review of: Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5Z5FEQEI}},
  note         = {Machine review of arXiv:2607.19219}
}
read the original abstract

Large language models (LLMs) have been widely applied to automated essay scoring (AES) and automated feedback generation (AFG). However, existing studies rely primarily on prompt engineering or supervised fine-tuning, while systematic research on reinforcement learning (RL) post-training and automated evaluation of feedback quality remains limited. We propose RLAES, a unified LLM framework that jointly optimizes essay scoring and feedback generation through RL. To make feedback quality measurable, interpretable, and usable for training, we introduce Rubric-based Feedback Evaluation (RFE), an essay-grounded feedback evaluation framework comprising 166 fine-grained binary rubric items and an LLM-as-judge. Building on RFE, we propose Adaptive Gated Feedback Optimization (AGFO), which activates rubric-based feedback rewards on demand during RL, reducing evaluation overhead while improving feedback quality. We also propose Adjacent Contrastive Reasoning (ACR) to improve ordinal score calibration by explicitly contrasting adjacent score levels. Experimental results show that the RFE framework captures essay-feedback consistency, exhibits strong pairwise discriminative power, and closely aligns with expert preferences. On the ASAP benchmark, RLAES-AGFO achieves the best scoring performance among LLM-based methods (QWK = 0.803), while maintaining feedback quality comparable to GPT-5.5 and avoiding the feedback degradation observed under score-only RL. Code and datasets are publicly available at https://github.com/hellomuyi/RLAES.

Figures

Figures reproduced from arXiv: 2607.19219 by the authors.

Figure 1
Figure 1. Construction of the RFE rubrics. Feedback Evaluation (RFE) framework, which supports au￾tomated feedback evaluation and converts feedback quality into a scalar reward for RL training. RFE comprises prompt￾specific, checklist-style rubrics and an LLM-as-judge. We will release these resources with the code [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. ACR for ordinal score calibration [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Mean predicted scores and one-standard-deviation ranges under different ACR configurations. Dashed markers show [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: QWK and rubric reward during RLAES-Score and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 12 canonical work pages

  1. [1]

    GPT-5.5 System Card

    OpenAI. GPT-5.5 System Card. 2026 , howpublished =

    2026

  2. [2]

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author=

  3. [3]

    The Blind Spots in Automated Feedback Generation for Academic Writing

    Sasaki, Toru and Conijn, Rianne and Willemsen, Martijn C. The Blind Spots in Automated Feedback Generation for Academic Writing. Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026). 2026. doi:10.1145/3785022.3785120

    arXiv 2026

  4. [4]

    Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoring

    Yida Cai and Kun Liang and Sanwoo Lee and Qinghan Wang and Yunfang Wu. Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoring. 2025. arXiv:2504.05736

    arXiv 2025

  5. [6]

    Can Large Language Models Automatically Score Proficiency of Written Essays?

    Mansour, Watheq Ahmad and Albatarni, Salam and Eltanbouly, Sohaila and Elsayed, Tamer. Can Large Language Models Automatically Score Proficiency of Written Essays?. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024

    2024

  6. [7]

    Enhancing automated essay scoring by leveraging LSTM networks with hyper-parameter tuned word embeddings and fine-tuned LLMs

    Johnsi, R and Kumar, G Bharadwaja. Enhancing automated essay scoring by leveraging LSTM networks with hyper-parameter tuned word embeddings and fine-tuned LLMs. Engineering Research Express. 2025. doi:10.1088/2631-8695/adcf74

    work page doi:10.1088/2631-8695/adcf74 2025

  7. [9]

    Automated Text Scoring in the Age of Generative AI for the GPU-poor

    Christopher Michael Ormerod and Alexander Kwako. Automated Text Scoring in the Age of Generative AI for the GPU-poor. 2024. arXiv:2407.01873

    arXiv 2024

  8. [10]

    Reflective Prompt Engineering for Assessment Rubric Optimization: An Empirical Study of Human--AI Alignment

    Norgaila, Edgaras and Daniela, Linda and Kalni \ n a, Daiga. Reflective Prompt Engineering for Assessment Rubric Optimization: An Empirical Study of Human--AI Alignment. Technology, Knowledge and Learning. 2026. doi:10.1007/s10758-026-09979-2

    work page doi:10.1007/s10758-026-09979-2 2026

Show all 49 references
  1. [11]

    Exploring zero-shot essay scoring: from feature-based to LLM-based approaches

    Choi, Hongseok and Kang, Myeong-Cheol and Seong, Jin and Huang, Jin-Xia. Exploring zero-shot essay scoring: from feature-based to LLM-based approaches. Data Mining and Knowledge Discovery. 2026. doi:10.1007/s10618-026-01193-z

    2026 doi

  2. [12]

    Argumentative essay assessment with LLMs: A critical scoping review

    Favero, Lucile and Gaudeau, Gabrielle and P \'e rez-Ortiz, Juan Antonio and K \"a ser, Tanja and Oliver, Nuria. Argumentative essay assessment with LLMs: A critical scoping review. 2026. doi:10.21203/rs.3.rs-8632843/v1

    2026 doi

  3. [13]

    Xu, Wenbo and Kassim, M. S. S. and Hoo, Wai Loy and Yang, Wudao and Xu, Tianrong. Explainable AI for education: Enhancing essay scoring via rubric-aligned chain-of-thought prompting. International Journal of Modern Physics C. 2025. doi:10.1142/S0129183125420136

    2025 doi

  4. [14]

    KAES: Multi-aspect Shared Knowledge Finding and Aligning for Cross-prompt Automated Scoring of Essay Traits

    Li, Xia and Pan, Wenjing. KAES: Multi-aspect Shared Knowledge Finding and Aligning for Cross-prompt Automated Scoring of Essay Traits. Proceedings of the AAAI Conference on Artificial Intelligence. 2025. doi:10.1609/aaai.v39i23.34626

    2025 doi

  5. [15]

    CEAES : Bidirectional Reinforcement Learning Optimization for Consistent and Explainable Essay Assessment

    Li, Xia and Pan, Wenjing. CEAES : Bidirectional Reinforcement Learning Optimization for Consistent and Explainable Essay Assessment. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-...

    2025 doi

  6. [16]

    Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading

    Jamiu Adekunle Idowu and Ahmed Almasoud. Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading. 2026. arXiv:2601.22386

    2026

  7. [17]

    An LLM-based hybrid approach for enhanced automated essay scoring

    Atkinson, John and Palma, Diego. An LLM-based hybrid approach for enhanced automated essay scoring. Scientific Reports. 2025. doi:10.1038/s41598-025-87862-3

    2025 doi

  8. [19]

    E ssay J udge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models

    Su, Jiamin and Yan, Yibo and Fu, Fangteng and Han, Zhang and Ye, Jingheng and Liu, Xiang and Huo, Jiahao and Zhou, Huiyu and Hu, Xuming. E ssay J udge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models. Findings o...

    2025 doi

  9. [20]

    AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition

    Wang, Yun and Ding, Zhaojun and Wu, Xuansheng and Sun, Siyue and Liu, Ninghao and Zhai, Xiaoming. AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition. Proceedings of the AAAI Conference on Artificial Intelligence. ...

    2026 doi

  10. [21]

    Making meta-learning solve cross-prompt automatic essay scoring

    Wang, Jiong and Zhang, Qing and Liu, Jie and Wang, Xiaoyi and Xu, Mingying and Yang, Liguang and Zhou, Jianshe. Making meta-learning solve cross-prompt automatic essay scoring. Expert Systems with Applications. 2025. doi:10.1016/j.eswa.2025.126710

    2025

  11. [22]

    From Automation to Augmentation: Large Language Models Elevating Essay Scoring Landscape

    Changrong Xiao and Wenxing Ma and Sean Xin Xu and Kunpeng Zhang and Yufang Wang and Qi Fu. From Automation to Augmentation: Large Language Models Elevating Essay Scoring Landscape. CoRR. 2024. doi:10.48550/arXiv.2401.06431. 2401.06431

    2024 doi arXiv

  12. [23]

    u rstenberg, Maurice and B \

    Se ler, Kathrin and F \"u rstenberg, Maurice and B \"u hler, Babette and Kasneci, Enkelejda. Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoring. Proceedings of the 15th International Learning Analytic...

    2025

  13. [24]

    Automated Essay Scoring via Pairwise Contrastive Regression

    Xie, Jiayi and Cai, Kaiwei and Kong, Li and Zhou, Junsheng and Qu, Weiguang. Automated Essay Scoring via Pairwise Contrastive Regression. Proceedings of the 29th International Conference on Computational Linguistics. 2022

    2022

  14. [25]

    Mixture of Ordered Scoring Experts for Cross-prompt Essay Trait Scoring

    Chen, Po-Kai and Tsai, Bo-Wei and Wei, Shao Kuan and Wang, Chien-Yao and Wang, Jia-Ching and Huang, Yi-Ting. Mixture of Ordered Scoring Experts for Cross-prompt Essay Trait Scoring. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume...

    2025 doi

  15. [26]

    Abudalfa and Jessie S

    Shadi I. Abudalfa and Jessie S. Barrot , keywords =. Generative artificial intelligence for automated writing evaluation: A systematic review of trends, efficacy, and challenges , journal =. 2026 , issn =. doi:https://doi.org/10.1016/j.asw.2026.101041 , url =

    2026

  16. [27]

    2012 , howpublished =

    Ben Hamner and Jaison Morgan and lynnvandev and Mark Shermis and Tom Vander Ark , title =. 2012 , howpublished =

    2012

  17. [28]

    A Neural Approach to Automated Essay Scoring

    Taghipour, Kaveh and Ng, Hwee Tou. A Neural Approach to Automated Essay Scoring. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016. doi:10.18653/v1/D16-1193

    2016 doi

  18. [29]

    2025 , eprint =

    Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains , author =. 2025 , eprint =

    2025

  19. [30]

    A survey of automated essay scoring: Challenges, advances, and future , volume =

    Sun, Jingbo and Song, Tianbao and Peng, Weiming and Song, Jihua , year =. A survey of automated essay scoring: Challenges, advances, and future , volume =. doi:10.1016/j.neucom.2025.130916 , journal =

    2025

  20. [31]

    Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Xiao Bi and Haowei Zhang and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024. arXiv:2402.03300

    2024 arXiv

  21. [32]

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning

    Guo, Daya and others. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature. 2025. doi:10.1038/s41586-025-09422-z

    2025 doi

  22. [33]

    LCES : Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models

    Shibata, Takumi and Miyamura, Yuichi. LCES : Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.1523

    2025 doi

  23. [34]

    and Collins, Penelope

    Li, Albert W. and Collins, Penelope. Formative feedback across sources: Student perceptions and writing outcomes with instructor, peer, and AI-generated feedback. Reading and Writing. 2026. doi:10.1007/s11145-026-10761-0

    2026 doi

  24. [35]

    Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking

    Yang, Ruosong and Cao, Jiannong and Wen, Zhiyuan and Wu, Youzheng and He, Xiaodong. Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking. Findings of the Association for Computational Linguistics: ...

    2020 doi

  25. [36]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale , url =

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and Yue, Yu and Dai, Weinan and Fan, Tiantian and Liu, Gaohong and Liu, Juncai and Liu, Lingjun and Liu, Xin and Lin, Haibin and Lin, Zhiqi and Ma, Bole and Sheng, Guangming and Tong, Yuxuan and Zha...

  26. [37]

    GEEF: A neural network model for automatic essay feedback generation by integrating writing skills assessment , volume =

    Liu, Yuanchao and Han, Jiawei and Sboev, Alexander and Makarov, Ilya , year =. GEEF: A neural network model for automatic essay feedback generation by integrating writing skills assessment , volume =. doi:10.1016/j.eswa.2023.123043 , journal =

    2023

  27. [38]

    Exploring structure-aware representation learning for automated essay scoring , volume =

    Cai, Kaiwei and Kong, Li and Zhou, Junsheng and Liang, Dandan and Qu, Weiguang , year =. Exploring structure-aware representation learning for automated essay scoring , volume =. Knowledge and Information Systems , publisher =. doi:10.1007/s10115-025-02530-2 , number =

    doi

  28. [39]

    DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report , publisher =

    Li, Ruizhe and Du, Mingxuan and Xu, Benfeng and Zhu, Chiwei and Wang, Xiaorui and Mao, Zhendong , keywords =. DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report , publisher =. doi:10.48550/ARXIV.2601.08536 , url =

    doi

  29. [40]

    International Conference on Artificial Intelligence in Education , pages=

    Has Automated Essay Scoring Reached Sufficient Accuracy? Deriving Achievable QWK Ceilings from Classical Test Theory , author=. International Conference on Artificial Intelligence in Education , pages=. 2026 , organization=

    2026

  30. [41]

    Science , volume =

    Judgment under Uncertainty: Heuristics and Biases , author =. Science , volume =. 1974 , doi =

    1974

  31. [42]

    Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education

    Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)

  32. [43]

    Classification Problem Solving

    Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence

  33. [44]

    , title =

    Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

    1980

  34. [45]

    New Ways to Make Microcircuits Smaller---Duplicate Entry

    Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science

  35. [46]

    Clancey and Glenn Rennels , abstract =

    Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

    1984 doi

  36. [47]

    and Rennels, Glenn R

    Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies

  37. [48]

    Poligon: A System for Parallel Problem Solving

    Rice, James. Poligon: A System for Parallel Problem Solving

  38. [49]

    Transfer of Rule-Based Expertise through a Tutorial Dialogue

    Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue

  39. [50]

    The Engineering of Qualitative Models

    Clancey, William J. The Engineering of Qualitative Models

  40. [51]

    2023 , eprint=

    Attention Is All You Need , author=. 2023 , eprint=

    2023

  41. [52]

    Pluto: The 'Other' Red Planet

    NASA. Pluto: The 'Other' Red Planet

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.