Pith. sign in

REVIEW 4 major objections 6 minor 55 references

SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM graders can match humans on overall scores but not on step-by-step or error-cause reasoning.

desk verdict A genuinely useful benchmark dataset with a real metric problem: the CCS definition in Eqs. (1)-(2) is underspecified for variable-length step vectors, so the paper's headline consistency scores are not reproducible as written. read the letter →

arxiv 2505.07247 v2 pith:AA4IEUNT submitted 2025-05-12 cs.CL cs.AI

Peichao Lai , Kexuan Zhang , Yi Lin , Linyihan Zhang , Feiyang Ye , Jinhao Yan , Yanwei Xu , Conghui He
show 3 more authors
Yilei Wang Wentao Zhang Bin Cui
This is my paper · ORCID
classification cs.CLcs.AI
keywords shortanswerscoringlargelanguagemodelsLLM-as-a-judgestep-wiseerrorcauseanalysisbenchmarkGaokaoconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Short answer scoring — grading short student responses against a reference answer — is central to education and testing, and large language models are increasingly used as automatic graders. This paper introduces SAS-Bench, a benchmark built from real Chinese college-entrance-exam questions across nine subjects, with 1,030 questions and 4,109 student responses annotated by experts step by step. The benchmark tests not only whether an LLM's overall score matches a human's, but whether it can assign per-step scores and name the cause of each error using a predefined error taxonomy. The paper also proposes two metrics — the Collaborative Consistency Score and the Errors Consistency Score — to quantify these finer-grained agreements. Experiments on sixteen LLMs find that science questions remain the hardest, step-wise consistency is systematically harder than overall-score agreement, and few-shot examples plus scoring guidelines usually improve performance.

What carries the argument

The central machinery is the SAS-Bench dataset together with its two evaluation metrics. The dataset supplies expert step annotations, error-cause labels, and scoring guidelines, which are what make fine-grained evaluation possible. The Collaborative Consistency Score (CCS) blends overall-score disagreement with per-step score disagreement into a weighted-kappa-style agreement measure, while the Errors Consistency Score (ECS) averages Spearman correlations, within score bands, between model-predicted and human-annotated error-cause frequencies. Together they convert explainability and step-level reasoning from qualitative concerns into measurable quantities.

What would settle it

Compute CCS by hand on a case where the model and human give the same overall score but disagree on step scores, using responses with different step counts; if the resulting number falls outside the zero-to-one agreement range, or changes when the same answer is split into steps differently, the metric is not measuring what the paper claims.

Watch

Extended reading notes

Core claim

The paper's central claim is that fine-grained evaluation of LLM-based short answer scoring is both possible and necessary, and that SAS-Bench is the first benchmark built specifically for it: each response is segmented into steps, each step carries an expert score and error-cause labels, and each question includes scoring guidelines. The results show a consistent gap: Collaborative Consistency Scores are generally lower than Quadratic Weighted Kappa scores across the sixteen models, meaning that matching a human's overall score is easier for LLMs than matching the human's step-by-step reasoning. Science-related questions, especially physics and mathematics short answers, produce the largest discrepancies, and error-cause consistency is markedly low even for the best models. The paper takes this as evidence that explainability, not just final-score accuracy, must be a first-class target in automated grading systems.

Load-bearing premise

The benchmark's central fine-grained results rest on the assumption that the Collaborative Consistency Score is a well-defined agreement measure when responses have different numbers of steps and different per-step score scales.

Editorial extensions

If this is right

  • Step-wise score consistency is generally lower than overall-score consistency across models, so fine-grained grading is a harder and distinct capability for LLM judges.
  • Science short-answer questions, chiefly physics and mathematics, are the largest failure area for LLM graders on both score consistency and error-cause consistency.
  • Adding few-shot human-scored examples and scoring guidelines usually improves overall scoring accuracy, but for some subjects step-scoring agreement worsens, showing that demonstrations can mislead step-level judgments.
  • Even models that detect the correct error causes at the response level (moderate Micro-F1) show low Errors Consistency Scores, indicating their step-level reasoning is not aligned with expert annotations.
  • By pairing step scores with error-cause labels, the benchmark turns explainability into a quantitative evaluation target for automated grading systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that its responses were synthesized by LLMs rather than collected from real students, so the reported gaps estimate behavior on real exam answers only insofar as synthetic responses resemble them; a follow-up with authentic student responses is a direct test of transfer.
  • The observed inverse trend between ECS and CCS suggests a grader may trade off local error detection against holistic leniency, and systems that separate diagnosis from scoring could avoid that trade-off.
  • Because CCS depends on how responses are segmented into steps, model rankings may be sensitive to the segmentation scheme; re-annotating a subset with a different step-boundary rule would show how stable the comparisons are.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SAS-Bench, a benchmark for short answer scoring (SAS) with large language models, constructed from Gaokao questions and containing 1,030 questions and 4,109 student responses annotated by experts with step-wise scores and error-cause labels. The authors propose two new metrics, the Collaborative Consistency Score (CCS) and the Errors Consistency Score (ECS), and report experiments on 16 LLMs, concluding that science-related questions are particularly challenging and that few-shot prompting with scoring guidelines improves overall performance. The dataset is released as open source.

Significance. If the proposed metrics were well-defined, this would be a valuable contribution: SAS-Bench is among the first SAS benchmarks to provide fine-grained step-wise annotations and structured error-cause labels, and the 16-LLM comparison is extensive. The authors are also transparent about the synthetic nature of the generated student responses in Appendix A. However, the central quantitative claims rest on two newly introduced metrics that are not mathematically pinned down as written, which currently prevents the experimental conclusions from being reproducible. The dataset itself, with its expert annotations, is a potentially useful resource regardless of the metric issues.

major comments (4)
  1. [§3.3, Eqs. (1)–(2)] The Collaborative Consistency Score is not well-defined as written. In Eq. (1), i and j are not specified; standard QWK uses i and j as score categories, but W_{i,j} here is a function of response-specific step scores s_{i,k} and s_{j,k} and of m, the number of steps. Since different responses with the same overall score can have very different step-score vectors, no unique W_{i,j} exists for a category pair, and the O and E matrices in Eq. (2) cannot be constructed from category marginals. The paper also does not state how responses with different numbers of steps are aligned, nor how m and N_{s_k} are defined when the human and model step segmentations differ. Without a worked example or code-level definition, the CCS values in Table 2 and the conclusions drawn from them (e.g., "incorporating step-wise consistency introduces additional challenges") are not reproducible.
  2. [§3.3, Eqs. (3)–(4)] The Errors Consistency Score is statistically fragile. With l in {4, ..., 10} error types, the Spearman correlation in Eq. (4) is computed over very short vectors, and no significance testing or tie handling is reported. Many entries in Table 3 are strongly negative (e.g., -45.19 for DeepSeek-R1-Distill-7B on Phy. (S.)), which suggests instability rather than meaningful negative agreement. Since m=3 intervals are used, each correlation is based on even smaller subsamples. The authors should report confidence intervals or a significance test, and justify the choice of interval count m=3.
  3. [§3.2, Human Annotation] No inter-annotator agreement statistic is reported for the human annotations, even though the benchmark's value depends on the reliability of step segmentation, step scores, and error-cause labels. The two-group annotation procedure is described, but the degree of agreement before consensus is unknown. Reporting at least Cohen's kappa or Krippendorff's alpha on a subset would substantiate the claim of expert-annotated reliability and would help users interpret the reference labels.
  4. [§4.1, Settings] The same model families used to generate the synthetic student responses are also evaluated as judges: GPT-4o-mini and Deepseek-V3 were used for data synthesis, and Deepseek-V3 (as well as GPT-4o-mini) appear in Tables 2 and 3. This overlap is not discussed or controlled for, and it could inflate the apparent performance of these models on the benchmark relative to models that did not contribute to the data distribution. The authors should analyze this contamination risk, for example by evaluating on a held-out set of human-written responses or by explicitly quantifying the distributional difference, rather than only acknowledging the general synthetic-data limitation in Appendix A.
minor comments (6)
  1. [Throughout] The term "LLM-as-a-Judger" is used throughout; the standard term is "LLM-as-a-Judge" (see §2.2 and Figure 2). Please standardize.
  2. [Table 1] The rows "Avg. Steps" and "Avg. Length" report per-subject averages, but the "Total" column shows sums (40.3 and 3578.9) that are not meaningful; report a weighted average across all responses instead.
  3. [§3.2, Data Statistics] The text says "each response includes six annotated error causes," but Table 1's "Error Causes" row lists per-subject numbers of error-cause types, not per-response counts; clarify which statistic is intended.
  4. [Appendix G, Figure 10] The caption of Figure 10 is identical to that of Figure 9; Figure 10 should describe predicted score distributions (Appendix G).
  5. [References] Reference [50] is truncated: "Superdistillation achieves near-r1 performance with just 5" is missing the rest of the title.
  6. [Appendix C, Prompts] The instruction "The cumulative value of 'step_score' must also be within the range of 0 to 'pred_score'" is unclear; presumably the sum of step scores should equal pred_score, or the constraint should be stated precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's benchmark results are direct measurements under defined metrics, not derivations that reduce to their inputs.

full rationale

SAS-Bench is an empirical benchmark paper; its central claims are measurements of LLM scoring behavior under the CCS, ECS, QWK, and F1 metrics. The CCS and ECS definitions in Eqs. (1)-(4) take as inputs human annotations and model predictions and compute agreement statistics; no parameter is fitted to the benchmark's headline conclusions, and no predicted quantity is defined in terms of the claim it is used to support. The use of LLMs (GPT-4o-mini, Deepseek-V3) to synthesize student responses and the same model family as evaluators creates a distributional-bias risk, not an equivalence-by-construction, and the paper explicitly flags this in Appendix A: 'the use of LLMs to simulate and generate student responses introduces inherent distributional differences compared to human responses, which constitutes a key limitation.' The sole self-citation, reference [22], appears in the related-work survey of SAS datasets and is not load-bearing for the benchmark's validity. A reader concern that Eq. (1) is under-specified (W_{i,j} mixing category indices with per-step vectors) is a correctness and reproducibility issue, not a circular reduction: even if the metric were ill-defined, the paper is not claiming to derive a prediction from its own assumption. Under the rule that circularity requires a quoted reduction of the result to its inputs, no such step is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 2 invented entities

The central claim rests on the coherence of two new metrics, the representativeness of LLM-synthesized student responses, and the reliability of expert annotations. The paper exposes some of these as limitations but provides no formal justification, sensitivity analysis, or inter-annotator agreement statistic.

free parameters (2)
  • CCS trade-off alpha = 0.5
    Equation (1), alpha controls the relative weight of overall versus step-wise discrepancy and is set to 0.5 without sensitivity analysis or justification.
  • ECS interval count m = 3
    Equation (4), m=3 divides samples into low, medium, and high scoring intervals without a robustness check or an explanation of why 3 is optimal.
assumptions (4)
  • ad hoc to paper The CCS formula defines a valid weighted kappa over responses with variable step counts.
    Eqs. (1)-(2) mix category-level observed and expected matrices with per-step weight terms; no derivation or worked example is provided, yet the metric underpins the paper's fine-grained evaluation claim.
  • domain assumption LLM-synthesized responses are representative enough to evaluate real SAS systems.
    Section 3.2 Data Synthesis and Appendix A acknowledge that simulated responses differ from human responses, but the benchmark's practical claims depend on this representativeness.
  • domain assumption Expert annotations are reliable ground truth.
    Section 3.2 Human Annotation describes two annotation groups and consensus discussion but reports no inter-annotator agreement statistic.
  • domain assumption The error-cause taxonomy is complete and unambiguous.
    Section 3.2 Error Causes Construction uses GPT-4o plus human consolidation, but no validation is given that the taxonomy covers all real student mistakes.
invented entities (2)
  • Collaborative Consistency Score (CCS)
    purpose: To combine overall-score and step-wise score agreement into a single kappa-style index.
    No simulation, external validation, or worked example demonstrates that CCS behaves as intended; its definition is ambiguous for variable-length responses.
  • Errors Consistency Score (ECS)
    purpose: To measure model-human agreement on error-cause frequencies within score intervals.
    The metric uses Spearman correlation on small error-frequency vectors, and the paper reports negative values without validating stability or significance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models." pith.science (2026). https://pith.science/paper/AA4IEUNT

@misc{pith2026250507247,
  author       = {Pith},
  title        = {Pith review of: SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AA4IEUNT}},
  note         = {Machine review of arXiv:2505.07247}
}
read the original abstract

Subjective Answer Grading (SAG) plays a crucial role in education, standardized testing, and automated assessment systems, particularly for evaluating short-form responses in Short Answer Scoring (SAS). However, existing approaches often produce coarse-grained scores and lack detailed reasoning. Although large language models (LLMs) have demonstrated potential as zero-shot evaluators, they remain susceptible to bias, inconsistencies with human judgment, and limited transparency in scoring decisions. To overcome these limitations, we introduce SAS-Bench, a benchmark specifically designed for LLM-based SAS tasks. SAS-Bench provides fine-grained, step-wise scoring, expert-annotated error categories, and a diverse range of question types derived from real-world subject-specific exams. This benchmark facilitates detailed evaluation of model reasoning processes and explainability. We also release an open-source dataset containing 1,030 questions and 4,109 student responses, each annotated by domain experts. Furthermore, we conduct comprehensive experiments with various LLMs, identifying major challenges in scoring science-related questions and highlighting the effectiveness of few-shot prompting in improving scoring accuracy. Our work offers valuable insights into the development of more robust, fair, and educationally meaningful LLM-based evaluation systems.

Figures

Figures reproduced from arXiv: 2505.07247 by the authors.

Figure 1
Figure 1. Comparison of existing benchmarks and our benchmark. To better understand how these challenges impact the SAS task, a benchmark specifically designed for generative language models is essential. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The workflow of our SAS-Bench. The results from the human judger are predefined during [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of ques￾tion types across different sub￾jects. 3.3 Evaluation Protocol Most existing benchmarks [35, 36, 16] provide only manually annotated overall scores, enabling model evaluation solely based on final scoring outcomes. However, they fall short in assessing step-wise scoring accuracy and the ability to identify specific error causes within responses. To more comprehensively evaluate LLM performance o… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Comparison of QWK scores across LLMs. Overall, we observe a positive correlation between CCS and QWK scores across all models, as well as between model size and average performance. Notably, Deepseek-V3 and Deepseek-R1 achieve the best average results in CCS and QWK me…
Figure 5
Figure 5. Figure 5: Performance changes of selected models across various subjects under settings of “few-shot” [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Prompts for model scoring and error cause prediction, where the descriptions are provided [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Example format of a math short-answer response annotated with both the score and [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: ECS and CCS comparison across low, medium, and high score intervals. The upper line [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Normalized distributions of total human-annotated scores and the number of steps in the [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Normalized distributions of total human-annotated scores and the number of steps in the [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 28 canonical work pages

  1. [1]

    Automated essay scoring using discourse external knowledge,

    N. A. Khayi and V . Rus, “Automated essay scoring using discourse external knowledge,” in Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024 . ijcai.org, 2024, pp. 7154–7160. [Online]. Available: https://www.ijcai.org/proceedings/2024/791

    work page 2024

  2. [2]

    Improve llm-based automatic essay scoring with linguistic features,

    Z. J. Hou, A. Ciuba, and X. L. Li, “Improve llm-based automatic essay scoring with linguistic features,” CoRR, vol. abs/2502.09497, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2502.09497

    arXiv doi:10.48550/arxiv.2502.09497 2025

  3. [3]

    An automated essay scoring systems: a systematic literature review,

    D. Ramesh and S. K. Sanampudi, “An automated essay scoring systems: a systematic literature review,” Artif. Intell. Rev. , vol. 55, no. 3, pp. 2495–2527, 2022. [Online]. Available: https://doi.org/10.1007/s10462-021-10068-2

    doi:10.1007/s10462-021-10068-2 2022

  4. [4]

    Prompt agnostic essay scorer: A domain generalization approach to cross-prompt automated essay scoring,

    R. Ridley, L. He, X. Dai, S. Huang, and J. Chen, “Prompt agnostic essay scorer: A domain generalization approach to cross-prompt automated essay scoring,” arXiv preprint arXiv:2008.01441, 2020

    arXiv 2008

  5. [5]

    Automated cross-prompt scoring of essay traits,

    R. Ridley, L. He, X.-y. Dai, S. Huang, and J. Chen, “Automated cross-prompt scoring of essay traits,” in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 15, 2021, pp. 13 745–13 753

    work page 2021

  6. [6]

    A short answer grading system in chinese by cnn,

    S.-H. Wu and C.-Y . Yeh, “A short answer grading system in chinese by cnn,” in2019 IEEE 10th Interna- tional Conference on Awareness Science and Technology (iCAST). IEEE, 2019, pp. 1–5

    work page 2019

  7. [7]

    A short answer grading system in chinese by support vector approach,

    S.-H. Wu and W.-F. Shih, “A short answer grading system in chinese by support vector approach,” in Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, 2018, pp. 125–129

    work page 2018

  8. [9]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,” CoRR, vol. abs/2303.08774, pp. 1–100, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.08774

    arXiv doi:10.48550/arxiv.2303.08774 2023

Show all 55 references
  1. [10]

    Deepseek-v3 technical report,

    DeepSeek-AI, “Deepseek-v3 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2412.19437

    2024 arXiv

  2. [11]

    Is llm-as-a-judge robust? investigating universal adversarial attacks on zero-shot LLM assessment,

    V . Raina, A. Liusie, and M. J. F. Gales, “Is llm-as-a-judge robust? investigating universal adversarial attacks on zero-shot LLM assessment,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16,...

    2024

  3. [12]

    Beyond yes and no: Improving zero-shot LLM rankers via scoring fine-grained relevance labels,

    H. Zhuang, Z. Qin, K. Hui, J. Wu, L. Yan, X. Wang, and M. Bendersky, “Beyond yes and no: Improving zero-shot LLM rankers via scoring fine-grained relevance labels,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguist...

    2024

  4. [13]

    GLIDER: grading LLM interactions and decisions using explainable ranking,

    D. Deshpande, S. S. Ravi, S. CH-Wang, B. Mielczarek, A. Kannappan, and R. Qian, “GLIDER: grading LLM interactions and decisions using explainable ranking,” CoRR, vol. abs/2412.14140, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2412.14140

    2024 doi arXiv

  5. [14]

    The hewlett foundation: Automated essay scoring,

    B. Hamner, J. Morgan, lynnvandev, M. Shermis, and T. V . Ark, “The hewlett foundation: Automated essay scoring,” https://kaggle.com/competitions/asap-aes, 2012, kaggle

    2012

  6. [15]

    Scaa: A dataset for automated short answer grading of children’s free-text answers in hindi and marathi,

    D. Agarwal, S. Gupta, and N. Baghel, “Scaa: A dataset for automated short answer grading of children’s free-text answers in hindi and marathi,” in Proceedings of the 17th International Conference on Natural Language Processing, ICON 2020, Indian Institute of Technology Patna, ...

    2020

  7. [16]

    Semeval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge,

    M. O. Dzikovska, R. D. Nielsen, C. Brew, C. Leacock, D. Giampiccolo, L. Bentivogli, P. Clark, I. Dagan, and H. T. Dang, “Semeval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge,” in Proceedings of the 7th International Workshop...

    2013

  8. [17]

    Automated long answer grading with ricechem dataset,

    S. Sonkar, K. Ni, L. T. Lu, K. Kincaid, J. S. Hutchinson, and R. G. Baraniuk, “Automated long answer grading with ricechem dataset,” in Artificial Intelligence in Education - 25th International Conference, AIED 2024, Recife, Brazil, July 8-12, 2024, Proceedings, Part I, ser. L...

    2024 doi

  9. [18]

    A new benchmark for automatic essay scoring in Portuguese,

    I. C. Silveira, A. Barbosa, and D. D. Mau’a, “A new benchmark for automatic essay scoring in Portuguese,” in Proceedings of the 16th International Conference on Computational Processing of Portuguese - Vol. 1, P. Gamallo, D. Claro, A. Teixeira, L. Real, M. Garcia, H. G. Olivei...

    2024

  10. [19]

    The hewlett foundation: Short answer scoring,

    Barbara, B. Hamner, J. Morgan, lynnvandev, and M. Shermis, “The hewlett foundation: Short answer scoring,” https://kaggle.com/competitions/asap-sas, 2012, kaggle

    2012

  11. [20]

    Automated short answer grading: A simple solution for a difficult task,

    S. Menini, S. Tonelli, G. D. Gasperis, and P. Vittorini, “Automated short answer grading: A simple solution for a difficult task,” in Proceedings of the Sixth Italian Conference on Computational Linguistics, Bari, Italy, November 13-15, 2019, ser. CEUR Workshop Proceedings, R....

    2019

  12. [21]

    Text-to-text semantic similarity for automatic short answer grading,

    M. Mohler and R. Mihalcea, “Text-to-text semantic similarity for automatic short answer grading,” in EACL 2009, 12th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, Athens, Greece, March 30 - April 3, 2009 , A...

    2009

  13. [22]

    M-sim: Multi-level semantic inference model for chinese short answer scoring in low-resource scenarios,

    P. Lai, F. Ye, Y . Fu, Z. Chen, Y . Wu, and Y . Wang, “M-sim: Multi-level semantic inference model for chinese short answer scoring in low-resource scenarios,” Comput. Speech Lang., vol. 84, p. 101575, 2024. [Online]. Available: https://doi.org/10.1016/j.csl.2023.101575

    2024

  14. [23]

    L-eval: Instituting standardized evaluation for long context language models,

    C. An, S. Gong, M. Zhong, X. Zhao, M. Li, J. Zhang, L. Kong, and X. Qiu, “L-eval: Instituting standardized evaluation for long context language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024...

    2024 doi

  15. [24]

    Is chatgpt a good NLG evaluator? A preliminary study,

    J. Wang, Y . Liang, F. Meng, H. Shi, Z. Li, J. Xu, J. Qu, and J. Zhou, “Is chatgpt a good NLG evaluator? A preliminary study,” CoRR, vol. abs/2303.04048, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.04048

    2023 doi arXiv

  16. [25]

    Large language models can accurately predict searcher preferences,

    P. Thomas, S. Spielman, N. Craswell, and B. Mitra, “Large language models can accurately predict searcher preferences,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-1...

    2024

  17. [26]

    Perspectives on large language models for relevance judgment,

    G. Faggioli, L. Dietz, C. L. A. Clarke, G. Demartini, M. Hagen, C. Hauff, N. Kando, E. Kanoulas, M. Potthast, B. Stein, and H. Wachsmuth, “Perspectives on large language models for relevance judgment,” in Proceedings of the 2023 ACM SIGIR International Conference on Theory of ...

    2023

  18. [27]

    Enhancing transfer learning of llms through fine- tuning on task - related corpora for automated short-answer grading,

    N. Kazi and I. Kahanda, “Enhancing transfer learning of llms through fine- tuning on task - related corpora for automated short-answer grading,” in International Conference on Machine Learning and Applications, ICMLA 2023, Jacksonville, FL, USA, December 15-17, 2023 . IEEE, 20...

    2023

  19. [28]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neur...

    2023

  20. [29]

    Judgelm: Fine-tuned large language models are scalable judges,

    L. Zhu, X. Wang, and X. Wang, “Judgelm: Fine-tuned large language models are scalable judges,”CoRR, vol. abs/2310.17631, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2310.17631

    2023 doi arXiv

  21. [30]

    Calibrating llm-based evaluator,

    Y . Liu, T. Yang, S. Huang, Z. Zhang, H. Huang, F. Wei, W. Deng, F. Sun, and Q. Zhang, “Calibrating llm-based evaluator,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 202...

    2024

  22. [31]

    Exploring LLM prompting strategies for joint essay scoring and feedback generation,

    M. Stahl, L. Biermann, A. Nehring, and H. Wachsmuth, “Exploring LLM prompting strategies for joint essay scoring and feedback generation,” inProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications, BEA 2024, Mexico City, Mexico, June 20,...

    2024

  23. [32]

    Knowledge distillation of llms for automatic scoring of science assessments,

    E. Latif, L. Fang, P. Ma, and X. Zhai, “Knowledge distillation of llms for automatic scoring of science assessments,” in Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Co...

    2024

  24. [33]

    Efficient LLM comparative assessment: A product of experts framework for pairwise comparisons,

    A. Liusie, V . Raina, Y . Fathullah, and M. J. F. Gales, “Efficient LLM comparative assessment: A product of experts framework for pairwise comparisons,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, Nove...

    2024

  25. [34]

    Evaluating the performance of large language models on GAOKAO benchmark,

    X. Zhang, C. Li, Y . Zong, Z. Ying, L. He, and X. Qiu, “Evaluating the performance of large language models on GAOKAO benchmark,” CoRR, vol. abs/2305.12474, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2305.12474

    2023 doi arXiv

  26. [35]

    Asag2024: A combined benchmark for short answer grading,

    G. Meyer, P. Breuer, and J. Fürst, “Asag2024: A combined benchmark for short answer grading,” in Proceedings of the 2024 on ACM Virtual Global Computing Education Conference V . 2, 2024, pp. 322–323

    2024

  27. [36]

    Learning to grade short answer questions using semantic similarity measures and dependency graph alignments,

    M. Mohler, R. C. Bunescu, and R. Mihalcea, “Learning to grade short answer questions using semantic similarity measures and dependency graph alignments,” in The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of th...

    2011

  28. [37]

    Short answer grading using one-shot prompting and text similarity scoring model,

    S. Yoon, “Short answer grading using one-shot prompting and text similarity scoring model,”CoRR, vol. abs/2305.18638, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2305.18638

    2023 doi arXiv

  29. [38]

    Improving the performance of automatic short answer grading using transfer learning and augmentation,

    S. Bonthu, S. R. Sree, and M. H. M. K. Prasad, “Improving the performance of automatic short answer grading using transfer learning and augmentation,” Eng. Appl. Artif. Intell., vol. 123, no. Part A, p. 106292,

  30. [39]

    Rankcse: Unsupervised sentence representations learning via learning to rank,

    J. Liu, J. Liu, Q. Wang, J. Wang, W. Wu, Y . Xian, D. Zhao, K. Chen, and R. Yan, “Rankcse: Unsupervised sentence representations learning via learning to rank,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2023 doi

  31. [40]

    Diffcse: Difference-based contrastive learning for sentence embeddings,

    Y . Chuang, R. Dangovski, H. Luo, Y . Zhang, S. Chang, M. Soljacic, S. Li, S. Yih, Y . Kim, and J. R. Glass, “Diffcse: Difference-based contrastive learning for sentence embeddings,” inProceedings of the 2022 Conference of the North American Chapter of the Association for Comp...

    2022

  32. [41]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Rozière, B. Biron, B. Ta...

    2024 doi arXiv

  33. [42]

    Qwen2 technical report,

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. ...

    2024 arXiv

  34. [43]

    Qwen2.5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y ...

    2024 arXiv

  35. [44]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools,

    T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Rojas, G. Feng, H. Zhao, H. Lai, H. Yu, H. Wang, J. Sun, J. Zhang, J. Cheng, J. Gui, J. Tang, J. Zhang, J. Li, L. Zhao, L. Wu, L. Zhong, M. Liu, M. Huang, P. Zhang, Q. Zheng, R. Lu, S. Duan, S. Zhang, S. Cao, S. Yang, W. L....

    2024

  36. [45]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T...

    2024 arXiv

  37. [46]

    Deepseek-prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition,

    Z. Z. Ren, Z. Shao, J. Song, H. Xin, H. Wang, W. Zhao, L. Zhang, Z. Fu, Q. Zhu, D. Yang, Z. F. Wu, Z. Gou, S. Ma, H. Tang, Y . Liu, W. Gao, D. Guo, and C. Ruan, “Deepseek-prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition,” ...

    2025 arXiv

  38. [47]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y . Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. L...

    2025 arXiv

  39. [48]

    Qwq-32b: Embracing the power of reinforcement learning,

    Q. Team, “Qwq-32b: Embracing the power of reinforcement learning,” March 2025. [Online]. Available: https://qwenlm.github.io/blog/qwq-32b/

    2025

  40. [49]

    [Online]

    ——, “Qwen3,” April 2025. [Online]. Available: https://qwenlm.github.io/blog/qwen3/

    2025

  41. [50]

    Team, “Superdistillation achieves near-r1 performance with just 5

    T. Team, “Superdistillation achieves near-r1 performance with just 5

  42. [51]

    Mimo: Unlocking the reasoning potential of language model – from pretraining to posttraining,

    Xiaomi LLM-Core Team, “Mimo: Unlocking the reasoning potential of language model – from pretraining to posttraining,” 2025. [Online]. Available: https://github.com/XiaomiMiMo/MiMo

    2025

  43. [52]

    Metamath: Bootstrap your own mathematical questions for large language models,

    L. Yu, W. Jiang, H. Shi, J. YU, Z. Liu, Y . Zhang, J. Kwok, Z. Li, A. Weller, and W. Liu, “Metamath: Bootstrap your own mathematical questions for large language models,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://o...

    2024

  44. [54]

    - Independently evaluate each step: * Determine correctness ('label')

    Step-by-step analysis: - Break down each step of the student's solution. - Independently evaluate each step: * Determine correctness ('label'). * If errors exist, select one or more primary causes from the error list ('errors'). - Single-step evaluation format: {{'step_score':...

  45. [55]

    - Provide an overall evaluation ('label’)

    Comprehensive assessment: - Summarize the scores of each step to calculate the total score. - Provide an overall evaluation ('label’)

  46. [56]

    id": "Math_ShortAns_3

    Result output: - Use standard JSON format: {{ 'total': total score, 'pred_score': evaluated total score, 'steps': [step-by-step evaluation results] }} - 'pred_score' must be within the range of 'total'. - The cumulative value of 'step_score' must also be within the range of 0 ...

  47. [2023]

    Available: https://doi.org/10.1016/j.engappai.2023.106292

    [Online]. Available: https://doi.org/10.1016/j.engappai.2023.106292

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.