{"id":"3eb4158c-e714-40f4-8e36-7987af782dbf","arxiv_id":"2505.07247","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper introduces a 1,030-question benchmark of 4,109 expert-annotated student responses with step-wise scores and error causes, along with two consistency metrics for evaluating LLM short answer scoring.","lead":"SAS-Bench is a new benchmark that evaluates how well large language models grade short student answers, using expert-annotated step-by-step scores and error categories from Chinese Gaokao exams. It could become a standard test for building more transparent automated grading systems, but the validity of its new metrics still needs checking.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CCS metric in Eqs. (1)-(2) is not well-defined because W_{i,j} requires a unique step-score vector per overall-score category, yet tied overall scores have different step vectors; the reported CCS values are therefore not reproducible.","rationale":"The reader's weakest_assumption correctly identifies the CCS definition as the key load-bearing point. The paper's contribution is a benchmark plus a set of quantitative evaluations; if the central evaluation metric is not formally well-defined, then the reported rankings and conclusions about step-wise scoring difficulty are not grounded. This is not an ad hominem or a disagreement with current consensus; it is an internal consistency problem at the level of the metric definition in §3.3. The concern is concrete: Eq. (1) mixes category-level QWK weights with per-step, per-response scores, and no construction of O and E is given that would resolve the ambiguity. The proposed concrete test settles the question by forcing an explicit construction on real data and checking whether the published numbers can be reproduced. I also note that the paper honestly acknowledges in Appendix A that student responses are LLM-simulated rather than real; that limitation is stated, so it is less damaging than the metric issue. The dataset itself, the expert annotation effort, and the Micro-F1 analysis in Appendix D provide independent value and are reasons not to reject the paper outright. The reader's CONDITIONAL verdict remains appropriate, with revision conditions focused on formalizing CCS and ECS, reporting inter-annotator agreement, and releasing code that reproduces Tables 2 and 3.","tokens_in":23336,"tokens_out":5273,"duration_ms":55315,"concrete_test":"Recompute the Math (S.) CCS for Deepseek-V3 from the released dataset and prompts, constructing O, E, and W explicitly for responses whose human-assigned overall scores are tied but whose step-score vectors differ. If W_{i,j} cannot be assigned a unique value for such tied categories, or if the recomputed value does not reproduce the published entry in Table 2, then the metric definition is the blocker and the reported CCS results are not interpretable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claims depend on the Collaborative Consistency Score described in §3.3, Eqs. (1)-(2), but this metric is not well-defined as written. In the standard QWK framework, i and j index score categories in a contingency table O (observed) and E (expected), and the weight is a fixed function of those categories. Eq. (1), however, defines W_{i,j} using r_i, r_j (overall scores) and s_{i,k}, s_{j,k} (step scores for the k-th step). If i,j are categories, then s_{i,k} is undefined: multiple responses with the same overall score can have very different step-score vectors, so no unique step vector exists for category i. If instead i,j index individual responses, then O and E cannot be the 'standard QWK' matrices, because QWK's expected matrix is built from category marginal distributions; no such distribution is defined over step-score vectors, and the formula would not be a chance-corrected agreement measure. The paper gives no worked example, no derivation, and no code-level definition of O, E, or W, and it does not state how responses with different numbers of steps m or different step-score scales N_{s_k} are aligned. Until this is pinned down, the CCS values in Table 2 and the resulting conclusions, such as 'incorporating step-wise consistency introduces additional challenges,' are not reproducible. The ECS metric in Eqs. (3)-(4) has a related issue: with only 4-10 error types and small per-interval samples, Spearman correlations over frequency vectors are unstable and can be undefined under ties, yet negative ECS values in Table 3 are never explained. Because the benchmark's headline claims are quantitative, under-specification of these metrics is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SAS-Bench, a benchmark for short answer scoring (SAS) with large language models, constructed from Gaokao questions and containing 1,030 questions and 4,109 student responses annotated by experts with step-wise scores and error-cause labels. The authors propose two new metrics, the Collaborative Consistency Score (CCS) and the Errors Consistency Score (ECS), and report experiments on 16 LLMs, concluding that science-related questions are particularly challenging and that few-shot prompting with scoring guidelines improves overall performance. The dataset is released as open source.","tokens_in":23684,"tokens_out":5148,"duration_ms":47064,"significance":"If the proposed metrics were well-defined, this would be a valuable contribution: SAS-Bench is among the first SAS benchmarks to provide fine-grained step-wise annotations and structured error-cause labels, and the 16-LLM comparison is extensive. The authors are also transparent about the synthetic nature of the generated student responses in Appendix A. However, the central quantitative claims rest on two newly introduced metrics that are not mathematically pinned down as written, which currently prevents the experimental conclusions from being reproducible. The dataset itself, with its expert annotations, is a potentially useful resource regardless of the metric issues.","major_comments":[{"comment":"The Collaborative Consistency Score is not well-defined as written. In Eq. (1), i and j are not specified; standard QWK uses i and j as score categories, but W_{i,j} here is a function of response-specific step scores s_{i,k} and s_{j,k} and of m, the number of steps. Since different responses with the same overall score can have very different step-score vectors, no unique W_{i,j} exists for a category pair, and the O and E matrices in Eq. (2) cannot be constructed from category marginals. The paper also does not state how responses with different numbers of steps are aligned, nor how m and N_{s_k} are defined when the human and model step segmentations differ. Without a worked example or code-level definition, the CCS values in Table 2 and the conclusions drawn from them (e.g., \"incorporating step-wise consistency introduces additional challenges\") are not reproducible.","section":"§3.3, Eqs. (1)–(2)"},{"comment":"The Errors Consistency Score is statistically fragile. With l in {4, ..., 10} error types, the Spearman correlation in Eq. (4) is computed over very short vectors, and no significance testing or tie handling is reported. Many entries in Table 3 are strongly negative (e.g., -45.19 for DeepSeek-R1-Distill-7B on Phy. (S.)), which suggests instability rather than meaningful negative agreement. Since m=3 intervals are used, each correlation is based on even smaller subsamples. The authors should report confidence intervals or a significance test, and justify the choice of interval count m=3.","section":"§3.3, Eqs. (3)–(4)"},{"comment":"No inter-annotator agreement statistic is reported for the human annotations, even though the benchmark's value depends on the reliability of step segmentation, step scores, and error-cause labels. The two-group annotation procedure is described, but the degree of agreement before consensus is unknown. Reporting at least Cohen's kappa or Krippendorff's alpha on a subset would substantiate the claim of expert-annotated reliability and would help users interpret the reference labels.","section":"§3.2, Human Annotation"},{"comment":"The same model families used to generate the synthetic student responses are also evaluated as judges: GPT-4o-mini and Deepseek-V3 were used for data synthesis, and Deepseek-V3 (as well as GPT-4o-mini) appear in Tables 2 and 3. This overlap is not discussed or controlled for, and it could inflate the apparent performance of these models on the benchmark relative to models that did not contribute to the data distribution. The authors should analyze this contamination risk, for example by evaluating on a held-out set of human-written responses or by explicitly quantifying the distributional difference, rather than only acknowledging the general synthetic-data limitation in Appendix A.","section":"§4.1, Settings"}],"minor_comments":[{"comment":"The term \"LLM-as-a-Judger\" is used throughout; the standard term is \"LLM-as-a-Judge\" (see §2.2 and Figure 2). Please standardize.","section":"Throughout"},{"comment":"The rows \"Avg. Steps\" and \"Avg. Length\" report per-subject averages, but the \"Total\" column shows sums (40.3 and 3578.9) that are not meaningful; report a weighted average across all responses instead.","section":"Table 1"},{"comment":"The text says \"each response includes six annotated error causes,\" but Table 1's \"Error Causes\" row lists per-subject numbers of error-cause types, not per-response counts; clarify which statistic is intended.","section":"§3.2, Data Statistics"},{"comment":"The caption of Figure 10 is identical to that of Figure 9; Figure 10 should describe predicted score distributions (Appendix G).","section":"Appendix G, Figure 10"},{"comment":"Reference [50] is truncated: \"Superdistillation achieves near-r1 performance with just 5\" is missing the rest of the title.","section":"References"},{"comment":"The instruction \"The cumulative value of 'step_score' must also be within the range of 0 to 'pred_score'\" is unclear; presumably the sum of step scores should equal pred_score, or the constraint should be stated precisely.","section":"Appendix C, Prompts"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is potentially useful, but the two new metrics need a rigorous formal definition and a worked example before the experimental claims can be evaluated. Please encourage the authors to provide code or a detailed algorithm for CCS and ECS, and to address the annotator-reliability and model-overlap concerns in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine benchmark contribution with a real, fixable metric problem. The dataset—1,030 Gaokao questions, 4,109 responses, nine subjects, expert step-wise scores and error-cause labels—fills a gap that existing SAS benchmarks leave open. The related work is honest about prior limits, and the appendix includes a clear limitation statement acknowledging that LLM-simulated responses are not the same distribution as real student writing. That matters, but it is not a fatal flaw; they are explicit about it.\n\nWhat the paper does well: the annotation design is meaningful. Giving each response segmented steps, per-step scores, and a small error-cause taxonomy lets you ask questions you cannot ask with ASAP-SAS or SciEntsBank. The 16-model sweep is reasonable, and the few-shot and ablation results in Figure 5 are a useful practical finding, even if the pattern is not uniform. I also credit them for including the GitHub link and describing the cleaning pipeline.\n\nThe soft spot is the load-bearing one. CCS in Eqs. (1)-(2) is not well-defined as written. If i and j are score categories, then s_{i,k} cannot be a single step-score vector, because multiple responses with the same overall score have different step scores. If i and j index individual responses, then O and E cannot be the standard QWK expected matrix, which is built from category marginals. Either way, the formula needs a worked example and an explicit statement of how O, E, and W are constructed. The paper gives none. Since CCS powers Table 2 and the 'step-wise consistency adds challenges' conclusion, this is not a stylistic issue. ECS has a related but smaller problem: Spearman on frequency vectors over 4-10 error types with no significance testing can produce unstable negative values, and those negatives are never explained. The lack of any inter-annotator agreement is also a notable omission for a benchmark whose value depends on annotation quality.\n\nI do not think the paper is circular or dishonest. The synthetic-data limitation is disclosed. The central issue is methodological reproducibility. A serious referee could send this back with a request for a precise metric definition, a worked example, code, and agreement statistics. That is standard and should be the outcome, not a desk rejection.\n\nVerdict: send to peer review. The dataset and annotation effort deserve a shot; the numbers just need to be made checkable.","headline":"A genuinely useful benchmark dataset with a real metric problem: the CCS definition in Eqs. (1)-(2) is underspecified for variable-length step vectors, so the paper's headline consistency scores are not reproducible as written.","tokens_in":24266,"tokens_out":2349,"would_cite":false,"duration_ms":23431,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM graders can match humans on overall scores but not on step-by-step or error-cause reasoning.","keywords":["short answer scoring","large language models","LLM-as-a-judge","step-wise scoring","error cause analysis","benchmark","Gaokao","scoring consistency"],"falsifier":"Compute CCS by hand on a case where the model and human give the same overall score but disagree on step scores, using responses with different step counts; if the resulting number falls outside the zero-to-one agreement range, or changes when the same answer is split into steps differently, the metric is not measuring what the paper claims.","tokens_in":23144,"feed_emoji":"📝","tokens_out":8630,"duration_ms":76108,"temperature":0.7,"pith_summary":"Short answer scoring — grading short student responses against a reference answer — is central to education and testing, and large language models are increasingly used as automatic graders. This paper introduces SAS-Bench, a benchmark built from real Chinese college-entrance-exam questions across nine subjects, with 1,030 questions and 4,109 student responses annotated by experts step by step. The benchmark tests not only whether an LLM's overall score matches a human's, but whether it can assign per-step scores and name the cause of each error using a predefined error taxonomy. The paper also proposes two metrics — the Collaborative Consistency Score and the Errors Consistency Score — to quantify these finer-grained agreements. Experiments on sixteen LLMs find that science questions remain the hardest, step-wise consistency is systematically harder than overall-score agreement, and few-shot examples plus scoring guidelines usually improve performance.","feed_headline":"LLM graders match humans on scores, not steps","feed_subtitle":"A 4,109-response exam benchmark shows models still trail humans at step-level and error-cause scoring.","key_machinery":"The central machinery is the SAS-Bench dataset together with its two evaluation metrics. The dataset supplies expert step annotations, error-cause labels, and scoring guidelines, which are what make fine-grained evaluation possible. The Collaborative Consistency Score (CCS) blends overall-score disagreement with per-step score disagreement into a weighted-kappa-style agreement measure, while the Errors Consistency Score (ECS) averages Spearman correlations, within score bands, between model-predicted and human-annotated error-cause frequencies. Together they convert explainability and step-level reasoning from qualitative concerns into measurable quantities.","core_discovery":"The paper's central claim is that fine-grained evaluation of LLM-based short answer scoring is both possible and necessary, and that SAS-Bench is the first benchmark built specifically for it: each response is segmented into steps, each step carries an expert score and error-cause labels, and each question includes scoring guidelines. The results show a consistent gap: Collaborative Consistency Scores are generally lower than Quadratic Weighted Kappa scores across the sixteen models, meaning that matching a human's overall score is easier for LLMs than matching the human's step-by-step reasoning. Science-related questions, especially physics and mathematics short answers, produce the largest discrepancies, and error-cause consistency is markedly low even for the best models. The paper takes this as evidence that explainability, not just final-score accuracy, must be a first-class target in automated grading systems.","pith_inferences":["An implication the paper leaves implicit is that its responses were synthesized by LLMs rather than collected from real students, so the reported gaps estimate behavior on real exam answers only insofar as synthetic responses resemble them; a follow-up with authentic student responses is a direct test of transfer.","The observed inverse trend between ECS and CCS suggests a grader may trade off local error detection against holistic leniency, and systems that separate diagnosis from scoring could avoid that trade-off.","Because CCS depends on how responses are segmented into steps, model rankings may be sensitive to the segmentation scheme; re-annotating a subset with a different step-boundary rule would show how stable the comparisons are."],"forward_implications":["Step-wise score consistency is generally lower than overall-score consistency across models, so fine-grained grading is a harder and distinct capability for LLM judges.","Science short-answer questions, chiefly physics and mathematics, are the largest failure area for LLM graders on both score consistency and error-cause consistency.","Adding few-shot human-scored examples and scoring guidelines usually improves overall scoring accuracy, but for some subjects step-scoring agreement worsens, showing that demonstrations can mislead step-level judgments.","Even models that detect the correct error causes at the response level (moderate Micro-F1) show low Errors Consistency Scores, indicating their step-level reasoning is not aligned with expert annotations.","By pairing step scores with error-cause labels, the benchmark turns explainability into a quantitative evaluation target for automated grading systems."],"supporting_citations":[{"why":"Supplies the source questions from China's Gaokao exam bank from which the benchmark is derived.","marker":"[34]"},{"why":"Prior short-answer scoring dataset with overall scores only; serves as the baseline the paper contrasts with its fine-grained annotations.","marker":"[19]"},{"why":"Prior benchmark with only overall-score annotations; motivates the need for step-wise and error-cause evaluation.","marker":"[16]"},{"why":"Documents LLM judges' sensitivity to wording and position of key phrases, motivating the benchmark's fine-grained design.","marker":"[11]"},{"why":"Shows LLM scoring bias depends on binary versus numeric scales, motivating the multi-dimensional consistency metrics.","marker":"[12]"},{"why":"Prior work on explainable LLM grading that underpins the error-cause consistency dimension.","marker":"[13]"},{"why":"Prior joint scoring-and-feedback work that informs the error-cause taxonomy and scoring guidelines.","marker":"[31]"},{"why":"Quadratic Weighted Kappa, the standard consistency metric that CCS extends.","marker":"[37]"},{"why":"Deepseek-R1, the top-scoring reasoning model in the experiments, used for conclusions about reasoning-based graders.","marker":"[47]"},{"why":"Deepseek-V3, the top-scoring RLHF-based model, used to compare model families.","marker":"[10]"}],"fun_headline_variants":["LLM scorers hit overall grades but miss step logic","New benchmark shows LLMs lag on step-level grading","SAS-Bench: LLMs match final scores, not reasoning","Grading gap: LLMs ace scores, flunk step analysis","LLMs score like humans, but not on reasoning steps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's central fine-grained results rest on the assumption that the Collaborative Consistency Score is a well-defined agreement measure when responses have different numbers of steps and different per-step score scales.","fun_headline_variants_meta":{"raw":{"variants":["LLM scorers hit overall grades but miss step logic","New benchmark shows LLMs lag on step-level grading","SAS-Bench: LLMs match final scores, not reasoning","Grading gap: LLMs ace scores, flunk step analysis","LLMs score like humans, but not on reasoning steps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000548,"raw_usage":{"total_tokens":2604,"prompt_tokens":918,"completion_tokens":1686,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":1612}},"tokens_in":534,"tokens_out":1686,"duration_ms":9900,"temperature":1.0,"reasoning_tokens":1612,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:21:34.477737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute CCS by hand on a case where the model and human give the same overall score but disagree on step scores, using responses with different step counts; if the resulting number falls outside the zero-to-one agreement range, or changes when the same answer is split into steps differently, the metric is not measuring what the paper claims.","supporting_citations":[{"cited_title":"The hewlett foundation: Short answer scoring,","cited_arxiv_id":null,"evidence_quote":"Prior short-answer scoring dataset with overall scores only; serves as the baseline the paper contrasts with its fine-grained annotations."},{"cited_title":"Semeval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge,","cited_arxiv_id":null,"evidence_quote":"Prior benchmark with only overall-score annotations; motivates the need for step-wise and error-cause evaluation."},{"cited_title":"Is llm-as-a-judge robust? investigating universal adversarial attacks on zero-shot LLM assessment,","cited_arxiv_id":null,"evidence_quote":"Documents LLM judges' sensitivity to wording and position of key phrases, motivating the benchmark's fine-grained design."},{"cited_title":"Exploring LLM prompting strategies for joint essay scoring and feedback generation,","cited_arxiv_id":null,"evidence_quote":"Prior joint scoring-and-feedback work that informs the error-cause taxonomy and scoring guidelines."}],"review_version":1}