{"id":"eb475fa2-5608-419e-9309-6bfc4c86d70f","arxiv_id":"2505.14577","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TRATES generates rubric-based trait-specific questions with an LLM, extracts high/medium/low answers as features, and trains a regression model that outperforms prior cross-prompt trait scorers.","lead":"This paper introduces TRATES, a system that uses an LLM to turn essay grading rubrics into specific questions, answers those questions for each essay, and feeds the answers into a small regression model to predict trait scores on unseen prompts. The authors report it beats previous state-of-the-art on all traits in the ASAP dataset and sets a first baseline on ELLIPSE.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'new SOTA across all traits' claim is supported only by cherry-picking the best LLM per trait from Table 1; no single TRATES variant outperforms all baselines on every trait.","rationale":"The reader identified the hand-chosen grade-level score scaling as the weakest assumption; I agree that this is a real limitation, but I see the multi-LLM per-trait selection as more load-bearing for the exact SOTA claim. The scaling is openly discussed by the authors and could be applied to baselines in a controlled rerun, whereas the test-set LLM selection is a property of the reported evidence itself: Table 1 shows no single TRATES row beating all baselines on all traits. A single fixed LLM (Starling) achieves SOTA on 6/8 traits, tying ORG and losing WC, so the abstract's 'across all traits' wording overstates what the experiments demonstrate. The framework's core contribution—using LLM-generated rubric-based questions as features for a simple regression model—is still plausible and the released artifacts (code and features) support further study. The appropriate verdict remains conditional: the paper should report a single-LLM result or a principled validation-based LLM selection rule, and soften the 'all traits' claim accordingly. This is why the reader's CONDITIONAL verdict is unchanged, but the specific condition should include fixing the LLM-selection ambiguity.","tokens_in":16760,"tokens_out":11094,"duration_ms":95381,"concrete_test":"Re-run the ASAP cross-prompt evaluation using one fixed LLM per trait selected only by inner validation on the source prompts (e.g., leave-one-prompt-out within the training folds), then report whether that single configuration outperforms ProTACT and Li & Ng on all traits; if not, the 'across all traits' SOTA claim should be revised to reflect the traits for which a single model achieves SOTA.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that TRATES 'establishes a new state-of-the-art performance across all traits' (Abstract, Section 6.2) is not supported by any single instantiation of the framework. Table 1 reports three variants (rows i-k: Starling, Llama, Gemma). The bolded best cells are taken across these rows: ORG best is Gemma (0.547), WC best is Gemma (0.622), while PA/NAR/LNG/CNV/CNT best is Starling. No row beats both ProTACT (row a) and Li & Ng (row b) on all eight traits. Starling loses WC (0.593 vs 0.599); Gemma loses PA (0.599 vs 0.619), NAR (0.600 vs 0.639), and LNG (0.521 vs 0.596); Llama loses on six traits. Thus 'across all traits' is a test-set union of per-trait best LLMs. The paper does not describe a validation-based rule for choosing the LLM on an unseen prompt, so this is not a single deployable model. This issue is independent of the score-scaling caveat the authors acknowledge in Section 8; even granting that scaling, the headline claim fails for a fixed LLM.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TRATES, a cross-prompt trait-scoring framework that uses an LLM to convert trait rubrics into sub-trait questions, answers those questions as High/Medium/Low, concatenates these trait-specific features with prompt-specific and generic writing-quality features, and trains a shallow neural network regressor per trait. Experiments on ASAP/ASAP++ report QWK across eight traits for three 7–9B LLMs, comparing against the ProTACT and Li & Ng baselines, with additional experiments on ELLIPSE and an ablation study of feature categories. The paper claims that TRATES establishes a new state of the art across all traits on ASAP and releases the generated trait-specific features.","tokens_in":17013,"tokens_out":4308,"duration_ms":40752,"significance":"If the claims are supported, the paper makes a useful contribution: it repurposes LLMs from direct graders to interpretable rubric-derived feature extractors, keeps the scoring model simple, and provides sub-trait questions that could serve as feedback. The public release of the generated features, the two-dataset evaluation, and the ablation study are explicit strengths that aid reproducibility. However, the headline claim of new state-of-the-art performance across all traits is not currently substantiated by a single instantiation of the framework, and the comparison with prior work mixes evaluation protocols. These issues are load-bearing for the central claim but appear fixable with a re-framed claim or additional experiments.","major_comments":[{"comment":"The claim that TRATES 'establishes a new state-of-the-art performance across all traits' is supported only by a union of the best per-trait result across the three LLM rows (i, j, k). No single TRATES row outperforms both ProTACT and Li & Ng on all eight traits: Starling loses on WC (0.593 vs. 0.599), Gemma loses on PA (0.599 vs. 0.619), NAR (0.600 vs. 0.639), and LNG (0.521 vs. 0.596), and Llama loses on six traits. Because the paper does not specify a validation-based rule for choosing the LLM per trait on an unseen prompt, the reported per-trait best is not a single deployable model. The abstract and Section 6.2 should be revised to describe the result as a per-trait best over LLM choices, or the authors should provide a principled model-selection procedure and report the resulting single-system performance.","section":"Abstract and Section 6.2, Table 1"},{"comment":"The grade-level score scaling is load-bearing for training a single cross-prompt model, but the assigned ranges [0,4] for grade 7, [0,5] for grade 8, and [0,6] for grade 10 are acknowledged in Section 8 to be 'intuitively determined after a thorough examination of the various rubrics.' This hand-chosen transformation changes the regression targets for different prompts, so the reported cross-prompt gains may be sensitive to the chosen ranges. The authors should provide a sensitivity analysis, for example comparing the grade-adjusted scaling to plain min-max scaling or other monotonic mappings, and show that the main conclusions are stable across reasonable choices.","section":"Section 4.6, Section 8, and Appendix I"},{"comment":"The comparison with ProTACT and Li & Ng is not apples-to-apples: those baseline scores are quoted from their original papers, while TRATES introduces a different feature-normalization protocol (train-set minimum and maximum values instead of per-prompt normalization) and the grade-level score-scaling described above. Since both protocol changes can affect QWK independently of the framework, the 'new SOTA' claim should be supported by re-running the baselines under the same normalization and scaling protocol, or by quantifying the effect of those protocol differences on the baseline scores.","section":"Section 5, Table 1"}],"minor_comments":[{"comment":"The caption says bold and underlined values indicate the most and second-most important categories, but no underlining is visible in the table as typeset.","section":"Table 2"},{"comment":"The hyperparameter numbering is inconsistent: 'Hidden layers' and 'Neurons per layer' are both numbered 3, so the list should be renumbered.","section":"Table 8"},{"comment":"The 'Arena Elo benchmark' is cited only by a Hugging Face URL; a formal citation with an access date would be more appropriate.","section":"Section 5, LLMs Selection"},{"comment":"The phrase 'by an average of 9 points' could be misread as 9 QWK units; consider writing '0.09 in QWK' or '9 points in the QWK scale used here.'","section":"Section 6.1"},{"comment":"The description of score-scaling should state explicitly how continuous predictions are mapped back to the original integer scales before QWK is computed, since rounding or clipping choices can affect the reported kappa values.","section":"Appendix I"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and the empirical material is real, but the headline SOTA claim overreaches Table 1, and the comparison protocol needs to be made fair or explicitly qualified. The authors should also address the sensitivity of the grade-level scaling, since it is the least principled component of an otherwise appealing pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely nice: instead of asking an LLM to score an essay directly, TRATES uses the rubric to generate a set of trait-specific assessment questions, has the LLM answer them as high/medium/low, and then feeds those answers, alongside generic and prompt-specific features, into a simple regression model. That reconceptualization gives you interpretable sub-traits and a natural route to cross-prompt generalization. The paper does that clearly, ships the code and features, runs a sensible ablation, and validates on a second dataset. For those reasons alone, this is not a desk-reject paper.\n\nWhat the paper does less well is support its central claim. The \"new state-of-the-art across all traits\" in the abstract and Section 6.2 is a union of per-trait best LLMs: in Table 1, Gemma wins ORG, WC, SF, CNV, and CNT, while Starling wins PA, NAR, and LNG. No single row beats both ProTACT and Li & Ng on every trait. So the SOTA claim describes a test-set oracle, not a deployable model. The authors never explain how a practitioner would choose the LLM for an unseen prompt. That is a substantive weakness, not a quibble.\n\nThe other soft spots are real but more contained. The score-scaling—reducing the max score by one point per grade level below the top—is a hand-picked transformation applied before training. The authors admit in the Limitations that it is debatable, and they are right to be cautious; it changes the regression targets and therefore every number in the paper. The baseline numbers for ASAP are quoted from previous papers, not re-run under TRATES's normalization and scaling protocol, so the comparison is not fully controlled. There are also no error bars or significance tests; QWK differences of one or two points could easily vanish with different random seeds.\n\nNone of this invalidates the central mechanism. The ablation study shows the LLM-generated features are the most important category on most traits, and the ELLIPSE results, while lower, still beat the available baselines. The paper would be strengthened by fixing a single LLM per trait using a validation-based rule, re-running the baselines under identical conditions, and reporting variance. As it stands, the contribution is a promising framework plus a set of reproducible features, not a clean SOTA result.\n\nFor a reading group, this is a good paper to argue about. I would cite it for the feature-generation trick. And I would send it to reviewers, because the idea is solid and all the major problems are fixable without changing the approach.","headline":"The core hybrid idea is clever and worth engaging, but the \"SOTA across all traits\" claim only holds if you let the authors pick a different LLM for each trait.","tokens_in":17534,"tokens_out":1868,"would_cite":true,"duration_ms":20608,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a hybrid pipeline—an LLM that turns grading rubrics into trait-specific assessment questions, plus a simple regression model—sets a new state of the art for cross-prompt essay trait scoring on the ASAP dataset.","keywords":["automated essay scoring","trait scoring","cross-prompt scoring","rubric-based features","large language models","LLM-based feature extraction","ASAP dataset","ELLIPSE dataset"],"falsifier":"Re-run the ASAP cross-prompt experiments with an alternative, explicit score-alignment rule (e.g., plain per-prompt min-max scaling or a mapping derived directly from the rubrics' descriptors) while keeping all TRATES components identical; if the average QWK then fails to exceed the ProTACT baseline on all traits, the new-SOTA claim is not robust to the chosen grade scaling.","tokens_in":1415,"feed_emoji":"📝","tokens_out":1408,"duration_ms":48786,"temperature":0.7,"pith_summary":"The paper argues that trait-based Automated Essay Scoring can be made both more accurate and more interpretable by reframing the LLM's role: instead of asking the LLM to score an essay directly, TRATES asks it to convert each trait's grading rubric into a set of assessment questions, then to answer each question for a given essay as high, medium, or low. These LLM-generated answers are combined with generic writing-quality features and prompt-specific metadata, and a simple regression model is trained to predict trait scores for essays from prompts the model has never seen. The authors report new state-of-the-art performance on all eight ASAP traits, and a first cross-prompt baseline on the ELLIPSE dataset.","feed_headline":"Rubric-to-question pipeline beats prior essay trait scores","feed_subtitle":"LLM-generated rubric questions feed a simple regression model, beating neural and feature baselines on every ASAP trait.","key_machinery":"The two-stage LLM feature chain: (1) rubric-to-question generation—an LLM is prompted to reformulate a trait rubric into a small set of short assessment questions, each targeting one scoring criterion; (2) question-to-answer extraction—the same LLM answers each question for an essay as high, medium, or low. These 3-valued answers are concatenated with a manually designed set of generic writing-quality features (length, readability, text variation, complexity, sentiment) and prompt-specific features (essay type, expected length, source length, grade level), and fed to a shallow neural-network regression model. A grade-level score scaling step maps different prompts' raw scores into a common [0–6] range before training, and predictions are mapped back for evaluation.","core_discovery":"On the ASAP dataset, TRATES improves average Quadratic Weighted Kappa over the strongest published baselines for cross-prompt trait scoring (ProTACT and Li & Ng), reaching new best scores on all eight traits when paired with the Starling or Gemma LLM. The ablation study shows that removing the trait-specific LLM features causes the largest drop in performance on most traits, indicating that these rubric-grounded features are the most informative of the three feature categories. The framework's success positions LLM-generated trait sub-questions as a practical feature source for cross-prompt scoring, rather than relying on LLMs as direct graders.","pith_inferences":["The grade-level score-scaling is hand-set; replacing it with a learned or data-driven mapping might change the cross-prompt results, and testing this would clarify how much of the SOTA gain depends on that choice.","The generated assessment questions themselves could be used as teaching feedback; the paper did not evaluate their pedagogical value, but if they are rubric-faithful, they offer a direct explanation of a trait score.","Because the framework only needs a rubric and an LLM, it could be adapted to holistic scoring or to non-ASAP rubrics, though holistic rubrics are more prompt-specific."],"forward_implications":["If correct, direct LLM prompting for essay scores is not the decisive factor; what matters is the feature representation derived from rubrics.","A simple regression model on rich features can match or beat complex neural trait-scoring models, shifting the field's emphasis toward feature design and rubric grounding.","The released trait-specific questions enable automated, interpretable sub-trait feedback for students and teachers.","The approach transfers to new datasets (ELLIPSE) and to new traits given a rubric, making it a generic pipeline for trait scoring.","Because the LLM features are the most significant category, further improving question generation—with larger LLMs or better prompting—should directly improve scoring."],"supporting_citations":[{"why":"Provides the ProTACT baseline, the strongest published cross-prompt trait scorer that TRATES claims to outperform on all ASAP traits.","marker":"(Do et al., 2023)"},{"why":"Supplies the feature-based neural baseline (Li & Ng) whose feature sets and results are compared against TRATES.","marker":"(Li and Ng, 2024b)"},{"why":"Introduces ASAP++, which extends the ASAP dataset with trait annotations for prompts P1–P6, enabling the cross-prompt trait-scoring evaluation.","marker":"(Mathias and Bhattacharyya, 2018)"},{"why":"Defines the cross-prompt trait-scoring task and provides the generic writing-quality feature set and code that TRATES reuses.","marker":"(Ridley et al., 2021)"},{"why":"Provides the ELLIPSE dataset used to test TRATES's generalizability to English-learner essays across 44 prompts.","marker":"(Crossley et al., 2023)"},{"why":"Defines the Quadratic Weighted Kappa (QWK) metric, the primary evaluation measure throughout the experiments.","marker":"(Cohen, 1968)"},{"why":"Provides evidence that direct LLM essay scoring falls short of baselines, motivating TRATES's feature-based use of LLMs.","marker":"(Mansour et al., 2024)"}],"fun_headline_variants":["LLM rubric questions power SOTA essay trait scoring","Trait-specific rubric questions boost essay scoring","Rubric-based LLM features top essay trait baselines","LLM turns rubrics into trait questions for better essay scoring"],"cache_read_input_tokens":19712,"weakest_assumption_plain":"The paper assumes that its hand-chosen grade-level score scaling—shaving one point off the maximum score for each grade below the top—makes scores from different ASAP prompts directly comparable; if that mapping is wrong, the cross-prompt training targets become misaligned and the reported gains could rest on an arbitrary transformation.","fun_headline_variants_meta":{"raw":{"variants":["LLM rubric questions power SOTA essay trait scoring","Trait-specific rubric questions boost essay scoring","Rubric-based LLM features top essay trait baselines","LLM turns rubrics into trait questions for better essay scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000491,"raw_usage":{"total_tokens":2346,"prompt_tokens":807,"completion_tokens":1539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":423,"completion_tokens_details":{"reasoning_tokens":1475}},"tokens_in":423,"tokens_out":1539,"duration_ms":11028,"temperature":1.0,"reasoning_tokens":1475,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:31:40.851250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ASAP cross-prompt experiments with an alternative, explicit score-alignment rule (e.g., plain per-prompt min-max scaling or a mapping derived directly from the rubrics' descriptors) while keeping all TRATES components identical; if the average QWK then fails to exceed the ProTACT baseline on all traits, the new-SOTA claim is not robust to the chosen grade scaling.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces ASAP++, which extends the ASAP dataset with trait annotations for prompts P1–P6, enabling the cross-prompt trait-scoring evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ELLIPSE dataset used to test TRATES's generalizability to English-learner essays across 44 prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Quadratic Weighted Kappa (QWK) metric, the primary evaluation measure throughout the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that direct LLM essay scoring falls short of baselines, motivating TRATES's feature-based use of LLMs."}],"review_version":1}