{"id":"aaa06f30-565e-43d8-8c95-d71feb4fc6ee","arxiv_id":"2505.23818","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RATAS decomposes rubrics into simplified rules, scores each rule with GPT-4o, and cascades scores to grade long textual exam answers with reported near-human accuracy.","lead":"This paper introduces RATAS, a system that turns grading rubrics into a knowledge tree and uses GPT-4o to score each small rule before combining results into a grade and explanation. The approach reports very low error on a new dataset of long student answers, but the dataset selection and evaluation choices leave open questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated LLM rubric decomposition is the load-bearing link; independent reconstruction test needed before MAE 0.0309 can be accepted.","rationale":"The reader's weakest_assumption correctly identifies the critical link: the cascade from leaf judgments to total scores is only as good as the rubric-to-SR decomposition. I considered whether the mismatch between Section 3's continuous SP/LQAP formulas and Algorithm 1's binary leaf scoring is a separate fatal issue; it is a real internal inconsistency, but it is best understood as a symptom of the same unvalidated decomposition, because binary leaves can realize partial credit only if the decomposition encodes it in the tree weights. The paper does not provide human agreement on ground-truth scores, an inter-rater reliability measure for the SR construction, an ablation with alternative decompositions, or a sensitivity analysis of leaf weights. These omissions leave the central accuracy claim plausible but unverified at its most load-bearing point. The proposed experiment—independent re-decomposition followed by re-scoring with the same leaf scorer—would settle whether the SRs genuinely preserve rubric semantics. If it passes, the claim gains substantial support; if it fails, the headline metrics are not interpretable as rubric-preserving accuracy. This does not change the reader's CONDITIONAL verdict; it makes the condition explicit: accept only after the decomposition is independently validated.","tokens_in":12474,"tokens_out":4455,"duration_ms":49113,"concrete_test":"Have two independent raters (or a second LLM, e.g., Claude or Gemini) decompose the same 34-criteria rubric into Simplified Rules with explicit influence weights, following Section 4.1. Then re-score all 417 answers by running the authors' leaf-level LLMSSR scorer (or an equivalent GPT-4o prompt) on each rater's tree and compute total scores. Compare the resulting MAE and R² against Table 3. If the independently decomposed trees yield MAE above 0.05 or R² below 0.95, or if RATAS's scores shift by more than 0.02 on average, the decomposition assumption is not stable enough to support the central claim. Additionally, verify on answers whose human score falls on an intermediate Level-of-Achievement that the binary leaf scores can produce that exact total; if not, Section 3's scoring model is not implemented by Algorithm 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline numbers (MAE=0.0309, RMSE=0.0443, R²=0.9627) are produced by cascading binary judgments on Simplified Rules (SRs) generated by GPT-4o. This cascade is trustworthy only if the rule-division process in Section 4.1 preserves rubric semantics: the SRs must jointly reconstruct each original Basic-Rule, be non-overlapping, and carry equal or correctly assigned influence. The paper asserts these three criteria but does not validate them with human annotations or an ablation. A related internal gap compounds this: the formal scoring model in Section 3 defines continuous SP and LQAP terms, while Algorithm 1 scores every leaf as 0 or 1 and propagates using fixed equal weights. If a rubric row's Level-of-Achievement contains intermediate percentages (e.g., la2 at 50%), binary leaves can express that partial credit only if the decomposition happens to split the row into appropriately weighted SRs; the paper does not demonstrate this mapping. Consequently, the reported accuracy could reflect favorable dataset selection or prompt behavior rather than semantic preservation of the rubric. The strongest claim should therefore be read as conditional on the decomposition being faithful, which is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RATAS (Rubric Automated Tree-based Answer Scoring), a framework that converts rubric rows into a tree of Simplified Rules (SRs), asks GPT-4o to judge each SR as met/unmet, and propagates binary leaf scores through the tree to produce final and partial scores with textual rationales. The authors formalize a rubric-based grading problem, describe the RKT construction and scoring pipeline, and evaluate on a self-constructed dataset of 417 selected responses from university project-based courses. They report strong accuracy for RATAS (MAE=0.0309, RMSE=0.0443, R²=0.9627) relative to direct GPT-4o (MAE=0.2355, RMSE=0.2923, R²=-0.6262), along with higher ICC. The central claim is that decomposing rubrics into simplified rules and scoring each rule with an LLM yields near-human, explainable grading on long real-world textual exams.","tokens_in":12652,"tokens_out":4479,"duration_ms":45909,"significance":"If the rubric-decomposition and the cascading scoring method are faithful to the original rubric semantics, RATAS would be a useful contribution to automated grading: it targets longer answers than most ASAG datasets, offers structured explainable feedback, and is subject-agnostic by construction. The authors also release code, which is a strength for reproducibility. However, the current evidence does not establish the central claim. The rule-division step is validated only by assertion, the formal scoring model is inconsistent with the implemented algorithm, and the evaluation is based on a self-selected dataset without confidence intervals or inter-rater reliability for the human ground truth. These gaps make the headline accuracy conditional on unverified design choices rather than a demonstrated property of the framework.","major_comments":[{"comment":"The load-bearing premise is that the LLMCTM rule-division process satisfies the three stated criteria (collective reconstruction, non-overlapping aspects, approximately equal scoring importance), but the paper provides no human annotation study, no reconstruction test, and no ablation in which the decomposition is replaced by a baseline. Because every leaf score is judged relative to the SRs generated by the same GPT-4o model, the reported MAE of 0.0309 in Table 3 is conditional on an unvalidated semantic-preservation assumption. Please report an independent verification where human raters confirm that the SRs reconstruct each Basic-Rule and that leaf influence weights reflect rubric intent, or at least an ablation with a different decomposition strategy.","section":"§4.1"},{"comment":"The formal model defines SP and LQAP as continuous values via arg max over [0,1], but Algorithm 1 scores every leaf node as 0 or 1 and propagates via N.nodeScore = Σ C.infl × C.nodeScore with fixed equal influence. The paper never shows how intermediate Level-of-Achievement percentages (e.g., 50% in Table 1) are realized by binary leaves, nor how the continuous SP/LQAP equations relate to the implementation. This is not merely a presentational mismatch: it prevents the reader from verifying that the claimed floating-point scores follow from the stated rubric logic. The formalism and the pseudocode need to be reconciled, or the paper must explicitly state that the SP/LQAP equations are a conceptual idealization and Algorithm 1 is the operative method.","section":"§3 and Algorithm 1"},{"comment":"The dataset consists of \"417 selected responses from approximately 1,500 answers\" with no description of the selection criteria. If the selection favors answers for which the rubric is unambiguous, the reported MAE underestimates deployment error. Moreover, no inter-rater reliability for the human ground-truth scores is reported; without it, the absolute scale of MAE=0.0309 cannot be interpreted, because human graders may themselves disagree by more than this amount. Please report the selection criteria, the number of raters, and the inter-rater agreement on the gold-standard scores.","section":"§6"},{"comment":"The headline metrics come from \"three runs for each approach,\" but no variance, confidence intervals, or significance tests are reported, and the subset rows for >600 words are based on an unreported number of responses. It is also unclear what the reported ICC values (0.9662 vs 0.5984) are computed between: RATAS vs human, GPT-4o vs human, or run-to-run reliability? Without this information, the reliability claim is not verifiable. Please report the exact ICC definition, per-subset sample sizes, and per-run or bootstrap intervals for the main metrics.","section":"§6 and Table 3"}],"minor_comments":[{"comment":"There are several typos and inconsistencies: \"nstructors\" in the abstract, \"RiceCharm\" for \"RiceChem\" in §6, \"RATASframework\" without a space in §4, and \"Tabl 1\" in §3. The acronym also alternates between RATG and RAGT; please choose one consistently.","section":"Abstract, §2, §4"},{"comment":"The displayed formula for AchievedScore lacks multiplication operators and is hard to parse; related_ls is also used before it is clearly defined. Please rewrite the formula with explicit operators and define all terms at first use.","section":"§3"},{"comment":"The definitions SP = arg max_{sp∈[0,1]} P(SP=sp|ri) and LQAP_{ij} = arg max over continuous values are formally ill-defined, since the probability of a specific continuous value is generally zero. If a modal or point estimate is intended, it should be stated explicitly.","section":"§3"},{"comment":"The text says \"Table 1 compares key criteria\" but the referenced table is Table 2; also, the prompts for LLMCTM, LLMCSC, and LLMSSR are not included in the paper, so the GitHub repository becomes essential for reproducibility. Please include the prompts or a detailed appendix, and clarify whether the evaluation dataset itself is released.","section":"§5.2 and Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from a clear statement on data availability, since the abstract promises code only while the dataset is central to the evaluation. The novelty relative to prior rubric-aware ASAG methods (e.g., Wang et al., 2019) should also be more sharply delineated. The core idea is promising, but the evaluation and the formal/algorithmic alignment need substantial work before the headline accuracy can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [name],\n\nThe one thing to know: the actual contribution is the rubric knowledge tree plus cascaded binary scoring of simplified rules, not the accuracy numbers. The MAE of 0.0309 is plausible but currently unbacked because the rubric decomposition is never validated.\n\nWhat's genuinely new: RATAS breaks rubric rows into simplified rules (SRs), scores each SR as 0/1 with GPT-4o, and propagates scores up the tree. That's a clean architecture, distinct from the usual 'grade the whole answer' prompt. The paper also contributes a new long-answer dataset (average 366 words, well beyond RiceChem's 120) and ships code. The structured reasoning output is a useful side effect. Compared to direct GPT-4o, the reported improvement is large, so the core idea deserves a serious look.\n\nWhere it goes soft:\n\n1. Dataset selection. The 417 responses were 'selected' from ~1,500 with no criteria. That is an open door to selection bias, and no amount of careful metric computation closes it.\n\n2. Ground truth. No inter-rater reliability for the human scores. If the humans are noisy, an MAE of 0.03 could be either impressive or accidentally on target.\n\n3. No uncertainty. Three runs, no confidence intervals. Fine for a pilot, not for 'outstanding accuracy.'\n\n4. The load-bearing step. The decomposition into SRs is asserted to be complete, non-overlapping, and equal-weight, but never checked. No human annotations of the SRs, no ablation, no reconstruction test. This is the real gap. The stress-test note about the binary leaf scores vs. the continuous SP/LQAP definitions in Section 3 is also accurate: Algorithm 1 uses equal weights and 0/1 leaves, and the paper never shows how a 50% level-of-achievement maps onto that.\n\n5. Baselines. Only direct GPT-4o. No comparison to fine-tuned BERT or other ASAG/ALAG methods, so the 'subject-agnostic and better than existing' claim is not established.\n\nThe math in Section 3 is mostly notational; the substance is the architecture. Minor typos throughout.\n\nBottom line: the paper is a solid proof-of-concept with a load-bearing validation hole. I'd send it to a serious referee, but the referee should be told to focus on decomposition fidelity and dataset selection. With those fixed, the paper could be a useful contribution to the educational technology community.\n\nMy take: worth engaging, but keep your skepticism about the headline numbers until the decomposition is independently tested.\n\nBest,\n[Name]","headline":"The rubric knowledge tree and cascaded SR scoring are a real architectural contribution, but the headline accuracy rests on a rubric decomposition the paper never validates.","tokens_in":13191,"tokens_out":3358,"would_cite":true,"duration_ms":32151,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Splitting a rubric into a tree of simple rules lets a large language model grade long exams with near-human accuracy.","keywords":["automated answer grading","rubric-based scoring","Rubric Knowledge Tree","large language models","GPT-4o","explainable feedback","long answer grading","generative AI"],"falsifier":"Grade a rubric that contains a conditional or capping rule (for example, 'if the formatting is wrong, the content score is capped at 50%') with RATAS; if the tree's additive weighted sum cannot reproduce the cap and the mean absolute error against human scores rises materially, the equal-weight, non-overlapping decomposition is where the method's validity stops.","tokens_in":12253,"feed_emoji":"🎓","tokens_out":10799,"duration_ms":90743,"temperature":0.7,"pith_summary":"This paper claims that automated grading of long, open-ended exam answers becomes accurate and explainable when the rubric is first rebuilt as a tree of tiny, independently checkable rules. The proposed framework, RATAS, has a large language model split every rubric row into simplified leaf rules, judge each leaf as met or unmet, and propagate those judgments up the tree using weights inherited from the rubric. On a new dataset of 417 university project answers (average 366 words, up to 1,285 words) graded with a 34-criterion rubric, RATAS reports a mean absolute error of 0.0309, RMSE of 0.0443, and $R^2$ of 0.9627 against human scores, while direct GPT-4o scores the same answers with MAE 0.2355 and a negative $R^2$. If correct, this would mean rubric decomposition alone—without fine-tuning or domain-specific training—can bring LLM grading close to human levels and make every mark traceable to specific rubric criteria.","feed_headline":"Breaking rubrics into a tree cuts grading error sevenfold","feed_subtitle":"A rubric tree lets GPT-4o grade 417 long exam answers with tiny error and per-criterion explanations.","key_machinery":"The load-bearing object is the Rubric Knowledge Tree (RKT), built by an iterative rule-division process that turns each rubric row into Simplified Rules (SRs). The division must satisfy three criteria: the simpler rules collectively reconstruct the original rule, cover non-overlapping aspects, and have approximately equal influence on scoring. Scores are leaf-level $0$ or $1$ judgments made by a language model, then cascaded upward through the tree using each node's influence_on_scoring and score_source weights, which yields both the final numeric score and a structured reason for every internal node. This is what turns a holistic, judgment-heavy grading problem into many small yes/no classification tasks.","core_discovery":"RATAS formalizes rubric-based grading as a scoring function over a Rubric Knowledge Tree (RKT), an object in which each row of an analytical rubric becomes a node and each node's rule is repeatedly decomposed until it reaches Simplified Rules (SRs) that cannot be simplified further. The decomposition is constrained to satisfy three criteria: the simpler rules must collectively reconstruct the original rule, cover distinct non-overlapping aspects, and carry approximately equal scoring importance. Scoring then reduces to asking a language model, for every SR, whether the answer fulfills it and which level of achievement is matched; the final score is obtained by cascading these $0$ or $1$ judgments up the tree, multiplying by each node's influence_on_scoring and score_source percentages. The paper reports that with GPT-4o handling all downstream NLP tasks through prompt engineering, RATAS achieves MAE 0.0309, RMSE 0.0443, $R^2$ 0.9627, and ICC 0.9662 on the 417-answer dataset, substantially outperforming direct GPT-4o while degrading less on answers longer than 600 words.","pith_inferences":["The decisive untested step is whether the automatic rule division truly preserves scoring semantics; validating the three decomposition criteria with human annotation across multiple rubrics would turn a single-dataset demonstration into a general recipe.","Because every leaf requires a separate language-model call, token cost grows with rubric breadth; one could test whether sampling leaves or routing easy leaves to a smaller model preserves accuracy at lower cost.","The architecture separates rubric structure from scoring judgments, so the same Rubric Knowledge Tree could be reused across thousands of answers and across courses sharing a rubric, enabling longitudinal consistency checks that direct grading does not offer.","A stronger comparison than direct GPT-4o would be a fine-tuned LLM or an LLM prompted with the full rubric plus chain-of-thought; if RATAS still wins, the tree decomposition itself, rather than better prompting, is the active ingredient."],"forward_implications":["If the reported accuracy holds, rubric-based grading of long textual exams can be carried out by decomposing any rubric into a tree and letting a general language model judge each leaf, with no fine-tuning on domain-specific data.","The framework yields a structured, node-level reason for every score, so students can see which rubric criteria were met, partially met, or missed, and instructors can audit each mark.","The method handles answers far longer than existing automated short-answer and long-answer datasets (average 366 words versus the previous 120 words), with performance declining only modestly for answers over 600 words.","Because RATAS relies on prompt engineering rather than retraining, adapting it to a new course or rubric requires only a new rubric table and incurs per-leaf API costs.","The flexible scoring logic lets instructors write fewer, broader rubric rows while the tree expansion restores fine-grained scoring behavior, so rubric design is simplified without losing score resolution."],"supporting_citations":[{"why":"Supplies the only existing long-answer grading dataset (average 120 words) and defines the previous state of the art that RATAS extends.","marker":"[8]"},{"why":"Introduces GPT-4o, the large language model used for every downstream scoring and reasoning task in RATAS.","marker":"[19]"},{"why":"Documents GPT-4's short-answer scoring performance and its variability across subjects, motivating the structured rubric-based approach.","marker":"[31]"},{"why":"Provides the comparison of GPT-4o with earlier GPT models that justifies choosing it as the framework's engine.","marker":"[45]"},{"why":"Supplies the chain-of-thought, few-shot, and instruction prompting techniques the framework combines to make its small downstream tasks reliable.","marker":"[49]"}],"fun_headline_variants":["RATAS rubric tree: 0.03 MAE grading 417 essays","GPT-4o on a rubric tree grades 417 answers with 0.03 MAE","Rubric-tree AI grading: 7x fewer errors on real exams","Tree-structured rubrics make AI grading explainable","RATAS: tree-based rubrics for transparent exam scoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework depends on the rule-division step preserving the original rubric's scoring semantics exactly: if a simplified rule misses part of the original requirement, overlaps with a sibling rule, or is given the wrong influence weight, the total score can be wrong even when every leaf judgment is individually correct.","fun_headline_variants_meta":{"raw":{"variants":["RATAS rubric tree: 0.03 MAE grading 417 essays","GPT-4o on a rubric tree grades 417 answers with 0.03 MAE","Rubric-tree AI grading: 7x fewer errors on real exams","Tree-structured rubrics make AI grading explainable","RATAS: tree-based rubrics for transparent exam scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001787,"raw_usage":{"total_tokens":7062,"prompt_tokens":982,"completion_tokens":6080,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":5994}},"tokens_in":598,"tokens_out":6080,"duration_ms":42682,"temperature":1.0,"reasoning_tokens":5994,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:22:03.176563+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Grade a rubric that contains a conditional or capping rule (for example, 'if the formatting is wrong, the content score is capped at 50%') with RATAS; if the tree's additive weighted sum cannot reproduce the cap and the mean absolute error against human scores rises materially, the equal-weight, non-overlapping decomposition is where the method's validity stops.","supporting_citations":[{"cited_title":"Automated long answer grading with ricechem dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the only existing long-answer grading dataset (average 120 words) and defines the previous state of the art that RATAS extends."},{"cited_title":"Gpt-4o: The cutting-edge advancement in multi- modal llm,","cited_arxiv_id":null,"evidence_quote":"Introduces GPT-4o, the large language model used for every downstream scoring and reasoning task in RATAS."},{"cited_title":"Short answer scoring with gpt-4,","cited_arxiv_id":null,"evidence_quote":"Documents GPT-4's short-answer scoring performance and its variability across subjects, motivating the structured rubric-based approach."},{"cited_title":"GPT-4o vs. GPT-4: How do they compare?","cited_arxiv_id":null,"evidence_quote":"Provides the comparison of GPT-4o with earlier GPT models that justifies choosing it as the framework's engine."}],"review_version":1}