{"id":"06e3893d-d867-4227-9f16-4045731638fc","arxiv_id":"2508.12922","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RUM is a rule-plus-LLM grader for software testing work that matches human scores on most dimensions while cutting grading time and cost.","lead":"This paper introduces RUM, a system for automatically grading software testing assignments by combining fixed rules for objective checks with large language models for subjective quality. In a national student contest it reported 80.77% faster grading and 97.38% lower cost than manual assessment, with scores strongly correlated with human experts on most dimensions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 shows RUM's Adequacy dimension is nearly uncorrelated with human grading (QWK 0.168, Kendall 0.024) and worse than the rule-based baseline; the headline Total QWK 0.889 is driven by other dimensions, so the 'high accuracy' claim is not established for the claimed comprehensive assessment.","rationale":"The paper's engineering contribution is real and partially supported: the rule+LLM pipeline is clearly described, the efficiency and cost numbers (80.77% time reduction, 97.38% cost reduction) are concrete and plausible, and the stability analysis over 25 repeated LLM runs shows controlled variation. I therefore would not reject the paper. However, the strongest claim — that RUM maintains high accuracy and consistency, and does so comprehensively — is undercut by a result visible in the paper's own Table 2. The Adequacy dimension has QWK=0.168 and Kendall=0.024, effectively no agreement with human graders, and it is worse than the rule-based baseline on both QWK and MAE. The paper's narrative cherry-picks the dimensions where RUM improves over rule-based scoring and omits the dimension where it fails. Because the total score QWK can be inflated by strong code-total agreement and by objective dimensions with perfect rule-based agreement, the aggregate metric does not establish accuracy for the subjective dimensions that are the paper's stated novelty. The missing human-human inter-rater reliability is related but secondary: it would help interpret whether low Adequacy agreement is a RUM failure or a human benchmark problem, but either way the paper's current reporting does not support 'high accuracy' for Adequacy. I agree with the reader's conditional verdict, but my primary pressure point is the internally reported Adequacy correlation rather than the missing inter-rater baseline, so my agreement with the reader's weakest-assumption framing is partial.","tokens_in":12457,"tokens_out":6248,"duration_ms":67540,"concrete_test":"Use the underlying 148-submission data to recompute the Total-Score QWK after removing the Adequacy indicator, and report per-indicator QWK with confidence intervals. If the total QWK remains near 0.889, the headline accuracy is carried by other dimensions and the comprehensive claim is unsupported; if it drops materially, the high accuracy is an artifact of unequal weighting. Then have two independent expert raters grade a random 30-submission subsample with the same rubric and report human-human QWK; if it is below 0.889, the human benchmark cannot validate RUM's accuracy claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"To support the central claim that RUM maintains 'high accuracy and consistency' while being comprehensive, RUM must agree with human graders on the subjective dimensions it adds beyond META. Table 2 reports the opposite for the Adequacy indicator: RUM-vs-human QWK=0.168, Kendall=0.024, Spearman=0.074, while the rule-based baseline achieves QWK=0.399, MAE=1.85 versus RUM's MAE=3.45. The paper's RQ2 narrative highlights improvements on code basics and standardization but never mentions Adequacy. The overall QWK of 0.889 is an aggregate dominated by Code Total (QWK=0.947) and objective rule dimensions (Timestamp QWK=1.000), so it can remain high while one of the six assessed dimensions is effectively unmeasured. Since RUM's stated contribution is comprehensive subjective assessment, this is not a minor subscore issue: it is the one place where the hybrid method is worse than the rule-only method while claiming superiority. The absence of a human-human inter-rater baseline (Section 4.2) compounds the problem: without knowing how much humans agree with each other, a low correlation on Adequacy could be blamed on human unreliability, but the paper cannot use human scores as ground truth for the headline QWK while discounting them for Adequacy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RUM, a hybrid rule-based and LLM-based system for assessing software testing skills in large-scale educational settings. RUM preprocesses test case documents, scripts, and screenshots into structured units; constructs assessment criteria from original rubrics and requirement documents with LLM assistance and human review; and then runs a dual-engine assessment in which objective indicators are scored by rules and subjective indicators by LLMs. The evaluation, based on 148 students from a national software testing contest, compares RUM against human assessment and against rule-only baselines. The authors report QWK 0.889 for total scores versus human assessors, a 80.77% reduction in assessment time, a 97.38% reduction in per-submission cost, and a 14x increase in daily capacity. The paper also reports a stability analysis over repeated LLM runs and describes a real deployment in the 2024 contest.","tokens_in":12724,"tokens_out":4173,"duration_ms":38887,"significance":"If the reported accuracy and efficiency results hold, RUM would be a valuable contribution to software testing education: it addresses a real gap by attempting to automate subjective assessment of test case documents and test reports, and it has been deployed at scale. The cost and efficiency model (Table 4), the stability analysis (Table 3), and the use of a real contest dataset are strengths. The paper also clearly separates rule-based and LLM-based responsibilities, which is a sensible design. However, the central accuracy claim is undermined by the near-zero correlation on one of the six assessment dimensions (Adequacy, Table 2) and by the absence of human-human inter-rater reliability. These issues need to be resolved before the claim of 'maintaining high accuracy and consistency' is fully supported.","major_comments":[{"comment":"The Adequacy row of Table 2 shows RUM-vs-human QWK=0.168, Kendall=0.024, and Spearman=0.074, while the rule-based baseline achieves QWK=0.399 and MAE=1.85 versus RUM's MAE=3.45. The RQ2 narrative (Section 4.2) reports improvements on code basics (MAE 1.40 vs 5.70) and standardization (QWK 0.881 vs 0.240) but does not mention Adequacy. Because Adequacy is one of the six assessment dimensions central to the claimed 'comprehensive assessment,' and because RUM is worse than the rule-only baseline on this dimension, the abstract's claim of 'maintaining high accuracy and consistency' is not supported for the full assessment. Please either provide an explanation for this failure, report a corrected aggregate, or temper the accuracy claim.","section":"§4.2, Table 2"},{"comment":"No human-human inter-rater reliability is reported. Without a measure of how much human graders agree with each other, the QWK=0.889 against a single human score cannot be interpreted as 'high accuracy'; if human graders disagree substantially, the low Adequacy correlation could be attributed to human unreliability, but then human scores cannot serve as ground truth for the headline QWK either. Please report inter-rater reliability on a subset of submissions scored by multiple independent graders.","section":"§4.2"},{"comment":"RUM's subjective indicators are constructed from the same 'original assessment criteria' (the rubric) that the human graders presumably used, and the evaluation uses those human scores as ground truth. This makes the high total correlation partly a measure of rubric alignment rather than independent validation of testing skill. The paper should discuss this construct-validity limitation and, ideally, provide evidence that RUM scores relate to external outcomes (e.g., future performance or expert judgment using a different rubric).","section":"§3.2 and §4.2"}],"minor_comments":[{"comment":"Table 2 uses the row label 'Adequacy,' but Table 1 uses 'Sufficiency' (SUFF1, SUFF2). Please define the correspondence between these names.","section":"Table 1 vs Table 2"},{"comment":"The notation in Eq. (1) uses R_i both for the rule and for the result set; please use distinct symbols for the rule and the assessment result to avoid ambiguity.","section":"§3.3, Eq. (1)"},{"comment":"The RQ1 answer states that 'MAE remains consistently low ... with most metrics showing MAE values below 2.0,' but Table 2 reports MAE values of 3.45, 4.30, and 5.50 for Adequacy, Test Case Total, and Total Score, respectively. Please correct the statement to match the data.","section":"§4.2, RQ1 answer"},{"comment":"The paper does not state how many human graders were involved, their background, or the grading instructions; adding this information would improve reproducibility.","section":"§4.2"},{"comment":"The panel labels in Figure 3 are abbreviated; a caption defining each abbreviation would help readers map the figure to Table 2.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a deployed system with a real contest dataset, which is a strength. The main concern is that the evaluation's headline accuracy is driven by aggregate scores while one dimension (Adequacy) shows near-zero correlation and underperforms the rule-based baseline; this needs to be fixed. Also, the authors are the creators of META and organizers of the contest, which may be worth disclosing as a conflict of interest in the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: RUM is a practical hybrid rule+LLM grader for software testing contest submissions, and the efficiency numbers are believable. The accuracy claim is only partly supported because one of the six assessed dimensions (Adequacy) is essentially uncorrelated with human grading, and the paper does not report human-human agreement.\n\nWhat's actually new: RUM extends the authors' earlier META system to subjectively grade test case documents and test reports using LLMs, alongside rule-based grading of test scripts. The evaluation uses 148 real finalists from a national contest, with scores compared against human assessors. That is a genuine deployment, not a toy. The three-stage design (preprocessing, criteria construction, dual-engine assessment) is sensible, and the checklist-constrained prompting is a reasonable way to stabilize LLM scoring. The efficiency claims (80.77% time reduction, 97.38% cost reduction, ~352 submissions/day capacity) are internally consistent and plausible given the reported per-submission cost of $0.186.\n\nWhere it wobbles: Table 2 shows RUM vs human on Adequacy at QWK=0.168, Kendall=0.024, Spearman=0.074, with MAE=3.45; the rule-only baseline does better (QWK=0.399, MAE=1.85). The RQ2 narrative emphasizes basic and standardization gains but never mentions Adequacy. Since Adequacy is one of the dimensions RUM adds beyond META, this is not a minor subscore. The aggregate total-score QWK of 0.889 is dominated by objective dimensions (Timestamp=1.000, Code Total=0.947) and by Test Case Total at 0.662. So \"high accuracy and consistency\" holds for the objective core, not for the subjective comprehensiveness claim. The stability analysis (Table 3) is over 25 experiments on only five students, which is thin. There is no human-human inter-rater reliability baseline, so we cannot tell whether the Adequacy gap is RUM's fault or the human graders' inconsistency. The authors built META and run the contest platform, so a skeptical reader will want artifacts and maybe an independent evaluation; the external human scores reduce but don't eliminate that concern.\n\nBottom line: the engineering contribution is real, and the efficiency gains are worth reporting. The accuracy claim needs either a fixed Adequacy dimension or a narrowed claim, plus an inter-rater baseline. This deserves a serious referee; I would accept it for review and ask for revision rather than desk-reject.","headline":"RUM's hybrid grading is a genuine deployment with believable efficiency gains, but the headline accuracy claim is undercut by a near-zero correlation on the Adequacy dimension and no human-human baseline.","tokens_in":13263,"tokens_out":2195,"would_cite":false,"duration_ms":21556,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RUM combines rule-based and LLM-based assessment to grade subjective software-testing work products, reporting 80.77% faster and 97.38% cheaper assessment than manual grading, with a quadratic weighted kappa of 0.889 against human total…","keywords":["Software testing education","Testing skill assessment","LLM-as-a-judge","Autograding","Rule-based assessment","Subjective assessment","Software testing contest"],"falsifier":"Have two independent teams of human graders score the same 148 contest submissions and compute their quadratic weighted kappa on total scores. If the human-human QWK is close to or below RUM's 0.889, then RUM's agreement with a single expert set does not establish that it measures testing skill more reliably than human graders do, and the accuracy claim would need to be restated as agreement with one rubric.","tokens_in":12251,"feed_emoji":"🧪","tokens_out":8435,"duration_ms":80076,"temperature":0.7,"pith_summary":"The paper proposes RUM, a hybrid assessment system that aims to close a gap in software testing education: objective aspects of test scripts can already be graded automatically, but subjective aspects such as test case design and test report quality still require slow, inconsistent human grading. RUM routes objective checks through deterministic rules and subjective judgment through large language models, and reports that in a national contest setting with 148 students it reaches a quadratic weighted kappa (an inter-rater agreement statistic) of 0.889 against human total scores. It also reports cutting average assessment time from 21:17 to 4:05 per submission (80.77%), reducing per-submission cost from $7.10 to $0.186 (97.38%), and raising daily capacity from 22.56 to 351.84 submissions. If these numbers hold, RUM would make comprehensive testing-skill assessment practical at the scale of large courses and national contests.","feed_headline":"Rule+LLM grader scores test skills at 3% of cost","feed_subtitle":"RUM combines rules with LLM judgment to scale subjective test-case and report grading in education.","key_machinery":"The central mechanism is the dual-engine assessment pipeline, which has three stages. Preprocessing parses heterogeneous student submissions (test case documents in Excel/Word/PDF, test script code, and screenshots) into unified structured units through document parsing, AST-plus-regex code analysis, and OCR. Criteria construction uses LLMs plus human review to turn the original assessment criteria and test requirement document into rule-based assessment code, subjective assessment indicators, and a checklist. The assessment stage then runs a rule-based engine of the form $R_i(I,S,F)=E_{\\text{Rule}}(C,R_i)$—each rule maps structured content $C$ to an indicator $I$, score $S$, and feedback $F$—and an LLM-based engine that scores subjective indicators using a four-part prompt (Input, Rules, Criteria, Output) whose checklist constraints are intended to keep LLM scores stable. This division of labor is what lets RUM assess both objective and subjective aspects in one pass.","core_discovery":"On the paper's own terms, the central discovery is that the objective/subjective boundary in testing-skill assessment is not a hard one: subjective indicators such as coverage, readability, and consistency can be scored by LLMs that are constrained by a checklist, while objective indicators such as naming, field completeness, and timestamp format can be compiled into executable rule code. The paper reports that this dual-engine design agrees with human assessors better than a pure rule-based baseline on almost every dimension—for example, test-case standardization agreement rises from a QWK of 0.240 to 0.881, and code-basics mean absolute error falls from 5.70 to 1.40. It further reports that the rule constraints stabilize the LLM: purely rule-governed metrics show zero score range across repeated runs, and total scores vary by an average of only 4.8 points. The result, as the authors state it, is that RUM maintains high accuracy and consistency while making subjective assessment efficient and scalable.","pith_inferences":["The paper validates RUM against one set of expert scores and reports no human-human inter-rater reliability; a fair reading is that RUM agrees with that particular grading set, and its accuracy relative to human disagreement remains untested.","Because the same assessment rubric guides both the human graders and the construction of RUM's prompts, the reported agreement may partly measure rubric alignment; an out-of-sample test with independently derived scoring criteria would separate these.","The cost and efficiency figures assume a specific LLM API and labor rate; with open-source models or different token pricing the 97.38% cost saving would shrink, though the time saving from parallelization would likely persist.","The same dual-engine pattern—deterministic checks plus checklist-constrained LLM judgment—could be applied to grading other professional artifacts such as design documents, code reviews, or bug reports, provided the criteria construction stage is repeated for each new domain."],"forward_implications":["Subjective testing-skill assessment becomes scalable: daily throughput rises from 22.56 to 351.84 submissions at $0.186 per submission, enabling national contests and large courses to grade test-case documents and reports automatically.","RUM extends META's objective-only, script-focused assessment to cover test-case design quality and test-report analysis, giving teachers per-indicator scores and feedback rather than a single grade.","The reported stability ranges (zero range for rule-governed metrics, average total-score range of 4.8) imply that repeated runs of the system produce nearly identical scores, which matters for contest fairness.","Compared with pure rule-based assessment, the addition of LLM judgment improves agreement with human experts on subjective dimensions, such as test-case standardization (QWK 0.240 to 0.881)."],"supporting_citations":[{"why":"Defines the META seven-dimensional objective assessment framework that RUM extends, providing the baseline and the contest infrastructure.","marker":"[40]"},{"why":"Supplies the national contest context and the large-scale assessment scenario in which RUM was applied.","marker":"[38]"},{"why":"Provides the rule-based competency assessment approach that RUM's rule engine builds on and compares against.","marker":"[37]"},{"why":"Demonstrates LLM-based automated assessment in e-learning, motivating RUM's use of LLMs for subjective grading.","marker":"[3]"},{"why":"Shows that LLM-based automatic short answer grading is feasible in medical education, supporting the LLM-as-judge component.","marker":"[16]"},{"why":"Evaluates an LLM for automatic short answer grading and notes the need for further research, a direct reference for RUM's LLM-judgment design.","marker":"[7]"}],"fun_headline_variants":["Rule+LLM grading cuts test assessment cost by 97%","LLM with rule checks scores test skills at 3% cost","RUM: Rules and LLM slash test grading cost 97%","Hybrid rule-LLM assessor: 80% faster, 97% cheaper","LLM-checked test grading: accurate at 3% of cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human expert scores used as ground truth are a stable, reliable measure of testing skill; the paper reports no human-human inter-rater reliability, and because the same rubric shapes both the human grading and RUM's prompts, high agreement could reflect rubric alignment rather than independent measurement of skill.","fun_headline_variants_meta":{"raw":{"variants":["Rule+LLM grading cuts test assessment cost by 97%","LLM with rule checks scores test skills at 3% cost","RUM: Rules and LLM slash test grading cost 97%","Hybrid rule-LLM assessor: 80% faster, 97% cheaper","LLM-checked test grading: accurate at 3% of cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000821,"raw_usage":{"total_tokens":3605,"prompt_tokens":971,"completion_tokens":2634,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2536}},"tokens_in":587,"tokens_out":2634,"duration_ms":17601,"temperature":1.0,"reasoning_tokens":2536,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:16:57.126891+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent teams of human graders score the same 148 contest submissions and compute their quadratic weighted kappa on total scores. If the human-human QWK is close to or below RUM's 0.889, then RUM's agreement with a single expert set does not establish that it measures testing skill more reliably than human graders do, and the accuracy claim would need to be restated as agreement with one rubric.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the META seven-dimensional objective assessment framework that RUM extends, providing the baseline and the contest infrastructure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the national contest context and the large-scale assessment scenario in which RUM was applied."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the rule-based competency assessment approach that RUM's rule engine builds on and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates LLM-based automated assessment in e-learning, motivating RUM's use of LLMs for subjective grading."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that LLM-based automatic short answer grading is feasible in medical education, supporting the LLM-as-judge component."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Evaluates an LLM for automatic short answer grading and notes the need for further research, a direct reference for RUM's LLM-judgment design."}],"review_version":1}