Pith. sign in

REVIEW 3 major objections 5 minor 41 references

RUM: Rule+LLM-Based Comprehensive Assessment on Testing Skills

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RUM combines rule-based and LLM-based assessment to grade subjective software-testing work products, reporting 80.77% faster and 97.38% cheaper assessment than manual grading, with a quadratic weighted kappa of 0.889 against human total…

desk verdict RUM's hybrid grading is a genuine deployment with believable efficiency gains, but the headline accuracy claim is undercut by a near-zero correlation on the Adequacy dimension and no human-human baseline. read the letter →

arxiv 2508.12922 v1 pith:V7SAAOFW submitted 2025-08-18 cs.SE

Yue Wang , Zhenyu Chen , Yuan Zhao , Chunrong Fang , Ziyuan Wang , Song Huang This is my paper · ORCID
classification cs.SE
keywords SoftwaretestingeducationskillassessmentLLM-as-a-judgeAutogradingRule-basedSubjectivecontest
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes RUM, a hybrid assessment system that aims to close a gap in software testing education: objective aspects of test scripts can already be graded automatically, but subjective aspects such as test case design and test report quality still require slow, inconsistent human grading. RUM routes objective checks through deterministic rules and subjective judgment through large language models, and reports that in a national contest setting with 148 students it reaches a quadratic weighted kappa (an inter-rater agreement statistic) of 0.889 against human total scores. It also reports cutting average assessment time from 21:17 to 4:05 per submission (80.77%), reducing per-submission cost from $7.10 to $0.186 (97.38%), and raising daily capacity from 22.56 to 351.84 submissions. If these numbers hold, RUM would make comprehensive testing-skill assessment practical at the scale of large courses and national contests.

What carries the argument

The central mechanism is the dual-engine assessment pipeline, which has three stages. Preprocessing parses heterogeneous student submissions (test case documents in Excel/Word/PDF, test script code, and screenshots) into unified structured units through document parsing, AST-plus-regex code analysis, and OCR. Criteria construction uses LLMs plus human review to turn the original assessment criteria and test requirement document into rule-based assessment code, subjective assessment indicators, and a checklist. The assessment stage then runs a rule-based engine of the form $R_i(I,S,F)=E_{\text{Rule}}(C,R_i)$—each rule maps structured content $C$ to an indicator $I$, score $S$, and feedback $F$—and an LLM-based engine that scores subjective indicators using a four-part prompt (Input, Rules, Criteria, Output) whose checklist constraints are intended to keep LLM scores stable. This division of labor is what lets RUM assess both objective and subjective aspects in one pass.

What would settle it

Have two independent teams of human graders score the same 148 contest submissions and compute their quadratic weighted kappa on total scores. If the human-human QWK is close to or below RUM's 0.889, then RUM's agreement with a single expert set does not establish that it measures testing skill more reliably than human graders do, and the accuracy claim would need to be restated as agreement with one rubric.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the objective/subjective boundary in testing-skill assessment is not a hard one: subjective indicators such as coverage, readability, and consistency can be scored by LLMs that are constrained by a checklist, while objective indicators such as naming, field completeness, and timestamp format can be compiled into executable rule code. The paper reports that this dual-engine design agrees with human assessors better than a pure rule-based baseline on almost every dimension—for example, test-case standardization agreement rises from a QWK of 0.240 to 0.881, and code-basics mean absolute error falls from 5.70 to 1.40. It further reports that the rule constraints stabilize the LLM: purely rule-governed metrics show zero score range across repeated runs, and total scores vary by an average of only 4.8 points. The result, as the authors state it, is that RUM maintains high accuracy and consistency while making subjective assessment efficient and scalable.

Load-bearing premise

The load-bearing premise is that the human expert scores used as ground truth are a stable, reliable measure of testing skill; the paper reports no human-human inter-rater reliability, and because the same rubric shapes both the human grading and RUM's prompts, high agreement could reflect rubric alignment rather than independent measurement of skill.

Editorial extensions

If this is right

  • Subjective testing-skill assessment becomes scalable: daily throughput rises from 22.56 to 351.84 submissions at $0.186 per submission, enabling national contests and large courses to grade test-case documents and reports automatically.
  • RUM extends META's objective-only, script-focused assessment to cover test-case design quality and test-report analysis, giving teachers per-indicator scores and feedback rather than a single grade.
  • The reported stability ranges (zero range for rule-governed metrics, average total-score range of 4.8) imply that repeated runs of the system produce nearly identical scores, which matters for contest fairness.
  • Compared with pure rule-based assessment, the addition of LLM judgment improves agreement with human experts on subjective dimensions, such as test-case standardization (QWK 0.240 to 0.881).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper validates RUM against one set of expert scores and reports no human-human inter-rater reliability; a fair reading is that RUM agrees with that particular grading set, and its accuracy relative to human disagreement remains untested.
  • Because the same assessment rubric guides both the human graders and the construction of RUM's prompts, the reported agreement may partly measure rubric alignment; an out-of-sample test with independently derived scoring criteria would separate these.
  • The cost and efficiency figures assume a specific LLM API and labor rate; with open-source models or different token pricing the 97.38% cost saving would shrink, though the time saving from parallelization would likely persist.
  • The same dual-engine pattern—deterministic checks plus checklist-constrained LLM judgment—could be applied to grading other professional artifacts such as design documents, code reviews, or bug reports, provided the criteria construction stage is repeated for each new domain.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes RUM, a hybrid rule-based and LLM-based system for assessing software testing skills in large-scale educational settings. RUM preprocesses test case documents, scripts, and screenshots into structured units; constructs assessment criteria from original rubrics and requirement documents with LLM assistance and human review; and then runs a dual-engine assessment in which objective indicators are scored by rules and subjective indicators by LLMs. The evaluation, based on 148 students from a national software testing contest, compares RUM against human assessment and against rule-only baselines. The authors report QWK 0.889 for total scores versus human assessors, a 80.77% reduction in assessment time, a 97.38% reduction in per-submission cost, and a 14x increase in daily capacity. The paper also reports a stability analysis over repeated LLM runs and describes a real deployment in the 2024 contest.

Significance. If the reported accuracy and efficiency results hold, RUM would be a valuable contribution to software testing education: it addresses a real gap by attempting to automate subjective assessment of test case documents and test reports, and it has been deployed at scale. The cost and efficiency model (Table 4), the stability analysis (Table 3), and the use of a real contest dataset are strengths. The paper also clearly separates rule-based and LLM-based responsibilities, which is a sensible design. However, the central accuracy claim is undermined by the near-zero correlation on one of the six assessment dimensions (Adequacy, Table 2) and by the absence of human-human inter-rater reliability. These issues need to be resolved before the claim of 'maintaining high accuracy and consistency' is fully supported.

major comments (3)
  1. [§4.2, Table 2] The Adequacy row of Table 2 shows RUM-vs-human QWK=0.168, Kendall=0.024, and Spearman=0.074, while the rule-based baseline achieves QWK=0.399 and MAE=1.85 versus RUM's MAE=3.45. The RQ2 narrative (Section 4.2) reports improvements on code basics (MAE 1.40 vs 5.70) and standardization (QWK 0.881 vs 0.240) but does not mention Adequacy. Because Adequacy is one of the six assessment dimensions central to the claimed 'comprehensive assessment,' and because RUM is worse than the rule-only baseline on this dimension, the abstract's claim of 'maintaining high accuracy and consistency' is not supported for the full assessment. Please either provide an explanation for this failure, report a corrected aggregate, or temper the accuracy claim.
  2. [§4.2] No human-human inter-rater reliability is reported. Without a measure of how much human graders agree with each other, the QWK=0.889 against a single human score cannot be interpreted as 'high accuracy'; if human graders disagree substantially, the low Adequacy correlation could be attributed to human unreliability, but then human scores cannot serve as ground truth for the headline QWK either. Please report inter-rater reliability on a subset of submissions scored by multiple independent graders.
  3. [§3.2 and §4.2] RUM's subjective indicators are constructed from the same 'original assessment criteria' (the rubric) that the human graders presumably used, and the evaluation uses those human scores as ground truth. This makes the high total correlation partly a measure of rubric alignment rather than independent validation of testing skill. The paper should discuss this construct-validity limitation and, ideally, provide evidence that RUM scores relate to external outcomes (e.g., future performance or expert judgment using a different rubric).
minor comments (5)
  1. [Table 1 vs Table 2] Table 2 uses the row label 'Adequacy,' but Table 1 uses 'Sufficiency' (SUFF1, SUFF2). Please define the correspondence between these names.
  2. [§3.3, Eq. (1)] The notation in Eq. (1) uses R_i both for the rule and for the result set; please use distinct symbols for the rule and the assessment result to avoid ambiguity.
  3. [§4.2, RQ1 answer] The RQ1 answer states that 'MAE remains consistently low ... with most metrics showing MAE values below 2.0,' but Table 2 reports MAE values of 3.45, 4.30, and 5.50 for Adequacy, Test Case Total, and Total Score, respectively. Please correct the statement to match the data.
  4. [§4.2] The paper does not state how many human graders were involved, their background, or the grading instructions; adding this information would improve reproducibility.
  5. [Figure 3] The panel labels in Figure 3 are abbreviated; a caption defining each abbreviation would help readers map the figure to Table 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RUM's agreement with human assessors is an external benchmark, not an input to the system.

full rationale

The central validation of RUM is correlation with human assessment scores (Table 2, QWK 0.889). Those human scores are external data collected from 148 contest finalists and are not used to fit or tune RUM's rule code, prompts, or weights. The assessment criteria are constructed from the original assessment criteria and test requirement document (Section 3.2), the same documents human graders plausibly used, but sharing a rubric is a standard criterion-referenced design rather than circularity: RUM's outputs are not defined in terms of the human scores, and no equation in the paper reduces a predicted score to a fitted parameter. The self-citations (e.g., META [40]) establish background and motivation about the prior contest system and are not load-bearing for the agreement claim. The weak Adequacy correlation (QWK 0.168) and absence of human-human inter-rater reliability are validity and quality concerns about the scope of the headline claim, not instances of the derivation being equivalent to its inputs. I therefore find no specific reduction that would warrant a circularity score above 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central evaluation rests on domain assumptions about indicator validity, ground-truth reliability, LLM prompt stability, and preprocessing accuracy. These are reasonable starting points for an engineering paper but are not independently verified.

assumptions (4)
  • domain assumption The 17 assessment indicators in Table 1 derived from expert experience and original contest criteria adequately capture software testing skill.
    Section 3.2 states the indicators were constructed based on expert experience and original assessment criteria, with no independent validation that these indicators measure testing skill.
  • domain assumption Human expert scores are treated as ground truth for accuracy.
    Section 4.2 compares RUM to human assessment using QWK, Pearson, Spearman, and MAE without reporting human-human agreement or the number and expertise of human graders.
  • domain assumption LLM-based scoring with structured checklists and prompts produces stable and valid subjective scores.
    Section 3.3 relies on LLMs for subjective indicators; stability is checked with only 25 runs on 5 students in Table 3, and prompt details are not fully specified.
  • domain assumption OCR output from screenshots is treated as accurate text for assessment.
    Section 3.1 uses Alibaba Cloud OCR to create image text units without reporting OCR error rates on this data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RUM: Rule+LLM-Based Comprehensive Assessment on Testing Skills." pith.science (2026). https://pith.science/paper/V7SAAOFW

@misc{pith2026250812922,
  author       = {Pith},
  title        = {Pith review of: RUM: Rule+LLM-Based Comprehensive Assessment on Testing Skills},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V7SAAOFW}},
  note         = {Machine review of arXiv:2508.12922}
}
read the original abstract

Over the past eight years, the META method has served as a multidimensional testing skill assessment system in the National College Student Contest on Software Testing, successfully assessing over 100,000 students' testing skills. However, META is primarily limited to the objective assessment of test scripts, lacking the ability to automatically assess subjective aspects such as test case and test report. To address this limitation, this paper proposes RUM, a comprehensive assessment approach that combines rules and large language models (LLMs). RUM achieves a comprehensive assessment by rapidly processing objective indicators through rules while utilizing LLMs for in-depth subjective analysis of test case documents, test scripts, and test reports. The experimental results show that compared to traditional manual testing skill assessment, RUM improves assessment efficiency by 80.77\% and reduces costs by 97.38\%, while maintaining high accuracy and consistency of assessment. By applying RUM on the contest on software testing, we find that it not only enhances the efficiency and scalability of skill assessment in software testing education, but also provides teachers with more comprehensive and objective evidence for student ability assessment, facilitating personalized teaching and learning. This study offers new insights into the assessment of testing skills, which are expected to promote further development in test process optimization and software quality assurance.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 32 canonical work pages

  1. [1]

    Deniz Akdur. 2022. Analysis of software engineering skills gap in the industry. ACM Transactions on Computing Education 23, 1 (2022), 1–28

    work page 2022

  2. [2]

    Andrea Arcuri. 2020. Teaching Software Testing in an Algorithms and Data Structures Course. In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 419–424

    work page 2020

  3. [3]

    Nursultan Askarbekuly and Nenad Aničić. 2024. LLM examiner: automating assessment in informal self-directed e-learning using ChatGPT. Knowledge and Information Systems (2024), 1–18

    work page 2024

  4. [4]

    Rishabh Balse, Viraj Kumar, Prajish Prasad, and Jayakrishnan Madathil Warriem

  5. [5]

    Ingrid A Buckley and Winston S Buckley. 2017. Teaching software testing using data structures. International Journal of Advanced Computer Science and Applications 8, 4 (2017)

    work page 2017

  6. [6]

    Felix Cammaerts and Monique Snoeck. 2024. Assessing the testing skills transfer of model-based testing on testing skill acquisition.Software and Systems Modeling (2024), 1–19

    work page 2024

  7. [7]

    Li-Hsin Chang and Filip Ginter. 2024. Automatic Short Answer Grading for Finnish with ChatGPT. In Proceedings of the AAAI Conference on Artificial Intelli- gence, Vol. 38. 23173–23181

    work page 2024

  8. [8]

    Rodrigues, Ewerson Luiz de S

    Jean Felipe P Cheiran, Elder de M. Rodrigues, Ewerson Luiz de S. Carvalho, and João Pablo S da Silva. 2017. Problem-based learning to align theory and practice in software testing teaching. In Proceedings of the XXXI Brazilian Symposium on Software Engineering. 328–337

    work page 2017

Show all 41 references
  1. [9]

    Zhenyu Chen. 2023. Education Reform of Software Engineering in the Age of AI: Keynote Address. In 2023 IEEE/ACIS 23rd International Conference on Computer and Information Science (ICIS) . IEEE, 2–2

    2023

  2. [10]

    Zhenyu Chen and Bin Luo. 2014. Quasi-crowdsourcing testing for educational projects. InCompanion Proceedings of the 36th International Conference on Software FSE ’25, June 23–27, 2025, Trondheim, Norway Yue Wang, Zhenyu Chen, Yuan Zhao, Chunrong Fang, Ziyuan Wang, and Song Hua...

    2014

  3. [11]

    Marian Daun and Jennifer Brings. 2023. How ChatGPT will change software engineering education. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 . 110–116

    2023

  4. [12]

    Lin Deng, Josh Dehlinger, and Suranjan Chakraborty. 2020. Teaching software testing with free and open source software. In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 412– 418

    2020

  5. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  6. [14]

    Haiyue Feng, Sixuan Du, Gaoxia Zhu, Yan Zou, Poh Boon Phua, Yuhong Feng, Haoming Zhong, Zhiqi Shen, and Siyuan Liu. 2024. Leveraging Large Language Models for Automated Chinese Essay Scoring. In International Conference on Artificial Intelligence in Education. Springer, 454–467

    2024

  7. [15]

    Gordon Fraser, Alessio Gambi, and José Miguel Rojas. 2020. Teaching software testing with the code defenders testing game: Experiences and improvements. In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 461–464

    2020

  8. [16]

    Christian Grévisse. 2024. LLM-based automatic short answer grading in under- graduate medical education. BMC Medical Education 24, 1 (2024), 1060

    2024

  9. [17]

    Neil Heffernan, Rose Wang, Christopher MacLellan, Arto Hellas, Chenglu Li, Candace Walkington, Joshua Littenberg-Tobias, David Joyner, Steven Moore, Adish Singla, et al. 2024. Leveraging Large Language Models for Next-Generation Educational Technologies. In Proceedings of the ...

    2024

  10. [18]

    Enosha Hettiarachchi, M Antonia Huertas, and Enric Mor. 2015. E-assessment system for skill and knowledge assessment in computer engineering education. International Journal of Engineering Education 31, 2 (2015), 529–540

    2015

  11. [19]

    Christina Julia Kohlbacher, Michael Vierhauser, and Iris Groher. 2023. Common code quality issues of novice Java programmers: a comprehensive analysis of student assignments. In 15th International Conference on Computer Supported Education

    2023

  12. [20]

    Milan Kostic, Hans Friedrich Witschel, Knut Hinkelmann, and Maja Spahic- Bogdanovic. 2024. LLMs in Automated Essay Evaluation: A Case Study. In Proceedings of the AAAI Symposium Series , Vol. 3. 143–147

    2024

  13. [21]

    Kyle Lam, Junhong Chen, Zeyu Wang, Fahad M Iqbal, Ara Darzi, Benny Lo, Sanjay Purkayastha, and James M Kinross. 2022. Machine learning for technical skill assessment in surgery: a systematic review. NPJ digital medicine 5, 1 (2022), 24

    2022

  14. [22]

    Stephen MacNeil, Juho Leinonen, Paul Denny, Natalie Kiesler, Arto Hellas, James Prather, Brett A Becker, Michel Wermelinger, and Karen Reid. 2024. Discussing the Changing Landscape of Generative AI in Computing Education. InProceedings of the 55th ACM Technical Symposium on Co...

    2024

  15. [23]

    Pedram Mohammadi, Abbas Ebrahimi-Moghadam, and Shahram Shirani. 2014. Subjective and objective quality assessment of image: A survey. arXiv preprint arXiv:1406.7799 (2014)

    2014 arXiv

  16. [24]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Im- proving Language Understanding by Generative Pre-Training. arXiv preprint arXiv:1801.06146 (2018)

    2018 arXiv

  17. [25]

    Nishat Raihan, Mohammed Latif Siddiq, Joanna Santos, and Marcos Zampieri

  18. [26]

    Mohamad Sattar Rasul, Rose Amnah Abd Rauf, Azlin Norhaini Mansor, and AP Puvanasvaran. 2012. Employability Skills Assessment Tool Development. International Education Studies 5, 5 (2012), 43–56

    2012

  19. [27]

    Stefan Reichhart, Tudor Gîrba, and Stéphane Ducasse. 2007. Rule-based Assess- ment of Test Quality. J. Object Technol. 6, 9 (2007), 231–251

    2007

  20. [28]

    Paola Ricchiardi and Federica Emanuel. 2018. Soft skill assessment in higher education. Journal of Educational, Cultural and Psychological Studies (ECPS Journal) 18 (2018), 21–53

    2018

  21. [30]

    Yaqing Shi and Song Huang. 2019. Research on Software Testing Technical Ability Training based on e-learning. In Proceedings of the 2019 4th International Conference on Distance Education and Learning . 30–34

    2019

  22. [31]

    Yishen Song, Qianta Zhu, Huaibo Wang, and Qinhua Zheng. 2024. Automated Essay Scoring and Revising Based on Open-Source Large Language Models. IEEE Transactions on Learning Technologies (2024)

    2024

  23. [32]

    Weisong Sun, Xingya Wang, Haoran Wu, Ding Duan, Zesong Sun, and Zhenyu Chen. 2019. MAF: method-anchored test fragmentation for test code plagiarism detection. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training (I...

    2019

  24. [33]

    Xiaoyi Tang, Hongwei Chen, Daoyu Lin, and Kexin Li. 2024. Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments. Heliyon 10, 14 (2024)

    2024

  25. [34]

    Bastian Tenbergen. 2024. A Tool to Facilitate Calibrated Peer Reviews in Soft- ware Engineering Education. In 2024 36th International Conference on Software Engineering Education and Training (CSEE&T) . IEEE, 1–3

    2024

  26. [35]

    Jingbai Tian, Jianghao Yin, and Liang Xiao. 2022. Software requirements en- gineer’s ability assessment method based on empirical software engineering. Wireless Communications and Mobile Computing 2022, 1 (2022), 3617140

    2022

  27. [36]

    Dave Towey and Tsong Yueh Chen. 2015. Teaching software testing skills: Metamorphic testing as vehicle for creativity and effectiveness in software testing. In 2015 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE). IEEE, 161–162

    2015

  28. [37]

    Zahid Ullah, Adidah Lajis, Mona Jamjoom, Abdulrahman H Altalhi, Jalal Shah, and Farrukh Saleem. 2019. A rule-based method for cognitive competency assessment in computer programming using Bloom’s taxonomy. IEEE Access 7 (2019), 64663–64675

    2019

  29. [38]

    Xingya Wang, Weisong Sun, Linghuan Hu, Yuan Zhao, W Eric Wong, and Zhenyu Chen. 2019. Software-testing contests: Observations and lessons learned. Com- puter 52, 10 (2019), 61–69

    2019

  30. [39]

    Markus Wieser, Klaus Schöffmann, Daniela Stefanics, Andreas Bollin, and Stefan Pasterk. 2023. Investigating the Role of ChatGPT in Supporting Text-Based Programming Education for Students and Teachers. In International Conference on Informatics in Schools: Situation, Evolution...

    2023

  31. [40]

    Tianqi Zhou, Jiawei Liu, Yifan Wang, and Zhenyu Chen. 2022. META: multi- dimensional evaluation of testing ability. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings . 139– 143

    2022

  32. [2023]

    In Proceedings of the 16th Annual ACM India Compute Conference

    Evaluating the Quality of LLM-Generated Explanations for Logical Errors in CS1 Student Programs. In Proceedings of the 16th Annual ACM India Compute Conference. 49–54

  33. [2024]

    arXiv preprint arXiv:2410.16349 (2024)

    Large Language Models in Computer Science Education: A Systematic Literature Review. arXiv preprint arXiv:2410.16349 (2024)

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.