The reviewed record of science sign in
Pith

arxiv: 2606.06301 · v1 · pith:MGV3ZCPY · submitted 2026-06-04 · cs.SE

More than a Judge: An Empirical Study of Agent-Human Interaction in Crowdsourced Testing Assessment

Yue Wang , Yuan Zhao , Shengcheng Yu , Zhenyu Chen , Qing Gu This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-28 00:08 UTCgrok-4.3pith:MGV3ZCPYrecord.jsonopen to challenge →

classification cs.SE
keywords crowdsourced testingagent feedbackLLM judgehuman-AI collaborationsoftware qualityreport assessmenttesting workflow
0
0 comments X

The pith

Agent-generated feedback from automated assessments improves how testers revise reports, perform on new tasks, and transfer practices in crowdsourced testing.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper examines whether feedback from a multi-agent LLM-based judge can enhance human performance in crowdsourced software testing. Through a controlled study with 20 testers on three applications, it tests effects on report revisions, subsequent task performance, and cross-application transfer. If true, this would mean assessment agents can reduce the review burden on developers by improving report quality at the source rather than only judging after submission. The study combines quantitative report analysis with participant questionnaires to track these effects.

Core claim

In a four-stage human-subject study, agent-generated feedback supported immediate improvements in revised reports, led to better first submissions on new tasks after prior exposure to feedback, and provided evidence of partial transfer of reporting practices to a later application. Questionnaire responses from 17 participants indicated the feedback was generally understandable and acted upon, though with some remaining issues in specificity.

What carries the argument

The multi-agent assessment backbone that generates actionable feedback along textuality, adequacy, and competitiveness dimensions for integration into the tester workflow.

If this is right

  • Immediate revisions to reports show measurable quality gains after receiving agent feedback.
  • First submissions on a subsequent task are stronger when testers have previously received such feedback.
  • Some reporting practices transfer to a different application in a later stage of the study.
  • Most participants found the feedback understandable and used it in their work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Scaling this feedback mechanism could lower the overall effort needed for developers to process crowdsourced reports.
  • Similar agent-human loops might be tested in other areas of software engineering involving report or code review.
  • Real-world deployment without study controls would help confirm if the benefits hold outside the lab setting.

Load-bearing premise

The improvements in tester performance are due to the agent feedback rather than other factors like the study structure or participant motivation.

What would settle it

Observing no difference in report quality between testers who receive agent feedback and those who do not, when the study is run in an uncontrolled real-world crowdsourced testing platform.

Figures

Figures reproduced from arXiv: 2606.06301 by Qing Gu, Shengcheng Yu, Yuan Zhao, Yue Wang, Zhenyu Chen.

Figure 1
Figure 1. Figure 1: Example of Crowdsourced Test Reports: The Good and The Bad. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The Multi-dimensional Assessment Agents and their workflow roles in this study. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Four-stage human-subject workflow for evaluating in-situ agent-generated feedback in crowdsourced [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Questionnaire evidence on external AI use during the four-stage tasks. Panel A summarizes the extent [PITH_FULL_IMAGE:figures/full_fig_p021_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Selected questionnaire items on feedback perception and use. The items are split into two panels for [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Semi-open questionnaire themes. The four panels summarize what respondents considered most [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
read the original abstract

Agentic AI is increasingly being integrated into software engineering workflows. In crowdsourced testing, however, the large volume and uneven quality of submitted reports still create a substantial review burden for developers. In prior work, we developed and validated a multi-agent assessment backbone based on the LLM-as-a-Judge paradigm. That backbone assesses reports along three dimensions--textuality, adequacy, and competitiveness--and was shown to align well with human consensus while substantially reducing assessment effort. Yet reliable automated judging does not by itself show whether agent outputs can improve human work when embedded into workflow. This paper studies that missing question in the context of crowdsourced testing. We investigate whether assessment-derived, actionable feedback can improve how testers revise reports, perform on later tasks, and transfer reporting practices across applications. To do so, we conducted a controlled four-stage human-subject study with 20 testers across three real-world applications. The results show that agent-generated feedback supports immediate improvements in revised reports, better first submissions on a new task after prior feedback exposure, and evidence of partial but meaningful transfer to a later application. A post-task questionnaire completed by 17 participants complements these artifact-based findings by suggesting that the feedback was generally understandable, acted upon in revision, and carried into later tasks, while also revealing remaining friction in specificity and execution. Overall, the study provides empirical evidence that, in the studied crowdsourced testing setting, assessment agents can serve not only as post-hoc judges but also as workflow-integrated feedback providers that support upstream report-quality improvement.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that agent-generated feedback from a multi-agent LLM-as-a-Judge assessment system (evaluating textuality, adequacy, and competitiveness) can improve human performance in crowdsourced testing workflows. A controlled four-stage human-subject study with 20 testers across three real-world applications shows immediate gains in revised reports, better first submissions on subsequent tasks after feedback exposure, partial transfer to a later application, and supporting questionnaire evidence from 17 participants that the feedback was understandable and actionable, while noting remaining friction in specificity.

Significance. If the results hold after addressing design concerns, the work meaningfully extends prior LLM-as-a-Judge validation by demonstrating workflow-integrated value: assessment agents can serve as feedback providers that improve upstream report quality rather than only reducing post-hoc review effort. The use of real applications, mixed artifact-based and questionnaire measures, and focus on transfer effects adds empirical grounding to agent-human collaboration claims in software engineering.

major comments (2)
  1. [Abstract and Methods (four-stage study description)] Abstract and Methods (four-stage study description): The within-subjects design with 20 testers across stages does not describe a no-feedback or sham-feedback control arm, randomization of feedback presentation order, or statistical modeling (e.g., mixed-effects regression) to isolate agent-feedback effects from repeated-task learning or motivation. This directly undermines the central causal claim that observed improvements in revisions, subsequent submissions, and transfer are attributable to the agent feedback rather than study artifacts or practice effects.
  2. [Results (n=20 and transfer claims)] Results (n=20 and transfer claims): With a modest sample size and no reported power analysis or effect-size quantification, the evidence for 'partial but meaningful transfer' to a later application rests on limited data; the paper should report specific metrics (e.g., report-quality deltas, statistical significance) and discuss generalizability limits to support the transfer finding.
minor comments (2)
  1. [Abstract] The abstract states that the feedback 'acted upon in revision' but provides no quantitative breakdown (e.g., percentage of suggestions incorporated or specific revision examples) to ground this claim.
  2. [Questionnaire results] Questionnaire results from 17 of 20 participants: the paper should report response rate, any demographic differences between responders and non-responders, and exact question wording to allow assessment of potential bias.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments on our four-stage human-subject study. We address each major point below, clarifying our design rationale while acknowledging limitations in causal isolation and quantitative reporting. We will incorporate revisions to strengthen the manuscript accordingly.

read point-by-point responses

  1. Referee: The within-subjects design with 20 testers across stages does not describe a no-feedback or sham-feedback control arm, randomization of feedback presentation order, or statistical modeling (e.g., mixed-effects regression) to isolate agent-feedback effects from repeated-task learning or motivation. This directly undermines the central causal claim that observed improvements in revisions, subsequent submissions, and transfer are attributable to the agent feedback rather than study artifacts or practice effects.

    Authors: We appreciate the referee's emphasis on rigorous causal inference. Our four-stage sequential design intentionally uses Stage 1 as a within-subject baseline (no feedback) to enable direct comparison with feedback-exposed stages for revision quality, subsequent submissions, and transfer. Randomization of feedback order was not feasible because the stages are progressive to study learning and transfer effects across applications. We relied on descriptive metrics from report artifacts and questionnaire data rather than mixed-effects models, given the study's exploratory focus on workflow integration. We agree this leaves room for practice-effect confounds and will revise the Methods and Limitations sections to explicitly discuss these choices, their rationale, and implications for the causal claims. We will also add basic within-stage statistical comparisons where data permit. revision: partial

  2. Referee: With a modest sample size and no reported power analysis or effect-size quantification, the evidence for 'partial but meaningful transfer' to a later application rests on limited data; the paper should report specific metrics (e.g., report-quality deltas, statistical significance) and discuss generalizability limits to support the transfer finding.

    Authors: We concur that additional quantitative detail would improve transparency. The Results section presents specific report-quality changes across the three applications and stages, including dimension-level improvements supporting the partial transfer observation. However, we did not include a priori power analysis or effect-size reporting. In revision, we will add effect sizes for key deltas, any applicable significance tests, and an expanded discussion of generalizability constraints given n=20 and the three applications studied. We will also moderate phrasing around transfer strength to better reflect the sample and exploratory design while retaining the mixed artifact-plus-questionnaire evidence. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical results derive from new human-subject experiment, not from self-referential definitions or fits.

full rationale

The paper reports outcomes from a controlled four-stage study with 20 testers across three applications. The sole reference to prior work is background on the multi-agent backbone used to produce the feedback under test; the central claims (improvements in revised reports, later submissions, and transfer) are measured directly from the new participant artifacts and questionnaires. No equations, parameter fits, uniqueness theorems, or ansatzes appear. The self-citation is not load-bearing for the reported findings, satisfying the criteria for an independent empirical result.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the validity of the controlled study design, the assumption that observed behavioral changes are attributable to the feedback, and the generalizability of findings from three applications to broader crowdsourced testing.

axioms (1)
  • domain assumption Participant behavior and self-reports in the controlled four-stage setting accurately reflect how testers would respond to agent feedback in real crowdsourced platforms.
    Invoked to support claims of immediate improvement, later-task gains, and transfer.

pith-pipeline@v0.9.1-grok · 5814 in / 1366 out tokens · 37308 ms · 2026-06-28T00:08:42.724142+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 2 canonical work pages

  1. [1]

    MoocTest

    2025. MoocTest. http://www.mooctest.net

    2025

  2. [2]

    Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can LLMs Replace Manual Annotation of Software Engineering Artifacts?. In2025 IEEE/ACM 22nd International Conference on Mining Software Repositories. 526–538

    2025

  3. [3]

    Anahita Alipour, Abram Hindle, and Eleni Stroulia. 2013. A contextual approach towards more accurate duplicate bug report detection. In10th Working Conference on Mining Software Repositories. 183–192

    2013

  4. [4]

    Lerina Aversano and Ermanno Tedeschi. 2016. Bug report quality evaluation considering the effect of submitter reputation. InInternational Conference on Software Engineering and Applications, Vol. 2. 194–201

    2016

  5. [5]

    Pamela Bhattacharya, Liudmila Ulanova, Iulian Neamtiu, and Sai Charan Koduru. 2013. An empirical analysis of bug reports and bug fixing in open source android apps. In17th European Conference on Software Maintenance and Reengineering. 133–143

    2013

  6. [6]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate. InThe Twelfth International Conference on Learning Representations

    2024

  7. [7]

    Li-Hsin Chang and Filip Ginter. 2024. Automatic Short Answer Grading for Finnish with ChatGPT. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 23173–23181

    2024

  8. [8]

    Xin Chen, He Jiang, Xiaochen Li, Tieke He, and Zhenyu Chen. 2018. Automated quality assessment for crowdsourced test reports of mobile applications. InIEEE 25th International Conference on Software Analysis, Evolution and Reengineering. 368–379

    2018

  9. [9]

    Xin Chen, He Jiang, Xiaochen Li, Liming Nie, Dongjin Yu, Tieke He, and Zhenyu Chen. 2020. A systemic framework for crowdsourced test report quality assessment.Empirical Software Engineering25 (2020), 1382–1418

    2020

  10. [10]

    O Fagbohun, NP Iduwe, M Abdullahi, A Ifaturoti, and OM Nwanna. 2024. Beyond traditional assessment: Exploring the impact of large language models on grading practices.Journal of Artificial Intelligence, Machine Learning and Data Science2, 1 (2024), 1–8

    2024

  11. [11]

    Xueyang Feng, Zhi-Yuan Chen, Yujia Qin, Yankai Lin, Xu Chen, Zhiyuan Liu, and Ji-Rong Wen. 2024. Large Language Model-based Human-Agent Collaboration for Complex Task Solving. InFindings of the Association for Computational Linguistics: EMNLP 2024. 1336–1357

    2024

  12. [12]

    Yang Feng, Zhenyu Chen, James A Jones, Chunrong Fang, and Baowen Xu. 2015. Test report prioritization to assist crowdsourced testing. InProceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. 225–236

    2015

  13. [13]

    Yang Feng, James A Jones, Zhenyu Chen, and Chunrong Fang. 2016. Multi-objective test report prioritization using image understanding. InProceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. 202–213

    2016

  14. [14]

    Ruizhi Gao, Yabin Wang, Yang Feng, Zhenyu Chen, and W Eric Wong. 2019. Successes, challenges, and rethinking–an industrial investigation on crowdsourced mobile application testing.Empirical Software Engineering24 (2019), 537–561

    2019

  15. [15]

    GitHub. 2025. GitHub Copilot. https://github.com/features/copilot

    2025

  16. [16]

    Shuchen Guo, Ehsan Latif, Yifan Zhou, Xuan Huang, and Xiaoming Zhai. 2024. Using Generative AI and Multi-Agents to Provide Automatic Feedback.arXiv preprint arXiv:2411.07407(2024)

    work page arXiv 2024

  17. [17]

    Rui Hao, Yang Feng, James A Jones, Yuying Li, and Zhenyu Chen. 2019. CTRAS: Crowdsourced test report aggregation and summarization. InProceedings of the 41st International Conference on Software Engineering. 900–911

    2019

  18. [18]

    Owen Henkel, Libby Hills, Adam Boxer, Bill Roberts, and Zach Levonian. 2024. Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability To Mark Short Answer Questions in K-12 Education. In Proceedings of the Eleventh ACM Conference on Learning @ Scale. 300–304

    2024

  19. [19]

    Abram Hindle, Anahita Alipour, and Eleni Stroulia. 2016. A contextual approach towards more accurate duplicate bug report detection and ranking.Empirical Software Engineering21 (2016), 368–410

    2016

  20. [20]

    Soodeh Hosseini and Hossein Seilani. 2025. The role of agentic ai in shaping a smart future: A systematic review. Array26 (2025), 100399. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:28 Wang et al

    2025

  21. [21]

    Breanna Jury, Angela Lorusso, Juho Leinonen, Paul Denny, and Andrew Luxton-Reilly. 2024. Evaluating llm-generated worked examples in an introductory programming course. InProceedings of the 26th Australasian Computing Education Conference. 77–86

    2024

  22. [22]

    Yuying Li, Yang Feng, Rui Hao, Di Liu, Chunrong Fang, Zhenyu Chen, and Baowen Xu. 2022. Classifying crowdsourced mobile test reports with image features: An empirical study.Journal of Systems and Software184 (2022), 111121

    2022

  23. [23]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2511–2522

    2023

  24. [24]

    Sanwal Manish. 2024. An autonomous multi-agent llm framework for agile software development.International Journal of Trend in Scientific Research and Development8, 5 (2024), 892–898

    2024

  25. [25]

    Ke Mao, Licia Capra, Mark Harman, and Yue Jia. 2017. A survey of the use of crowdsourcing in software engineering. Journal of Systems and Software126 (2017), 57–84

    2017

  26. [26]

    Anh Tuan Nguyen, Tung Thanh Nguyen, Tien N Nguyen, David Lo, and Chengnian Sun. 2012. Duplicate bug report detection with a combination of information retrieval and topic modeling. InProceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. 70–79

    2012

  27. [27]

    Zeeshan Rasheed, Malik Abdul Sami, Muhammad Waseem, Kai-Kristian Kemell, Xiaofeng Wang, Anh Nguyen, Kari Systä, and Pekka Abrahamsson. 2024. AI-powered Code Review with LLMs: Early Results.arXiv preprint arXiv:2404.18496(2024)

    work page arXiv 2024

  28. [28]

    Chamuditha Senanayake and Dinesh Asanka. 2024. Rubric Based Automated Short Answer Scoring using Large Language Models (LLMs). In2024 International Research Conference on Smart Computing and Systems Engineering, Vol. 7. 1–6

    2024

  29. [29]

    Chengnian Sun, David Lo, Siau-Cheng Khoo, and Jing Jiang. 2011. Towards more accurate retrieval of duplicate bug reports. In26th IEEE/ACM International Conference on Automated Software Engineering. 253–262

    2011

  30. [30]

    Chengnian Sun, David Lo, Xiaoyin Wang, Jing Jiang, and Siau-Cheng Khoo. 2010. A discriminative model approach for accurate duplicate bug report retrieval. InProceedings of the 32nd International Conference on Software Engineering, Vol. 1. 45–54

    2010

  31. [31]

    Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawendé Bissyandé. 2024. CodeAgent: Autonomous Communicative Agents for Code Review. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 11279–11313

    2024

  32. [32]

    Huynh Khanh Vi Tran, Nauman bin Ali, Michael Unterkalmsteiner, Jürgen Börstler, and Panagiota Chatzipetrou. 2025. Quality attributes of test cases and test suites–importance & challenges from practitioners’ perspectives.Software Quality Journal33, 1 (2025), 9

    2025

  33. [33]

    Haoming Wang, Chengliang Wang, Zhan Chen, Fa Liu, Chunjia Bao, and Xianlong Xu. 2025. Impact of AI-agent- supported collaborative learning on the learning outcomes of University programming courses.Education and Information Technologies(2025), 1–33

    2025

  34. [34]

    Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1955–1977

    2025

  35. [35]

    Yue Wang, Yuan Zhao, Shengcheng Yu, and Zhenyu Chen. 2025. Multi-dimensional Assessment of Crowdsourced Testing Reports via LLMs. In2025 40th IEEE/ACM International Conference on Automated Software Engineering. 1794– 1806

    2025

  36. [36]

    Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, and Jianghong Ma. 2025. RocketEval: Efficient automated LLM evaluation via grading checklist. InThe Thirteenth International Conference on Learning Representations

    2025

  37. [37]

    Yang Wu, Yao Wan, Zhaoyang Chu, Wenting Zhao, Ye Liu, Hongyu Zhang, Xuanhua Shi, Hai Jin, and Philip S. Yu. 2025. Can Large Language Models Serve as Evaluators for Code Summarization?IEEE Transactions on Software Engineering 51, 12 (2025), 3205–3217

    2025

  38. [38]

    Shengcheng Yu, Chunrong Fang, Zhenfei Cao, Xu Wang, Tongyu Li, and Zhenyu Chen. 2021. Prioritize crowdsourced test reports via deep screenshot understanding. InIEEE/ACM 43rd International Conference on Software Engineering. 946–956

    2021

  39. [39]

    Shengcheng Yu, Chunrong Fang, Quanjun Zhang, Mingzhe Du, Jia Liu, and Zhenyu Chen. 2024. Semi-supervised Crowd- sourced Test Report Clustering via Screenshot-Text Binding Rules.Proceedings of the ACM on Software Engineering1, FSE (2024), 1540–1563

    2024

  40. [40]

    Huan Zhang, Yuan Zhao, Shengcheng Yu, and Zhenyu Chen. 2022. Automated Quality Assessment for Crowdsourced Test Reports Based on Dependency Parsing. In9th International Conference on Dependable Systems and Their Applications. 34–41

    2022

  41. [41]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2026. More than a Judge: An Empirical Study of Agent-Human Interaction in Crowdso...

    2023

  42. [42]

    Thomas Zimmermann, Rahul Premraj, Nicolas Bettenburg, Sascha Just, Adrian Schroter, and Cathrin Weiss. 2010. What makes a good bug report?IEEE Transactions on Software Engineering36, 5 (2010), 618–643. Received November 2025; revised March 2026; accepted June 2026 J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2026

    2010