Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

You Don't Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluation

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that LLM-generated applications can be judged automatically by an agent that clicks through their GUI, matching expert human assessments at 0.92 accuracy and 0.85 correlation.

desk verdict A promising agent-as-a-judge GUI testing benchmark for LLM apps, but the 0.92/0.85 headline numbers cannot be trusted until the full methodology, including human rater reliability and baselines, is on the table. read the letter →

arxiv 2508.14104 v1 pith:2YGT4HD5 submitted 2025-08-17 cs.SE cs.AI

Yutong Bian , Xianhao Lin , Yupeng Xie , Tianyang Liu , Mingchen Zhuge , Siyuan Lu , Haoming Tang , Jinlin Wang
show 6 more authors
Jiayi Zhang Jiaqi Chen Xiangru Tang Yongxin Ni Sirui Hong Chenglin Wu
This is my paper · ORCID
classification cs.SEcs.AI
keywords LLM-generatedsoftwareGUItestingagent-as-a-judgeevaluationruntimebehaviorvisualfidelityhuman-alignedopen-endedtasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLMs and code agents are starting to produce complete applications with graphical interfaces, but existing benchmarks mostly check static code or binary pass/fail scripts and miss what happens when a person actually uses the app. RealDevWorld is an evaluation framework built around the idea that you do not know if an app works until you click through it: it supplies 194 open-ended software tasks (RealDevBench) and an agent-as-a-judge system (AppEvalPilot) that simulates GUI interaction and assesses functionality, visual fidelity, and runtime behavior. The paper reports that this automated judge agrees with expert human assessments at 0.92 accuracy with a 0.85 correlation, while providing fine-grained diagnostic feedback and reducing manual review. If that result holds, it gives a scalable way to evaluate and improve production-ready software generated by LLMs.

What carries the argument

AppEvalPilot is the mechanism, an agent-as-a-judge that simulates realistic GUI-based user interactions instead of comparing code statically. It takes a task description and a generated repository, launches the app, performs UI actions, and produces fine-grained, task-specific diagnostics covering functional correctness, visual fidelity, and runtime behavior. RealDevBench supplies the 194 open-ended tasks with multimodal elements that define what the judge must check. The load-bearing move is replacing pass/fail scripts with a dynamic interaction loop, because the evaluation signal is the app's observed response to use.

What would settle it

Have real end users, not expert reviewers, use a sample of RealDevBench apps and compare their quality ratings with AppEvalPilot's scores; if the correlation with end users is far below the reported 0.85, the human-alignment claim does not generalize beyond expert labels.

Watch

Extended reading notes

Core claim

The central claim is that end-to-end evaluation of LLM-generated software can be automated without losing human alignment. The paper introduces RealDevWorld, whose judge component AppEvalPilot acts as a simulated user: it launches the generated application, clicks through its interface, observes the responses, and scores functional correctness, visual fidelity, and runtime behavior. On the 194 tasks of RealDevBench the framework achieves an accuracy of 0.92 and a correlation of 0.85 with expert human assessments, meaning the automated scores track expert judgments closely enough to replace much of the manual review. The paper positions this as closing the 'blind spot' of static benchmarks: interactive behavior and runtime dynamics only emerge when an application is actually used.

Load-bearing premise

The load-bearing premise is that expert human scores are the correct ground truth for production-ready quality and that AppEvalPilot's simulated clicks match how real users actually interact; if either fails, the reported agreement may not hold in real deployments.

Editorial extensions

If this is right

  • Evaluation of LLM-generated software can move from static checks and pass/fail scripts to automated, interaction-based assessment that catches crashes and dynamic behavior.
  • AppEvalPilot's fine-grained diagnostic feedback can guide code agents through revision cycles without waiting for manual review.
  • RealDevBench's 194 open-ended tasks give a common testbed for comparing code agents across domains, including multimodal inputs.
  • Manual review effort can be reduced while keeping assessments aligned with expert human judgment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editorial extension: the same agent-as-a-judge loop could be adapted to evaluate web, mobile, or desktop apps outside this benchmark, as long as the interaction harness and task descriptions are rebuilt for those platforms.
  • An open question the paper does not settle is whether the 0.85 correlation with expert reviewers also holds for non-expert end users; one could test this by recruiting end users to rate the same apps.
  • Because the benchmark tasks are open-ended, shifting to more subjective criteria, such as visual appeal or accessibility, might change the measured accuracy and reveal where GUI simulation stops tracking human judgment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces RealDevWorld, a framework for automated end-to-end evaluation of LLM-generated, production-ready software. It combines RealDevBench, a benchmark of 194 open-ended software engineering tasks with multimodal elements, and AppEvalPilot, an LLM agent that simulates GUI interactions to judge functional correctness, visual fidelity, and runtime behavior. The abstract reports that this framework achieves 0.92 accuracy and 0.85 correlation with expert human assessments, while reducing manual review burden. The central claim is that RealDevWorld enables scalable, human-aligned evaluation of production-level LLM-generated applications. The abstract, however, provides no methodological details on how the human ground truth was constructed, how the reported metrics were computed, or how the simulated user was calibrated, leaving the strength of the empirical claim unverifiable from the submitted text.

Significance. If the reported results hold, this is a genuinely useful contribution: it addresses a real blind spot in LLM evaluation by moving beyond static checks and binary pass/fail scripts to interactive GUI-level assessment. The proposed benchmark is large-scale (194 tasks) and domain-diverse, and the open-source release is a concrete asset to the community. The agent-as-a-judge approach is timely, and the claim of fine-grained diagnostic feedback goes beyond typical success/failure scoring. However, the significance is conditional on the validity of the human-alignment evidence. The paper's potential value is high, but the abstract alone does not establish that AppEvalPilot's judgments are actually human-aligned rather than aligned with a single rater or with a majority-class bias.

major comments (4)
  1. [Abstract] The central claim 'achieving an accuracy of 0.92 and a correlation of 0.85 with expert human assessments' is not interpretable without a description of the gold standard. The paper must report how many experts scored each of the 194 RealDevBench tasks, whether the raters were independent, what rubric/instructions they received, and inter-rater agreement (e.g., Cohen's kappa or ICC). Because the tasks are explicitly open-ended, there is no objective ground-truth label, so the expert scores are the sole reference; if they are noisy or idiosyncratic, the reported agreement does not establish human alignment.
  2. [Abstract] The accuracy and correlation figures are reported without baselines or class-balance information. Please provide the label distribution (e.g., fraction of tasks whose generated apps pass human review), a confusion matrix, and the accuracy of a trivial majority-class predictor. If most generated applications fail, a classifier that always predicts 'fail' can achieve high accuracy while being useless for ranking. The abstract also does not state whether the accuracy is a task-level binary decision or a fine-grained score, nor how the decision threshold was selected.
  3. [Abstract] The claim that AppEvalPilot 'simulates realistic, GUI-based user interactions' is load-bearing but is not supported by any reported comparison with real user behavior. The paper should describe how the simulated user was calibrated, for instance by comparing its interaction traces with those of human users on a subset of tasks, and should discuss how systematic differences between the simulated user and real users might affect the generalization of the 0.85 correlation to actual deployments.
  4. [Abstract] There is a potential circularity concern that should be addressed explicitly: AppEvalPilot is an LLM-based agent used to judge software produced by LLMs. If the judge shares systematic biases with the evaluated models (e.g., similar blind spots in GUI understanding or in following natural-language instructions), the agreement with human experts reported here could be inflated or could fail to detect common failure modes. The paper should report the judge model, the exact prompts used, and a sensitivity analysis with at least one alternative judge model to show the results are not an artifact of a particular LLM judge.
minor comments (4)
  1. [Abstract] The term 'production-ready' is used without an operational definition; please clarify whether it means installable, runnable, passing a smoke test, or something stronger, and specify the evaluation criterion precisely.
  2. [Abstract] The phrase 'significantly reducing the reliance on manual review' is presented as a contribution but no quantitative comparison of manual vs. automated review cost is given in the abstract; a numerical measure (e.g., time or number of hours saved) would strengthen the claim.
  3. [Abstract] Please report error bars or confidence intervals for the accuracy and correlation, and state whether the correlation is Pearson or Spearman, since the two have different interpretations for ordinal task scores.
  4. [Abstract] The abstract says the benchmark incorporates 'multimodal elements'; it would be helpful to state explicitly how these elements are used in the evaluation (e.g., are they present in the task specification, in the expected output, or in the judge's interaction loop?).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified in the abstract-level evidence; validation is against external expert human assessments.

full rationale

The abstract presents AppEvalPilot as an agent-as-a-judge system that automates GUI-based evaluation of LLM-generated software. The reported evidence is an accuracy of 0.92 and a correlation of 0.85 with expert human assessments. Circularity would require showing that the judge's outputs were derived from those human labels, that the human labels were themselves produced by the same agent, or that a fitted parameter was renamed as a prediction. No such construction is visible in the provided text. The judge and the human ground truth are presented as independent sources, and the benchmark tasks are open-ended rather than generated from the judge's scoring rule. Concerns about inter-rater reliability, class balance, and the fidelity of simulated GUI interactions are substantive evaluation-validity questions, but they are not circularity: they do not make the reported result equivalent to its inputs by definition. There is also no load-bearing self-citation chain in the abstract. Therefore, based on the available evidence, the derivation is self-contained and the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The abstract does not introduce free parameters or invented entities. Three domain assumptions are inferred from the evaluation design.

assumptions (3)
  • domain assumption Expert human assessments are the ground truth for production-ready software quality.
    The paper validates AppEvalPilot by comparing it to human experts, implying human judgment is the baseline. Abstract only; no discussion of inter-human agreement.
  • domain assumption Automated GUI interaction by an LLM agent can substitute for real human user behavior.
    The simulation of clicks and typed inputs is assumed to capture real-world usability. This is a core premise of AppEvalPilot.
  • domain assumption The 194 tasks in RealDevBench are representative of production-ready software development.
    The abstract claims diversity across domains but does not describe how tasks were sampled or selected.

how reviews work

0 comments
Cite this review

Pith. "Pith review of You Don't Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluation." pith.science (2026). https://pith.science/paper/2YGT4HD5

@misc{pith2026250814104,
  author       = {Pith},
  title        = {Pith review of: You Don't Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YGT4HD5}},
  note         = {Machine review of arXiv:2508.14104}
}
read the original abstract

Large Language Models (LLMs) and code agents in software development are rapidly evolving from generating isolated code snippets to producing full-fledged software applications with graphical interfaces, interactive logic, and dynamic behaviors. However, current benchmarks fall short in evaluating such production-ready software, as they often rely on static checks or binary pass/fail scripts, failing to capture the interactive behaviors and runtime dynamics that define real-world usability - qualities that only emerge when an application is actively used. This is the blind spot of current evaluation: you don't know if an app works until you click through it, interact with it, and observe how it responds. To bridge this gap, we introduce RealDevWorld, a novel evaluation framework for automated end-to-end assessment of LLMs' ability to generate production-ready repositories from scratch. It features two key components: (1) RealDevBench, a diverse collection of 194 open-ended software engineering tasks across multiple domains, incorporating multimodal elements to reflect real-world complexity; and (2) AppEvalPilot, a new agent-as-a-judge evaluation system that simulates realistic, GUI-based user interactions to automatically and holistically assess software functional correctness, visual fidelity, and runtime behavior. The framework delivers fine-grained, task-specific diagnostic feedback, supporting nuanced evaluation beyond simple success/failure judgments. Empirical results show that RealDevWorld delivers effective, automatic, and human-aligned evaluations, achieving an accuracy of 0.92 and a correlation of 0.85 with expert human assessments, while significantly reducing the reliance on manual review. This enables scalable, human-aligned assessment of production-level software generated by LLMs. Our code is available on GitHub.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A new heterogeneous-workspace benchmark for data agents, built from transformed Text-to-SQL tasks, with complete-table deterministic evaluation and a best measured accuracy of 66.34%.

Reference graph

Works this paper leans on

46 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Software and systems engineering — Software testing — Part 1: General concepts

    2022. Software and systems engineering — Software testing — Part 1: General concepts

    work page 2022

  4. [4]

    Anthropic. 2024. Claude 3.5 Sonnet. https://www.anthropic.com/news/claude-3-5-sonnet. Accessed on March 28, 2025

    work page 2024

  5. [5]

    Anthropic. 2025. Claude 3.7 Sonnet . https://www.anthropic.com/claude/sonnet

    work page 2025

  6. [6]

    Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  7. [7]

    Beyer, D. 2024. State of the Art in Software Verification and Witness Validation: SV-COMP 2024. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems, Lecture Notes in Computer Science. Springer

    work page 2024

  8. [8]

    S.; Chowdhury, N.; Jaffe, O.; Aung, J.; Sherburn, D.; Mays, E.; Starace, G.; Liu, K.; Maksin, L.; Patwardhan, T.; et al

    Chan, J. S.; Chowdhury, N.; Jaffe, O.; Aung, J.; Sherburn, D.; Mays, E.; Starace, G.; Liu, K.; Maksin, L.; Patwardhan, T.; et al. 2024. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095

    arXiv 2024

Show all 46 references
  1. [9]

    Cheng, K.; Sun, Q.; Chu, Y.; Xu, F.; Li, Y.; Zhang, J.; and Wu, Z. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935

    2024 arXiv

  2. [10]

    Comanici, G.; Bieber, E.; Schaekermann, M.; Pasupat, I.; Sachdeva, N.; Dhillon, I.; Blistein, M.; Ram, O.; Zhang, D.; Rosen, E.; et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv ...

    2025 arXiv

  3. [11]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437

    2024 arXiv

  4. [12]

    K.; Nallapati, R.; Bhatia, P.; Roth, D.; et al

    Ding, Y.; Wang, Z.; Ahmad, W.; Ding, H.; Tan, M.; Jain, N.; Ramanathan, M. K.; Nallapati, R.; Bhatia, P.; Roth, D.; et al. 2023. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems, 36: 46701--46723

    2023

  5. [13]

    Fan, A.; Gokkaya, B.; Harman, M.; Lyubarskiy, M.; Sengupta, S.; Yoo, S.; and Zhang, J. M. 2023. Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoS...

    2023

  6. [14]

    He, H.; Yao, W.; Ma, K.; Yu, W.; Dai, Y.; Zhang, H.; Lan, Z.; and Yu, D. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919

    2024 arXiv

  7. [15]

    Hou, X.; Zhao, Y.; Liu, Y.; Yang, Z.; Wang, K.; Li, L.; Luo, X.; Lo, D.; Grundy, J.; and Wang, H. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8): 1--79

    2024

  8. [16]

    Hu, Y.; Cai, Y.; Du, Y.; Zhu, X.; Liu, X.; Yu, Z.; Hou, Y.; Tang, S.; and Chen, S. 2025. Self-Evolving Multi-Agent Networks for Software Development. In The Thirteenth International Conference on Learning Representations

    2025

  9. [17]

    Jain, N.; Han, K.; Gu, A.; Li, W.-D.; Yan, F.; Zhang, T.; Wang, S.; Solar-Lezama, A.; Sen, K.; and Stoica, I. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Representations

    2025

  10. [18]

    E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K

    Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. R. 2024. SWE -bench: Can Language Models Resolve Real-world Github Issues? In The Twelfth International Conference on Learning Representations

    2024

  11. [19]

    Jin, H.; Huang, L.; Cai, H.; Yan, J.; Li, B.; and Chen, H. 2024. From llms to llm-based agents for software engineering: A survey of current, challenges and future. arXiv preprint arXiv:2408.02479

    2024 arXiv

  12. [20]

    Laskar, M. T. R.; Alqahtani, S.; Bari, M. S.; Rahman, M.; Khan, M. A. M.; Khan, H.; Jahan, I.; Bhuiyan, A.; Tan, C. W.; Parvez, M. R.; et al. 2024. A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations. In Pro...

    2024

  13. [21]

    Li, J.; Li, G.; Zhang, X.; Zhao, Y.; Dong, Y.; Jin, Z.; Li, B.; Huang, F.; and Li, Y. 2025. Evocodebench: An evolving code generation benchmark with domain-specific evaluations. Advances in Neural Information Processing Systems, 37: 57619--57641

    2025

  14. [22]

    Liu, T.; Xu, C.; and McAuley, J. 2024. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems. In The Twelfth International Conference on Learning Representations

    2024

  15. [23]

    Lu, Z.; Yang, Y.; Ren, H.; Hou, H.; Xiao, H.; Wang, K.; Shi, W.; Zhou, A.; Zhan, M.; and Li, H. 2025. WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch. arXiv:2505.03733

    2025 arXiv

  16. [24]

    Miserendino, S.; Wang, M.; Patwardhan, T.; and Heidecke, J. 2025. SWE-Lancer: Can Frontier LLMs Earn \ 1 Million from Real-World Freelance Software Engineering? arXiv preprint arXiv:2502.12115

    2025 arXiv

  17. [25]

    Müller, M.; and Žunič, G. 2024. Browser Use: Enable AI to control your browser

    2024

  18. [26]

    OpenBMB. 2024. SRDD. https://github.com/OpenBMB/ChatDev/tree/main/SRDD. Accessed: 2025-03-29

    2024

  19. [27]

    Qin, Y.; Ye, Y.; Fang, J.; et al. 2025. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv preprint arXiv:2501.12326

    2025 arXiv

  20. [28]

    StackBlitz. 2024. Bolt: AI-Powered Development Platform. https://bolt.new

    2024

  21. [29]

    Team, K.; Bai, Y.; Bao, Y.; Chen, G.; Chen, J.; Chen, N.; Chen, R.; Chen, Y.; Chen, Y.; Chen, Y.; Chen, Z.; Cui, J.; Ding, H.; Dong, M.; Du, A.; Du, C.; Du, D.; Du, Y.; Fan, Y.; Feng, Y.; Fu, K.; Gao, B.; Gao, H.; Gao, P.; Gao, T.; Gu, X.; Guan, L.; Guo, H.; Guo, J.; Hu, H.; H...

    2025 arXiv

  22. [30]

    Team, L. 2024 a . Lovable: AI Development Solution. https://lovable.dev

    2024

  23. [31]

    Team, M. 2024 b . MGX: AI Software Development Platform. https://mgx.dev

    2024

  24. [32]

    Team, Q. 2025. Qwen3 Technical Report. arXiv:2505.09388

    2025 arXiv

  25. [33]

    N.; Chiang, W.-L.; Tang, K.; and Manolache, L

    Vichare, A.; Angelopoulos, A. N.; Chiang, W.-L.; Tang, K.; and Manolache, L. 2025. WebDev Arena: A Live LLM Leaderboard for Web App Development

    2025

  26. [34]

    Wang, L.; et al. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091

    2023 arXiv

  27. [35]

    F.; Tang, X.; Zhuge, M.; Pan, J.; Song, Y.; Li, B.; Singh, J.; Tran, H

    Wang, X.; Li, B.; Song, Y.; Xu, F. F.; Tang, X.; Zhuge, M.; Pan, J.; Song, Y.; Li, B.; Singh, J.; Tran, H. H.; Li, F.; Ma, R.; Zheng, M.; Qian, B.; Shao, Y.; Muennighoff, N.; Zhang, Y.; Hui, B.; Lin, J.; Brennan, R.; Peng, H.; Ji, H.; and Neubig, G. 2025. OpenHands: An Open Pl...

    2025

  28. [36]

    T.; and Ni, L

    Wang, Y.; Yao, Q.; Kwok, J. T.; and Ni, L. M. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys, 53(3): 1--34

    2020

  29. [37]

    Xu, J.; Guo, K.; Gong, W.; and Shi, R. 2024. OSAgent: Copiloting Operating System with LLM-based Agent. In 2024 International Joint Conference on Neural Networks (IJCNN), 1--9. IEEE

    2024

  30. [38]

    Xu, K.; Mao, Y.; Guan, X.; and Feng, Z. 2025. Web-bench: A llm code benchmark based on web standards and frameworks. arXiv preprint arXiv:2505.07473

    2025 arXiv

  31. [39]

    Zhang, F.; Chen, B.; Zhang, Y.; Keung, J.; Liu, J.; Zan, D.; Mao, Y.; Lou, J.-G.; and Chen, W. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processin...

    2023

  32. [40]

    Zhang, S.; Zhao, H.; Liu, X.; Zheng, Q.; Qi, Z.; Gu, X.; Dong, Y.; and Tang, J. 2024. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user queries. In Findings of the Association for Computational Linguistics ACL 2024, 7907--7928

    2024

  33. [41]

    T.; Cardie, C.; Gall \'e , M.; and Rush, A

    Zhao, W.; Jiang, N.; Lee, C.; Chiu, J. T.; Cardie, C.; Gall \'e , M.; and Rush, A. M. 2024. Commit0: Library generation from scratch. arXiv preprint arXiv:2412.01769

    2024 arXiv

  34. [42]

    E.; and Stoica, I

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023 a . Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and...

    2023

  35. [43]

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023 b . Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 46595--46623

    2023

  36. [44]

    Zhu, H.; Zhang, Y.; Zhao, B.; Ding, J.; Liu, S.; Liu, T.; Wang, D.; Liu, Y.; and Li, Z. 2025. FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation. arXiv preprint arXiv:2506.13832

    2025 arXiv

  37. [45]

    Zhuge, M.; Zhao, C.; Ashley, D.; Wang, W.; Khizbullin, D.; Xiong, Y.; Liu, Z.; Chang, E.; Krishnamoorthi, R.; Tian, Y.; et al. 2024. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934

    2024 arXiv

  38. [46]

    Y.; Vu, M

    Zhuo, T. Y.; Vu, M. C.; Chim, J.; Hu, H.; Yu, W.; Widyasari, R.; Yusuf, I. N. B.; Zhan, H.; He, J.; Paul, I.; et al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.