{"id":"46cc5b41-3386-464f-bf77-3034f5314047","arxiv_id":"2508.14104","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"RealDevWorld, an automated GUI-testing evaluation system, matches human expert judgments (0.92 accuracy, 0.85 correlation) on assessing LLM-generated production-ready software.","lead":"This paper introduces RealDevWorld, an evaluation framework that uses an AI agent to click through and interact with software generated by large language models, scoring it on functionality, look, and behavior. The framework achieved 0.92 accuracy and 0.85 correlation with expert human assessments on 194 software-building tasks, offering a way to judge production-ready code automatically.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation of the automated judge depends on a human ground truth whose reliability is not reported; without inter-rater agreement, accuracy 0.92 and correlation 0.85 may reflect alignment with a single rater.","rationale":"The paper's strongest claim is that RealDevWorld provides effective, automatic, and human-aligned evaluation. The abstract gives two aggregate numbers, 0.92 accuracy and 0.85 correlation, as evidence. For the claim to hold, the expert labels must be a reliable ground truth and the GUI simulation must approximate real user behavior. Neither is supported in the abstract. I do not treat this as a positive objection to the method; rather, it is the minimal information needed to interpret the headline numbers. A reliability study on a small subset would settle whether this concern is real. I did not find an internal inconsistency in the abstract; the concern is an unverified precondition. Hence the reader's UNVERDICTED status is appropriate. Credit: the paper does include a constructed benchmark with 194 tasks and open-source code, and the design of AppEvalPilot is falsifiable, which is a point in its favor.","tokens_in":857,"tokens_out":3934,"duration_ms":40877,"concrete_test":"Run an inter-rater reliability study: recruit at least three independent expert raters to score a held-out random subset of about 30 tasks from RealDevBench using the paper's rubric, and compute Krippendorff's alpha or ICC. If alpha is below 0.7, the reported 0.92/0.85 agreement with a single expert label set cannot be interpreted as evidence of human-aligned evaluation. Also report per-class accuracy and the pass/fail base rate to rule out trivial majority-class predictions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract reports 0.92 accuracy and 0.85 correlation with expert human assessments, but it does not state how many experts scored each of the 194 tasks, whether raters were independent, what rubric they used, or how the accuracy decision threshold was set. These details are load-bearing because RealDevBench tasks are explicitly open-ended; for such tasks there is no single objective label, so the expert scores are the only ground truth. If the ground truth is noisy or idiosyncratic, a high aggregate agreement does not establish human alignment; it may only show that AppEvalPilot reproduces one rater's preferences. In particular, if the label distribution is imbalanced (for example, most generated apps fail), a classifier that always predicts the majority class can achieve high accuracy while being useless for ranking. The abstract also does not demonstrate that the simulated GUI interactions match real user behavior, so even a high correlation with expert labels would not guarantee that the framework predicts actual deployment outcomes. The absence of inter-rater reliability, class-balance information, and user-behavior calibration is the weakest point in the central argument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RealDevWorld, a framework for automated end-to-end evaluation of LLM-generated, production-ready software. It combines RealDevBench, a benchmark of 194 open-ended software engineering tasks with multimodal elements, and AppEvalPilot, an LLM agent that simulates GUI interactions to judge functional correctness, visual fidelity, and runtime behavior. The abstract reports that this framework achieves 0.92 accuracy and 0.85 correlation with expert human assessments, while reducing manual review burden. The central claim is that RealDevWorld enables scalable, human-aligned evaluation of production-level LLM-generated applications. The abstract, however, provides no methodological details on how the human ground truth was constructed, how the reported metrics were computed, or how the simulated user was calibrated, leaving the strength of the empirical claim unverifiable from the submitted text.","tokens_in":1036,"tokens_out":2861,"duration_ms":29501,"significance":"If the reported results hold, this is a genuinely useful contribution: it addresses a real blind spot in LLM evaluation by moving beyond static checks and binary pass/fail scripts to interactive GUI-level assessment. The proposed benchmark is large-scale (194 tasks) and domain-diverse, and the open-source release is a concrete asset to the community. The agent-as-a-judge approach is timely, and the claim of fine-grained diagnostic feedback goes beyond typical success/failure scoring. However, the significance is conditional on the validity of the human-alignment evidence. The paper's potential value is high, but the abstract alone does not establish that AppEvalPilot's judgments are actually human-aligned rather than aligned with a single rater or with a majority-class bias.","major_comments":[{"comment":"The central claim 'achieving an accuracy of 0.92 and a correlation of 0.85 with expert human assessments' is not interpretable without a description of the gold standard. The paper must report how many experts scored each of the 194 RealDevBench tasks, whether the raters were independent, what rubric/instructions they received, and inter-rater agreement (e.g., Cohen's kappa or ICC). Because the tasks are explicitly open-ended, there is no objective ground-truth label, so the expert scores are the sole reference; if they are noisy or idiosyncratic, the reported agreement does not establish human alignment.","section":"Abstract"},{"comment":"The accuracy and correlation figures are reported without baselines or class-balance information. Please provide the label distribution (e.g., fraction of tasks whose generated apps pass human review), a confusion matrix, and the accuracy of a trivial majority-class predictor. If most generated applications fail, a classifier that always predicts 'fail' can achieve high accuracy while being useless for ranking. The abstract also does not state whether the accuracy is a task-level binary decision or a fine-grained score, nor how the decision threshold was selected.","section":"Abstract"},{"comment":"The claim that AppEvalPilot 'simulates realistic, GUI-based user interactions' is load-bearing but is not supported by any reported comparison with real user behavior. The paper should describe how the simulated user was calibrated, for instance by comparing its interaction traces with those of human users on a subset of tasks, and should discuss how systematic differences between the simulated user and real users might affect the generalization of the 0.85 correlation to actual deployments.","section":"Abstract"},{"comment":"There is a potential circularity concern that should be addressed explicitly: AppEvalPilot is an LLM-based agent used to judge software produced by LLMs. If the judge shares systematic biases with the evaluated models (e.g., similar blind spots in GUI understanding or in following natural-language instructions), the agreement with human experts reported here could be inflated or could fail to detect common failure modes. The paper should report the judge model, the exact prompts used, and a sensitivity analysis with at least one alternative judge model to show the results are not an artifact of a particular LLM judge.","section":"Abstract"}],"minor_comments":[{"comment":"The term 'production-ready' is used without an operational definition; please clarify whether it means installable, runnable, passing a smoke test, or something stronger, and specify the evaluation criterion precisely.","section":"Abstract"},{"comment":"The phrase 'significantly reducing the reliance on manual review' is presented as a contribution but no quantitative comparison of manual vs. automated review cost is given in the abstract; a numerical measure (e.g., time or number of hours saved) would strengthen the claim.","section":"Abstract"},{"comment":"Please report error bars or confidence intervals for the accuracy and correlation, and state whether the correlation is Pearson or Spearman, since the two have different interpretations for ordinal task scores.","section":"Abstract"},{"comment":"The abstract says the benchmark incorporates 'multimodal elements'; it would be helpful to state explicitly how these elements are used in the evaluation (e.g., are they present in the task specification, in the expected output, or in the judge's interaction loop?).","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the abstract for arXiv:2508.14104. My take: this is a promising idea that we cannot yet assess, because the full text did not come with the review pack and the abstract's headline numbers are unverified.\n\nWhat's genuinely new: most LLM software benchmarks use static checks or binary pass/fail. RealDevWorld targets interactive GUI behavior, which is a real blind spot. The 194-task RealDevBench is concrete, and AppEvalPilot's agent-as-a-judge approach that simulates clicking through an app to score correctness, visual fidelity, and runtime behavior is a logical way to fill that gap. If the benchmark construction is sound, this would be a useful contribution to LLM software evaluation.\n\nWhat worries me: the two validation numbers, 0.92 accuracy and 0.85 correlation with expert human assessments, are load-bearing but entirely unspecified. The tasks are open-ended, so expert scores are the only ground truth. The abstract doesn't say how many experts scored each task, whether they were independent, what rubric they used, or whether their ratings agreed enough to be a reliable gold standard. It also doesn't report class balance, baselines, error bars, or how the simulated GUI interactions were calibrated against real user behavior. Without those details, 0.92 could be inflated by a skewed label distribution, and 0.85 could just mean the agent reproduces one rater's preferences. That doesn't make the paper bad; it makes the abstract insufficient to judge it.\n\nThe stress-test note makes the same point, and I think it lands. The authors may well have done all this work in the full paper; we just can't see it.\n\nWhere does that leave us? If the full paper exists and reports inter-rater reliability, baselines, and calibration, this deserves serious refereeing. The idea is good enough that a desk reject would be premature. For my own work, I wouldn't cite the numbers until I can inspect the methodology.\n\nRecommendation: send the full manuscript to peer review if it hasn't been; if the submission is only the abstract, ask the authors for the full text before deciding.","headline":"A promising agent-as-a-judge GUI testing benchmark for LLM apps, but the 0.92/0.85 headline numbers cannot be trusted until the full methodology, including human rater reliability and baselines, is on the table.","tokens_in":1616,"tokens_out":2324,"would_cite":false,"duration_ms":22860,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM-generated applications can be judged automatically by an agent that clicks through their GUI, matching expert human assessments at 0.92 accuracy and 0.85 correlation.","keywords":["LLM-generated software","GUI testing","agent-as-a-judge","software evaluation","runtime behavior","visual fidelity","human-aligned evaluation","open-ended tasks"],"falsifier":"Have real end users, not expert reviewers, use a sample of RealDevBench apps and compare their quality ratings with AppEvalPilot's scores; if the correlation with end users is far below the reported 0.85, the human-alignment claim does not generalize beyond expert labels.","tokens_in":659,"feed_emoji":"🖱️","tokens_out":4742,"duration_ms":45861,"temperature":0.7,"pith_summary":"LLMs and code agents are starting to produce complete applications with graphical interfaces, but existing benchmarks mostly check static code or binary pass/fail scripts and miss what happens when a person actually uses the app. RealDevWorld is an evaluation framework built around the idea that you do not know if an app works until you click through it: it supplies 194 open-ended software tasks (RealDevBench) and an agent-as-a-judge system (AppEvalPilot) that simulates GUI interaction and assesses functionality, visual fidelity, and runtime behavior. The paper reports that this automated judge agrees with expert human assessments at 0.92 accuracy with a 0.85 correlation, while providing fine-grained diagnostic feedback and reducing manual review. If that result holds, it gives a scalable way to evaluate and improve production-ready software generated by LLMs.","feed_headline":"Automated GUI testing scores generated apps at 0.92 accuracy","feed_subtitle":"It clicks through 194 tasks and matches expert human judgment 85 percent of the time.","key_machinery":"AppEvalPilot is the mechanism, an agent-as-a-judge that simulates realistic GUI-based user interactions instead of comparing code statically. It takes a task description and a generated repository, launches the app, performs UI actions, and produces fine-grained, task-specific diagnostics covering functional correctness, visual fidelity, and runtime behavior. RealDevBench supplies the 194 open-ended tasks with multimodal elements that define what the judge must check. The load-bearing move is replacing pass/fail scripts with a dynamic interaction loop, because the evaluation signal is the app's observed response to use.","core_discovery":"The central claim is that end-to-end evaluation of LLM-generated software can be automated without losing human alignment. The paper introduces RealDevWorld, whose judge component AppEvalPilot acts as a simulated user: it launches the generated application, clicks through its interface, observes the responses, and scores functional correctness, visual fidelity, and runtime behavior. On the 194 tasks of RealDevBench the framework achieves an accuracy of 0.92 and a correlation of 0.85 with expert human assessments, meaning the automated scores track expert judgments closely enough to replace much of the manual review. The paper positions this as closing the 'blind spot' of static benchmarks: interactive behavior and runtime dynamics only emerge when an application is actually used.","pith_inferences":["An editorial extension: the same agent-as-a-judge loop could be adapted to evaluate web, mobile, or desktop apps outside this benchmark, as long as the interaction harness and task descriptions are rebuilt for those platforms.","An open question the paper does not settle is whether the 0.85 correlation with expert reviewers also holds for non-expert end users; one could test this by recruiting end users to rate the same apps.","Because the benchmark tasks are open-ended, shifting to more subjective criteria, such as visual appeal or accessibility, might change the measured accuracy and reveal where GUI simulation stops tracking human judgment."],"forward_implications":["Evaluation of LLM-generated software can move from static checks and pass/fail scripts to automated, interaction-based assessment that catches crashes and dynamic behavior.","AppEvalPilot's fine-grained diagnostic feedback can guide code agents through revision cycles without waiting for manual review.","RealDevBench's 194 open-ended tasks give a common testbed for comparing code agents across domains, including multimodal inputs.","Manual review effort can be reduced while keeping assessments aligned with expert human judgment."],"supporting_citations":[],"fun_headline_variants":["Simulated clicks score AI apps: 92% human match","GUI click-test judges LLM apps: 0.92 vs human experts","Click-through evaluation: 0.92 accuracy on AI-generated apps","You know when you click: GUI judge hits 0.92"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that expert human scores are the correct ground truth for production-ready quality and that AppEvalPilot's simulated clicks match how real users actually interact; if either fails, the reported agreement may not hold in real deployments.","fun_headline_variants_meta":{"raw":{"variants":["Simulated clicks score AI apps: 92% human match","GUI click-test judges LLM apps: 0.92 vs human experts","Click-through evaluation: 0.92 accuracy on AI-generated apps","You know when you click: GUI judge hits 0.92"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000594,"raw_usage":{"total_tokens":2807,"prompt_tokens":995,"completion_tokens":1812,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":1736}},"tokens_in":611,"tokens_out":1812,"duration_ms":13628,"temperature":1.0,"reasoning_tokens":1736,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:23:14.584995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have real end users, not expert reviewers, use a sample of RealDevBench apps and compare their quality ratings with AppEvalPilot's scores; if the correlation with end users is far below the reported 0.85, the human-alignment claim does not generalize beyond expert labels.","supporting_citations":[],"review_version":2}