{"id":"ef5fb7e9-e696-427f-b50f-b89aa66d92c4","arxiv_id":"2608.11434","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MobileJudgeBench, a new 931-trajectory benchmark, shows a simple screenshot-based LLM judge matches purpose-built judges and that benchmark metrics predict evaluation and RL-training utility.","lead":"Researchers built MobileJudgeBench, 931 human-labeled Android agent trajectories, to test whether AI judges score task completion correctly. They find simple screenshot-based judges match or beat elaborate purpose-built pipelines, and that judge accuracy predicts how well judges rank agents and support RL training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RL leg of the central claim is under-powered: four reward conditions, one environment, and a best-checkpoint convention that turns into a tie at the final checkpoint do not establish that benchmark metrics reliably predict training outcomes.","rationale":"I read the paper in good faith. It is a substantial empirical contribution: 931 human-annotated trajectories across 6 benchmarks, a unified evaluation of 30 judge variants, careful bootstrap and leave-one-out uncertainty quantification, a useful null result for precision, and a double-coded failure taxonomy. The strongest claim, however, has two legs: evaluation-side utility and training-side utility. The evaluation leg is well supported. The training leg is not. The reader's weakest assumption points exactly at §5.2.2's four reward conditions and the convention-dependent precision comparison, and I agree. The decisive evidence is in the paper itself: Appendix E, Table 10 shows GPT-5.2 and Qwen tie at the fixed final checkpoint, and the Limitations section states the training experiments are limited to one environment and one judge method. With N=4 and no variation in judge method, the monotone ordering between judge accuracy and training success cannot be distinguished from chance, and the abstract's phrase 'reliably predict' overstates what the evidence supports. I do not see a more load-bearing concern: the meta-correlation results are robust to the checks reported, and the SPA-Bench coarse-stage adaptation, while worth noting, does not threaten the central metric-to-utility claim. The appropriate verdict remains CONDITIONAL, unchanged from the reader: the paper is acceptable if the abstract is aligned with the strength of the RL evidence and the artifacts are released.","tokens_in":21797,"tokens_out":6072,"duration_ms":55362,"concrete_test":"Expand the RL study to at least 8 reward conditions crossing at least two judge methods (e.g., SPA-Bench fine stage and the simple baseline) with four LLM backends plus the rule-based oracle, all on AndroidWorld. Report primary outcome at the fixed final checkpoint (step 300) rather than best checkpoint, and compute the Spearman correlation between judge accuracy on the AndroidWorld subset and final-task success rate. If the ordering is not preserved at the fixed final checkpoint, or the rank correlation is not meaningfully positive (say ρ<0.7), the abstract should be revised to describe the training result as a preliminary case study rather than a reliable prediction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's second claim—that benchmark quality metrics reliably predict downstream performance when judges serve as reward signals for on-policy RL—rests entirely on §5.2.2. That section compares four reward conditions: one rule-based oracle and three backends of the same simple-baseline judge, all on AndroidWorld's easy task set. With N=4, the monotone ordering (rule-based 54.6% > GPT-5-mini 45.4% > GPT-5.2 42.6% > Qwen 39.9%) is consistent with chance variation; the paper's own Limitations section concedes that training comparisons across judge methods are narrower than the offline study. More damaging, the matched-accuracy comparison between GPT-5.2 and Qwen—the only evidence that precision matters for training—holds only under the best-checkpoint convention: Appendix E, Table 10 shows the two conditions tie at 25.9% at the fixed final checkpoint on the full task set. The paper itself labels this 'suggestive rather than conclusive.' Because the central claim in the abstract is stated as a reliable prediction, not a suggestive trend, the training leg is the least secure part of the central claim. The evaluation-side meta-correlation (§5.2.1), by contrast, has 30 variants, bootstrap CIs, and leave-one-out checks, and I do not see an equally serious problem there.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MobileJudgeBench, a benchmark of 931 human-annotated mobile-agent trajectories drawn from six existing benchmarks, four agent models, and 68 apps. It evaluates six LLM-as-judge methods (five adapted from prior benchmarks plus a simple baseline) across five LLM backbones, producing 30 judge variants. The main findings are: (1) a simple screenshot-based baseline is competitive with or better than purpose-built judge pipelines; (2) judge quality metrics, especially F1 and balanced accuracy, correlate with agent-ranking fidelity in a meta-correlation across 30 variants; (3) an on-policy RL experiment on AndroidWorld with four reward conditions suggests that judge accuracy orders downstream training performance; and (4) a failure analysis identifies contrasting conservative and permissive failure profiles for GPT- and Qwen-based judges. The authors release trajectories, annotations, and code.","tokens_in":22000,"tokens_out":3756,"duration_ms":35012,"significance":"If the central claims hold, this is a valuable contribution: it addresses an underexamined but important issue, provides a reusable judge-evaluation benchmark, and offers an unusually careful evaluation-side validation with task-cluster bootstrap confidence intervals, leave-one-out checks over backbones and methods, and a nondeterminism sensitivity analysis. The failure taxonomy is also well grounded, with high double-coded inter-rater agreement (Cohen's kappa = 0.957). The evaluation-side meta-correlation (Section 5.2.1, Table 4) is the strongest part of the paper and is not circular: judges are scored against human labels, and downstream evaluation reliability is measured with a rule-based checker. The training-side claim in the abstract, however, is substantially stronger than the evidence in Section 5.2.2, which rests on only four reward conditions, one environment, one judge method, and a best-checkpoint convention that collapses at the fixed final checkpoint.","major_comments":[{"comment":"The abstract states that benchmark quality metrics 'reliably predict real-world judge utility' including 'downstream performance when judges serve as reward signals for on-policy reinforcement learning.' Section 5.2.2 supports this with only four reward conditions (one rule-based oracle and three backends of the same simple-baseline judge), all on a single environment (AndroidWorld easy set). With N=4, a monotone ordering is weak evidence, and the paper's own Limitations section concedes that training-time comparisons across judge methods are narrower than the offline study. Either add additional reward conditions or judge methods/environments, or reword the abstract and conclusion to describe the training-side evidence as suggestive and preliminary.","section":"Abstract and §5.2.2"},{"comment":"The conclusion states that 'the judge with stronger false-positive control reached a higher best-checkpoint success rate.' This is internally inconsistent with the appendix: at the fixed final checkpoint (step 300), GPT-5.2 and Qwen tie at 25.9% on the full 116-task suite. The separation of 2.7pp on the easy set exists only under the best-checkpoint convention, which the paper itself labels 'suggestive rather than conclusive.' The conclusion and abstract should carry this caveat explicitly, and the best-checkpoint convention should be stated wherever the precision-based ordering is discussed.","section":"§7 and Appendix E, Table 10"},{"comment":"The rule-based reward condition coincides with the evaluation signal used to measure training success, so its top ranking is an upper-bound control rather than evidence that judge benchmark metrics predict training outcomes. Among the three LLM conditions, all use the same simple baseline judge and differ only in backbone. This design cannot support a general claim that judge quality metrics predict downstream training performance across judge methods. The paper acknowledges some of this in the Limitations, but the abstract's unqualified wording should be revised to reflect the narrow scope of the RL validation.","section":"§5.2.2 and Appendix E"}],"minor_comments":[{"comment":"The ablation table states that 16 screenshots at 1/4 resolution, 48 at 1/16, and 192 at 1/64 are 'approximately equal cost,' but the cost proxy is not defined. If the proxy is total input pixels or tokens, that should be stated explicitly; otherwise the iso-cost comparison is hard to interpret.","section":"§4.2 and Table 3"},{"comment":"The hard-core failure definition requires at least 4 of 5 judge methods to err, but no sensitivity analysis is given for this threshold. Because the failure sets are small (48 and 78 cases), reporting the taxonomy under a 3-of-5 threshold would help establish that the qualitative conclusions are not threshold artifacts.","section":"Appendix F.2"},{"comment":"The text reports the best judge's agreement with human success rates as 'rho = 0.97' without consistently distinguishing Spearman rho from the rho_s used elsewhere. Please define the correlation coefficient in the caption or text.","section":"§5.2.1 and Figure 2"},{"comment":"The labels 'GT:SuccessPred:Failure' are missing a space after 'Success'; minor formatting issue that should be corrected.","section":"Figures 8 and 9"}],"recommendation":"major_revision","confidential_remarks":"The RL leg of the central claim is likely to attract reviewer skepticism because of the small number of reward conditions and the best-checkpoint issue. The paper's own limitations are candid, but the abstract overstates the training-side evidence. I would ask for either additional training experiments or a clear downgrade of the training claim to 'suggestive' throughout, including the abstract and conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid benchmark paper with a careful evaluation-side validation and an overreaching abstract. The evaluation claim—judge quality metrics predict ranking fidelity—is well supported. The training claim is not; four reward conditions don't establish reliable prediction. Both problems are fixable.\n\nWhat's actually new: first mobile-specific judge benchmark, spanning 6 Android benchmarks, 931 human-annotated trajectories, 30 judge variants. The meta-correlation analysis is the real contribution: F1 and balanced accuracy predict agent-ranking fidelity and success-rate estimation error across 24 agents, with task-cluster bootstrap CIs and leave-one-out checks; precision is convincingly non-predictive. That's a useful result for anyone building leaderboards. The simple baseline matching or beating purpose-built judges is also a clean finding, and the ablation is honest.\n\nSoft spots: the RL leg in §5.2.2 rests on four reward conditions (rule-based + three backends of one judge method), one environment, and the claimed ordering holds only under the best-checkpoint convention. At the fixed final checkpoint on the full suite, GPT-5.2 and Qwen tie at 25.9% (Appendix E, Table 10). The paper itself calls it suggestive; the abstract says 'reliably predict.' That's a mismatch. The SPA-Bench judge is evaluated without its coarse stage, so 'purpose-built methods' is generous wording. Also, no code or data link in the preprint; for a benchmark paper that's a meaningful gap, though likely a submission issue.\n\nThe citation pattern looks fair; they cite AgentRewardBench and explain the difference. No sign of circularity: judge predictions are scored against human labels, and the RL comparison uses the rule-based checker, not the judge itself.\n\nWho this is for: the mobile/GUI agent community, especially people designing leaderboards or RL rewards. The evaluation-side meta-correlation alone is worth a serious referee. Recommend: accept for peer review with the expectation that the authors align the abstract with the actual evidence and release artifacts.","headline":"A solid mobile-agent judge benchmark with an overreaching abstract: the evaluation-side meta-correlation is convincing, but the RL validation is too underpowered to support the paper's headline claim.","tokens_in":22612,"tokens_out":1899,"would_cite":true,"duration_ms":16794,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Judge quality scores predict how well LLM judges evaluate mobile agents.","keywords":["LLM-as-a-judge","mobile agents","GUI agent evaluation","benchmark","reward model","reinforcement learning","judge reliability","multimodal trajectories"],"falsifier":"Train an agent with a fifth reward signal that scores high on F1 but is deliberately biased toward one agent class, and check whether ranking fidelity or RL success still tracks F1; if a low-F1, high-precision judge outperforms a high-F1 judge in training, the paper's ordering claim collapses. A lighter check: add another reward condition to the RL study and see whether the easy-set success-rate ordering still follows judge accuracy.","tokens_in":21534,"feed_emoji":"📱","tokens_out":6303,"duration_ms":61883,"temperature":0.7,"pith_summary":"Mobile agent benchmarks increasingly trust LLM judges to decide whether a phone task was completed, and this paper asks whether those judges can be trusted and whether their quality can be predicted cheaply. To answer, it builds MobileJudgeBench, 931 human-annotated trajectories drawn from six existing benchmarks, four agent models, and 68 apps, and evaluates six judge methods across five LLM backbones. Its central thesis is that benchmark quality metrics—especially F1 and balanced accuracy—predict real-world judge utility: they track how reliably a judge ranks agents (F1 vs. ranking fidelity, Spearman $\\rho_s=0.90$) and how well the judge behaves as a reward signal in on-policy reinforcement learning. A second thesis is that elaborate judge pipelines are not worth much: a simple baseline that samples screenshots matches or beats purpose-built methods, with the LLM backbone driving most of the remaining difference. If these claims hold, judge evaluation becomes a first-class benchmarking concern rather than an assumed background detail.","feed_headline":"A simple judge keeps pace with purpose-built LLM evaluators for mobile agents","feed_subtitle":"New benchmark of 931 human-annotated trajectories shows quality metrics predict real-world judge utility.","key_machinery":"The load-bearing object is MobileJudgeBench itself: 931 human-annotated success/failure labels over trajectories from six mobile benchmarks, four agent models, and 68 apps, with every trajectory stored as a sequence of screenshots, actions, UI trees, and agent reasoning. Its mechanism is a two-stage evaluation: each judge variant is first scored with trajectory-level classification metrics (accuracy, precision, recall, F1, balanced accuracy), and then those metrics are validated against two downstream uses of judges—ranking fidelity and success-rate error for agent evaluation, and success rate of a GRPO-trained agent when the judge is the reward. A meta-correlation across the 30 variants is what turns the benchmark from a static dataset into a predictive instrument.","core_discovery":"The paper's central claim is that MobileJudgeBench scores predict how useful a judge is in actual mobile-agent evaluation and training. Across 30 judge variants (6 methods × 5 backbones), F1 correlates with agent-ranking fidelity at 0.90 and balanced accuracy correlates with success-rate estimation error at −0.79, while precision alone has no predictive power. In an on-policy RL study with four reward sources on AndroidWorld's easy tasks, judge accuracy orders the final agent performance: the rule-based checker reaches 54.6% success, followed by a GPT-5-mini judge at 45.4%, a GPT-5.2 judge at 42.6%, and a Qwen judge at 39.9%. The paper also claims that judge failure modes are backbone-specific rather than method-specific: GPT-based judges over-reject (false-negative-heavy, driven by last-frame anchoring and unfamiliar success states), while Qwen-based judges over-accept (false-positive-heavy, driven by constraint violations and partial completion), with visual surface matching a shared weakness.","pith_inferences":["If the meta-correlation generalizes, judge benchmarks like this one could be used as a cheap filter for reward-model selection: score candidate judges on a small annotated set before spending GPU-hours on RL, an extension the paper does not run.","The opposite failure profiles suggest a testable calibration strategy: choose or adjust judges by precision-recall target depending on whether the use is leaderboard ranking (balanced) or reward signaling (false-positive averse); the paper points this way but does not test it directly.","The RL validation compares only one judge method across three backbones plus an oracle in one environment; a natural next step would vary judge methods and environments to see whether the accuracy-to-training-success ordering is stable.","The shared 'surface UI match' failure suggests that verification-style judges that check actual system state, rather than screenshots alone, could fix a common failure mode; this is implied by the taxonomy but not implemented here."],"forward_implications":["Benchmarks should report judge reliability alongside agent scores, since judge accuracy between 76% and 91% can shift agent rankings by up to 13 positions.","Simple, cheap judges—uniformly sampled screenshots with a strong backbone—are a defensible default for mobile-agent evaluation; elaborate pipelines demonstrated no consistent advantage.","F1 and balanced accuracy, not precision alone, are the metrics to report for judge quality in evaluation settings.","When judges are used as RL rewards, false-positive control matters: the equal-accuracy judge with higher precision yielded higher best-checkpoint training success.","Because fault profiles depend on the backbone, a judge's precision-recall character should be matched to its application, not chosen by method alone."],"supporting_citations":[{"why":"Supplies the SPA-Bench judge method adapted into MobileJudgeBench and 220 trajectories from its benchmark.","marker":"Chen et al. 2025"},{"why":"Supplies the A3 judge with final-state and essential-state modes, among the six benchmarked methods.","marker":"Chai et al. 2025"},{"why":"Supplies the AndroidArena text-based judge method and 151 trajectories in the benchmark.","marker":"Xing et al. 2024"},{"why":"AgentRewardBench is the closest prior judge benchmark and the source of the adapted four-dimension judge prompt.","marker":"Lù et al. 2025"},{"why":"AndroidWorld provides the environment, 205 trajectories, the rule-based checker, and the easy-task training suite.","marker":"Rawles et al. 2025"},{"why":"Establishes that a VLM evaluator can serve as an RL reward signal, motivating the training-side validation.","marker":"Bai et al. 2024"},{"why":"Supplies the claim that false positives in rewards corrupt training, used to interpret the precision comparison.","marker":"Huang et al. 2024"},{"why":"Provides UI-TARS, the agent model behind the generated trajectories and the base model in RL training.","marker":"Qin et al. 2025"}],"fun_headline_variants":["LLM backbone, not pipeline, drives mobile judge quality","Simple judge rivals purpose-built LLM evaluators on mobile","MobileJudgeBench: metrics predict real-world judge utility","F1 score predicts LLM judge fidelity for mobile agents","For LLM judges, backbone beats pipeline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that judge accuracy predicts training outcomes rests on only four reward conditions, three of them the same judge method with different backbones, in a single Android environment, so the ordering could change with a different judge method or app suite.","fun_headline_variants_meta":{"raw":{"variants":["LLM backbone, not pipeline, drives mobile judge quality","Simple judge rivals purpose-built LLM evaluators on mobile","MobileJudgeBench: metrics predict real-world judge utility","F1 score predicts LLM judge fidelity for mobile agents","For LLM judges, backbone beats pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000314,"raw_usage":{"total_tokens":1797,"prompt_tokens":977,"completion_tokens":820,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":743}},"tokens_in":593,"tokens_out":820,"duration_ms":56110,"temperature":1.0,"reasoning_tokens":743,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:17.743830+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an agent with a fifth reward signal that scores high on F1 but is deliberately biased toward one agent class, and check whether ranking fidelity or RL success still tracks F1; if a low-F1, high-precision judge outperforms a high-F1 judge in training, the paper's ordering claim collapses. A lighter check: add another reward condition to the RL study and see whether the easy-set success-rate ordering still follows judge accuracy.","supporting_citations":[],"review_version":1}