{"id":"4021833f-b323-40e7-9d83-6d72b10830a3","arxiv_id":"2505.19095","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A VLM trained with GRPO and a world-model curiosity reward explores a real desktop GUI more diversely than larger frozen models, but the diversity metric is nearly identical to its training reward.","lead":"ScreenExplorer trains a small vision-language model with reinforcement learning to explore a real Linux desktop, using a world model to reward curiosity and distilling past experience back into the model. The paper claims this helps a 3-billion-parameter model explore more broadly than much larger static models, which matters for building agents that improve from their own experience.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation metric is the training objective: reported exploration gains may be by construction, not validated by independent measures.","rationale":"The reader's weakest assumption is correct but can be sharpened: the problem is not only that cosine similarity is unvalidated; it is that the evaluation metrics in Eqs. (5)-(6) are nearly identical to the reward terms in Eq. (1). RL training thus optimizes the exact quantity later reported as evidence of exploration, so the headline comparison against frozen baselines largely measures reward optimization rather than exploration quality. My proposed test would settle this by measuring diversity through OS-level state coverage and human judgment, neither of which appears in the training reward. If the advantage persists on those independent metrics, the central claim survives; if not, the paper's main evidence is an artifact of metric-reward alignment. I would keep the reader's REJECT verdict because the current submission does not provide the required independent validation.","tokens_in":19614,"tokens_out":3915,"duration_ms":33708,"concrete_test":"Re-run the evaluation with an independent, non-embedding state metric. Instrument the same environment to log OS-level state at every step (active window title, open application, focused URL/domain, and a canonical state hash from the accessibility tree). Compute trajectory- and group-level diversity as the number/proportion of distinct OS-level states visited, and compare ScreenExplorer-3B-E1/-Distill against all baselines under the identical 20x10 protocol with 95% CIs. Also have two independent annotators rate a sample of 50 trajectories per model for 'meaningful exploration' (distinct applications/websites/tasks, not repeated clicks on news/video). If the trained model does not exceed the best baselines on these independent metrics, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that RL training produces a GUI agent with better 'diverse exploration' than frozen baselines. The only evidence is Table 1, computed from Eqs. (5)-(6): average pairwise cosine dissimilarity of VAE image embeddings and OCR text embeddings across states. But the reward in Eq. (1) contains the same quantities: rvis_seq and rtext_seq are exactly within-trajectory average pairwise dissimilarities of post-action embeddings, and the full reward also includes instantaneous dissimilarity and world-model prediction dissimilarity in the same embedding spaces. GRPO therefore directly maximizes the metric used for evaluation; the improvement from 0.21/0.17 to 0.53/0.54 can be a reward-hacking artifact rather than evidence of meaningful exploration. This is not merely 'unvalidated similarity': even if cosine similarity were a good diversity measure, the evaluation is not independent of the training signal. The paper's own Section 5 admits agents get stuck on perpetually novel stimuli (news/video pages), which the change rewards are exactly designed to incentivize. No human judgment, OS-level state coverage, or held-out metric is reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ScreenExplorer, a 3B-parameter vision-language model trained with GRPO in a real Linux GUI environment. The training reward combines a format reward, visual and textual state-change rewards, a world-model prediction-error curiosity reward, and intent-state alignment terms (Eq. (1)). After RL, the authors also distill the collected experience streams via SFT. The central claim is that this framework improves diverse exploration in open GUI environments, supported by Table 1, which reports trajectory-level and group-level diversity computed from cosine similarity of VAE image embeddings and OCR text embeddings (Eqs. (5)-(6)), plus qualitative case studies and ablations.","tokens_in":19807,"tokens_out":4628,"duration_ms":44880,"significance":"If the central claim were established, the contribution would be notable: it would show that a small 3B VLM, trained with a world-model curiosity reward and experience-stream distillation in a live GUI environment, can explore more diversely than much larger frozen models. The paper also ships source code, builds a real GUI RL environment, and includes ablation studies on reward components, which are concrete strengths. However, the quantitative evidence for the central claim is currently compromised because the evaluation metrics in Section 4.1 use the same cosine-similarity definitions that appear in the training reward in Eq. (1). The reported exploration gains may therefore reflect optimization of the training objective rather than meaningful exploration, so the significance of the claimed result is not yet established.","major_comments":[{"comment":"The evaluation metrics d_vis_seq, d_text_seq, D_vis_grp, and D_text_grp are average pairwise cosine dissimilarities of the same VAE image embeddings and OCR text embeddings that appear in the reward terms r_vis_seq and r_text_seq. Because GRPO directly maximizes Eq. (1), the improvement in Table 1 from average diversity 0.21/0.17 to 0.53/0.54 is partially by construction and does not by itself demonstrate meaningful exploration. Please report independent evaluation metrics, such as OS-level state coverage, number of distinct applications or windows visited, human diversity ratings, or held-out exploration tasks, and provide error bars over multiple seeds.","section":"Section 3.2, Eq. (1) and Section 4.1, Eqs. (5)-(6)"},{"comment":"The main comparison in Table 1 is against frozen, statically deployed models; the only RL-trained control appears in the ablation study of Section 4.4, but that control is not included in Table 1. Without a baseline trained under the same RL loop with, for example, only the format reward, the reported gains cannot be attributed to the exploration rewards rather than to RL in general or to reward hacking of the similarity metric.","section":"Table 1 and Section 4.4"},{"comment":"The paper admits that agents get stuck on perpetually novel stimuli such as news and video pages, which is exactly the behavior incentivized by reward terms that maximize pairwise dissimilarity of screen embeddings. This admission means the high d_seq and D_grp scores in Table 1 may reflect trivial novelty rather than useful exploration breadth. The paper should quantify the distribution of visited applications, domains, and action types, and separate useful exploration from the noisy-TV trap.","section":"Section 5"},{"comment":"The evaluation is based on only 20 episodes of 10 steps each (200 frames) per model, and no standard deviations, confidence intervals, or significance tests are reported. Given the high variance of GUI exploration trajectories, single-run point estimates are not sufficient to support the claim that trained models outperform baselines across the board.","section":"Section 4.2, Table 1"}],"minor_comments":[{"comment":"The text refers to 'Equation 3.2' when discussing format reward penalties; the correct reference is Eq. (1).","section":"Section 4.3"},{"comment":"The caption says 'All 9 Rewards' but the list in Section 3.2 contains six bullet categories; please clarify how the nine individual reward terms map to the six categories.","section":"Figure 2 caption"},{"comment":"The caption mentions 'box indicates the lowest scores,' but no boxes are visible in the rendered table; please check the formatting or remove the reference.","section":"Table 1"},{"comment":"The 'Avg. Diversity' column is not defined; please state explicitly that it is the mean of the four preceding diversity columns.","section":"Table 1"},{"comment":"The GRPO advantage formula uses group normalization but the notation omits the group index in the mean and standard deviation; please clarify the grouping notation.","section":"Section 3.4, Eq. (3)"},{"comment":"The claim that a high and oscillating world-model loss 'indicates sustained curiosity' is not self-evident; a persistently high reconstruction loss could also mean the world model is failing to learn, which should be discussed.","section":"Section 4.3 and Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The core issue is that the evaluation metric is effectively the training objective, so the headline numbers in Table 1 are not independently meaningful. The paper has promising components—real-environment RL training, a world-model reward, and experience-stream distillation—but the current evidence does not support the abstract's claim of enhanced exploration. I would be willing to reconsider after a major revision that adds independent diversity metrics, RL-trained baseline controls, and multiple-seed statistics; without such additions, the central claim remains unsubstantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe real news here is the training setup, not the result as reported. The team built an actual Linux GUI environment and trained a 3B VLM with GRPO, a world-model curiosity reward, and experience-stream distillation. That is a genuinely useful infrastructure contribution, and the qualitative case studies show the model progressing from near-random clicks to opening applications and navigating web pages. The ablation suggesting the world model helps overcome cold-start is plausible and worth taking seriously.\n\nThat said, the central claim—that training improves diverse exploration—is not supported by the evidence. The evaluation metrics in Section 4.1 are the same cosine-similarity distances that appear in the reward function of Eq. (1). The model is rewarded via GRPO to maximize pairwise dissimilarity of VAE/OCR embeddings, and then scored on that same dissimilarity. The jump from 0.21/0.17 to 0.53/0.54 in Table 1 is therefore at least partly by construction; it is not an independent measure of meaningful exploration. The paper also lacks any RL-trained baseline with a different reward, reports no error bars, and samples only 20 short episodes per setting. And the paper's own Section 5 admits the agents get stuck on perpetually novel stimuli—exactly what the change rewards are designed to incentivize. Without human judgment, OS-level state coverage, or a held-out task-oriented metric, we cannot separate genuine exploration from reward hacking.\n\nThe method deserves engagement, though. The evaluation flaw is fixable: compare against GRPO-trained baselines without curiosity rewards, add a task-completion benchmark, and use state coverage or human annotation. The authors also honestly discuss limitations, which is a point in their favor.\n\nMy recommendation: send this to peer review, but with the expectation that the evaluation will need to be redone. It is not ready for acceptance as is. For my own work, I would not cite it as evidence of improved exploration, but I would bring it to a reading group to discuss evaluation transparency in RL.\n\nBest,\n[Your name]","headline":"A real-world RL training pipeline for GUI agents with a serious evaluation flaw: the diversity metric is the training objective.","tokens_in":20306,"tokens_out":2953,"would_cite":false,"duration_ms":27282,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ScreenExplorer trains a 3B vision-language model to explore GUI environments more diversely than much larger static models, using a curiosity reward from a learned world model and experience stream distillation.","keywords":["GUI agents","Vision-language model","Reinforcement learning","Exploration","World model","Curiosity-driven reward","Experience stream distillation","GRPO"],"falsifier":"A direct test is to evaluate the same models on diverse downstream tasks (e.g., file navigation, web search, software web-based tasks) and compare task success rates with the reported diversity metrics. If the cosine-similarity-based diversity scores do not correlate positively with meaningful task completion or human preference, the equivalence between these metrics and true exploration would be broken.","tokens_in":19415,"feed_emoji":"🖥️","tokens_out":2335,"duration_ms":22041,"temperature":0.7,"pith_summary":"ScreenExplorer claims that a small vision-language model can learn to explore a real, open-ended desktop environment more diversely than much larger frozen models if it is trained online with reinforcement learning in that environment. The paper builds a GUI environment, rewards the agent for both valid interactions and state novelty, and pairs GRPO training with world-model curiosity and distillation of past exploration experience. A 3B-parameter model improved its average diversity score from 0.21 to 0.53, surpassing static deployments of gpt-4o, Qwen2.5-VL-72B, OpenAI Computer Use, and a 3B base. The value of the claim lies in a scalable path toward self-improving agents that gradually need less human-labeled data.","feed_headline":"ScreenExplorer: a 3B model out-explores much larger static ones in GUI environments","feed_subtitle":"A world-model curiosity reward and experience distillation push exploration diversity up after online RL.","key_machinery":"A LLaMA-style world model with a VAE image tokenizer and text embeddings predict the next GUI state from the current state and chosen action. The world model's prediction discrepancy forms the core curiosity reward, combined with additional rewards for formatted actions, instantaneous and subsequent state changes, environment description, and intent interpretation. This reward design is paired with GRPO for policy optimization, then followed by experience stream distillation with manual or automated filtering to fine-tune the base model for the next generation.","core_discovery":"The paper's central claim is that a 3B-parameter VLM, trained in a real open GUI desktop environment with GRPO, a world-model curiosity reward, and experience stream distillation, develops better environmental adaptation and sustained exploration than static deployment models. Training rewards diverse, novel state changes and meaningful interaction, and the results show a notable jump in diversity scores versus the base Qwen2.5-VL-3B and versus larger static models such as gpt-4o, Qwen2.5-VL-72B, and GUI-specific agents like OpenAI Computer Use. The paper also finds that the world model's predicted-state discrepancy is the most critical reward component, accelerating effective exploration during the cold-start phase and smoothing gradient differentiation in GRPO.","pith_inferences":["The underlying mechanism might transfer to other open-ended environments beyond GUI, such as 3D worlds or web browsing, where a world model can predict next-state embeddings and quantify novelty from prediction error.","A testable extension is to compare the cosine-similarity-based diversity scores with human judgments or task-completion statistics, to see whether the reported diversity improvements align with truly meaningful exploration.","The paper's argument implies a major shift: for exploration-focused open environments, a small trainable model can outperform large frozen ones, which casts doubt on the assumption that raw scale alone is the main bottleneck in such settings."],"forward_implications":["A 3B model can match or exceed exploration diversity of much larger models in a real desktop environment when trained online with a curiosity-driven reward and RL.","World-model curiosity is a key ingredient for overcoming the cold-start exploration phase in open GUI environments.","Self-collected exploration experience, without additional labels, can be distilled to improve exploration capabilities and sustain improvement.","The training and distillation cycle offers a general recipe for reducing reliance on carefully curated human-annotated data in GUI agent development."],"supporting_citations":[{"why":"Provides the GRPO algorithm used for policy optimization in the screen-exploration setting.","marker":"[37]"},{"why":"Inspires the experience stream distillation method that reuses exploration experience for future fine-tuning.","marker":"[16]"},{"why":"Inspires the world-model curiosity mechanism in the exploration reward.","marker":"[17]"},{"why":"Inspires the world-model-based intrinsic reward for exploration in open-world environments.","marker":"[23]"},{"why":"Accelerates VLM decoding during RGB sampling and training in the GUI environment.","marker":"[40]"}],"fun_headline_variants":["3B VLM out-explores 72B models in open GUI worlds","Curiosity reward makes 3B agent explore GUI better than 72B","World-model curiosity beats cold-start in GUI exploration","GRPO + world-model curiosity drives diverse GUI exploration in 3B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that cosine similarity between VAE image embeddings and OCR text embeddings measures meaningful exploration diversity is load-bearing: it is used both inside the reward function and inside the evaluation metrics, and the paper does not validate it against human judgment or an external benchmark.","fun_headline_variants_meta":{"raw":{"variants":["3B VLM out-explores 72B models in open GUI worlds","Curiosity reward makes 3B agent explore GUI better than 72B","World-model curiosity beats cold-start in GUI exploration","GRPO + world-model curiosity drives diverse GUI exploration in 3B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001958,"raw_usage":{"total_tokens":7610,"prompt_tokens":856,"completion_tokens":6754,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":6678}},"tokens_in":472,"tokens_out":6754,"duration_ms":40698,"temperature":1.0,"reasoning_tokens":6678,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:20:06.913836+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to evaluate the same models on diverse downstream tasks (e.g., file navigation, web search, software web-based tasks) and compare task success rates with the reported diversity metrics. If the cosine-similarity-based diversity scores do not correlate positively with meaningful task completion or human preference, the equivalence between these metrics and true exploration would be broken.","supporting_citations":[{"cited_title":"Exploration by random network distillation, 2018","cited_arxiv_id":null,"evidence_quote":"Inspires the world-model curiosity mechanism in the exploration reward."},{"cited_title":"Open-world reinforcement learning over long short-term imagination, 2025","cited_arxiv_id":null,"evidence_quote":"Inspires the world-model-based intrinsic reward for exploration in open-world environments."},{"cited_title":"Space\") Key(","cited_arxiv_id":null,"evidence_quote":"Accelerates VLM decoding during RGB sampling and training in the GUI environment."}],"review_version":1}