{"id":"973b0d88-f53d-4a56-9e52-65ec7ab38edd","arxiv_id":"2412.10047","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A four-phase training pipeline converts a 7B language model into a Windows GUI action model that reaches 81.2% offline and 71.0% online task success on the authors' Word test set, beating text-only GPT-4o.","lead":"Microsoft researchers describe a step-by-step recipe for turning a language model into an action-taking computer agent that operates Windows applications, using Microsoft Word as a test case. It is a practical blueprint for building GUI agents, though its headline result, that the trained 7B model beats GPT-4o, rests on GPT-4o-generated data and GPT-4o-based evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set contamination: §7.1 evaluates on the same 435 tasks used during training, and §§4.3–4.4 never exclude those IDs from self-boosting or PPO pools; without a task-level split audit, the reported TSR advantage is unverifiable.","rationale":"I agree with the reader that the conditional verdict is appropriate. The paper's central empirical claim is a comparison of task completion rates, and the integrity of that comparison hinges on the test set being truly held out. The internal text is ambiguous: Section 7.1's phrase 'used during LAM training' is at best sloppy and at worst an admission of leakage; in either case, the paper never documents an exclusion filter for the failure pools used in Phases 3 and 4. Without a task-ID-level audit, any reviewer must treat the reported superiority as unverified. The proposed test is cheap and decisive: intersect task IDs across phases, and if overlap exists, recompute metrics on the clean subset. I do not see internal inconsistency in the four-phase pipeline itself; the concern is about the measurement, not the mechanism. The paper does release data-collection code and detailed prompts, which is credit-worthy, but that does not resolve the split ambiguity. Therefore the verdict remains CONDITIONAL: accept only after the split audit and, ideally, a human-label agreement check on a subsample. This is the same conclusion as the reader, so no verdict change is needed.","tokens_in":34107,"tokens_out":6987,"duration_ms":66800,"concrete_test":"Extract all task IDs from the 435-test split (Section 4.2) and intersect them with the task IDs in: (a) the 2,284 GPT-4o failed tasks used in Phase 3, (b) the 496 LAM-success trajectories added in Phase 3, (c) the 1,788 failure trajectories used in Phase 4 offline PPO, and (d) the 4,476 RM training trajectories. If the intersection is non-empty, re-run Tables 3 and 4 on the subset of test tasks with zero training-phase overlap; the reported TSR gaps (81.2 vs 67.2 and 71.0 vs 63.0) must persist on that clean subset. If the intersection is empty, publish the per-task split as a supplement to confirm. Additionally, to address the GPT-4o-judge issue, have a human annotator independently label 50 randomly sampled online trajectories from Table 4 as success/failure and compute agreement with GPT-4o's labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LAM4 beats GPT-4o on Windows GUI tasks, with offline TSR 81.2% vs 67.2% (Table 3) and online TSR 71.0% vs 63.0% (Table 4). The validity of these numbers depends entirely on the 435 test tasks being unseen during Phases 3 and 4. Section 4.2 splits 2,192 GPT-4o trajectories into 1,757 training and 435 test trajectories, but Section 7.1 says online evaluation uses 'the same set of 435 test requests used during LAM training,' and the templates are 'identical to the training setup.' More importantly, §4.3 samples 2,284 GPT-4o failed tasks and §4.4 uses 1,788 LAM failure trajectories for offline PPO and 4,476 trajectories for RM training; the paper never states that the 435 test task IDs were excluded from these failure pools. If any test task appears in the Phase 3 successes or Phase 4 rewards, the model was trained on the test set, and both Table 3 and Table 4 overstate generalization. The reader's concern is therefore the most load-bearing: it attacks the measurement of the central claim directly, not a peripheral assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a four-phase pipeline for converting Mistral-7B into a Windows GUI agent (LAM4), consisting of task-plan pretraining, supervised fine-tuning on GPT-4o trajectories, self-boosting exploration on GPT-4o failure cases, and offline PPO with a learned reward model. Using Microsoft Word as a testbed and the UFO agent for grounding, the authors report offline TSR of 81.2% versus 67.2% for GPT-4o (Table 3) and online TSR of 71.0% versus 63.0% for text-only GPT-4o (Table 4), with substantially lower latency. The paper also positions the workflow as a generalizable blueprint for LAM development and releases the data collection code.","tokens_in":34345,"tokens_out":6247,"duration_ms":64116,"significance":"The staged data collection and training recipe, detailed prompts, and public dataflow code are useful engineering contributions, and the paper is clearly organized. The headline result—a 7B text-only model beating GPT-4o on Windows GUI task completion with roughly 2–3x lower latency—would be practically significant if the evaluation is valid. However, the current evidence does not establish that result, because the test split may have been used in Phases 3 and 4 and because GPT-4o serves as data generator, filter, and judge throughout. The manuscript's strengths are its explicit phasewise data counts, the concrete description of grounding in the UFO agent, and the reproducibility of the data collection pipeline.","major_comments":[{"comment":"The test set appears to contaminate Phases 3 and 4. Section 4.2 splits the 2,192 GPT-4o successes into 1,757 training and 435 test trajectories, but Section 4.3 states that LAM3 is fine-tuned on the original 2,192 GPT-4o successes plus 496 new successes, and Section 4.4.1 trains the reward model on GPT-4o's successful trajectories (2,192) as well as 1,788 LAM failures. Section 7.1 then says online evaluation uses the same set of 435 test requests used during LAM training. Unless the authors provide a task-level audit showing that the 435 test task IDs were excluded from the Phase 3 training pool, the Phase 3 failure pool, and the Phase 4 RM/PPO pools, Tables 3 and 4 can be explained by memorization rather than generalization. This is the most load-bearing issue and must be fixed by re-running Phases 3 and 4 with an explicit held-out split and reporting the resulting numbers.","section":"§4.2, §4.3, §4.4, §7.1"},{"comment":"GPT-4o is used both to generate the data the model is trained on and to judge whether tasks are successfully completed. Specifically, GPT-4o constructs task-plan data (Section 3.1.3), instantiates task-action trajectories (Section 3.2.1), filters which execution trajectories are retained (Section 3.2.3), computes the planning metrics for LAM1 (Section 5.2.1), and determines online task success (Section 7.4). Because LAM is trained to imitate GPT-4o's style and then evaluated by a GPT-4o judge, the judge may systematically favor LAM's outputs over GPT-4o's own zero-shot outputs. The authors should validate at least a sample of the online evaluations with human judgment or with an independent deterministic success criterion (e.g., final document state), and report agreement or disagreement with the GPT-4o judge.","section":"§3.1.3, §3.2.1, §3.2.3, §5.2.1, §7.4"},{"comment":"All headline comparisons are single point estimates over 435 tasks with no confidence intervals, significance tests, or multiple runs. Given the relatively small test set and the additional variance in online execution, the claim that LAM4 'outperforms' GPT-4o requires more than point estimates; please report bootstrap confidence intervals or per-task breakdowns for the TSR differences in Tables 3 and 4.","section":"§5.3, §7.4 (Tables 3 and 4)"},{"comment":"The offline decision-making metrics compare predicted actions against a ground-truth action history that was itself generated by GPT-4o. This exact-match style of evaluation penalizes valid alternative solutions and therefore favors models fine-tuned on that exact distribution. The offline TSR should be interpreted as a measure of imitation fidelity rather than task-completion quality, and this distinction should be stated explicitly in the paper.","section":"§5.3.1"}],"minor_comments":[{"comment":"The caption says 'two-phrase data collection'; this should be 'two-phase data collection.'","section":"Figure 4 caption"},{"comment":"The data augmentation numbers are not fully consistent: 29,182 original pairs plus 47,490 evolved pairs gives 76,672, but 'increased by 150%' suggests 43,773 additional pairs; please clarify the exact augmentation ratio and how the 47,490 figure is derived.","section":"§3.1.4"},{"comment":"The sentence beginning 'we first The reward model' is incomplete and should be rewritten.","section":"§4.4"},{"comment":"The heading says 'The instantiation prompt used in the evaluation phase,' but the prompt is the evaluation prompt, not the instantiation prompt; please correct the heading.","section":"Appendix B.2 heading"},{"comment":"The evaluation in Table 2 is reported on 15,334 tasks, which is not reconciled with the 76,672 training pairs described in Section 3.1.4; please state the split and whether any evolution-derived tasks appear in the evaluation set.","section":"§5.2.2 and Table 2"}],"recommendation":"major_revision","confidential_remarks":"The test-contamination issue is serious but potentially fixable: if the authors can show a clean task-level split and re-run Phases 3 and 4 with the 435 test IDs excluded, the paper could become publishable. If they cannot, or if the re-run changes the headline results, the paper should not be accepted. I would also recommend that the editor require at least a small human-validated subset of the online evaluation before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real engineering contribution with a serious evaluation flaw. The four-phase pipeline—plan SFT, expert imitation, self-boosting from GPT-4o failures, and RM-based offline PPO—is a sensible, well-described way to turn a 7B Mistral into a Windows GUI agent, and the authors ship the data collection code. That part is solid.\n\nThe weak point is the measurement of the headline claim. Section 7.1 says online evaluation uses the same 435 test requests used during LAM training, with templates identical to the training setup. The paper never states that those 435 task IDs were excluded from Phase 3 self-boosting or Phase 4 RM/PPO pools. If any of them appear in those pools, both Table 3 and Table 4 overstate generalization. The stress-test note is right: this is load-bearing, not a peripheral worry.\n\nSeparately, GPT-4o is the data generator, the success judge during data collection, the planning evaluator, and the online success judge, so the model is trained and scored against one model's notion of correctness. That is not automatically fatal—a strong teacher can be a reasonable proxy—but combined with no error bars and no external benchmark, the reported 81.2% versus 67.2% TSR gap is not established.\n\nWhat is genuinely new: the instantiate-execute-evaluate loop that converts task-plan data into executable action trajectories with minimal human labor, and the staged protocol that deliberately mines GPT-4o failures for self-boosting. The components are known; the integration for desktop GUI agents is not, and the code release makes the recipe usable.\n\nMinor weaknesses worth naming: the offline decision metrics compare predicted objects, operations, and status against a GPT-4o-generated trajectory, so they measure imitation fidelity more than true task success. The planning evaluation in Table 2 uses GPT-4o as a soft judge, which is fine. The online latency comparison is slightly apples-to-oranges because LAM is text-only with UIA control information while GPT-4o with visuals receives screenshots, but the text-only GPT-4o comparison is fairer and LAM still wins.\n\nWho this is for: practitioners building GUI agents or action models. The paper is a useful blueprint, and the method deserves serious referee time, but the evaluation needs an explicit task-level split audit, a subset of human-verified success labels, and ideally a run against OSWorld or WindowsArena. As written, cite it for the pipeline, not for the performance numbers.","headline":"A genuinely useful engineering recipe for turning a 7B model into a Windows GUI agent, but the headline performance claim rests on an unverified test-set split and a circular GPT-4o-as-teacher-and-judge setup.","tokens_in":725,"tokens_out":1066,"would_cite":true,"duration_ms":30682,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A four-phase training pipeline converts a 7B language model into a Large Action Model that the paper reports surpasses GPT-4o on Windows GUI task completion, reaching 81.2% offline task success versus 67.2% for GPT-4o, and 71.0% online…","keywords":["Large Action Models","GUI agents","supervised fine-tuning","self-boosting exploration","reward model","offline reinforcement learning","Windows OS automation","task-plan data"],"falsifier":"Re-run the offline and online evaluation on a disjoint, human-annotated set of Windows GUI tasks never touched by any training phase, scoring success by document-state verification such as XML diff rather than GPT-4o judgment; if the reported gap over text-only GPT-4o disappears or inverts, the paper's core claim is refuted.","tokens_in":33861,"feed_emoji":"🤖","tokens_out":4653,"duration_ms":49400,"temperature":0.7,"pith_summary":"This paper argues that a general-purpose language model can be turned into a Large Action Model—a model that emits executable actions in a real environment—through a carefully staged training pipeline. Using a 7-billion-parameter Mistral model fine-tuned for Windows GUI tasks, it reports that the resulting LAM4 completes 81.2% of offline test tasks, ahead of GPT-4o's 67.2%, and 71.0% of online text-only tasks, ahead of GPT-4o's 63.0%, while also finishing tasks in about a third of the time. The authors frame the contribution as a generalizable blueprint: task-plan pretraining, imitation of expert trajectories, self-boosting exploration on tasks the expert failed, and reward-model-based policy optimization. A sympathetic reader would take the central point to be that a small specialized action model can outperform much larger general-purpose LLMs within a constrained environment, and that the same workflow can be ported to other domains.","feed_headline":"Four-stage pipeline lifts a 7B model past GPT-4o on Windows GUI tasks","feed_subtitle":"Offline task success reaches 81.2 percent versus 67.2 percent for GPT-4o, with text-only online wins at 71.0 percent.","key_machinery":"The central object is the four-phase LAM training pipeline. Phase 1, task-plan pretraining, uses supervised fine-tuning on task-plan pairs drawn from documentation, WikiHow, search queries, and evolved data to teach the model to decompose requests into structured plans. Phase 2, learning from experts, fine-tunes on GPT-4o-generated action trajectories where each step maps a UI state to a concrete action, grounding the plans in an executable form. Phase 3, self-boosting exploration, uses the ReAct mechanism to let the model attempt tasks that GPT-4o failed, collecting its own successful trajectories as new training data. Phase 4, learning from a reward model, trains a scalar reward model with uniform +1/−1 labels on successful and failed trajectories and uses offline PPO to optimize the policy on failure trajectories. This staged progression—from planning, to imitation, to self-exploration, to reward-based refinement—is what carries the argument, along with the automated data-collection pipeline that converts abstract task-plan data into concrete, execution-validated action sequences.","core_discovery":"The paper's central claim is that its four-phase training recipe yields a specialized action model whose performance exceeds the teacher model, GPT-4o, that supplied most of its training data, in both offline and online evaluation. The phase-wise trajectory is concrete: LAM1, trained only on 76,672 task-plan pairs, reaches 82.2% planning task success; LAM2, which imitates 2,192 GPT-4o-generated action trajectories, jumps to 76.8% decision-making task success; LAM3, which adds 496 self-generated successes from tasks GPT-4o failed, reaches 79.3%; and LAM4, refined with a reward model and offline PPO on 1,788 failure trajectories, reaches 81.2%. Online, LAM's text-only task success rate of 71.0% surpasses GPT-4o's text-only 63.0%, while GPT-4o with visual input reaches 75.5% but takes 96.48 seconds versus LAM's 30.42 seconds. The paper presents this as evidence that a domain-specialized LAM trained from scratch can be both more accurate and far more efficient than a state-of-the-art generalist model, and that the workflow transfers beyond the Windows/Word case study.","pith_inferences":["A testable consequence of the pipeline's reliance on environment-specific action grounding is that performance will likely drop more sharply on Windows applications not seen in training (e.g., Excel or PowerPoint) than on unseen Word tasks; the paper reports no cross-application generalization results.","Because the 435 online test tasks are the same set used for training, the reported online TSR of 71.0% is best read as an upper bound; a held-out task set scored by human verification or document-state checks would give a cleaner, likely smaller, estimate of the pipeline's true benefit.","The reward model assigns the same +1 to every step of a successful trajectory and −1 to every step of a failed one, ignoring intermediate progress; a step-level or progress-aware reward could plausibly extract more from the failure trajectories than the reported 1.9-point gain.","The efficiency edge of a text-only, 7B-parameter LAM suggests the approach could extend to resource-constrained or edge deployment, but the pipeline's dependence on GPT-4o for data annotation and evaluation leaves open how it would bootstrap in a domain without a strong teacher model."],"forward_implications":["A 7-billion-parameter specialized model can outperform a much larger generalist LLM on bounded GUI tasks, both in task success rate and in execution speed, when trained with environment-grounded data.","The jump from 76.8% to 79.3% offline task success shows that a model can learn from its own successes on tasks its teacher failed, making self-boosting a viable data source beyond expert demonstrations.","Adding reward-model-guided offline RL on failure trajectories yields a further improvement to 81.2%, indicating that mistakes, when properly scored, are useful training signal rather than noise to discard.","The authors claim the same data-collection and training workflow carries over to other environments and applications, offering a template for building LAMs beyond the Windows and Word testbed."],"supporting_citations":[{"why":"Mistral-7B is the base model that the four-phase training pipeline transforms into a Large Action Model.","marker":"[25]"},{"why":"ReAct is the mechanism used in Phase 3 self-boosting exploration to let the model attempt and solve tasks GPT-4o failed.","marker":"[80]"},{"why":"Proximal Policy Optimization (PPO) is the offline RL algorithm used in Phase 4 with the reward model to refine policy on failure trajectories.","marker":"[56]"},{"why":"UFO is the GUI agent framework into which the trained LAM is integrated and grounded for Windows OS interaction.","marker":"[83]"},{"why":"WizardLM's approach to data evolution is adapted to generate more complex task-plan training data.","marker":"[74]"},{"why":"AgentGen's data evolution method is adapted to enrich the task-plan dataset with additional complexity and constraints.","marker":"[23]"}],"fun_headline_variants":["Four-phase LAM recipe hits 81.2% offline, beats GPT-4o","Specialized LAM surpasses GPT-4o accuracy and speed","81.2% offline success: LAM outperforms GPT-4o","Windows agent trained in four phases beats GPT-4o","LAM beats GPT-4o with 71% online text-only rate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the same 435 test requests used during training as the online test set and relies on GPT-4o to judge whether tasks succeeded, so the claim that LAM4 beats GPT-4o depends on those test tasks not leaking into the self-boosting and reward-model training data and on the judge being unbiased.","fun_headline_variants_meta":{"raw":{"variants":["Four-phase LAM recipe hits 81.2% offline, beats GPT-4o","Specialized LAM surpasses GPT-4o accuracy and speed","81.2% offline success: LAM outperforms GPT-4o","Windows agent trained in four phases beats GPT-4o","LAM beats GPT-4o with 71% online text-only rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000777,"raw_usage":{"total_tokens":3519,"prompt_tokens":1112,"completion_tokens":2407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":728,"completion_tokens_details":{"reasoning_tokens":2309}},"tokens_in":728,"tokens_out":2407,"duration_ms":19132,"temperature":1.0,"reasoning_tokens":2309,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:24:41.380148+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the offline and online evaluation on a disjoint, human-annotated set of Windows GUI tasks never touched by any training phase, scoring success by document-state verification such as XML diff rather than GPT-4o judgment; if the reported gap over text-only GPT-4o disappears or inverts, the paper's core claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"UFO is the GUI agent framework into which the trained LAM is integrated and grounded for Windows OS interaction."}],"review_version":1}