{"id":"84827ba8-014b-4ce8-8348-49f992a32ecd","arxiv_id":"2411.13451","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adding 1 to 2 human demonstrations improves web agent task success by 3.36 to 7.21 absolute percentage points on Mind2Web and VisualWebArena, though the best adapted agents still succeed less than 25% of the time.","lead":"The paper tests whether giving web agents one or two human-demonstrated task examples before they tackle a new website improves their success rate. On two standard benchmarks, the best agents still complete under 25% of tasks, but the demonstrations add a few percentage points of success, which is a useful but limited gain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Open-weights gain is confounded by unequal adaptation protocol: CogAgent-FOMAML receives 2 target-site demonstrations before evaluation while CogAgent-FT and CogAgent-FT (DE) do not, so the gap may be plain fine-tuning rather than meta-learning.","rationale":"The reader's weakest assumption identifies exactly the load-bearing flaw: the open-weights comparison is not between a meta-learned model and a conventionally trained model receiving the same target-site adaptation, so the improvement cannot be attributed to meta-learning. The reader's REJECT verdict is appropriate for the current version because the central two-part claim is only partially supported: the in-context learning results for SeeAct are direct and plausible, but the meta-learning results for CogAgent rest on an unequal protocol. I agree with the reader's framing and would not soften the verdict without the missing control. Secondary issues, such as omitted error bars and the unpublished Mind2Web cross-task amendment, also weaken comparability but are not necessary for the rejection. The core fix is straightforward: add a conventional fine-tuning baseline that receives the same 2 demonstration tasks before evaluation. If that baseline matches CogAgent-FOMAML, the paper should be reframed as showing that target-site fine-tuning helps, not that meta-learning provides a superior prior. The rationale for keeping the verdict at REJECT rather than CONDITIONAL is that the central empirical claim, as written, is not adequately supported and would require new experiments rather than a reanalysis of existing tables.","tokens_in":19623,"tokens_out":4186,"duration_ms":46946,"concrete_test":"Run the missing control: take the CogAgent-FT and CogAgent-FT (DE) checkpoints and fine-tune each on the same 2 target-site or target-domain tasks used to adapt CogAgent-FOMAML, using the same LoRA hyperparameters (rank 20, learning rate 1e-5, one inner gradient step per task), then evaluate on the remaining tasks. Repeat across the same 5 random task selections and compare overall success rate in the cross-website and cross-domain settings. If adapted CogAgent-FT or CogAgent-FT (DE) matches CogAgent-FOMAML within noise, the reported gains are due to the target-site demonstrations rather than to meta-learning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim for open-weights models is that meta-learning (CogAgent-FOMAML) yields a better prior for few-shot adaptation than conventional fine-tuning. Section 4 states that after meta-learning, CogAgent-FOMAML is adapted to each new website or domain by fine-tuning on 2 tasks from that target website/domain, then evaluated on remaining tasks. In contrast, CogAgent-FT and CogAgent-FT (DE) receive no target-site adaptation before evaluation. Table 1 then compares the adapted FOMAML model against these unadapted baselines and attributes the improvement to the meta-learned prior. That attribution is unsupported: any model, meta-learned or not, could improve from fine-tuning on the same 2 target-site demonstrations. The missing control is the same base model, or CogAgent-FT/CogAgent-FT (DE), fine-tuned on the same 2 demonstration tasks and then evaluated. Without it, the cross-website and cross-domain gains for CogAgent-FOMAML could be explained entirely by the target-site adaptation data, not by FOMAML. This is not merely a nuance: on the cross-task split, where no target adaptation is applied, CogAgent-FOMAML (11.89% overall SR) does not beat CogAgent-FT (13.36%), indicating that the meta-learned prior alone is not demonstrably superior. The proprietary ICL results are not affected by this confound, but the abstract and conclusions bundle both model classes into the headline claim, overstating the evidence for meta-learning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdaptAgent, a framework for adapting multimodal web agents with few-shot human demonstrations. For proprietary models (SeeAct with GPT-4o), it adds one multimodal in-context demonstration to the prompt; for open-weights models (CogAgent), it meta-trains with FOMAML and then adapts on two demonstrations from the target website or domain. The authors evaluate on Mind2Web and VisualWebArena, reporting absolute gains of 3.36 to 7.21 percentage points in task success rate over non-adapted baselines, and they provide ablations on demonstration modality, meta-learning data-selection strategies, and the number of demonstrations.","tokens_in":19910,"tokens_out":4572,"duration_ms":47656,"significance":"If the results hold, the in-context-learning component is a practically useful and inexpensive way to adapt proprietary web agents to new websites and domains, and the multimodal-versus-text demonstration comparison is informative. The paper also provides useful implementation details, including prompts and LoRA hyperparameters. However, the claim that FOMAML meta-learning produces a better starting point than conventional fine-tuning for open-weights agents is not supported by the current evaluation because the comparison is confounded by unequal adaptation protocols. The central meta-learning contribution therefore needs additional experimental work before the headline claims can be accepted.","major_comments":[{"comment":"The comparison between CogAgent-FOMAML and the CogAgent-FT baselines is confounded by the adaptation protocol. Section 4 states that after meta-learning, CogAgent-FOMAML is fine-tuned on 2 tasks from each target website (or domain) before evaluation, while CogAgent-FT and CogAgent-FT (DE) receive no target-site adaptation. The gains in Table 1 (e.g., cross-website Overall SR 16.96 vs. 12.42) could therefore be caused entirely by the two adaptation demonstrations rather than by the meta-learned prior. The required control is a baseline, ideally CogAgent-FT and CogAgent-FT (DE), fine-tuned on the same two target-site tasks and then evaluated on the remaining tasks. This is not a minor nuance: on the cross-task split, where no target adaptation is applied, CogAgent-FOMAML (11.89) does not beat CogAgent-FT (13.36), so the claim that the meta-learned prior itself is superior to conventional fine-tuning is unsupported.","section":"Section 4, Table 1"},{"comment":"The statement that CogAgent-FOMAML outperforms CogAgent-FT 'across all tests except for Mind2Web cross-task' is misleading because every non-cross-task comparison includes the two target-site adaptation demonstrations for FOMAML but none for FT. The only unconfounded evidence about the learned prior is the cross-task row, where FOMAML is worse. The data-selection-strategy analysis in Table 3 has the same issue: all FOMAML variants receive adaptation data on the target website or domain, while the FT baselines do not, so the differences among intra-website, inter-website, and hybrid strategies cannot be attributed to meta-learning without the matching fine-tuned-adaptation control.","section":"Section 5, footnote 2 and Table 3"}],"minor_comments":[{"comment":"The phrase 'date-equivalence' appears to be a typo for 'data-equivalence'; please correct it.","section":"Section 5, first paragraph"},{"comment":"The sentence 'For VisualWebArena, we adapt the meta-trained model on the Mind2Web training set, using 2 tasks from each of the 3 websites' is confusing; it should presumably say that adaptation is performed on the VisualWebArena websites, not on the Mind2Web training set.","section":"Section 4, Experimental Protocol"},{"comment":"The claim that GPT-4o has 175B parameters is not supported by the cited GPT-4 technical report; the parameter count should be removed or attributed to an official source.","section":"Table 1 footnote"},{"comment":"The visual-difficulty labels are generated by GPT-4o with chain-of-thought and then validated by humans; since the same model family is then evaluated, this is a potential source of bias in the difficulty-stratified analysis and should be acknowledged as a limitation or replaced with independent labels.","section":"Appendix A.5"},{"comment":"The numbers inset in the bars are difficult to read; consider adding a table or larger annotations for readability.","section":"Figure 2 (right)"}],"recommendation":"major_revision","confidential_remarks":"The ICL half of the paper is sound and could stand on its own, but the meta-learning half needs a missing control: fine-tune CogAgent-FT and CogAgent-FT (DE) on the same two target-site demonstrations and then compare. If the control shows no advantage for FOMAML, the paper should be revised to present the ICL result as the main contribution and to describe the meta-learning result as exploratory or negative. The current abstract and conclusions overstate the evidence for meta-learning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The in-context learning half of this paper is legitimately useful. Adding one multimodal demonstration to SeeAct improves overall success rate on both Mind2Web and VisualWebArena, and the multimodal-versus-text ablation is a clean, fair comparison that gives practitioners a concrete recommendation. The data selection ablation for meta-learning is also thoughtful, even if the main meta-learning result doesn't hold up. The paper is honest about the low absolute success rates, and the amended cross-task split is a reasonable attempt to clean up a known benchmark weakness, though it means the numbers aren't directly comparable to prior published results on the original split.\n\nThe load-bearing problem is the open-weights comparison. CogAgent-FOMAML is fine-tuned on two tasks from each target website or domain before evaluation, while the CogAgent-FT baselines receive no such adaptation. So the gap in cross-website and cross-domain success rates could come entirely from having seen two in-distribution examples, not from any meta-learned prior. The cross-task row confirms this: without target-site adaptation, FOMAML (11.89) does not beat plain fine-tuning (13.36). The missing control is simple: fine-tune the same base model, or CogAgent-FT, on the same two demonstration tasks and then evaluate. Until that is done, the abstract's claim that meta-learning boosts open-weights agents is unsupported.\n\nTwo smaller issues. The ICL results have no error bars; the conclusions would be more robust with variance across demonstration selections. And the visual difficulty labeling in Appendix A.5 uses GPT-4o with human validation — that's defensible, but it's a self-referential step and should be flagged more prominently, not buried in an appendix.\n\nWho gets value from this? Practitioners who want a cheap way to adapt a proprietary model like GPT-4o to a new site can take the ICL recipe and run with it. Researchers working on meta-learning for web agents should treat the open-weights results as preliminary and wait for the revised protocol.\n\nThis deserves a serious referee, not a desk reject. The ICL half is publishable; the meta-learning half is fixable with one additional baseline. I'd recommend sending it to review with a clear request for that control, error bars, and a toned-down abstract.","headline":"The ICL results are solid and useful, but the meta-learning claim is confounded by an unequal adaptation protocol and should not be taken as supported in the current draft.","tokens_in":20463,"tokens_out":1778,"would_cite":false,"duration_ms":19249,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Few human demonstrations adapt web agents to new sites, lifting success by up to 7.21 percentage points.","keywords":["web agents","multimodal large language models","few-shot learning","human demonstrations","in-context learning","meta-learning","Mind2Web","VisualWebArena"],"falsifier":"Take CogAgent-FT (DE), the conventionally fine-tuned model trained on the same 292 examples, and give it the identical two-demonstration fine-tuning step on each Mind2Web cross-website and cross-domain test site before evaluation. If it matches or exceeds CogAgent-FOMAML's success rates, the paper's central attribution to meta-learning is unsupported.","tokens_in":19406,"feed_emoji":"🤖","tokens_out":6434,"duration_ms":59917,"temperature":0.7,"pith_summary":"This paper claims that web agents can be adapted to unseen websites and domains with just one or two human demonstrations, rather than requiring more large-scale pre-training or fine-tuning. On the Mind2Web and VisualWebArena benchmarks, adding 1 or 2 multimodal demonstrations improves task success rate by 3.36 to 7.21 percentage points, a relative increase of 21.03% to 65.75% over baseline agents. The AdaptAgent framework packages demonstrations in two ways: in-context examples for proprietary multimodal models like GPT-4o used through SeeAct, and meta-learning for open-weights models like CogAgent. The broader claim is that few-shot adaptability is a practical complement to scale-based generalization for proprietary and unfamiliar web platforms.","feed_headline":"Two demos teach web agents new sites, lifting success up to 7.2%","feed_subtitle":"A handful of human demonstrations beats relying on scale alone on Mind2Web and VisualWebArena.","key_machinery":"The central objects are the human demonstration and the meta-learning update that turns it into adaptation. A demonstration is decomposed into (visual snapshot, filtered top-50 HTML elements, action) triples and, for proprietary models, inserted into the SeeAct prompt as an in-context example. For open-weights models, the machinery is first-order model-agnostic meta-learning: an inner loop fine-tunes the agent on two tasks per website, an outer loop updates the original parameters using loss on held-out tasks from that website, and training on 292 such tasks produces a prior that adapts to a new website with two demonstrations. The paper also varies how inner-loop and outer-loop tasks are paired—intra-website, inter-website, or hybrid—to steer whether adaptation generalizes within a site or across a domain.","core_discovery":"At its core, the paper reports that a small number of multimodal human demonstrations—each recorded as a sequence of page snapshots, filtered HTML elements, and the human's chosen action—can shift a web agent's success on unseen sites and domains. For proprietary agents, AdaptAgent inserts one such demonstration into the SeeAct prompt for GPT-4o and observes consistent gains across Mind2Web's cross-task, cross-website, and cross-domain sets as well as VisualWebArena. For open-weights agents, it applies first-order model-agnostic meta-learning (FOMAML) over tasks sampled from 73 websites, then adapts the meta-learned model with two demonstrations per new website or domain; the resulting CogAgent-FOMAML beats a conventionally fine-tuned CogAgent trained on the same amount of data and, in most settings, the larger fine-tuned baseline. Additional ablations show that multimodal demonstrations outperform text-only ones, that the pairing of inner-loop and outer-loop tasks during meta-learning changes whether generalization favors cross-website or cross-domain settings, and that adding more demonstrations yields diminishing returns.","pith_inferences":["A natural next step is to select demonstrations automatically—by coverage of common actions or by difficulty—since the paper's meta-learning data-selection ablation shows that which tasks are paired changes the kind of generalization achieved.","The same demonstration format could encode compliance or accessibility rules for enterprise sites by recording one example of the desired behavior, an application the paper leaves implicit.","The live-environment gains on VisualWebArena suggest adaptation helps not just offline trajectory matching but interactive success; deploying adapted agents on additional live sites would test that directly."],"forward_implications":["If the gains hold, web agents can be tailored to proprietary or enterprise websites with a single recorded human session, reducing the need for task-specific annotation campaigns.","Multimodal demonstrations, not just text transcripts, are the effective unit of adaptation, so recording screenshots along with clicks is worth the extra prompt cost.","Meta-learning's data-selection strategy becomes a tunable choice: intra-website pairing favors cross-website generalization, inter-website pairing favors cross-domain, and hybrid balances the two.","The saturating gains with more demonstrations suggest a practical operating point of one to two demonstrations per target site, balancing accuracy against inference cost.","Even with adaptation, best task success remains below 25%, so few-shot adaptation is an improvement within a still-limited range, not a full solution."],"supporting_citations":[{"why":"Supplies the SeeAct prompting strategy and the proprietary-agent baseline that is extended with in-context demonstrations.","marker":"[62]"},{"why":"Supplies the CogAgent open-weights multimodal agent that is meta-learned and adapted in the main experiments.","marker":"[22]"},{"why":"Supplies the Mind2Web benchmark, its train/test splits, and the HTML element filtering procedure.","marker":"[14]"},{"why":"Supplies the VisualWebArena benchmark and the live-environment success evaluation.","marker":"[28]"},{"why":"Supplies the model-agnostic meta-learning update that the open-weights adaptation builds on.","marker":"[16]"},{"why":"Justifies using the first-order approximation of MAML for billion-parameter models.","marker":"[34]"},{"why":"Establishes in-context learning as the mechanism for proprietary-model adaptation with demonstrations.","marker":"[8]"},{"why":"Supplies the Set-of-Mark prompting augmentation used to construct the SeeAct* baseline.","marker":"[58]"},{"why":"Provides the GPT-4o model used as the proprietary multimodal backbone.","marker":"[2]"}],"fun_headline_variants":["2 demos adapt web agents to new sites, boosting success up to 7.2%","Few-shot human demos teach web agents new sites and domains","Meta-learned from 73 sites, two demos per site adapt web agents","Human demos outperform scale: 2 samples raise agent success up to 7.2%","AdaptAgent: two human demos boost web agent success on unseen sites"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The open-weights result assumes that the improvement attributed to meta-learning is not actually just the effect of fine-tuning on two demonstration tasks before evaluation, because the conventionally fine-tuned baseline receives no such adaptation.","fun_headline_variants_meta":{"raw":{"variants":["2 demos adapt web agents to new sites, boosting success up to 7.2%","Few-shot human demos teach web agents new sites and domains","Meta-learned from 73 sites, two demos per site adapt web agents","Human demos outperform scale: 2 samples raise agent success up to 7.2%","AdaptAgent: two human demos boost web agent success on unseen sites"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1563,"prompt_tokens":1091,"completion_tokens":472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":707,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":707,"tokens_out":472,"duration_ms":5001,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:23:29.057351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take CogAgent-FT (DE), the conventionally fine-tuned model trained on the same 292 examples, and give it the identical two-demonstration fine-tuning step on each Mind2Web cross-website and cross-domain test site before evaluation. If it matches or exceeds CogAgent-FOMAML's success rates, the paper's central attribution to meta-learning is unsupported.","supporting_citations":[],"review_version":1}