{"id":"0a11affe-6d8c-4eb5-ba53-8a9ac9ba7d2d","arxiv_id":"2507.00938","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"WebArXiv is a time-invariant 275-task benchmark for multimodal web agents on arXiv, plus a dynamic-reflection prompting method that modestly improves success rates.","lead":"A new benchmark, WebArXiv, tests web-browsing AI agents on 275 fixed tasks built from frozen arXiv pages, so answers never change over time. The authors also propose a 'dynamic reflection' trick that helps agents pick the most useful past screen to guide the next click.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing premise is that WebArXiv is actually anchored in fixed snapshots with verified time-invariant answers, but the paper supplies no snapshot hosting, versioning, or re-verification protocol; that premise is asserted rather than demonstrated.","rationale":"The reader's weakest assumption is temporal invariance of the 275 tasks and fidelity of the frozen snapshots, and that is exactly the load-bearing concern I find. The abstract's promise of 'fixed web snapshots with deterministic ground truths and standardized action trajectories' is the benchmark's reason for existing; if the snapshot layer is absent, unversioned, or unreachable, the benchmark is not materially different from a static QA set over a live website, and the reproducibility guarantee collapses. The paper contains no machinery to enforce or even describe this layer: Section 3.1 stops at manual verification of answers, and Section 4.1's evaluation protocol compares final responses with gold answers without saying what environment the agents were pointed at. I therefore agree with the reader's conditional judgment rather than pushing to reject: the idea is sound, the task set is plausibly stable, and the gap is addressable by publishing snapshots, hashes, and a replay path. I would not downgrade the verdict to ACCEPT without that evidence. I also note the numerical inconsistencies in Table 2 and the Section 3.2 vs. Section 4.1 human-annotation contradiction as additional symptoms of under-specified evaluation, but the temporal-invariance gap is the single most load-bearing issue because it attacks the benchmark's foundational promise rather than a specific reported number.","tokens_in":9395,"tokens_out":4397,"duration_ms":53038,"concrete_test":"Ask the authors for the snapshot archive and replay harness (e.g., WARC files or Memento captures with content hashes), then independently re-run a stratified random subset of at least 50 tasks twice: once against the frozen snapshot in a hermetic local environment and once against live arXiv, using identical prompts and action space. If any gold answer or required trajectory step differs between the two runs, or if any snapshot URL no longer resolves, temporal invariance is falsified. A lighter but still decisive check is to re-verify all 275 gold answers against live arXiv six months after publication; any changed answer demonstrates that the time-invariance premise requires an explicit re-verification protocol rather than an assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that WebArXiv 'ensures reproducible and reliable evaluation by anchoring tasks in fixed web snapshots with deterministic ground truths and standardized action trajectories' (Abstract). For this claim to hold, two conditions must be true: (1) every task's content is frozen in a retrievable snapshot, and (2) the frozen content matches the verified gold answers and the interface the agents actually see. Section 3.1 describes task generation, semantic filtering, and manual verification, but it never says how snapshots are captured, stored, versioned, or served; no hashes, no Memento/WARC endpoints, no replay harness, and no refresh schedule are given. Section 4.1 compares final responses with gold answers, yet the paper never establishes that agents browse the snapshot rather than live arXiv. If agents browse live arXiv, 'time-invariant' silently reduces to 'the arXiv pages happen not to have changed since verification,' which is known to fail over time for help pages, submission rules, and account-management instructions. This is not a cosmetic omission: without a replayable frozen environment, the deterministic ground truths and standardized trajectories that the reproducibility claim rests on cannot be independently checked. The paper also asserts machine-verifiable evaluation with no manual inspection (Section 1) while Section 3.2 describes three human annotators labeling trajectories Correct/Partial/Incorrect; this internal inconsistency makes the evaluation protocol, not just the content, in need of verification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WebArXiv, a benchmark of 275 arXiv-based tasks across five categories of 55 tasks each, and claims that it provides reproducible and reliable evaluation by anchoring tasks in fixed web snapshots with deterministic ground truths and standardized action trajectories. It also proposes a lightweight dynamic reflection mechanism that selects the most relevant of the last three interaction steps before each action, and evaluates ten multimodal web agents (GPT-4o, GPT-o1, Gemini variants, SeeAct, LiteWebAgent, OpenWebAgent, etc.) on the benchmark. Results are reported as category-wise and overall task success rates, with an ablation comparing dynamic reflection to fixed-history baselines.","tokens_in":9494,"tokens_out":6730,"duration_ms":68833,"significance":"If properly substantiated, WebArXiv would be a useful stable evaluation instrument for web agents, and the dynamic reflection strategy is a simple, parameter-light intervention with potential to transfer to other navigation benchmarks. The paper's strengths include a concrete task construction pipeline, manual verification by three annotators, a broad comparison across ten models, and an open-source code release. However, the manuscript does not yet demonstrate the load-bearing reproducibility premise (snapshot availability, versioning, and refresh protocol), and it contains internal inconsistencies in the reported numbers and evaluation protocol that must be resolved before the headline claims can be accepted.","major_comments":[{"comment":"The abstract and contribution list claim that WebArXiv is anchored in fixed web snapshots, but the paper never specifies how snapshots are captured, stored, versioned, or served. No hashes, Memento/WARC endpoints, or replay harness are provided, and the anonymous link in the abstract is described only as 'code.' Without a retrievable frozen environment, the deterministic ground truths and standardized action trajectories cannot be independently checked, and the 'time-invariant' premise reduces to an assertion that live arXiv pages happen not to have changed after verification. Please describe the snapshot artifacts (or a restore/replay mechanism) and provide a re-verification or refresh protocol.","section":"Section 3.1 and Abstract"},{"comment":"Several reported totals are inconsistent with the category values under the stated design of 55 tasks per category. For example, the GPT-4o + dynamic reflection row lists category values 63.6%, 60.0%, 38.2%, 34.5%, and 52.7%, which average to 49.8%, not the reported total of 38.4%. Similarly, GPT-o1's total of 56.7% exceeds its category average of 55.2%, and LiteWebAgent's total of 44.0% exceeds its category average of 42.4%. Please correct the tables or explain why the totals are weighted differently; as printed, these errors undermine confidence in all reported comparisons.","section":"Tables 1 and 2"},{"comment":"Section 1 states that all answers are machine-verifiable, 'eliminating the need for manual inspection,' yet Section 3.2 describes three annotators reviewing each agent's full trajectory and assigning Correct/Incorrect/Partial-Correct labels. Section 4.1 then says success is determined by strict matching to gold-standard answers. Please clarify whether the reported success rates come from automatic string matching or from human annotation, and what role the Partial-Correct label and human review play in the reported numbers. If both are used, the protocol and the claimed 'no manual inspection' property need to be reconciled.","section":"Sections 1, 3.2, and 4.1"},{"comment":"The ablation compares dynamic reflection against the 'last 3 steps' baseline and concludes that the mechanism is effective, but Table 3 also includes a 'last step only' baseline. For GPT-4-Turbo, dynamic reflection achieves 40.2%, which is lower than the 43.6% success of simply using the last step. For GPT-o1, the improvement over last-step-only is only 1.8 points (61.8% vs 60.0%). With three runs per task and no variance or significance testing, these differences are not sufficient to support the claim that dynamic reflection is uniformly superior. Please report per-run results and statistical comparisons, and discuss why dynamic reflection underperforms the last-step-only baseline for GPT-4-Turbo.","section":"Section 4.3 and Table 3"}],"minor_comments":[{"comment":"The manuscript contains multiple typos and grammatical errors: 'to to improve' (Contributions bullet 2), 'webagents' and 'most webagents handles' (Section 3.3), 'provides well-aligned' (Section 1), 'failed is on the right track' (Section 3.2), and inconsistent capitalization 'WebArxiv' vs 'WebArXiv'.","section":"Throughout"},{"comment":"The arrow notation in the header 'Successful (↑) Partial (↓) Failed (↓)' is never defined; please clarify what the arrows denote.","section":"Table 3"},{"comment":"Figure 7 appears without an in-text citation, and several example figures (Figures 2-6) are not referenced in the body text; all figures should be cited in order.","section":"Figures"},{"comment":"The reference list contains a duplicated OpenWebAgent entry (the same Iong et al. 2024 work is listed twice) and inconsistent formatting for the Gemini 1.5 series, which appears under two different technical-report titles.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper's main risk is verifiability rather than novelty. The authors should be required to supply the actual benchmark data and snapshot artifacts during revision, because the current anonymous link appears to point to code only. The numeric inconsistencies in Table 2 (especially the GPT-4o + dynamic reflection row) and the contradiction between automatic 'machine-verifiable' evaluation and human annotation in Section 3.2 should be resolved before external reviewers can assess the reported results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe one thing to know: WebArXiv is a reasonable idea that is currently underspecified in exactly the place that matters. The paper proposes a 275-task arXiv benchmark with claimed \"time-invariant\" tasks and deterministic ground truths, plus a lightweight \"dynamic reflection\" tweak for web agents. If the snapshot claim holds, it would be a useful reproducibility instrument for agent evaluation. But the paper never shows how snapshots are captured, hosted, or refreshed; the agents might just be browsing live arXiv. That is the load-bearing wall, and it is not demonstrated.\n\nWhat is genuinely new: a static benchmark on a widely used real site, with five task categories, manually verified answers, and standardized action trajectories. That fills a real gap, since Mind2Web, WebArena, WebVoyager, and WebCanvas use live or simulated content. The dynamic reflection mechanism is a small but neat idea: instead of using the last k steps uniformly, ask the model to pick the most relevant prior step. The trajectory examples look sensible.\n\nWhat is good: task coverage is practical, the manual verification process is described, the limitation section honestly mentions the English-only interface, and the paper is clearly written. The comparison across ten agents is a useful baseline.\n\nNow the soft spots, in order:\n\n1. The snapshot anchoring is asserted, not shown. No hosting, versioning, replay harness, Memento/WARC endpoint, or refresh schedule. Without that, \"time-invariant\" is just \"haven't noticed the pages changing.\" This is the core of the paper and it is missing.\n\n2. The evaluation protocol contradicts itself. Abstract and intro say machine-verifiable with no manual inspection, but Section 3.2 describes three human annotators labeling trajectories Correct/Partial/Incorrect. That needs reconciling.\n\n3. The numbers do not add up. In Table 1, GPT-o1's category rates average to 55.2% but the total says 56.7%. In Table 2, GPT-4o + dynamic reflection's category rates average to 49.8% but the total says 38.4%. That is not rounding; it is an error. It makes the main results table unreliable until fixed.\n\n4. The experimental reporting is thin. Three runs, no variance, no confidence intervals, no significance tests. Some improvements are a couple of points, well within noise.\n\n5. The dynamic reflection results are mixed. GPT-o4-mini gets worse with it. And in Table 3, the last-step-only baseline beats dynamic reflection for GPT-4-Turbo (43.6% vs 40.2%). The paper highlights only the last-3-steps comparison, which is cherry-picking.\n\nBottom line: the benchmark idea is worth pursuing, and the paper identifies a real reproducibility problem. But as it stands, the core reproducibility premise is unverified and the reported numbers need correction. A serious editor could send it out with the expectation of heavy revision, but the current version is not ready to be trusted as a benchmark paper.\n\nWho this is for: people building web-agent benchmarks or doing agent evaluation will find the task design useful, but only after the snapshot issue is resolved. My recommendation: engage with it, but treat the results as provisional. If this crosses your desk, ask for the snapshot infrastructure first.","headline":"A useful benchmark idea whose core reproducibility premise is asserted but never demonstrated, and the result tables contain errors that need correction.","tokens_in":10227,"tokens_out":3569,"would_cite":false,"duration_ms":37009,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WebArXiv proposes a time-invariant benchmark of 275 arXiv tasks with fixed snapshots and deterministic answers, plus a dynamic reflection mechanism that improves top agents by up to 8.9 points.","keywords":["web agents","multimodal agents","benchmark","arXiv","time-invariant tasks","dynamic reflection","web navigation","deterministic ground truth"],"falsifier":"Reopen a subset of tasks after a documented arXiv help-page or interface change, for example re-asking 'How can I package my submission files?' on the current help page; if the live instruction differs from the frozen gold answer 'Create tar.gz and zip Files', the time-invariance premise is violated.","tokens_in":8976,"feed_emoji":"📚","tokens_out":7047,"duration_ms":73494,"temperature":0.7,"pith_summary":"This paper argues that web-agent evaluation can be made reproducible by grounding tasks in fixed snapshots of a stable site, arXiv, with exact answers that machines can verify without manual inspection. It offers 275 such tasks across five categories—site information, rules and accounts, paper retrieval, advanced search and filtering, and deep paper content extraction—together with standardized prompts, reference action trajectories, and deterministic ground truths. The same study identifies a common failure mode it calls Rigid History Reflection, where agents treat their most recent interaction steps as equally relevant, and proposes a lightweight dynamic reflection mechanism that selects the most useful prior step before each action. Across ten multimodal web agents, the reflection variant consistently improves on the base model, with the largest gains on GPT-o1 (from 56.7% to 61.8%) and Gemini-2.5 (from 51.1% to 60.0%). If the time-invariance premise holds, the field gains a stable measuring stick for navigation agents and a simple prompting mechanism that improves them.","feed_headline":"275 frozen arXiv tasks put web agents to a stable test","feed_subtitle":"Selective past-step recall lifts Gemini-2.5 from 51 to 60 percent on the new benchmark.","key_machinery":"The central object is WebArXiv, a static, time-invariant benchmark of 275 arXiv tasks divided into five temporally stable categories, each with a frozen snapshot context, standardized prompts, reference trajectories, and deterministic ground truths. It carries the argument by removing answer drift as a confounding factor, so differences in success rates can be attributed to agent behavior rather than changing web content. The second mechanism is dynamic reflection, a prompting procedure that, at every interaction step, asks the model to choose the most relevant of the last three screenshots, then combines that chosen screenshot with the current view to generate the next action; this replaces the uniform last-three-steps memory that most agents use.","core_discovery":"The paper claims that a benchmark can be made time-invariant by anchoring each task to fixed arXiv webpage content, manually verifying gold answers for uniqueness and stability, and enforcing strict machine-checkable matching so that task success is unaffected by web drift or API changes. The second claim is that most web agents fail not from lack of context but from rigid history use: they keep a fixed number of recent interaction steps and weigh them all equally, which causes loops and missed information on dense pages. Dynamic reflection addresses this by asking the model to identify the single most relevant of the last three visual observations and combining that chosen observation with the current view before generating the next action. The evidence is a ten-agent evaluation, each run three times per task, showing that dynamic reflection beats uniform three-step memory and single-step memory, and lifts top models such as Gemini-2.5 from 51.1% to 60.0% and GPT-o1 from 56.7% to 61.8%.","pith_inferences":["The strongest test of the time-invariance premise would be to re-run the same 275 tasks after a known arXiv layout or help-page change and show that success rates are unchanged.","Dynamic reflection is a prompt-time intervention, so it should transfer to other static-knowledge domains, but the paper only demonstrates it on arXiv.","Because all tasks use arXiv's English interface, the benchmark measures English-only navigation; multilingual help pages may present different navigation behaviors and are an untested extension.","If the frozen snapshots are publicly accessible, a later agent could shortcut the navigation by memorizing the answer-bearing page, so an extension would be to rotate or randomize tasks to prevent memorization."],"forward_implications":["WebArXiv can be re-run on new models as they ship without re-labeling, since answers are designed to be deterministic and time-invariant.","The dynamic reflection prompt is a zero-training intervention that lifts success rates for both frontier and smaller models, making it a cheap component to add to existing agent stacks.","The gap between categories, with Advanced Search and Filtering staying below 50 percent for almost all agents, points navigation research toward dense, form-heavy interfaces.","Overall scores separate agents clearly enough to rank them, giving the community a usable baseline for tracking progress over time.","Standardized prompts, reference trajectories, and strict matching make the benchmark usable for reinforcement learning training, not just evaluation."],"supporting_citations":[{"why":"Supplies WebVoyager, the live-website benchmark whose volatile answers motivate the need for static, time-invariant tasks.","marker":"He et al., 2024b"},{"why":"Provides Mind2Web, the simplified-simulator style of benchmark that WebArXiv contrasts with.","marker":"Deng et al., 2023"},{"why":"Provides WebArena, a realistic web environment benchmark with fixed action traces that WebArXiv extends toward static grounding.","marker":"Zhou et al., 2024"},{"why":"Defines the SeeAct agent, one of the ten evaluated baselines on WebArXiv.","marker":"Zheng et al., 2023"},{"why":"Defines LiteWebAgent, the DOM-parsing baseline that leads the Deep Paper Extraction category.","marker":"Zhang et al., 2025"},{"why":"Defines OpenWebAgent, a modular DOM-based baseline evaluated on the benchmark.","marker":"Iong et al., 2024"},{"why":"Underlies GPT-4o, one of the multimodal agents used in the main experiments and reflection ablation.","marker":"OpenAI, 2024a"},{"why":"Underlies the Gemini model family, including Gemini-2.5 whose dynamic-reflection variant reaches 60.0 percent.","marker":"DeepMind, 2025"},{"why":"Supplies the earlier Reflexion method that motivates the paper's dynamic reflection contrast.","marker":"Shinn et al., 2023"}],"fun_headline_variants":["Frozen arXiv snapshots expose web agents' rigid history flaw","Reflection boosts web agents on 275 static arXiv tasks","Time-invariant tasks on arXiv freeze web agent evaluation","Selective recall fixes web agents' rigid history on arXiv tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is temporal invariance: gold answers are verified once at construction time, and the paper does not specify how snapshots are hosted, refreshed, or re-checked, so if arXiv content drifts or the frozen snapshots diverge from the described interface, the deterministic ground truths go stale and the reproducibility claim breaks.","fun_headline_variants_meta":{"raw":{"variants":["Frozen arXiv snapshots expose web agents' rigid history flaw","Reflection boosts web agents on 275 static arXiv tasks","Time-invariant tasks on arXiv freeze web agent evaluation","Selective recall fixes web agents' rigid history on arXiv tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000593,"raw_usage":{"total_tokens":2758,"prompt_tokens":903,"completion_tokens":1855,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":1787}},"tokens_in":519,"tokens_out":1855,"duration_ms":15284,"temperature":1.0,"reasoning_tokens":1787,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:02:14.651700+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reopen a subset of tasks after a documented arXiv help-page or interface change, for example re-asking 'How can I package my submission files?' on the current help page; if the live instruction differs from the frozen gold answer 'Create tar.gz and zip Files', the time-invariance premise is violated.","supporting_citations":[{"cited_title":"LiteWebAgent: The Open-Source Suite for VLM-Based Web-Agent Applications","cited_arxiv_id":"2503.02950","evidence_quote":"Defines LiteWebAgent, the DOM-parsing baseline that leads the Deep Paper Extraction category."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Underlies the Gemini model family, including Gemini-2.5 whose dynamic-reflection variant reaches 60.0 percent."}],"review_version":1}