{"id":"d2fbc5d0-eaac-438d-a1cf-6a1f65b53eed","arxiv_id":"2608.12429","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A website-prior data synthesis framework that jointly refines task and trajectory during generation produces more executable, semantically aligned supervision and improves web agent success rates over prior synthesis baselines.","lead":"SynWeaver is a three-stage pipeline that builds a map of a website, fine-tunes a UI-aware model on that map, then generates task-trajectory training pairs by jointly refining the task and the execution trace when they conflict. On WebArena and WebVoyager, agents fine-tuned on SynWeaver data outperform agents trained on data from existing synthesis methods, using fewer trajectories.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The crawler's state-equivalence rule can classify a freshly opened menu as the same state and skip the revealed subtree, so the website map's coverage—the foundation of every synthesized task—is unvalidated.","rationale":"The reader's weakest assumption and my concern coincide: the website map's functional topology is the foundation of the whole method. I sharpen it with a concrete failure mode drawn from the paper's own definitions. Algorithm 3's accessibility-tree comparison can miss row insertions or reorderings in a menu-open state, and the VLM fallback is explicitly instructed to ignore open/closed dropdowns, so the post-click state is merged with the pre-click state. In Algorithm 1, a non-new state does not trigger recursion, and the newly exposed elements are never visited. This is not speculative: it follows from the published prompt and control flow. The claim that SynWeaver 'covers a broad set of functionally distinct page states and executable interactions' is exactly the load-bearing assertion, and it is unvalidated. The empirical gains over baselines could still be real even with this flaw, so I do not change the reader's CONDITIONAL verdict; I would add the map audit as an explicit acceptance condition. Secondary concerns such as missing error bars and website-level reversals are real but less fundamental to the mechanism.","tokens_in":29864,"tokens_out":9858,"duration_ms":106107,"concrete_test":"Construct a minimal test site (or use one WebArena site with a collapsed user menu) where a functionally distinct page is reachable only through a hamburger menu or a hover-revealed submenu, with no direct link to it. Run SynWeaver's crawler with the released code and check (1) whether the menu-reachable page appears as a node in G, and (2) whether any validated task-trajectory pair targets it. Then rerun with StateEquiv modified to treat 'menu opened' as a new state; if new states/transitions appear and downstream WebArena SR changes by more than about 2 points, the equivalence assumption in Algorithm 3 is load-bearing. As a cheap supplementary check, log all StateEquiv calls on menu-opening actions in the existing five WebArena maps and count how many revealed elements are never subsequently explored.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 3 defines state equivalence by identical URL plus accessibility-tree similarity, with a VLM fallback whose prompt (Prompt 3) explicitly lists 'dropdown menu open/closed states', 'hover effects or focus states', and 'popup hints or tooltips' as acceptable differences to ignore. A collapsed navigation menu is precisely such a case: clicking or hovering to open it need not change the URL, and the accessibility tree may differ mainly by the newly revealed items. If StateEquiv returns 'same', Algorithm 1 does not recurse into the post-click state, so none of the menu-only elements are ever added as executable candidates; the entire subtree reachable only through that menu is absent from G. Stages 2 and 3 then derive the UI prior, task proposals, and trajectory anchors exclusively from the recorded transitions of G, so this blind spot propagates into the training data. The paper reports map sizes and a scaling study, but no check that G actually contains states reachable only via menus, popups, or hover interactions—and no precision/recall audit of StateEquiv decisions. If even one common menu pattern is skipped, the central 'broad coverage' premise is weakened, and the comparison against random-walk baselines (which can still stumble into such pages) becomes a comparison of exploration completeness, not of co-synthesis quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SynWeaver, a three-stage framework for synthesizing web-agent training data. Stage 1 performs DFS-based website exploration to construct a 'website map' of functionally distinct states and transitions, using progressive state comparison and duplicate-trigger detection to prune redundant exploration. Stage 2 derives page-level and transition-level UI supervision from this map and fine-tunes the target backbone with LoRA to obtain a UI-aware model. Stage 3 performs collaborative task-trajectory synthesis: reverse task synthesis from map transitions, online execution with joint refinement of task and trajectory when they become inconsistent, and post-verification with reconstruction of failing trajectories. On a filtered WebArena set (226 tasks) and a filtered WebVoyager set (388 tasks), SynWeaver with Qwen3-VL-8B-Instruct reaches 19.91% and 27.06% success rate, outperforming NNetNav, OS-Genesis, and SynthAgent. Ablations show contributions from the website map, website prior, collaborative refinement, and post-verification, and a scaling study shows monotonic aggregate gains with more synthesized trajectories.","tokens_in":30058,"tokens_out":5225,"duration_ms":54565,"significance":"If the claims hold, SynWeaver is a valuable contribution to web-agent data synthesis: it places explicit website-specific knowledge before task synthesis, introduces a joint refinement mechanism for task-trajectory consistency, and provides a relatively complete pipeline with public code. The paper's strengths include clearly described algorithms, consistent component-level ablations, a held-out UI knowledge test, and a data-scaling study. The central empirical claim, however, rests on the validity of the website map and on evaluations that lack statistical reliability measures. The most serious risk is that the state-equivalence rule may silently omit subtrees reachable only through menus, popups, or hover interactions, which would propagate to all downstream supervision. The paper is publishable in principle, but the coverage concern and the absence of error bars or significance tests need to be addressed before the performance claims can be accepted at face value.","major_comments":[{"comment":"The state-equivalence rule used to build the website map treats dropdown-menu open/closed states, hover/focus states, and popup hints or tooltips as acceptable differences (Prompt 3), and Algorithm 3 returns 'same' whenever the URL is identical and the accessibility-tree similarity passes the tolerance. Since Algorithm 1 recurses only when StateEquiv returns false (lines 31-35), any subtree reachable only after expanding a collapsed menu or hovering over a submenu is never added to G. The UI prior, reverse task synthesis, and collaborative trajectory synthesis are all derived from G, so this coverage blind spot propagates to the entire training set. The manuscript reports map sizes and a scaling study but provides no precision/recall audit of StateEquiv decisions and no check that G contains states reachable via menus, popups, or hover interactions. I ask for such an audit (e.g., compare G against a manually annotated interaction inventory for at least one WebArena site, or measure how many evaluation tasks have their target state or a required intermediate state reachable in G) and, if the blind spot is confirmed, a revised equivalence rule that treats newly revealed actionable elements as a new state.","section":"§3.1, Algorithm 3, Prompt 3"},{"comment":"All success rates are reported as point estimates with no error bars, confidence intervals, or significance tests, and the text does not indicate that evaluation was repeated over multiple runs or seeds. Given only 226 WebArena evaluation tasks and per-site counts of roughly 34–61 tasks, several headline differences (e.g., 19.91 vs 16.81 overall in Table 2, or CMS 14.04 vs 15.79) are within plausible sampling noise. The WebVoyager per-site cells are even smaller. Since the central claim is that SynWeaver 'consistently outperforms' strong baselines, the paper should provide repeated evaluation runs or bootstrap confidence intervals, and ideally per-site intervals, so that the reader can distinguish genuine gains from noise.","section":"§4.2, Tables 2–4"},{"comment":"The comparison is defined on filtered benchmark subsets and filtered maps, which makes it difficult to separate method quality from exploration coverage. WebVoyager drops 5 of 15 sites and 255 of 643 tasks; WebArena keeps only 226 of 812 tasks; and the CMS and GitLab maps are aggressively filtered (646→169 states and 606→203 transitions after capping and hop sampling). A random-walk baseline can still reach pages that the filtered map omits, so the difference between Ours and −Map in Table 4 could reflect map completeness rather than co-synthesis quality. Please report sensitivity of the main results to the filtering choices (e.g., minimally filtered or unfiltered WebArena evaluation, and unfiltered map synthesis for at least one site), or at minimum quantify how many evaluation tasks require a page or interaction that is absent from G.","section":"§4.1, Appendix D.2/D.3"}],"minor_comments":[{"comment":"The abstract reads 'reaches 27.06 success rate'; this should be '27.06%' for consistency with the tables.","section":"Abstract"},{"comment":"Figure 1 is very dense, and the labels for the website map, state processor, and collaborative refinement stages are difficult to read at normal zoom; a higher-resolution version or a simplified schematic would help.","section":"Figure 1"},{"comment":"The cost comparison in Table 6 is not apples-to-apples because SynthAgent uses GPT-4.1 while DR and CR use Gemini-3-Flash plus Gemini-3.1-Pro; please state this caveat in the text and, if possible, report API prices using comparable model tiers.","section":"Table 6"},{"comment":"The trajectory validity check refers to a predefined step budget B but does not state its value; please give the concrete budget used in the experiments.","section":"Appendix B"},{"comment":"The original map construction uses Qwen3-VL-235B-A22B-Instruct for representative-item sampling and Qwen3-VL-Plus as fallback for state equivalence, but these models are not listed in the main model description; a one-sentence clarification in §3.1 or §4.1 would improve reproducibility.","section":"Appendix D.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution with a clearly described pipeline and a reasonable ablation design. The main risk is not novelty but verification: the state-equivalence coverage blind spot, if real, undermines the 'broad coverage' premise, and the lack of error bars weakens the comparative claim. I would be comfortable with acceptance after the authors provide a map-coverage audit and statistical reliability measures. The manuscript also sits well within cs.SE scope as a systems and data-engineering contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SynWeaver is worth a serious look. The genuinely new piece is the collaborative refinement: when the agent stalls, the stronger teacher jointly rewrites the task and edits the execution prefix (delete, reorder, update reasoning), rather than decoupling task and trajectory fixes as in SynthAgent. The ablations are the strongest part of the paper: each component (map, collaborative refinement, website-prior proposer, post-verification) contributes consistently to the overall success rate, and the scaling study shows a plausible monotonic gain from 413 to 2,581 trajectories. The authors are also honest about limitations (anti-bot walls, teacher bound, no RL).\n\nThe soft spots are mostly in the evaluation protocol. The main comparisons lack error bars or significance tests; on InternVL3 the edge over SynthAgent is 1.33 points on 226 tasks, which is within sampling noise. On Qwen the margin is more comfortable, but the 'consistently outperforms' claim is not supported at the website level: SynthAgent still wins on CMS and Maps, and ties on Reddit for InternVL3. The WebVoyager filtering (388 of 643 tasks) is aggressive, though the rationale for each exclusion is stated.\n\nThe deeper concern is the crawler's state-equivalence rule. Algorithm 3 and Prompt 3 explicitly list dropdown open/closed states, hover effects, and tooltips as acceptable differences to ignore. That means a collapsed navigation menu that reveals items on click can be classified as 'same state,' and the newly visible subtree is never added to the map. Since all downstream synthesis (UI prior, task proposals, trajectories) draws from the map, this could bias the comparison toward pages reachable without menus. The paper does not audit StateEquiv decisions or check that menu-only interactions are covered. This is a real gap, though not necessarily fatal: the ablations show map vs random walk matters, and collaborative refinement matters even within the map, so the co-synthesis claim is not entirely riding on map completeness.\n\nWho is this for? Anyone building web-agent training data from synthetic interaction: the collaborative refinement idea is worth borrowing, and the cost comparison ($0.085 per pair) is useful. The paper deserves a serious referee. I would ask for an audit of crawler coverage (especially menu-only states), error bars or significance tests on the main comparisons, and a less sweeping wording of 'consistently outperforms.' That is a conditional accept, not a reject.","headline":"SynWeaver is a solid, clearly-written synthesis pipeline with a genuinely new collaborative refinement step and an honest scaling study; the main caveats are evaluation noise and an unvalidated state-equivalence assumption in the crawler.","tokens_in":30628,"tokens_out":3824,"would_cite":true,"duration_ms":33472,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training web agents on 822 co-synthesized task–trajectory pairs from a website map beats exploration-based baselines that use 1,000 trajectories, on WebArena and WebVoyager alike.","keywords":["web agents","synthetic supervision","website map","UI priors","task-trajectory co-synthesis","collaborative refinement","WebArena","WebVoyager"],"falsifier":"Crawl a site whose pages share a URL and near-identical accessibility trees but render different content or actions depending on session state, and check whether progressive state comparison merges them into one node; if it does, tasks synthesized from that node are infeasible on one of the two states, and an agent fine-tuned on SynWeaver's data should fail exactly those tasks while passing single-state variants.","tokens_in":29606,"feed_emoji":"🗺️","tokens_out":12917,"duration_ms":110377,"temperature":0.7,"pith_summary":"This paper tries to establish that the bottleneck in synthetic training data for web agents is grounding, not quantity. Tasks proposed without website-specific knowledge tend to be hallucinated or infeasible, and refining a task separately from its execution trajectory drifts the supervision away from natural user intent. SynWeaver addresses both problems by crawling a website into a map of functionally distinct states and executable transitions, fine-tuning a UI-aware model on supervision derived from that map, and then co-refining the task and trajectory together whenever execution stalls or contradicts the stated intent. With 822 validated task–trajectory pairs, fewer than the 1,000 trajectories used for each baseline, the trained Qwen3-VL-8B-Instruct agent reaches 19.91 percent success on WebArena and 27.06 percent on WebVoyager; ablations attribute the gains to the map, the website-prior proposer, collaborative refinement, and post-verification. If the claim holds, structured exploration and joint refinement can substitute for brute-force data collection in web agent training.","feed_headline":"822 co-synthesized examples beat 1,000 baseline trajectories","feed_subtitle":"A website map plus joint task–trajectory repair lifts WebArena success to 19.91 and WebVoyager to 27.06.","key_machinery":"The load-bearing object is the website map $G=(S,T)$: nodes are page states judged functionally distinct, and edges are executable interactions with recorded actions. A depth-first crawler builds the map using a state processor whose progressive comparison checks URL equality, then row-wise accessibility-tree similarity after stripping digits, with an optional vision-model fallback, and whose duplicate-trigger detection prunes elements, lists, and forms whose functional signatures or XPaths were already explored. Two assets come out of the map: five formats of UI supervision (page description, page QA, element description, forward transition description, inverse transition description) used to LoRA-tune a UI-aware task proposer, and transition triplets from which tasks are reverse-synthesized. The second mechanism is collaborative refinement: when execution stalls or the page contradicts the task, a stronger teacher either rewrites the task alone or jointly edits task and trajectory prefix by deleting steps, swapping commutable steps, and updating affected reasoning, preserving natural intent while restoring executability. Post-verification then filters or repairs the collected pairs.","core_discovery":"The paper's central claim is that high-quality web agent supervision couples task intent with executable trajectories, grounded in an explicit map of the target website, and that this yields stronger in-domain and out-of-domain generalization than exploration-based synthesis. On its own terms, SynWeaver 'consistently outperforms strong synthesis baselines and yields more effective supervision.' The evidence: 822 validated pairs fine-tuned into Qwen3-VL-8B-Instruct give 19.91 percent WebArena success versus 16.81 for the strongest baseline SynthAgent, and 27.06 percent WebVoyager success versus 22.42; InternVL3-8B shows the same ordering (14.16 against 12.83). Ablations locate the sources of the gain: replacing the map with a random walk costs 4.42 points, replacing collaborative refinement with decoupled refinement costs 4.87 points, replacing the website-prior proposer with a general-purpose model costs 2.21 points, and dropping post-verification costs 5.31 points. The paper also claims the website prior transfers, since the model trained only on five WebArena websites posts its best score on seven of nine unseen WebVoyager sites.","pith_inferences":["An implication the paper leaves implicit: the map-then-co-refine pipeline is not web-specific, since any GUI with a stable DOM (desktop applications, mobile interfaces, IDEs) could be crawled into a functional map, and the five UI-supervision formats would transfer with little change.","Testable extension: the map could serve as a coverage certificate, reporting what fraction of map transitions a synthesized dataset or an agent's rollouts actually exercise; the paper reports state and transition counts but never checks coverage of the generated trajectories against the map.","The hierarchical teacher design (a cheap teacher for routine synthesis, a strong teacher only for repairs) is reported as a cost figure but not analyzed; a natural experiment is to lower the strong teacher's capability until the co-refinement loop begins producing repairs that preserve the original errors."],"forward_implications":["Data efficiency: 822 validated task–trajectory pairs beat 1,000-trajectory baselines, and the advantage persists when the baselines are trained on their full released datasets (NNetNav 17.26 and SynthAgent 18.58 at full scale, versus SynWeaver 19.91).","Out-of-domain transfer: UI priors and trajectories learned on five self-hosted WebArena websites carry over to unseen live websites, with the best score on seven of nine WebVoyager sites.","Scaling inside a map: resampling transitions from the same website map continues to raise success rates (14.16 to 19.91 to 20.80 to 23.89 as the trajectory budget grows from 0.5× to 3.12×), so the map is a reusable synthesis asset rather than a one-shot crawl.","Task quality: the map-grounded UI-aware proposer yields tasks that need fewer refinements and convert into higher downstream success than tasks from a stronger general-purpose proposer, and the UI prior lifts a held-out UI-knowledge test by 14.5 points.","Synthesis cost: collaborative refinement retains 99.52 percent of collected trajectories instead of discarding them, lowering the estimated per-trajectory cost below both the decoupled variant and the SynthAgent reference."],"supporting_citations":[{"why":"Supplies the reverse task synthesis strategy that SynWeaver adopts and is the random-walk baseline it must beat.","marker":"Sun et al., 2025a"},{"why":"The strongest baseline (16.81 WebArena, 22.42 WebVoyager) and the source of the delete/reorder trajectory edits that collaborative refinement extends.","marker":"Wang et al., 2025b"},{"why":"The multi-step LLM exploration baseline compared in the main evaluations.","marker":"Murty et al., 2024"},{"why":"WebArena, the self-hostable benchmark whose five websites provide the in-domain evaluation and the synthesis targets.","marker":"Zhou et al., 2024"},{"why":"WebVoyager, the online benchmark that provides the out-of-domain evaluation on unseen websites.","marker":"He et al., 2024"},{"why":"Mind2Web's human-annotated training tasks serve as the reference set for the task-diversity comparison.","marker":"Deng et al., 2023"},{"why":"Set-of-Mark annotation defines the marked-element screenshots used in element and transition-level UI supervision.","marker":"Yang et al., 2023"},{"why":"LoRA is the fine-tuning method for both the UI-aware initialization and the final agent, the matched training setup that makes the comparison fair.","marker":"Hu et al., 2022"}],"fun_headline_variants":["Website map drives web agent gains over brute-force data","SynWeaver's site map yields stronger web agents, fewer samples","Co-synthesis with site priors beats raw trajectory scaling","Map-guided task-trajectory synthesis boosts web agent skill","Fewer but grounded examples: SynWeaver's recipe for web agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything downstream inherits the crawler's notion of functionally distinct state: if Algorithm 3's state comparison merges two pages that behave differently, or if duplicate-trigger detection prunes an interaction with a distinct effect, the UI prior, the proposed tasks, and the trajectory supervision all inherit that blind spot.","fun_headline_variants_meta":{"raw":{"variants":["Website map drives web agent gains over brute-force data","SynWeaver's site map yields stronger web agents, fewer samples","Co-synthesis with site priors beats raw trajectory scaling","Map-guided task-trajectory synthesis boosts web agent skill","Fewer but grounded examples: SynWeaver's recipe for web agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000594,"raw_usage":{"total_tokens":2820,"prompt_tokens":1024,"completion_tokens":1796,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":1711}},"tokens_in":640,"tokens_out":1796,"duration_ms":11319,"temperature":1.0,"reasoning_tokens":1711,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:20:41.040288+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Crawl a site whose pages share a URL and near-identical accessibility trees but render different content or actions depending on session state, and check whether progressive state comparison merges them into one node; if it does, tasks synthesized from that node are infeasible on one of the two states, and an agent fine-tuned on SynWeaver's data should fail exactly those tasks while passing single-state variants.","supporting_citations":[],"review_version":1}