{"id":"adaa7d0d-c94d-45d0-9bda-03af66d56ed1","arxiv_id":"2507.15061","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"WebShaper formalizes web information-seeking tasks as set-theoretic queries, synthesizes training questions by layer-wise expansion, and uses the resulting data to train open-source agents that outperform prior open-source systems on GAIA and WebWalkerQA.","lead":"WebShaper is a new way to create training data for AI agents that search the web: it first writes every question as a small set-theory equation, then expands those equations into harder questions while checking the answers. The trained agent beats other open-source web-research agents on the GAIA and WebWalkerQA benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Answer-set preservation in layer-wise expansion is asserted, not verified, and the paper's own validation logic confirms the gap.","rationale":"The reader's weakest_assumption identifies precisely Section 3.2.2's unproven answer-preservation claim and Section 3.2.3's explicitly weak validation. I agree this is the most load-bearing concern: the strongest claim is about downstream benchmark performance, but that performance is only causally attributable to the formalization if the synthesized labels are actually correct. The paper's own text admits the validator does not check strict answer identity, which is a direct admission of the gap. This is not an outside-consensus disagreement; it is an internally acknowledged limitation that undermines the stated invariant. The concern does not immediately invalidate the empirical results: the synthesized dataset could still be useful even with some label noise, and the benchmark gains across three backbones are real evidence in favor. But the formal mechanism claimed to drive those gains is unverified, so the correct verdict is CONDITIONAL with a request for a targeted audit of answer-set preservation, rather than a full rejection. The concrete check I propose would settle the question with modest effort using the released dataset and a retrieval agent.","tokens_in":16504,"tokens_out":2385,"duration_ms":20273,"concrete_test":"Use the released WebShaper dataset to sample 200 expanded questions with depth l >= 2, recover their KP graphs, and for each leaf expansion check whether the natural-language sub-question uniquely determines the original constant. Concretely: instantiate a baseline web agent (e.g., the same search tools) to independently answer the sub-question without seeing the original constant, and compare its answer set with the intended constant C. Then re-answer the full q_n and q_{n+1} questions with a strong retrieval model. If q_{n+1} is answered correctly but q_n is not (or vice versa) on a substantial fraction, or if independent sub-question answers exceed the intended constant on even 10% of samples, the answer-preservation invariant fails and the label noise is material.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that formalization-driven synthesis systematically improves IS agents rests on the layer-wise expansion preserving the target answer set T. Section 3.2.2 states that q_{n+1}(T) always has the same answer as q_n(T), but this is asserted without proof or per-step verification. Section 3.2.3's Validate tool explicitly does not check whether the constant C is strictly the answer to the constructed sub-question, only whether the type of C satisfies the sub-question. Thus, a sub-question that is satisfied by C but also by other entities yields an expanded question whose true answer set is a superset of T. Training labels are then wrong for those entities, injecting systematic noise into SFT/RL. The reported filtering by final trajectory correctness with a judge LLM operates on natural-language answers and does not repair the formal answer-set semantics; a confidently wrong expanded label can pass. This is load-bearing because the key ablation (Figure 7a) attributes gains to the formalization, yet the formalization's core invariant is unverified. The risk is partially mitigated by the agentic expansion and consistent benchmark gains across backbones, but the causal attribution to formalization depends on label correctness, not just benchmark averages.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WebShaper, a formalization-driven pipeline for synthesizing training data for information-seeking (IS) agents. The core idea is to represent IS tasks through set-theoretic Knowledge Projections (KPs), compose seed questions, and then iteratively expand them with an agentic Expander that retrieves web knowledge, constructs sub-questions, and validates them. The resulting 5,000 trajectories are used to train agents via SFT and GRPO-based RL. The authors report state-of-the-art results among open-source IS agents on GAIA and WebWalkerQA across three backbones, and ablations attributing the gains to the formalization and to the layer-wise expansion strategy.","tokens_in":16767,"tokens_out":4374,"duration_ms":46001,"significance":"If the claims hold, WebShaper is a valuable contribution: it provides one of the first set-theoretic formalizations for IS data synthesis, releases the dataset and code, and demonstrates consistent gains over strong baselines on three backbones. The formalization-versus-NL and layer-wise-versus-sequential ablations are the right kinds of controlled experiments and support the central mechanism. However, the load-bearing invariant of answer-set preservation is asserted rather than verified, and the empirical evaluation lacks variance and contamination analysis. These issues are fixable and do not invalidate the approach, but they need to be addressed before the benchmark claims can be fully trusted.","major_comments":[{"comment":"The claim that q_{n+1}(T) always has the same answer as q_n(T) is asserted without proof, and it is not ensured by the Expander's validation procedure. Section 3.2.3 states that Validate checks only whether the type of constant C satisfies the sub-question, not whether C is strictly the answer. If a sub-question is satisfied by multiple entities of the same type, the expanded target set is a superset of the intended T, making the natural-language question ambiguous and the training label incorrect. This invariant is load-bearing because Figure 7a attributes the gains to the formalization. Please provide either a proof of uniqueness under the layer-wise construction, a per-expansion verification step that checks strict answer identity, or a quantitative analysis of how often the current validation admits non-unique sub-questions and how that noise affects downstream performance.","section":"Section 3.2.2, Eq. (11); Section 3.2.3"},{"comment":"All reported scores are single point estimates without error bars, number of seeds, or significance tests. Some margins over strong baselines are small (e.g., QwQ-32B: 53.3 vs. 51.5 for WebDancer on GAIA; 49.7 vs. 47.9 on WebWalkerQA), so the state-of-the-art claim and the formalization attribution are not yet statistically supported. Please report multiple runs or confidence intervals, and state explicitly whether the GAIA evaluation is on the public validation split or the hidden test split and how the judge-based Pass@1 metric was calibrated.","section":"Table 1; Figure 6; Figure 7"},{"comment":"The seed questions are generated by random walks over Wikipedia, and WebWalkerQA is also constructed from web traversal data. The paper reports no contamination analysis between the WebShaper training set and the GAIA or WebWalkerQA evaluation sets. Since the benchmark improvements are the main evidence for the method, please include an n-gram or embedding-based overlap analysis, a deduplication step against the evaluation sets, and a discussion of the remaining overlap risk.","section":"Section 3.1; Table 1; Section 4.2"},{"comment":"The formalization-versus-NL ablation is performed only at the SFT stage, while the main results in Table 1 are reported after SFT plus RL. It would strengthen the central claim to show whether the formalization advantage persists after RL, or to justify why the SFT-only comparison is sufficient to attribute the final RL improvements to the formalization.","section":"Section 4.3.4; Figure 7a"}],"minor_comments":[{"comment":"The sentence \"the agent first first produces a Thought\" contains a duplicated word; please revise.","section":"Section 3.3"},{"comment":"There are typos: \"severl queries\" should be \"several queries\" and \"R-Union oepration\" should be \"R-Union operation.\"","section":"Section 3.2.3"},{"comment":"The text claims \"statistically significantly higher proportions\" but no statistical test is reported; either add a test or soften the wording to \"substantially higher proportions.\"","section":"Section 4.3.6"},{"comment":"The legend labels such as \"Pass@1-Fl\" and \"Pass@1-NL\" are hard to parse at a glance; consider using full names such as \"Pass@1 with formalization\" and \"Pass@1 with natural language.\"","section":"Figure 7"},{"comment":"The sentence \"WebShaper is currently the only open source method with a score of more than 60 points\" should specify that this refers to the GAIA average of the Qwen-2.5-72B model.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper releases its dataset and code, which is commendable and will help reproducibility. I do not see grounds for rejection: the central approach is plausible and the ablations are informative. The main concerns are the unverified answer-set preservation invariant, missing variance estimates, and lack of contamination analysis; all are addressable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper. First, it is a real empirical advance: training on the WebShaper dataset gives consistent gains over prior open-source IS agents across three backbones and two benchmarks, with GAIA 60.1 for Qwen-2.5-72B, and the authors release code and data. Second, the mechanism they credit for the gains—the set-theoretic formalization and the layer-wise expansion—has a gap in its correctness argument that the paper itself implicitly acknowledges. That gap is not fatal, but it needs to be addressed before I'd take the causal claim at face value.\n\nThe new bit is the Knowledge Projection triplet representation and the idea of expanding tasks by replacing leaf constants with sub-questions that are themselves formalized. It's a simple idea—the math is just relational algebra, essentially projection, union, and intersection—but applying it to data synthesis for web agents is novel, and the ablations (formalization vs. natural language, layer-wise vs. sequential) are well designed. Consistent improvements across models and benchmarks suggest the effect is real, not a fluke of one configuration.\n\nThe soft spot is the answer-set preservation claim. Section 3.2.2 states that q_{n+1}(T) always has the same answer as q_n(T), but the Expander's Validate tool only checks whether the type of the constant C satisfies the sub-question, not whether C is the unique answer. If a sub-question has multiple answers, the expanded question's label is wrong for the extra entities. That injects systematic noise into the training data. The final trajectory filtering may remove many bad cases, but it operates on natural-language answers and won't catch a confidently wrong expanded label. So the formalization's core invariant is unverified, and the causal attribution in Figure 7a depends on that invariant.\n\nOther issues are more standard: no error bars or significance tests, no contamination analysis between the synthesized data and GAIA/WebWalkerQA, and the teacher-student self-referentiality (QwQ generates, validates, and is then used for training). These are worth flagging but would not by themselves change the conclusion.\n\nWho should read this: anyone working on synthetic data for LLM agents, or on improving web-based IS agents. It deserves a serious referee—the empirical contribution is substantial and the method is clearly described—but the authors should be pushed to verify the answer-preservation property (e.g., automated checks or human evaluation of expanded sub-questions), report variance across seeds, and speak to contamination.\n\nMy recommendation: send it to peer review, but condition acceptance on the authors closing the validation gap and adding robustness analysis. If the invariant holds, this is a solid contribution; if it doesn't, the gains may still be real but the formalization story needs rethinking.","headline":"A genuinely new formalization-driven recipe for web-agent training data, with consistent benchmark gains, but the paper's central correctness invariant is asserted rather than verified and the evaluation lacks error bars and contamination checks.","tokens_in":17326,"tokens_out":2693,"would_cite":true,"duration_ms":28225,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WebShaper claims that formalizing web information-seeking tasks as set-theoretic Knowledge Projections yields training data that lifts open-source agents to state-of-the-art results on GAIA and WebWalkerQA.","keywords":["information-seeking agents","data synthesis","formalization","Knowledge Projections","set theory","layer-wise expansion","web agents","GAIA benchmark"],"falsifier":"Take a random sample of final expanded questions, hide the formal graph, and have independent annotators (or a stronger oracle) answer each question with web access; then compare each recorded answer with the synthesized one. If a substantial fraction diverge, the claimed answer-preserving property fails at the validation step. A cheaper version is to rerun the synthesis with strict uniqueness validation, requiring the sub-question's answer to be exactly the constant, and check whether downstream GAIA scores drop or rise.","tokens_in":16331,"feed_emoji":"🕸️","tokens_out":7781,"duration_ms":78034,"temperature":0.7,"pith_summary":"WebShaper argues that the bottleneck in training web information-seeking agents is how the training questions are generated: information-driven pipelines collect web content first and then write questions around it, which can make the reasoning structure of the question diverge from the structure of the answer. The paper proposes to invert that order by formalizing every sought answer as a set-theoretic expression built from Knowledge Projections, then synthesizing questions by expanding that expression layer by layer. On this basis it constructs a dataset, trains agents on it, and reports that the resulting models outperform previous open-source information-seeking systems on GAIA and WebWalkerQA. The claim a sympathetic reader should take away is that a controllable formal representation of the task, rather than more data or more natural-language rewriting, is what improves the synthesized training signal.","feed_headline":"Train on formalized web tasks: open-source agents top GAIA","feed_subtitle":"Formal, set-theoretic synthetic questions outperform natural-language pipelines on GAIA and WebWalkerQA.","key_machinery":"The load-bearing object is the Knowledge Projection: for a relation $R\\subseteq E\\times E$ and an entity set $S\\subseteq E$, $R(S)=\\{u \\mid \\exists v\\in S,\\ (u,v)\\in R \\text{ or } (v,u)\\in R\\}$. A target set $T$ is written as an intersection of projections, $T=\\bigcap_i R_i(S_i)$, and R-union is flattened into a merged set using the distributive law, so the formal language only needs intersections of triplets $[X, r, S]$. This representation converts an information-seeking question into a graph whose leaf constants are the only inputs, which is what makes the layer-wise expansion strategy possible: each leaf constant is expanded into a sub-question that has that constant as its answer, so the whole question deepens without attaching extra constants to the target. The Expander agent, built on the ReAct loop with Search, Summarize, and Validate tools, performs the actual expansion and validation at each layer.","core_discovery":"The paper's central claim is that the quality of training data for web-based information-seeking agents is set by the formal structure of the question, not by the richness of the retrieved content. It formalizes an information-seeking task as a query $q(T) \\triangleq \\,?T$ for an entity set $T$ built by composing Knowledge Projections, with intersections of projections carrying multiple conditions and R-union absorbed through the distributive law $R(S_1)\\cup R(S_2)=R(S_1\\cup S_2)$. The synthesis pipeline starts from seed questions, represents each one as a list of triplets $[X, r, S]$ (variables prefixed by `V@`, constants by `C@`), and repeatedly expands the formal graph: at every layer an agentic Expander takes a leaf constant, searches the web, forms a sub-question whose answer type matches that constant, validates it, and splices it into the question. The authors claim this layer-wise expansion preserves the answer set of the original question while preventing reasoning shortcuts, and they report that models trained on the resulting dataset reach state-of-the-art scores among open-source information-seeking agents on GAIA and WebWalkerQA, with an ablation showing the formal-language version beats a natural-language expansion version.","pith_inferences":["If the answer-preserving property holds in practice, the same Knowledge Projection formalization could be applied to other entity-seeking tasks with verifiable relations, such as database queries, knowledge-graph retrieval, or structured fact extraction, not just open-web search.","The type-level validation is the likeliest point of label drift; a testable extension is to require the validator to produce the exact constant or to list all valid answers before accepting a sub-question.","The paper's ablations isolate formal language and layer-wise structure, but they do not isolate seed-question quality; one could train on information-driven datasets that share the same seed pool to see how much of the gain is due to expansion structure alone.","The framework's claim that formalization prevents reasoning shortcuts suggests a further test: measure whether trained agents can still answer questions when the constants are pruned from the prompt, where shortcut-based models should fail more often."],"forward_implications":["The synthesized dataset transfers across base models: WebShaper improves Qwen-2.5-32B, Qwen-2.5-72B, and QwQ-32B relative to training on WebWalkerQA, E2HQA, and MHQA, so the formalization's benefit is not tied to one architecture.","After supervised fine-tuning, reinforcement learning with GRPO further raises GAIA and WebWalkerQA scores, so the data also supports RL-based strategy learning.","Layer-wise expansion produces longer and more uniform tool-use chains than sequential expansion, which should make trained agents more persistent on multi-hop web tasks.","The formalization enables deliberate control of question difficulty and coverage by setting the number of expansion layers, which is impossible when questions are expanded in natural language."],"supporting_citations":[{"why":"WebDancer supplies the seed-question completion pipeline and the SFT-then-RL training recipe that WebShaper adapts, and contributes the E2HQA comparison dataset.","marker":"(Wu et al., 2025a)"},{"why":"GAIA is one of the two benchmarks on which WebShaper reports state-of-the-art open-source results.","marker":"(Mialon et al., 2023)"},{"why":"WebWalkerQA provides the other evaluation benchmark and an information-driven baseline dataset.","marker":"(Wu et al., 2025b)"},{"why":"ReAct is the thought-action-observation loop underlying both the Expander agent and the trajectory construction.","marker":"(Yao et al., 2023)"},{"why":"GRPO is the reinforcement-learning algorithm used to train the final agents.","marker":"(Shao et al., 2024)"},{"why":"WebThinker is a primary open-source comparison baseline and source of the LLM-as-judge evaluation paradigm.","marker":"(Li et al., 2025c)"},{"why":"QwQ is used as the seed-question generator, validator, trajectory agent, and one of the base backbones.","marker":"(Team, 2025)"},{"why":"Qwen2.5 models are the backbones on which dataset comparisons and final training are run.","marker":"(Yang et al., 2024)"}],"fun_headline_variants":["Formalizing web tasks yields SOTA open-source agents","Set-theoretic questions train better info-seeking agents","Formal data synthesis powers top open-source web agents","WebShaper: formal synthesis for state-of-the-art web agents","Agentic expansion with formal structure beats natural language"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The expansion step assumes that replacing a constant with a sub-question leaves the answer set unchanged, but the Validate tool only checks that the constant's type satisfies the sub-question, not that the constant is the unique answer, so an underspecified sub-question can silently change the true answer.","fun_headline_variants_meta":{"raw":{"variants":["Formalizing web tasks yields SOTA open-source agents","Set-theoretic questions train better info-seeking agents","Formal data synthesis powers top open-source web agents","WebShaper: formal synthesis for state-of-the-art web agents","Agentic expansion with formal structure beats natural language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000327,"raw_usage":{"total_tokens":1859,"prompt_tokens":1009,"completion_tokens":850,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":772}},"tokens_in":625,"tokens_out":850,"duration_ms":8706,"temperature":1.0,"reasoning_tokens":772,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:41:34.084468+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of final expanded questions, hide the formal graph, and have independent annotators (or a stronger oracle) answer each question with web access; then compare each recorded answer with the synthesized one. If a substantial fraction diverge, the claimed answer-preserving property fails at the validation step. A cheaper version is to rerun the synthesis with strict uniqueness validation, requiring the sub-question's answer to be exactly the constant, and check whether downstream GAIA scores drop or rise.","supporting_citations":[],"review_version":1}