{"id":"e777aaa3-6a31-4737-a556-215712f34686","arxiv_id":"2608.02163","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An automatic pipeline evolves simple QA questions into 500 deep-research tasks with DAG-structured, fact-grounded rubrics that discriminate between models.","lead":"This paper builds a benchmark of 500 deep-research tasks by automatically evolving simple questions into harder ones with checkable answer rubrics, using a three-role LLM pipeline. It matters because it offers a cheaper, verifiable alternative to expert-written deep-research benchmarks and gives pointwise, not relative, evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stopping criterion measures Explorer saturation, not task saturation; checkpoints inherit a single GPT-5.5 trajectory's blind spots, so coverage completeness is unverified.","rationale":"The reader's weakest assumption—that a single LLM's exploration is a sufficient and reliable oracle—is exactly the load-bearing concern, and I sharpen it by pointing to the stopping criterion's circularity. The pipeline treats DAG stability as evidence of task saturation, but the DAG is generated from the Explorer's own trajectory, so stability can simply mean the Explorer has run out of ideas. This is not a minor caveat; it determines whether the rubrics are complete enough to support absolute, pointwise evaluation. The paper's empirical evidence—human review of 100 tasks, high judge-human correlations, per-round difficulty curves—is real and credible, but it does not settle the completeness question for the 400 unreviewed tasks. The proposed test directly checks whether content coverage is robust to the choice of Explorer, which would either confirm the concern (low overlap / high absent-fact fraction) or disconfirm it (high overlap across models). I do not think this warrants rejection: the benchmark may still be useful for relative ranking, and the human-review evidence is positive. But conditional acceptance is appropriate until the oracle-saturation concern is addressed. My read does not change the reader's conditional verdict; hence UNCHANGED.","tokens_in":26221,"tokens_out":3701,"duration_ms":34466,"concrete_test":"Sample 30–50 converged tasks. For each seed query, rerun the full Explorer-Formalizer-Challenger pipeline with a different frontier Explorer model (e.g., Claude Sonnet 5 or Gemini 3.5 Flash) using identical prompts and stopping rules, then compute the Jaccard overlap between the original and new checkpoint sets and the fraction of new checkpoints that are factual, relevant, and absent from the original rubric. If the independent exploration adds more than ~10% relevant checkpoints or the overlap falls below ~0.6, the stopping criterion reflects Explorer saturation rather than task saturation, and the benchmark's pointwise scores are not absolute. A complementary check: for 20 tasks, have human experts independently list the essential facts a strong answer should cover and compute the recall of the original checkpoints against that list.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that automatically constructed tasks with fact-grounded pointwise rubrics can substitute for expert-authored benchmarks and provide absolute quality scores. The load-bearing assumption is that the gold content of each task—the DAG and its checkpoints—covers what a deep-research answer must include. The construction pipeline derives this content from a single GPT-5.5 Explorer trajectory, and the stopping criterion (Eq. 1) halts evolution when the DAG is unchanged for two rounds. This creates a circularity: the DAG is produced by the Formalizer from the Explorer's trajectory, so the DAG stops changing when the Explorer stops finding new information, not necessarily when the task's information space is saturated. If the Explorer has search blind spots, systematically misses relevant sources, or prematurely abandons a direction, the DAG stabilizes and the task is declared converged even though a more thorough exploration would reveal additional essential content. The resulting rubrics are then incomplete in an undetectable way. Human review of 100 tasks checked verifiability and apparent correctness, not independent completeness against the full space of relevant facts, and the remaining 400 tasks have no such check. Consequently, a model that covers facts outside the rubrics receives no credit, and a model that misses those omitted facts can still score highly. The pointwise score therefore measures coverage of one model's exploration, not absolute deep-research quality. This threatens the central validity claim even if every checkpoint is factually correct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a fully automatic pipeline (Explorer–Formalizer–Challenger) that evolves simple seed questions into 500 deep-research tasks, each represented as a DAG of atomic evidential/analytical steps with associated checkpoints and pointwise rubrics. The authors report three query forms (hinted, without hints, assigned topic), human review of 100 tasks, evaluations of ten models, and comparisons with existing benchmarks. The central claim is that automatically constructed, source-grounded, pointwise rubrics can substitute for expert-authored rubrics and provide absolute, human-aligned, stable evaluation of open-ended deep-research responses.","tokens_in":26551,"tokens_out":3426,"duration_ms":32161,"significance":"If the construction method is valid, this is a substantial contribution: it removes the expert-authoring bottleneck for deep-research benchmarks, provides traceable DAG-structured rubrics, and ships a public 500-task benchmark with code and data. The human-review evidence (Fleiss κ=0.81, no poor ratings on 100 tasks) and judge–human correlations (0.88–0.92 across judges) are meaningful external validity checks. The three-query-form design is also a useful diagnostic contribution. The main risk is that the gold content is generated from a single LLM's exploration trajectory, which may encode that model's search blind spots; the paper's claims of 'absolute' pointwise quality scores depend on resolving this concern.","major_comments":[{"comment":"The central validity claim rests on the rubric covering what a deep-research answer must include. The paper's current evidence addresses verifiability of checkpoints, not completeness of coverage.","section":"Construction Pipeline, Formalizer and Stopping Criterion (Eq. 1)"},{"comment":"Load-bearing? It affects the evaluation-stability claim but the human correlations are strong. Still, the overlap between judge and tested models is a concrete, addressable confound.","section":"Benchmark Evaluation, Table 4 and Table 6"},{"comment":"This is a secondary but still load-bearing point for the paper's claim that the pipeline produces controlled difficulty escalation.","section":"Construction Analysis, Figure 5"}],"minor_comments":[{"comment":"The table formatting is garbled in the arXiv text (e.g., 'Pearsonr–0.73 0.6024–0.79/0.76 0.899'). Please replace with a clean table with separate rows for each benchmark and metric, and define all abbreviations in the caption.","section":"Table 5"},{"comment":"The text says 'all 10,695 checkpoints that encode task knowledge were manually checked as reliable,' but Table 2 reports a mean of 104.85 checkpoints per task; 100 tasks would yield about 10,485 checkpoints. Please reconcile the number or clarify whether the 10,695 figure includes additional per-task metadata.","section":"Human Review (Table 3)"},{"comment":"The reviser is described as using 'at most 100,000 output tokens' per construction step. This seems excessively high for the prompts shown and likely a typo; please confirm the intended token limit.","section":"Hyperparameters (Appendix)"},{"comment":"Top-500 selection by DAG node count is a design choice that biases the benchmark toward broader tasks; please state explicitly in the main text that the benchmark is not a representative sample of seed queries, but a difficulty-filtered subset.","section":"Benchmark Overview"},{"comment":"In Eq. (1), the notation G_t = G_{t-1} = G_{t-2} is used; please clarify that equality is on both the node set and edge set, as stated in the prose, and note whether node IDs must match exactly across rounds.","section":"Notation"}],"recommendation":"major_revision","confidential_remarks":"The core idea is valuable and the empirical effort is substantial, but the manuscript currently overclaims absolute pointwise quality given the single-trajectory derivation of gold content. The fix is achievable: an independent completeness audit on a stratified sample, plus a non-tested judge, would likely resolve my main reservations. I would not support acceptance without that evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the artifact is real and the pipeline is worth engaging with. The main contribution—automatically evolving simple QA into deep-research tasks with DAG-structured pointwise rubrics—is a genuine improvement over relative-ranking benchmarks like AgentDisCo and QUEST. The three query forms and the fact that query, graph, and rubrics co-evolve are new relative to the cited automatic benchmarks. The human review of 100 tasks (Fleiss kappa 0.81, no poor ratings) is honest evidence that the rubrics are usable, and the judge-human correlations are decent. I would trust this benchmark to rank models and to reveal query-type effects.\n\nThe weak spot is the one the stress-test flags. The gold content of every task comes from a single GPT-5.5 Explorer trajectory. The stopping criterion only checks that the DAG is unchanged for two rounds, which means the process stops when the Explorer stops finding new information, not when the task's information space is saturated. The 100-task human audit checked verifiability and apparent correctness, not completeness against the full space of relevant facts. So the pointwise scores are coverage of one model's exploration, not an absolute measure of deep-research quality. That is the key limitation, and it should be stated more directly in the paper.\n\nTwo smaller things. The evaluator Qwen3.7-Max is also one of the ten tested models in Table 4. That is a self-evaluation confound for the model-ranking claims, even though the judge-human correlations suggest the judge is reasonable. And the iterative query-hardening closely resembles the Evol-Instruct/WizardLM pattern, which the paper does not cite; that overstates the novelty. Neither issue is fatal.\n\nI would send this to peer review. The pipeline, the artifact, and the human-checked rubric set deserve a serious referee. The authors should either soften the absolute-score claim or add an independent completeness audit on a sample of tasks—e.g., have a second model or human annotation build the DAG from scratch and measure recall against it.","headline":"The DAG-evolving task pipeline is a real step forward for automatic deep-research benchmarks, but the gold content derives from a single model's exploration, so the pointwise scores measure coverage of that trajectory, not absolute research quality.","tokens_in":637,"tokens_out":1695,"would_cite":true,"duration_ms":36435,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fully automatic pipeline turns simple questions into 500 deep-research tasks with rubrics grounded in web evidence.","keywords":["deep research","benchmark construction","automatic evaluation","rubrics","DAG","task evolution","checkpoints","LLM agents"],"falsifier":"Have independent domain experts write gold answers for a random subset of the 500 tasks, then check whether every expert-identified essential point corresponds to a checkpoint in the automatic rubric and whether the checkpoints themselves are factually accurate against authoritative sources. If sizable gaps or errors appear in that sample, the rubric-based scores do not measure deep-research quality.","tokens_in":26088,"feed_emoji":"🔍","tokens_out":3633,"duration_ms":29938,"temperature":0.7,"pith_summary":"The paper claims that deep-research benchmarks — tasks that require multi-source search, integration, and open-ended answers — can be built fully automatically, without expert-authored questions or rubrics. It introduces a pipeline that starts from simple factual queries and iteratively evolves them into hard research tasks, representing each task as a directed acyclic graph of atomic steps. Every step carries checkpoints grounded in evidence gathered during exploration, and these checkpoints become the scoring rubric. The claim is that this produces 500 tasks with rubrics that give fine-grained, absolute, human-aligned scores, not just relative rankings. If right, it removes the main bottleneck to scalable deep-research evaluation: dependable pointwise scoring without human authoring.","feed_headline":"Automatic pipeline builds 500 verifiable deep-research tasks","feed_subtitle":"An Explorer-Formalizer-Challenger loop grounds every rubric checkpoint in web evidence for pointwise scoring.","key_machinery":"The load-bearing mechanism is the task graph: a DAG whose nodes are atomic research steps (evidential nodes retrieve traceable facts; analytical nodes reason over them), and whose checkpoints are concrete, independently verifiable claims derived from the Explorer's trajectory. The Formalizer builds and maintains this DAG, and the rubric set is exactly one rubric per node, with weights propagated from leaves to parents so that steps supporting more downstream conclusions count more. Evaluation scores a response by whether its content satisfies the checkpoints under these weights — without requiring the response to reproduce the graph's solving path. The iterative loop (Explorer → Formalizer →","core_discovery":"On the paper's own terms, the central discovery is that the task-specific knowledge needed to evaluate a deep-research answer — what evidence and reasoning a good response must include — can be revealed incrementally by an agent exploring the web, and that this same exploration can be converted into a verifiable scoring rubric. An Explorer solves a simple query and records a trajectory; a Formalizer distills the trajectory into a minimal DAG of evidential and analytical steps with independently checkable checkpoints; a Challenger uses explored-but-unused clues to pose a harder next query. Repeating this loop grows a simple question into a professional-level research task while the query, DAG","pith_inferences":["The paper's own limitation about construction cost (about 168 hours and $10,000) suggests a practical ceiling: scaling to thousands of tasks is expensive, although the cost is paid per benchmark, not per evaluation.","Checkpoint-based scoring likely rewards coverage of the constructor's chosen evidence and may undervalue a response that reaches correct conclusions through novel sources the Explorer never visited; a test would be whether accepted alternative answers score well.","The stability of scores across judges is reported for 100 tasks; checking all 500 and testing against independent expert rubrics would reveal whether the automatic rubrics generalize beyond their origin trajectory.","Because the Challenger can only use clues already in the trajectory, the benchmark's difficulty is bounded by the Explorer's search behavior; a stronger Explorer would produce harder tasks, so the method couples benchmark quality to the constructor model's competence."],"forward_implications":["Benchmark construction in new domains can be automated: given any seed corpus of simple questions, the pipeline yields research tasks with traceable rubrics in roughly 14 rounds and about 265 tool calls per task.","Scores from this benchmark are absolute pointwise measures, not relative rankings; a model's score does not depend on which other models are in the comparison.","Distinguishing query forms (with hints, without hints, assigned-topic) isolates separate deep-research abilities, so failures can be attributed to retrieval, planning, or argument formation.","Because checkpoints are fact-grounded, the same rubric can be re-applied by different judges (LLM or human) with high agreement, enabling stable longitudinal comparisons.","If the pipeline works, it could be extended to professional domains, turning sparse long-tail knowledge into verifiable research tasks without expert labor."],"fun_headline_variants":["Iterative loop turns simple questions into deep research benchmarks","500 deep-research tasks built by self-improving pipeline","From simple QA to deep research via iterative task evolution","Verifiable deep-research benchmark from iterative task growth","Pipeline evolves simple queries into graded deep-research tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that one language model's web exploration is a sufficient and reliable oracle for the task-specific knowledge a deep-research answer must cover, so checkpoints derived from that single trajectory are treated as gold.","fun_headline_variants_meta":{"raw":{"variants":["Iterative loop turns simple questions into deep research benchmarks","500 deep-research tasks built by self-improving pipeline","From simple QA to deep research via iterative task evolution","Verifiable deep-research benchmark from iterative task growth","Pipeline evolves simple queries into graded deep-research tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000591,"raw_usage":{"total_tokens":2582,"prompt_tokens":695,"completion_tokens":1887,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":1810}},"tokens_in":439,"tokens_out":1887,"duration_ms":10635,"temperature":1.0,"reasoning_tokens":1810,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T13:27:44.552099+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent domain experts write gold answers for a random subset of the 500 tasks, then check whether every expert-identified essential point corresponds to a checkpoint in the automatic rubric and whether the checkpoints themselves are factually accurate against authoritative sources. If sizable gaps or errors appear in that sample, the rubric-based scores do not measure deep-research quality.","supporting_citations":[],"review_version":1}