{"id":"102dc660-6726-4589-b639-e6266e71fd0d","arxiv_id":"2505.15859","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AutoData, a multi-agent system with a hypergraph message cache, automates web dataset collection from a sentence instruction and outperforms general agent baselines on the new Instruct2DS benchmark.","lead":"AutoData is a multi-agent AI system that turns a natural-language instruction into a ready-to-use dataset by having eight agents plan, browse, code, and validate the data. The paper also introduces a new benchmark, Instruct2DS, for open web data collection, and reports that AutoData outperforms general-purpose agent baselines in accuracy, speed, and cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Manus and human baselines were evaluated on an unspecified subset of Instruct2DS tasks, while AutoData was scored on all 234 tasks; Table 1's 'surpasses all baselines' claim is not established without matched task sets.","rationale":"The reader identified task sampling and ground-truth completeness as the weakest assumption. The baseline-subset issue is a distinct and more direct threat to the central relative claim: even with a perfect ground truth, comparing full-benchmark AutoData scores to subset-evaluated Manus and human scores cannot establish superiority. The paper itself acknowledges the subset in Appendix C, so this is a concrete internal limitation, not speculation. AutoData's margin over the strongest baselines is small in the sport domain (about 2.6 F1 over Manus), so task-selection bias or stochastic run-to-run variation could plausibly flip that result. The paper could still be correct, but the evidence does not yet support the unqualified 'surpasses all baseline methods' statement. The appropriate verdict remains CONDITIONAL, with the condition now explicitly requiring matched task sets and repeated runs.","tokens_in":21498,"tokens_out":6906,"duration_ms":63685,"concrete_test":"Require the authors to release the exact list of task IDs used in the Manus and human baseline runs, rerun AutoData on only that list, and verify that AutoData still achieves the highest F1 in each domain on that matched subset. If it does, the concern is resolved; if not, Table 1 is invalid. Ideally, also run all baselines on the full 234-task set with at least three independent runs and report mean and standard deviation per domain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix C states that Manus experiments were run on 'a representative subset of tasks' due to cost, and human programmers similarly 'leverage a representative subset.' Table 1 reports AutoData on the full Instruct2DS benchmark (234 tasks, per Appendix D.4) and gives single F1/precision/recall/time/expense numbers for every method. If the Manus and human rows are computed over a different, smaller task sample, then the headline claim that AutoData 'surpasses all baseline methods in every domain' is not a comparison on the same evaluation set. The paper provides no task IDs, no subset size, no sampling procedure, and no variance information, so the magnitude or even the direction of the gap could be an artifact of task selection. Because the central claim is explicitly relative, the comparison set must be identical across methods; otherwise the claim is unverified. The same concern applies to the time and expense comparisons, since those aggregate quantities are also computed over possibly different task sets.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AutoData, a multi-agent system that takes a natural-language instruction and produces a dataset by orchestrating eight specialized agents in research and development squads, coordinated by a central manager and communicating through a novel 'oriented hypergraph cache system' (OHCache). The authors also introduce Instruct2DS, a new benchmark for open web data collection with 234 tasks in academic, finance, and sports domains. The system is evaluated on Instruct2DS, SWDE, EXTENDED SWDE, and HumanEval, with additional case studies on picture-book collection and paper collection from surveys. The central claim in Section 5.1 is that AutoData surpasses all baseline methods in every domain of Instruct2DS while requiring less implementation time and lower expense.","tokens_in":21655,"tokens_out":4300,"duration_ms":41625,"significance":"If the reported results hold, the paper would make a useful contribution: a low-cost, instruction-driven data-collection system, a new live-web benchmark, and a communication architecture that addresses token-cost and information-overload problems in multi-agent systems. The authors are to be credited for evaluating against external ground truth, comparing with a wide range of baselines including human programmers and commercial agents, and providing source-code and dataset links. The case studies, ablation experiments, and LLM-backbone sensitivity analysis (Appendix E.1) also add useful evidence. However, the significance is currently contingent on the fairness and completeness of the evaluation, which has several load-bearing weaknesses discussed below.","major_comments":[{"comment":"The headline comparison is not made on a matched task set. AutoData is scored on all 234 Instruct2DS tasks (Appendix D.4), while Appendix C states that Manus and the human-programmer baseline were run on 'a representative subset of tasks' with no subset size, task IDs, or sampling procedure. Since the central claim in Section 5.1 is explicitly relative ('surpasses all baseline methods in every domain'), the comparison is unverified unless all rows of Table 1 are computed over the same task set. The time and expense comparisons are affected equally because they are aggregate quantities. The authors should either evaluate all baselines on the full 234-task set or restrict all methods, including AutoData, to a single matched subset, and report the exact tasks used.","section":"Section 5.1, Table 1, Appendix C, Appendix D.4"},{"comment":"The Instruct2DS evaluation samples only 'valid tasks,' defined as instructions that yield non-empty ground truth, which excludes instructions for which the web source does not contain the requested data or for which extraction is difficult. This is a form of selection bias in the benchmark itself: the reported F1 scores are conditional on data being accessible, and the excluded cases could be systematically harder for AutoData than for other methods. The authors should quantify how many generated instructions were discarded as invalid, report task-level success/failure rates, and show that the conclusions are robust when invalid tasks are included or when the validity criterion is applied equally across all methods.","section":"Appendix D.4"},{"comment":"All results are reported as single point estimates without error bars, confidence intervals, or significance tests. Because AutoData and the baselines are LLM-based and stochastic, and because the reported differences are often small (for example, Table 2 shows AutoData and Manus within 0.03 F1 on SWDE), single-run numbers do not support the claim of superiority. The authors should provide multiple runs (at least three to five) with means and standard deviations, or bootstrap confidence intervals, and where feasible paired tests over the same task set. The same issue applies to the ablation study in Figure 3, which is presented only graphically and without numerical values or variance information.","section":"Tables 1, 2, 4, 5 and Figure 3"},{"comment":"The ablation study is not reported quantitatively. The text states that removing components or agent squads leads to 'notable decline in performance' or 'marginal' changes, but Figure 3 contains no numerical values, no task-level breakdown, and no indication of how many tasks were used. Since the OHCache architecture is a central claimed contribution, its contribution should be documented with a table of F1, precision, recall, time, and cost for each ablation condition on the same task set used in Table 1.","section":"Section 5.4, Figure 3"}],"minor_comments":[{"comment":"The Human row reports a time value but no expense, while the surrounding text claims AutoData incurs 'lower expenses' than existing methods; please clarify the expense basis for the Human baseline and for each method.","section":"Table 1"},{"comment":"Several numeric cells appear to be malformed, for example the COT row shows '2.447.32', which is likely two numbers (2.44 and 7.32) concatenated without a separator. Please fix the table formatting.","section":"Table 2"},{"comment":"There are typos such as 'Baseketball' for 'Basketball' and, in Figure 6, 'corporate' appears as 'corporte' and 'distributed' as 'distrbuted'. Please proofread the appendices.","section":"Appendix A.4"},{"comment":"The experiments are said to use four NVIDIA A40 GPUs, but the method is implemented via LLM API calls with a GPT-4o backbone; please clarify what computation is actually performed on the local GPUs.","section":"Appendix D.1"},{"comment":"The definition of 'time' excludes program execution time, so the reported time values reflect only the interval from instruction release to the start of data collection; this should be stated prominently in the main text because it materially affects the interpretation of the efficiency comparison.","section":"Appendix D.2"}],"recommendation":"major_revision","confidential_remarks":"The central weakness is the unmatched evaluation set for Table 1; this is not a matter of style but of whether the paper's main claim can be verified from the reported data. The authors should be encouraged to release the exact task lists used for every method, or to reframe all comparisons to a common subset. The valid-task sampling in D.4 is also a form of label selection that should be disclosed prominently and analyzed for robustness. There is no indication of misconduct or circularity; the issues are standard evaluation-rigor problems that are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The Instruct2DS benchmark is the real contribution here: the first live open-web data collection benchmark with ground truth across three stable domains, and that alone deserves serious attention. AutoData also looks like a serious engineering effort, with a clear eight-agent architecture and an OHCache mechanism that plausibly cuts token cost. The ablation, though only graphical, suggests OHCache helps; the HumanEval result is credible; and the limitations appendix is honest about proprietary LLM dependence and anti-bot walls.\n\nBut the central comparison in Table 1 is compromised. Appendix C says Manus and the human programmers were run on \"a representative subset of tasks\" for cost reasons, while AutoData ran on all 234 tasks. There are no task IDs, no subset sizes, no sampling procedure. The paper's headline claim — \"surpasses all baseline methods in every domain\" — is a relative claim, and it cannot be verified when the methods are not scored on the same task set. The direction and even the existence of the gap could be an artifact of task selection. The same problem poisons the time and expense comparisons, since those aggregates are computed over possibly different sets. This is not a minor reporting gap; it is load-bearing for the paper's main claim.\n\nThe other weaknesses are more standard. Single runs with no error bars or significance tests. The \"valid task\" definition in Appendix D.4 excludes cases where the web source lacks the requested data, which could bias the benchmark toward easier tasks. Ground truth correctness is asserted through manual cross-validation, but neither code nor data is released at the moment, so the benchmark cannot yet be used or audited.\n\nWho is this paper for? Researchers building LLM-based agents for web scraping, and anyone designing benchmarks for data collection agents. It deserves a serious referee, but not in its current form. The fix is straightforward: run all baselines on the same, precisely specified task set (even if smaller), report per-task results or variance, and release the code, benchmark, and ground truth construction pipeline. With matched evaluation and released artifacts, the paper could be solid. As it stands, the headline claim is unverified, and I would not cite the results in my own work until the data is available.\n\nRecommendation: conditional acceptance, pending matched-set evaluation and release of code and data. Send to peer review, but flag the matched-set issue as the top reviewer concern.","headline":"Useful system and a genuinely new benchmark, but the headline comparison on Instruct2DS is not matched-set, so the central claim isn't established yet.","tokens_in":22242,"tokens_out":1754,"would_cite":false,"duration_ms":16583,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoData claims that a coordinated multi-agent system can turn a one-sentence data request into an accurate, ready-to-use dataset collected live from open web sources, outperforming general-purpose agents, programming assistants, and…","keywords":["multi-agent system","open web data collection","natural language instruction","oriented hypergraph cache","LLM agents","web scraping","Instruct2DS benchmark","token cost reduction"],"falsifier":"Independently rebuild the Instruct2DS ground truth for a random 50-task sample using the paper's published instruction templates, run AutoData on those same instructions, and check whether the reported F1 scores reproduce; separately, include tasks where the target web source is sparse or locked down, since the 'valid task' definition removes such cases from the reported averages.","tokens_in":21280,"feed_emoji":"🤖","tokens_out":13542,"duration_ms":100588,"temperature":0.7,"pith_summary":"AutoData is designed to prove that open web data collection can be fully automated: the user gives a single natural-language instruction describing the dataset they want, and the system returns a ready-to-use dataset. The paper argues that existing solutions are either brittle wrappers requiring manual per-site engineering or expensive LLM-based agents that burn tokens, and it introduces a multi-agent system with eight specialized agents split into research and development squads, coordinated by a manager and a communication layer called OHCache. On a new benchmark, Instruct2DS, spanning academic, finance, and sports domains, AutoData reports the highest F1, precision, and recall in every domain while taking less implementation time and lower monetary expense than every baseline, including human programmers and the strongest general-purpose agents. If these results hold, dataset acquisition becomes much cheaper and more accessible for researchers, educators, journalists, and other non-experts.","feed_headline":"One prompt in, a clean dataset out: AutoData tops baselines","feed_subtitle":"Eight coordinated agents beat general-purpose rivals on accuracy, time, and cost.","key_machinery":"The carrying mechanism is OHCache, the oriented hypergraph cache system for agent communication. In OHCache, messages are modeled as oriented hyperedges of an eight-node hypergraph, each hyperedge directed from a single source agent to a targeted subset of receivers, so no message is broadcast to everyone and no agent processes an unbounded history. A formatter turns each agent output into a structured message before insertion, and a local cache node stores large artifacts, broadcasting only a cache identifier so raw HTML or API payloads never clog the message channel. This coordination layer is what keeps the multi-agent pipeline cheap enough to run, and the paper's ablations show that removing OHCache, the formatter, or the cache degrades performance and raises cost.","core_discovery":"The central claim is that a single instruction can be turned end-to-end into an accurate, ready-to-use dataset from live web sources, and that the proposed system, AutoData, does this better and more cheaply than the alternatives it is compared with. The workflow is a two-stage division of labor: a research squad plans, browses the web, and consolidates a development blueprint, then a development squad implements, debugs, executes, and validates the collection program. The innovation that makes this efficient is OHCache, which replaces broadcast messaging among agents with an oriented message hypergraph—each message is a directed hyperedge from one source agent to a chosen set of recipients—and moves bulky artifacts such as HTML files into a local cache that agents fetch on demand. The paper reports that on its new Instruct2DS benchmark across academic, finance, and sports domains, AutoData surpasses every baseline in every domain with less implementation time and lower expense, and it presents supporting results on SWDE, EXTENDEDSWDE, HUMANEVAL, and two case studies to show the approach transfers to legacy extraction benchmarks, code generation, and specialized collection tasks.","pith_inferences":["Because the paper defines a 'valid task' as one with non-empty ground truth, the reported F1 numbers cover only cases where the web source contains the requested data; a stress test that deliberately includes sparse, login-walled, or JavaScript-rendered sources could expose a different ranking.","The oriented-hypergraph routing idea is general: any multi-agent workflow where a few agents share an artifact—software development, literature review, data curation—could adopt the targeted-hyperedge plus local-cache pattern to cut token costs, independent of web scraping.","The cost figures are tied to proprietary LLM pricing; the paper's own ablations show an open-weight backbone performs markedly worse, so the dollar-and-minute advantages should be treated as conditional on the chosen model and vendor rates.","Releasing the Instruct2DS database construction algorithm and the full sampled task list would let others independently recompute the ground truth; the paper asserts manual cross-validation but publishes neither the database nor the algorithm."],"forward_implications":["Routine dataset requests—such as all accepted papers of a conference in a given year—can be executed by non-programmers, removing per-site wrapper engineering.","On Instruct2DS, the reported average is 5.58 minutes and $0.57 per task, versus 15.37 minutes and $2.49 for the runner-up general agent, making large-scale collection plausible.","Instruct2DS provides a public live-web benchmark with 234 sampled tasks over three domains, giving future agent systems a standard test rather than ad hoc scrapes.","The same architecture achieves 92.5 Pass@1 on HUMANEVAL with a GPT-4o backbone, suggesting the development squad is a capable general-purpose coder, not just a scraper.","Case studies on picture-book collection and survey-paper BibTeX show the system handles multi-level HTML crawling and per-source quirks with higher human-evaluated accuracy and lower cost than the best general agent."],"supporting_citations":[{"why":"The proprietary general-purpose agent that serves as the strongest baseline on Instruct2DS and as the comparison point in both case studies.","marker":"[42]"},{"why":"A fully automated zero-code agent framework that AutoData is compared against on Instruct2DS.","marker":"[43]"},{"why":"An open-source general multi-agent system whose results establish the baseline level for open agents on the new benchmark.","marker":"[44]"},{"why":"An automatic agent-generation framework included in the Instruct2DS baseline comparison.","marker":"[45]"},{"why":"Supplies the ReAct reasoning-and-acting loop that every AutoData agent follows, making it the behavioral foundation of the system.","marker":"[32]"},{"why":"Provides the evaluation metrics and seed-page experiment design used to compare on SWDE and EXTENDEDSWDE.","marker":"[50]"},{"why":"The multi-agent coding baseline whose setup is followed for HUMANEVAL, supporting the claim that the development squad is a capable programmer.","marker":"[26]"},{"why":"The SWDE structured-web extraction corpus used as a secondary evaluation to show AutoData generalizes beyond the new benchmark.","marker":"[15]"},{"why":"The HumanEval code-generation benchmark used to evaluate the development squad's coding ability.","marker":"[52]"}],"fun_headline_variants":["One prompt, one dataset: AutoData tops web-collection baselines","AutoData: single instruction, accurate datasets, lower costs","AutoData: from natural language to a ready dataset in one go","AutoData: multi-agent system turns one prompt into datasets","AutoData: beat baselines on live web data with one instruction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation assumes that the manually constructed Instruct2DS ground truth is complete and correct, and that restricting the benchmark to 'valid tasks' whose ground truth is non-empty does not systematically exclude precisely the hard cases where AutoData's advantage could shrink.","fun_headline_variants_meta":{"raw":{"variants":["One prompt, one dataset: AutoData tops web-collection baselines","AutoData: single instruction, accurate datasets, lower costs","AutoData: from natural language to a ready dataset in one go","AutoData: multi-agent system turns one prompt into datasets","AutoData: beat baselines on live web data with one instruction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1742,"prompt_tokens":1033,"completion_tokens":709,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":621}},"tokens_in":649,"tokens_out":709,"duration_ms":6419,"temperature":1.0,"reasoning_tokens":621,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:23:32.238536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently rebuild the Instruct2DS ground truth for a random 50-task sample using the paper's published instruction templates, run AutoData on those same instructions, and check whether the reported F1 scores reproduce; separately, include tasks where the target web source is sparse or locked down, since the 'valid task' definition removes such cases from the reported averages.","supporting_citations":[{"cited_title":"Leave it to manus, 2025","cited_arxiv_id":null,"evidence_quote":"The proprietary general-purpose agent that serves as the strongest baseline on Instruct2DS and as the comparison point in both case studies."},{"cited_title":"Autoagent: A fully-automated and zero-code framework for llm agents.arXiv e-prints, pages arXiv–2502, 2025","cited_arxiv_id":null,"evidence_quote":"A fully automated zero-code agent framework that AutoData is compared against on Instruct2DS."},{"cited_title":"Autoagents: The automatic agents generation framework","cited_arxiv_id":null,"evidence_quote":"An automatic agent-generation framework included in the Instruct2DS baseline comparison."},{"cited_title":"Autoscraper: A progressive understanding web agent for web scraper generation","cited_arxiv_id":null,"evidence_quote":"Provides the evaluation metrics and seed-page experiment design used to compare on SWDE and EXTENDEDSWDE."},{"cited_title":"From one tree to a forest: a unified solution for structured web data extraction","cited_arxiv_id":null,"evidence_quote":"The SWDE structured-web extraction corpus used as a secondary evaluation to show AutoData generalizes beyond the new benchmark."}],"review_version":1}