{"id":"6bb31164-4883-4edc-a00c-61e13634363a","arxiv_id":"2506.10055","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TaskCraft automatically generates verifiable, multi-tool agentic tasks with execution trajectories via atomic task construction and depth/width extension, and fine-tuning on the resulting 36,000-task dataset improves multi-hop QA.","lead":"This paper introduces TaskCraft, a pipeline that automatically generates thousands of tool-using agent tasks (search, PDF, image) with step-by-step execution traces, without human annotation. The authors show that fine-tuning a small language model on these synthetic traces improves its multi-hop question answering, suggesting a cheaper path to training and evaluating AI agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Golden answers for depth-extended tasks are never verified by agent execution; the SFT gains and 'verifiable' claim rest on uncalibrated linguistic-only checks.","rationale":"The reader identified the unmeasured judge-LLM verification as the weakest assumption, specifically that depth-extended tasks are validated only by linguistic checks and never executed. I agree and sharpen the concern: the SFT trajectories used in Table 3 come from these unexecuted tasks, so their golden answers are unverified. This is the most load-bearing issue because it directly threatens both halves of the central claim—verifiability and the value of the training data. While the missing alternative-data baseline and internal inconsistencies are real, they are secondary; if the trajectories contain wrong or unreachable answers, the SFT results lose their evidential foundation. The concern is testable and addressable, so the existing CONDITIONAL verdict is appropriate; no change is needed.","tokens_in":14939,"tokens_out":3908,"duration_ms":45510,"concrete_test":"Sample 200 depth-extended tasks from the released dataset (or regenerate via the published pipeline). Execute each task end-to-end with the same agent setup used for atomic verification (or Smolagents) using the real tools/retriever, and record whether the final answer matches the golden answer. If more than, say, 20% of tasks fail to reach the golden answer—or if a human audit of 100 judge-LLM decisions shows verifier accuracy below, say, 90%—the 'verifiable' claim and the reliability of the SFT trajectories are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that TaskCraft generates 'verifiable' agentic tasks and that 3,202 generated multi-hop trajectories improve SFT (Table 3). The load-bearing weakness is that the depth-extended tasks, which produce those SFT trajectories, are never executed by an agent. Section 3.3 states that extension verification is performed 'purely through linguistic analysis without agent involvement'; Appendix B.3.1/B.3.2 rely on judge-LLM checks for superset validity and information concealment only. No step confirms that the golden answer is reachable by actual tool calls, and judge-LLM accuracy is never measured against a human-audited sample. If the superset index is hallucinated, retrieval fails, or the chain is leaky in ways the LLM judge accepts, the trajectory's intermediate and final answers can be wrong. Training on such trajectories could still yield Table 3 gains by teaching general search-and-reason patterns, but the 'verifiable' property—and the quality of the SFT data—is unestablished. The alternative-data baseline issue is secondary; the integrity of the training signal is primary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces TaskCraft, an automated pipeline that generates agentic tasks by sampling atomic tasks from an unlabeled corpus via tool contexts, extending them with depth-based and width-based transformations, and filtering candidates with judge-LLM verification. The authors report that the generated tasks improve prompt optimization in the generation workflow (Table 2) and that supervised fine-tuning on 3,202 generated multi-hop trajectories improves exact match on HotpotQA, Musique, and Bamboogle (Table 3), with the largest gain for Qwen2.5-3B-Base (average 0.034 to 0.174, and Search-R1+SFT to 0.245). They release a dataset of approximately 36,000 tasks.","tokens_in":14960,"tokens_out":7705,"duration_ms":87695,"significance":"If the verification pipeline were properly calibrated, TaskCraft would be a useful contribution: it addresses a real scalability bottleneck in agentic benchmark construction, and the held-out SFT results on three standard multi-hop QA benchmarks are genuinely positive evidence that the generated trajectories teach useful tool-use and reasoning patterns. The paper also provides a public code/data link and a detailed workflow appendix. However, the central 'verifiable' claim is not yet supported, because the verifier's accuracy is unmeasured and depth-extended tasks are never executed by an agent. The SFT gains therefore cannot yet be attributed to correct, reachable golden answers. With additional verification experiments, the released dataset could become a valuable community resource.","major_comments":[{"comment":"The 'verifiable' guarantee is not established for the depth-extended tasks that supply the SFT trajectories. Section 3.3 states that task-extension verification is 'conducted purely through linguistic analysis without agent involvement,' and Appendix B restricts checks to judge-LLM superset validation and information-leakage filtering; no agent executes an extended task, no judge-LLM accuracy against a human-audited sample is reported, and no human sample audit is described. Since Table 3's SFT uses 3,202 multi-hop trajectories from this extension pipeline, a judge that accepts pseudo-supersets, wrong intermediate answers, or leaky chains would inject incorrect targets into training. Please report judge-LLM agreement with human labels and run an executor agent on a random sample of retained depth-extended tasks to verify that the golden answer is reachable by actual tool calls.","section":"Section 3.3; Appendix B.3.1, B.3.2"},{"comment":"The 'ideal search engine' assumption (Section 2) is load-bearing for golden-answer reachability. The workflow assumes that an index iT retrieves precise content, but Section 4.4 and Appendix D evaluate agents with a fixed retrieval depth of 3 over a 2018 Wikipedia dump, which need not contain the superset indices (e.g., i^{n+1}_T) used in the generated trajectories. If retrieval fails for an intermediate index, the recorded intermediate answer and final answer cannot be reproduced by the trained agent. Please report retrieval success rates for the generated indices in the actual evaluation environment, or explicitly restrict the reachability claim to an oracle-retrieval setting.","section":"Section 2; Appendix D"},{"comment":"The definition of 'atomic task' conflicts with the measured behavior. Section 2 defines an atomic task as solvable with 'a single target tool invocation,' yet Section 3.3 relaxes atomic verification to allow up to three tool-use steps, and Table 4 reports an average of 2.1 tool uses per query for the retained atomic tasks (and 2.8 for the LLM-only baseline). This suggests that many retained atomic tasks require multiple tool calls, weakening the atomicity guarantee and the claim that tool requirements are 'regulated.' Please report the distribution of tool-use counts and ideally enforce the single-tool criterion during verification.","section":"Section 3.2; Table 4"}],"minor_comments":[{"comment":"The dataset size is reported inconsistently: the abstract and conclusion say approximately 36,000 tasks, the contributions list says about 32k tasks, and Appendix A reports 22,053 atomic tasks. Please reconcile these numbers and define what counts as a task.","section":"Abstract; Contributions; Appendix A"},{"comment":"Table 1 compares GAIA levels with tool-modality categories, which is not an apples-to-apples difficulty comparison. Please clarify the intended claim and add per-level or per-modality error bars or descriptions.","section":"Table 1"},{"comment":"The 'Base workflow' is described as 'Search-R1 without training,' which is confusing because Search-R1 is itself an RL-trained workflow. Please rename or clarify what 'without training' means.","section":"Section 4.4"},{"comment":"Table 3 reports single-run point estimates without confidence intervals or significance tests; given that the Instruct variant improves over Search-R1 by only +0.6% on average, please report variance across seeds or runs.","section":"Table 3"},{"comment":"The definitions of 'pass rate' differ between Table 2 and Table 4 (54.9% vs. 43.0% for atomic tasks); please state the exact numerator and denominator for each table.","section":"Tables 2 and 4"},{"comment":"References [10]/[11] (Bamboogle) and [31]/[32] (HotpotQA) are duplicated; please consolidate them into single citations.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from an explicit statement that the released trajectories are generated traces, not necessarily verified executions. If the authors can supply a human-audited verification study and retrieval-success measurements, the paper's contribution would be substantially stronger. The paper may be better received at a data or benchmark venue than as a methods paper, given that the core novelty is the data-generation workflow."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is solid: generate atomic tasks grounded in real tool contexts, then extend them depth- and width-wise to build multi-tool agentic tasks with trajectories. The tool-grounded construction (q = f(i_T, R)) and the strict-superset and information-leakage checks for depth extension are genuinely new relative to WizardLM-style rule-based evolution. The released dataset of ~32k-36k tasks with trajectories is a useful community resource, assuming the repo checks out.\n\nThe empirical core is the SFT result in Table 3: training on 3,202 generated multi-hop trajectories lifts Qwen2.5-3B-Base exact match on HotpotQA, Musique, and Bamboogle from 0.034 to 0.174, and Search-R1+SFT reaches 0.245. That is held-out evidence, not circular, and the Table 4 ablation does support the tool-context construction over direct LLM prompting. The prompt-optimization gains in Table 2 are plausible but less interesting. Credit where due: the workflow is a legitimate extension of the Self-Instruct/Evol-Instruct program into the agentic setting, and the SFT gains are non-trivial.\n\nNow the soft spots, in proportion. The load-bearing weakness is the 'verifiable' claim. Atomic tasks are filtered by judge-LLM comparison with a bare LLM, but judge accuracy is never measured, and depth-extended tasks—which produce the SFT trajectories—are verified only by linguistic checks, never by agent execution. Appendix B.3 describes exactly this. So the golden answers for the training trajectories are not confirmed reachable. If judges accept pseudo-supersets or leaky chains, the trajectories can contain wrong intermediate or final answers. Table 3 gains might survive such noise, but the 'verifiable' property and the data quality guarantee are unestablished. That is the main issue.\n\nSecondary issues: the SFT experiment lacks a same-size alternative-data baseline, so the specific benefit of TaskCraft's mechanism is not isolated from just having more multi-hop data; the abstract says ~36k tasks while the contributions and data stats say ~32k, and the GAIA average in Table 1 (44.20) does not match the computed average from the three levels (41.55); and AgentInstruct, the closest prior work on agentic task generation, is not cited. All addressable, none fatal.\n\nWho gets value: people working on synthetic data for agent training, especially SFT/RL data pipelines. It deserves a serious referee, but the 'verifiable' claim needs to be either backed by a human-audited verifier accuracy measurement and agent-execution spot-checks of extended tasks, or softened. I would not reject it; I would send it to review with a request for major revision on the verification evidence.","headline":"TaskCraft is a genuinely useful extension of the self-instruct line into tool-using agents, but the 'verifiable' claim rests on uncalibrated judge-LLMs and depth-extended tasks are never executed by an agent, so the training-signal integrity is unestablished.","tokens_in":15759,"tokens_out":1097,"would_cite":true,"duration_ms":13987,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TaskCraft is an automated workflow that turns unlabeled corpora into difficulty-scaled, multi-tool agentic tasks with execution trajectories, and shows that fine-tuning on those tasks improves agent performance on multi-hop QA.","keywords":["agentic tasks","synthetic data generation","multi-hop reasoning","tool use","supervised fine-tuning","task verification","difficulty scaling","execution trajectories"],"falsifier":"Sample a random set of depth-extended tasks from the released dataset, have fresh agents execute them without seeing the golden trajectories, and check whether each final answer is reachable, unique, and only findable through the intended tool chain. If a significant fraction fail, because supersets are pseudo-supersets, answers are leaked in the question, or no trajectory reaches the golden answer, the verifiability guarantee at the core of the workflow is falsified.","tokens_in":14552,"feed_emoji":"🤖","tokens_out":8865,"duration_ms":92072,"temperature":0.7,"pith_summary":"TaskCraft is an automated workflow for producing agentic tasks—multi-step problems that require an AI to call external tools and reason over the results—together with the exact step-by-step trajectories that solve them. The paper argues that by starting from simple atomic tasks that each need one tool call and then extending them in depth, where each answer becomes the key to the next question, and in width, by merging independent subtasks, a pipeline can generate large numbers of tasks of controlled difficulty without human annotation. The empirical payoff is that supervised fine-tuning on 3,202 generated multi-hop trajectories lifts a 3-billion-parameter base model's average exact match on HotpotQA, Musique, and Bamboogle from 0.034 to 0.174, and adding reinforcement learning raises it to 0.245. The workflow also self-improves: using its own generated tasks to optimize its prompts raises the atomic-task pass rate from 54.9% to 68.1% and the depth-extension pass rate from 41.0% to 51.2%. About 36,000 tasks are released to support further agent tuning and evaluation.","feed_headline":"Machine-generated tasks teach agents to use tools better","feed_subtitle":"TaskCraft turns web pages, PDFs, and images into verified multi-hop tasks with solved trajectories, then trains on them.","key_machinery":"The load-bearing object is the atomic-task schema $q = f(i_T, R) \\to a$ together with its recursive extension rule. A single tool call with input index $i_T$ retrieves context $C$; the LLM infers the relation $R$ and produces answer $a$. Depth-based extension builds $q_{n+1} = f(\\hat{q}_{n+1}, R_n) \\to a$, where the new sub-question $\\hat{q}_{n+1}$ must name a strict superset of the previous index so that solving it yields $i_T^n$; width-based extension composes $q_1$ and $q_2$ into $q_{\\text{width}} = q_1 + q_2$ with answer $a_1 + a_2$. The machinery that keeps this honest is verification: judge-LLM comparison for atomic tasks, and for extended tasks a strict-superset check plus an answer-concealment check, both purely linguistic, so tasks can be made harder than current agents can solve while still carrying ground-truth trajectories.","core_discovery":"On its own terms, the paper's central claim is that high-quality agentic training data can be manufactured from unlabeled corpora rather than written by humans. The generative formula is an atomic task $q = f(i_T, R) \\to a$, where $i_T$ is a tool input index such as a document name, webpage title, or image path, $R$ is the relationship between the retrieved context and the answer, and $f$ is an LLM sampler that turns that pair into a natural-language question. Depth extension recursively replaces $i_T$ with a new sub-question $\\hat{q}_{n+1} = f(i_T^{n+1}, R_{n+1}) \\to i_T^n$, requiring a superset search and verification so each hop genuinely depends on the previous one; width extension merges two task questions into one multi-part problem. Verification keeps an atomic task only when a judge model scores a tool-using agent above a bare LLM, and keeps extended tasks only when linguistic checks confirm strict superset structure and answer concealment. The paper reports that this controlled process yields ground-truth trajectories, and that training on them improves planning and tool invocation in downstream agent models across three multi-hop QA benchmarks.","pith_inferences":["Editorial inference: the same depth/width recipe should transfer to private or domain-specific tools such as code interpreters, database query interfaces, and API collections, because the verification logic is tool-agnostic and only the corpus and index extraction need to change.","Editorial inference: since extended tasks are verified linguistically rather than by execution, an execution-based re-verification of a random sample would be a natural cheap addition and would show how many golden answers are actually reachable by agents.","Editorial inference: the hop-count difficulty gradient makes the dataset a ready-made curriculum; ordering tasks by depth for sequential fine-tuning is an experiment the paper does not report.","Editorial inference: the improved prompt pass rates imply the generated tasks and trajectories could be used to optimize other agent workflows, not just the generator itself."],"forward_implications":["Human annotation is not required to build large agentic training sets: unlabeled web pages, PDFs, and images can be converted into tool-use tasks at scale.","Supervised fine-tuning on generated trajectories can substitute for part of the engineering in agent workflows, since a small base model moves from near-zero exact match to usable multi-hop QA performance.","The generated tasks feed back into the generator itself: optimizing the workflow prompts with its own examples raises the atomic-task pass rate from 54.9% to 68.1% and the depth-extension pass rate from 41.0% to 51.2%.","The dataset's difficulty gradient, from single-tool atomic tasks to deep multi-hop chains, offers a calibration curve for measuring agent capability beyond current benchmarks.","The released trajectories provide a ready-made supervisory signal for reinforcement learning, not just supervised fine-tuning, and the paper shows RL on top of the SFT initialization gives the highest overall accuracy."],"supporting_citations":[{"why":"Supplies the reinforcement-learning workflow and training setup that the SFT results are combined with and measured against.","marker":"[4]"},{"why":"Supplies the bootstrap few-shot mechanism used to optimize the generation prompts.","marker":"[5]"},{"why":"Provides the human-annotated benchmark whose difficulty levels are used to calibrate the synthetic tasks.","marker":"[8]"},{"why":"Provides the agent used to execute sampled tasks and produce the difficulty score distribution.","marker":"[14]"},{"why":"Is the prior automatic instruction-data method that motivates the need for tool-interactive tasks.","marker":"[24]"},{"why":"Provides the HotpotQA evaluation set used to measure multi-hop QA performance.","marker":"[31]"},{"why":"Provides the Musique evaluation set used to measure multi-hop QA performance.","marker":"[22]"},{"why":"Provides the Bamboogle evaluation set used to measure multi-hop QA performance.","marker":"[11]"},{"why":"Supplies the content-masking scheme applied to search-tool contexts during supervised fine-tuning.","marker":"[37]"}],"fun_headline_variants":["Automated pipeline churns out trainable agentic tasks","AI-made tasks sharpen tool-using agents","TaskCraft: synthetic tasks, real agent gains","From corpus to tasks: training data on autopilot","36k verifiable tasks to level up agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the judge model's this-task-is-verified verdict is trustworthy enough to guarantee the golden answer is reachable, unique, and not leaked, but the paper never measures how often that verdict is correct.","fun_headline_variants_meta":{"raw":{"variants":["Automated pipeline churns out trainable agentic tasks","AI-made tasks sharpen tool-using agents","TaskCraft: synthetic tasks, real agent gains","From corpus to tasks: training data on autopilot","36k verifiable tasks to level up agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000475,"raw_usage":{"total_tokens":2349,"prompt_tokens":927,"completion_tokens":1422,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":1347}},"tokens_in":543,"tokens_out":1422,"duration_ms":12434,"temperature":1.0,"reasoning_tokens":1347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:37:27.759433+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample a random set of depth-extended tasks from the released dataset, have fresh agents execute them without seeing the golden trajectories, and check whether each final answer is reachable, unique, and only findable through the intended tool chain. If a significant fraction fail, because supersets are pseudo-supersets, answers are leaked in the question, or no trajectory reaches the golden answer, the verifiability guarantee at the core of the workflow is falsified.","supporting_citations":[{"cited_title":"Cohen, Ruslan Salakhutdinov, and Christo- pher D","cited_arxiv_id":null,"evidence_quote":"Provides the HotpotQA evaluation set used to measure multi-hop QA performance."},{"cited_title":"Smith, and Mike Lewis","cited_arxiv_id":null,"evidence_quote":"Provides the Bamboogle evaluation set used to measure multi-hop QA performance."}],"review_version":1}