Pith. sign in

REVIEW 3 major objections 6 minor 8 cited by

TaskCraft: Automated Generation of Agentic Tasks

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TaskCraft is an automated workflow that turns unlabeled corpora into difficulty-scaled, multi-tool agentic tasks with execution trajectories, and shows that fine-tuning on those tasks improves agent performance on multi-hop QA.

desk verdict TaskCraft is a genuinely useful extension of the self-instruct line into tool-using agents, but the 'verifiable' claim rests on uncalibrated judge-LLMs and depth-extended tasks are never executed by an agent, so the training-signal integrity is unestablished. read the letter →

arxiv 2506.10055 v2 pith:A5AHTY4Z submitted 2025-06-11 cs.CL

Dingfeng Shi , Jingyi Cao , Qianben Chen , Weichen Sun , Weizhen Li , Hongxuan Lu , Fangchen Dong , Tianrui Qin
show 9 more authors
King Zhu Minghao Liu Jian Yang Ge Zhang Jiaheng Liu Changwang Zhang Jun Wang Yuchen Eleanor Jiang Wangchunshu Zhou
This is my paper · ORCID
classification cs.CL
keywords agentictaskssyntheticdatagenerationmulti-hopreasoningtoolusesupervisedfine-tuningtaskverificationdifficultyscalingexecutiontrajectories
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TaskCraft is an automated workflow for producing agentic tasks—multi-step problems that require an AI to call external tools and reason over the results—together with the exact step-by-step trajectories that solve them. The paper argues that by starting from simple atomic tasks that each need one tool call and then extending them in depth, where each answer becomes the key to the next question, and in width, by merging independent subtasks, a pipeline can generate large numbers of tasks of controlled difficulty without human annotation. The empirical payoff is that supervised fine-tuning on 3,202 generated multi-hop trajectories lifts a 3-billion-parameter base model's average exact match on HotpotQA, Musique, and Bamboogle from 0.034 to 0.174, and adding reinforcement learning raises it to 0.245. The workflow also self-improves: using its own generated tasks to optimize its prompts raises the atomic-task pass rate from 54.9% to 68.1% and the depth-extension pass rate from 41.0% to 51.2%. About 36,000 tasks are released to support further agent tuning and evaluation.

What carries the argument

The load-bearing object is the atomic-task schema $q = f(i_T, R) \to a$ together with its recursive extension rule. A single tool call with input index $i_T$ retrieves context $C$; the LLM infers the relation $R$ and produces answer $a$. Depth-based extension builds $q_{n+1} = f(\hat{q}_{n+1}, R_n) \to a$, where the new sub-question $\hat{q}_{n+1}$ must name a strict superset of the previous index so that solving it yields $i_T^n$; width-based extension composes $q_1$ and $q_2$ into $q_{\text{width}} = q_1 + q_2$ with answer $a_1 + a_2$. The machinery that keeps this honest is verification: judge-LLM comparison for atomic tasks, and for extended tasks a strict-superset check plus an answer-concealment check, both purely linguistic, so tasks can be made harder than current agents can solve while still carrying ground-truth trajectories.

What would settle it

Sample a random set of depth-extended tasks from the released dataset, have fresh agents execute them without seeing the golden trajectories, and check whether each final answer is reachable, unique, and only findable through the intended tool chain. If a significant fraction fail, because supersets are pseudo-supersets, answers are leaked in the question, or no trajectory reaches the golden answer, the verifiability guarantee at the core of the workflow is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that high-quality agentic training data can be manufactured from unlabeled corpora rather than written by humans. The generative formula is an atomic task $q = f(i_T, R) \to a$, where $i_T$ is a tool input index such as a document name, webpage title, or image path, $R$ is the relationship between the retrieved context and the answer, and $f$ is an LLM sampler that turns that pair into a natural-language question. Depth extension recursively replaces $i_T$ with a new sub-question $\hat{q}_{n+1} = f(i_T^{n+1}, R_{n+1}) \to i_T^n$, requiring a superset search and verification so each hop genuinely depends on the previous one; width extension merges two task questions into one multi-part problem. Verification keeps an atomic task only when a judge model scores a tool-using agent above a bare LLM, and keeps extended tasks only when linguistic checks confirm strict superset structure and answer concealment. The paper reports that this controlled process yields ground-truth trajectories, and that training on them improves planning and tool invocation in downstream agent models across three multi-hop QA benchmarks.

Load-bearing premise

The load-bearing premise is that the judge model's this-task-is-verified verdict is trustworthy enough to guarantee the golden answer is reachable, unique, and not leaked, but the paper never measures how often that verdict is correct.

Editorial extensions

If this is right

  • Human annotation is not required to build large agentic training sets: unlabeled web pages, PDFs, and images can be converted into tool-use tasks at scale.
  • Supervised fine-tuning on generated trajectories can substitute for part of the engineering in agent workflows, since a small base model moves from near-zero exact match to usable multi-hop QA performance.
  • The generated tasks feed back into the generator itself: optimizing the workflow prompts with its own examples raises the atomic-task pass rate from 54.9% to 68.1% and the depth-extension pass rate from 41.0% to 51.2%.
  • The dataset's difficulty gradient, from single-tool atomic tasks to deep multi-hop chains, offers a calibration curve for measuring agent capability beyond current benchmarks.
  • The released trajectories provide a ready-made supervisory signal for reinforcement learning, not just supervised fine-tuning, and the paper shows RL on top of the SFT initialization gives the highest overall accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same depth/width recipe should transfer to private or domain-specific tools such as code interpreters, database query interfaces, and API collections, because the verification logic is tool-agnostic and only the corpus and index extraction need to change.
  • Editorial inference: since extended tasks are verified linguistically rather than by execution, an execution-based re-verification of a random sample would be a natural cheap addition and would show how many golden answers are actually reachable by agents.
  • Editorial inference: the hop-count difficulty gradient makes the dataset a ready-made curriculum; ordering tasks by depth for sequential fine-tuning is an experiment the paper does not report.
  • Editorial inference: the improved prompt pass rates imply the generated tasks and trajectories could be used to optimize other agent workflows, not just the generator itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper introduces TaskCraft, an automated pipeline that generates agentic tasks by sampling atomic tasks from an unlabeled corpus via tool contexts, extending them with depth-based and width-based transformations, and filtering candidates with judge-LLM verification. The authors report that the generated tasks improve prompt optimization in the generation workflow (Table 2) and that supervised fine-tuning on 3,202 generated multi-hop trajectories improves exact match on HotpotQA, Musique, and Bamboogle (Table 3), with the largest gain for Qwen2.5-3B-Base (average 0.034 to 0.174, and Search-R1+SFT to 0.245). They release a dataset of approximately 36,000 tasks.

Significance. If the verification pipeline were properly calibrated, TaskCraft would be a useful contribution: it addresses a real scalability bottleneck in agentic benchmark construction, and the held-out SFT results on three standard multi-hop QA benchmarks are genuinely positive evidence that the generated trajectories teach useful tool-use and reasoning patterns. The paper also provides a public code/data link and a detailed workflow appendix. However, the central 'verifiable' claim is not yet supported, because the verifier's accuracy is unmeasured and depth-extended tasks are never executed by an agent. The SFT gains therefore cannot yet be attributed to correct, reachable golden answers. With additional verification experiments, the released dataset could become a valuable community resource.

major comments (3)
  1. [Section 3.3; Appendix B.3.1, B.3.2] The 'verifiable' guarantee is not established for the depth-extended tasks that supply the SFT trajectories. Section 3.3 states that task-extension verification is 'conducted purely through linguistic analysis without agent involvement,' and Appendix B restricts checks to judge-LLM superset validation and information-leakage filtering; no agent executes an extended task, no judge-LLM accuracy against a human-audited sample is reported, and no human sample audit is described. Since Table 3's SFT uses 3,202 multi-hop trajectories from this extension pipeline, a judge that accepts pseudo-supersets, wrong intermediate answers, or leaky chains would inject incorrect targets into training. Please report judge-LLM agreement with human labels and run an executor agent on a random sample of retained depth-extended tasks to verify that the golden answer is reachable by actual tool calls.
  2. [Section 2; Appendix D] The 'ideal search engine' assumption (Section 2) is load-bearing for golden-answer reachability. The workflow assumes that an index iT retrieves precise content, but Section 4.4 and Appendix D evaluate agents with a fixed retrieval depth of 3 over a 2018 Wikipedia dump, which need not contain the superset indices (e.g., i^{n+1}_T) used in the generated trajectories. If retrieval fails for an intermediate index, the recorded intermediate answer and final answer cannot be reproduced by the trained agent. Please report retrieval success rates for the generated indices in the actual evaluation environment, or explicitly restrict the reachability claim to an oracle-retrieval setting.
  3. [Section 3.2; Table 4] The definition of 'atomic task' conflicts with the measured behavior. Section 2 defines an atomic task as solvable with 'a single target tool invocation,' yet Section 3.3 relaxes atomic verification to allow up to three tool-use steps, and Table 4 reports an average of 2.1 tool uses per query for the retained atomic tasks (and 2.8 for the LLM-only baseline). This suggests that many retained atomic tasks require multiple tool calls, weakening the atomicity guarantee and the claim that tool requirements are 'regulated.' Please report the distribution of tool-use counts and ideally enforce the single-tool criterion during verification.
minor comments (6)
  1. [Abstract; Contributions; Appendix A] The dataset size is reported inconsistently: the abstract and conclusion say approximately 36,000 tasks, the contributions list says about 32k tasks, and Appendix A reports 22,053 atomic tasks. Please reconcile these numbers and define what counts as a task.
  2. [Table 1] Table 1 compares GAIA levels with tool-modality categories, which is not an apples-to-apples difficulty comparison. Please clarify the intended claim and add per-level or per-modality error bars or descriptions.
  3. [Section 4.4] The 'Base workflow' is described as 'Search-R1 without training,' which is confusing because Search-R1 is itself an RL-trained workflow. Please rename or clarify what 'without training' means.
  4. [Table 3] Table 3 reports single-run point estimates without confidence intervals or significance tests; given that the Instruct variant improves over Search-R1 by only +0.6% on average, please report variance across seeds or runs.
  5. [Tables 2 and 4] The definitions of 'pass rate' differ between Table 2 and Table 4 (54.9% vs. 43.0% for atomic tasks); please state the exact numerator and denominator for each table.
  6. [References] References [10]/[11] (Bamboogle) and [31]/[32] (HotpotQA) are duplicated; please consolidate them into single citations.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the empirical claims are checked against external benchmarks, and the only self-references are non-load-bearing citations.

full rationale

The derivation chain does not exhibit circularity. Atomic-task questions are constructed from tool contexts extracted from an unlabeled corpus (Section 3.1), and the central empirical evaluation is external: supervised fine-tuning on 3,202 generated multi-hop trajectories is measured against HotpotQA, Musique, and Bamboogle (Table 3), and the atomic-generation ablation in Table 4 compares against a direct LLM-generation baseline. No fitted parameter is renamed as a prediction, and no result is forced by construction. The depth-based extension equations (1)-(3) define a recursive task construction, but the paper does not use them to claim a benchmark prediction; the recursive chain's correctness is a data-quality concern (judge-LLM linguistic verification in Appendix B.3), not circular reasoning. The only self-referential elements are citations to the corresponding authors' earlier work ([38]-[40]) in a survey list of language agents, which is not load-bearing for the paper's main claims. Accordingly, no circular step is identified.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The workflow's claims rest on three unverified premises rather than fitted parameters: ideal retrieval (Section 2), uncalibrated LLM judges (Section 3.3), and LLM-extracted golden answers (Section 3.1). Hand-chosen thresholds (3 tool-step budget, 6 extension attempts, 3,202-task SFT subset) shape what counts as a validated task and what gets trained on, so the empirical numbers in Tables 2-4 should be read as conditional on those choices. The only new constructs (atomic task, depth/width extension, superset index) are working definitions, not postulated objects requiring independent evidence, so invented_entities is empty.

free parameters (3)
  • atomic verification tool-step budget = 3
    Atomic tasks are retained only if the tool-using agent's answer beats the bare LLM's within about 3 tool-use steps (Section 3.3); this hand-chosen budget operationally defines 'tool necessity' and shapes which tasks enter the dataset.
  • extension attempts per depth-wise task (nk) = 6
    Depth-wise extension pass rates are measured over 6 attempts (Section 4.3); this choice bounds the achievable depth and the difficulty ceiling of extended tasks.
  • SFT multi-hop subset size = 3,202 tasks
    Only 3,202 of the ~36k tasks are used for supervised fine-tuning (Section 4.4, Appendix D), so the reported gains in Table 3 depend on this hand-chosen subset and its content-masking recipe.
assumptions (3)
  • domain assumption Ideal search engine: invoking tool T with input index i_T always retrieves the precise needed context C.
    Section 2 states 'we assume an ideal search engine capable of retrieving precise data based on iT.' Task construction, golden answers, and trajectories all depend on retrieval succeeding.
  • domain assumption Judge-LLM verification is accurate: judges correctly detect answer inconsistency, pseudo-supersets, and information leakage.
    Section 3.3 and Appendix B.3 delegate all extension verification and most atomic verification to LLMs; no error-rate measurement or human audit of judge decisions is reported.
  • domain assumption LLM-extracted golden answers from tool context C are correct and unique.
    Section 3.1 samples answers a by prompting an LLM on retrieved content C; these answers become ground truth for verification and trajectories, but are never human-validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TaskCraft: Automated Generation of Agentic Tasks." pith.science (2026). https://pith.science/paper/A5AHTY4Z

@misc{pith2026250610055,
  author       = {Pith},
  title        = {Pith review of: TaskCraft: Automated Generation of Agentic Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A5AHTY4Z}},
  note         = {Machine review of arXiv:2506.10055}
}
read the original abstract

Agentic tasks, which require multi-step problem solving with autonomy, tool use, and adaptive reasoning, are becoming increasingly central to the advancement of NLP and AI. However, existing instruction data lacks tool interaction, and current agentic benchmarks rely on costly human annotation, limiting their scalability. We introduce \textsc{TaskCraft}, an automated workflow for generating difficulty-scalable, multi-tool, and verifiable agentic tasks with execution trajectories. TaskCraft expands atomic tasks using depth-based and width-based extensions to create structurally and hierarchically complex challenges. Empirical results show that these tasks improve prompt optimization in the generation workflow and enhance supervised fine-tuning of agentic foundation models. We present a large-scale synthetic dataset of approximately 36,000 tasks with varying difficulty to support future research on agent tuning and evaluation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories

    cs.CL 2026-06 conditional novelty 7.0 of 10

    ISE creates 23,132 execution-grounded multi-turn OS agent trajectories via intent simulation and live execution, improving agent performance on ClawEval from 19.3 to 37.7 pass@1 with Qwen3-8B.

  2. NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Requirement-driven synthesis generates executable agent training tasks and trajectories from a capability description, improving a 35B model from 22.5% to 52.0% on Terminal-Bench 2.0 and from 813 to 1338 GDPval Elo.

  3. Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL

    cs.AI 2025-08 conditional novelty 6.0 of 10

    A single LLM is trained with multi-agent distilled trajectories plus agentic RL, and the resulting Chain-of-Agents models set state-of-the-art Pass@1 scores among tool-integrated reasoning methods on GAIA, BrowseComp,...

  4. WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization

    cs.CL 2025-07 conditional novelty 6.0 of 10

    WebShaper formalizes web information-seeking tasks as set-theoretic queries, synthesizes training questions by layer-wise expansion, and uses the resulting data to train open-source agents that outperform prior open-s...

  5. HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better

    cs.CV 2026-07 conditional novelty 5.0 of 10

    HunyuanOCR-1.5 makes a lightweight end-to-end OCR VLM the fastest among peers via DFlash speculative decoding while expanding long-tail capabilities through agent-driven data construction.

  6. LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent

    cs.AI 2026-04 unverdicted novelty 5.0 of 10

    LiteResearcher uses a lite virtual world to make agentic RL training scalable and stable, enabling a 4B model to achieve 71.3% on GAIA and 78.0% on Xbench, outperforming larger open-source and commercial systems.

  7. OAgents: An Empirical Study of Building Effective Agents

    cs.AI 2025-06 reject novelty 5.0 of 10

    OAgents reports a framework and ablation study claiming open-source state-of-the-art on GAIA, but the empirical evidence is weakened by benchmark fitting and missing statistics.

  8. Efficient Agents: Building Effective Agents While Reducing Cost

    cs.AI 2025-07 conditional novelty 4.0 of 10

    A GAIA-tuned agent using GPT-4.1, 8 steps, simple memory, and multiple search sources keeps 51.52% accuracy while lowering cost-of-pass from 0.75 to 0.55 versus OWL.

Reference graph

Works this paper leans on

65 extracted references · 28 canonical work pages · cited by 8 Pith papers

  1. [1]

    Tapeagents: a holistic framework for agent development and optimization, 2024

    Dzmitry Bahdanau, Nicolas Gontier, Gabriel Huang, Ehsan Kamalloo, Rafael Pardinas, Alex Piché, Torsten Scholak, Oleh Shliazhko, Jordan Prince Tremblay, Karam Ghanem, Soham Parikh, Mitul Tiwari, and Quaizar Vohra. Tapeagents: a holistic framework for agent development and optimization, 2024. URLhttps://arxiv. org/abs/2412.08445

    arXiv 2024

  2. [2]

    Magentic-one: A generalist multi-agent system for solving complex tasks.arXiv preprint arXiv:2411.04468, 2024

    Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. Magentic-one: A generalist multi-agent system for solving complex tasks.arXiv preprint arXiv:2411.04468, 2024

    arXiv 2024

  3. [3]

    Autonomous agentic ai: execute multi-step workflows autonomously

    H2O.ai. Autonomous agentic ai: execute multi-step workflows autonomously. [Online], 2024.https://h2o.ai/ platform/enterprise-h2ogpte/#AgenticAI

    2024

  4. [4]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025. URL https://arxiv.org/abs/2503.09516

    arXiv 2025

  5. [5]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-improving pipelines. 2024

    2024

  6. [6]

    Camel: Communicative agents for" mind" exploration of large language model society.Advances in Neural Information Processing Systems, 36:51991–52008, 2023

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society.Advances in Neural Information Processing Systems, 36:51991–52008, 2023

    2023

  7. [7]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repository, 13:9, 2024

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repository, 13:9, 2024

    2024

  8. [8]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InThe TwelfthInternational Conference on Learning Representations, 2023

    2023

Show all 65 references
  1. [9]

    Humanity’s last exam, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, et al. Humanity’s last exam, 2025...

    2025 arXiv

  2. [10]

    Measuring and narrowing the compositionality gap in language models.arXiv preprint arXiv:2210.03350, 2022

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models.arXiv preprint arXiv:2210.03350, 2022

    2022 arXiv

  3. [11]

    Smith, and Mike Lewis

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Findings of the Association for Computational Linguistics: EMNLP 2023, S...

    2023

  4. [12]

    AutoAct: Automatic agent learning from scratch for QA via self-planning

    Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Jiang, Chengfei Lv, and Huajun Chen. AutoAct: Automatic agent learning from scratch for QA via self-planning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual M...

    2024 doi

  5. [13]

    Tool learning with foundation models.ACM Computing Surveys, 57(4):1–40, 2024

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang, Chaojun Xiao, et al. Tool learning with foundation models.ACM Computing Surveys, 57(4):1–40, 2024

    2024

  6. [14]

    ‘smola- gents‘: a smol library to build great agentic systems.https://github.com/huggingface/smolagents, 2025

    Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. ‘smola- gents‘: a smol library to build great agentic systems.https://github.com/huggingface/smolagents, 2025

    2025

  7. [15]

    Significant-Gravitas. Autogpt. [Online], 2023.https://github.com/Significant-Gravitas/AutoGPT

    2023

  8. [16]

    Llm-planner: Few-shot grounded planning for embodied agents with large language models

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. InProceedings of the IEEE/CVF international conference on computer vision, pages 2998–3009, 2023

    2023

  9. [17]

    Autoagent: A fully-automated and zero-code framework for llm agents

    Jiabin Tang, Tianyu Fan, and Chao Huang. Autoagent: A fully-automated and zero-code framework for llm agents. arXiv e-prints, pages arXiv–2502, 2025. 13

    2025

  10. [18]

    Mader: Trajectory planner in multiagent and dynamic environments

    Jesus Tordesillas and Jonathan P How. Mader: Trajectory planner in multiagent and dynamic environments. IEEE Transactions on Robotics, 38(1):463–476, 2021

    2021

  11. [19]

    Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data

    Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024

    2024 arXiv

  12. [20]

    Meet trase systems

    Trase. Meet trase systems. [Online], 2024.https://www.trasesystems.com/

    2024

  13. [21]

    Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    2022

  14. [22]

    Interleaving retrieval with chain- of-thought reasoning for knowledge-intensive multi-step questions

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain- of-thought reasoning for knowledge-intensive multi-step questions. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st AnnualMe...

    2023 doi

  15. [23]

    Executable code actions elicit better llm agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. InForty-firstInternational Conference on Machine Learning, 2024

    2024

  16. [24]

    Self-instruct: Aligning language models with self-generated instructions.arXiv preprint arXiv:2212.10560, 2022

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Ha- jishirzi. Self-instruct: Aligning language models with self-generated instructions.arXiv preprint arXiv:2212.10560, 2022

    2022 arXiv

  17. [25]

    Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025

    2025 arXiv

  18. [26]

    Autogen: Enabling next-gen llm applications via multi-agent conversation.arXiv preprint arXiv:2308.08155, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation.arXiv preprint arXiv:2308.08155, 2023

    2023 arXiv

  19. [27]

    Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  20. [28]

    Baize: An open-source chat model with parameter-efficient tuning on self-chat data.arXiv preprint arXiv:2304.01196, 2023

    Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data.arXiv preprint arXiv:2304.01196, 2023

    2023 arXiv

  21. [29]

    How2comm: Communication-efficient and collaboration-pragmatic multi-agent perception.Advances in Neural Information Processing Systems, 36:25151–25164, 2023

    Dingkang Yang, Kun Yang, Yuzheng Wang, Jing Liu, Zhi Xu, Rongbin Yin, Peng Zhai, and Lihua Zhang. How2comm: Communication-efficient and collaboration-pragmatic multi-agent perception.Advances in Neural Information Processing Systems, 36:25151–25164, 2023

    2023

  22. [30]

    What2comm: Towards communication-efficient collaborative perception via feature decoupling

    Kun Yang, Dingkang Yang, Jingyu Zhang, Hanqi Wang, Peng Sun, and Liang Song. What2comm: Towards communication-efficient collaborative perception via feature decoupling. In Proceedings of the 31st ACM international conference on multimedia, pages 7686–7695, 2023

    2023

  23. [31]

    Cohen, Ruslan Salakhutdinov, and Christo- pher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors,...

    2018 doi

  24. [32]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering.arXiv preprint arXiv:1809.09600, 2018

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering.arXiv preprint arXiv:1809.09600, 2018

    2018 arXiv

  25. [33]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023. 14

    2023

  26. [34]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023

    2023 arXiv

  27. [35]

    Mammoth2: Scaling instructions from the web.Advances in Neural Information Processing Systems, 37:90629–90660, 2024

    Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web.Advances in Neural Information Processing Systems, 37:90629–90660, 2024

    2024

  28. [36]

    Star: Self-taught reasoner bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. Star: Self-taught reasoner bootstrapping reasoning with reasoning. InProc. the 36th International Conference on Neural Information Processing Systems, volume 1126, 2024

    2024

  29. [37]

    Agent models: Internalizing chain-of- action generation into reasoning models, 2025

    Yuxiang Zhang, Yuqi Yang, Jiangming Shu, Xinyan Wen, and Jitao Sang. Agent models: Internalizing chain-of- action generation into reasoning models, 2025. URLhttps://arxiv.org/abs/2503.06580

    2025 arXiv

  30. [38]

    Recurrentgpt: Interactive generation of (arbitrarily) long text, 2023

    Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. Recurrentgpt: Interactive generation of (arbitrarily) long text, 2023. URL https: //arxiv.org/abs/2305.13304

    2023 arXiv

  31. [39]

    Agents: An open-source framework for autonomous language agents

    Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Xiangru Tang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An open-source framework fo...

  32. [40]

    Travel Trends 2025 | Our Annual Report

    Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, Huajun Chen, and Yuchen Eleanor Jiang. Symbolic learning enables self-evolving agents. 2024. URLhttps://arxiv.org/abs/2406.18532. 15 Appendix A Data...

    2024 arXiv

  33. [43]

    A increased by 5% and B decreased by 2%

    Atomicity: Extract only indivisible basic facts (no combined conclusions, e.g., split “A increased by 5% and B decreased by 2%” into two separate conclusions)

  34. [44]

    Performance has improved

    Verifiability: Include at least one definite identifier (numeric value, time, unique name) and reject vague expressions (e.g., “Performance has improved”)

  35. [45]

    Global GDP grew by 3.0% in 2023

    Timeliness Handling: Explicitly mark time ranges for time-sensitive information (e.g., “Global GDP grew by 3.0% in 2023” instead of “Recent GDP growth of 3.0%”)

    2023

  36. [46]

    as stated in (2)

    Citation Integrity: Embed complete content of cited references (e.g., expand “as stated in (2)” to include the full text of (2) in the conclusion) Valid Examples: • Example 1:3D deconvolution microscopy illumination optimization for refractive index tomogra- phy (Optics Expres...

    2021

  37. [47]

    Paragraph/sentence: Its belonging text content

  38. [48]

    Specific term: Corresponding discipline/category

  39. [49]

    Specific date: Date range it’s in (e.g., its week/month)

  40. [50]

    Short event: Complete specific event it’s part of

  41. [51]

    Page: Referencing pages or parent page

  42. [52]

    Generate only one relationship, avoiding strongly specific proper nouns Relationship expression guidelines:

  43. [53]

    Indicate position for series sub-items; clarify ownership for parts of a superset

    Clearly show hierarchical/ownership. Indicate position for series sub-items; clarify ownership for parts of a superset

  44. [54]

    Specify input content’s positioning (e.g., time range, publication field, role in superset)

  45. [55]

    Use research/industry standard wording

  46. [56]

    Provide only necessary associations Notes:

  47. [57]

    Return the superset’s unique identifier (e.g., attribute name, page title, paper title)

  48. [58]

    Obtain superset content via tool (web, PDF, image)

  49. [59]

    Concisely describe the relationship, listing unique qualification conditions

  50. [60]

    Use≤3 search keywords per search; do multiple searches if needed

  51. [61]

    Derive the identifier from search results, excluding the input content

  52. [62]

    Output Format:

    Prioritize reading PDF content with tools if the input is a PDF Valid Examples: •Example 1: –Input:Avatar 3: Fire and Ash –Superset Identifier:Avatar film series –Relation:The third film •Example 2: –Input:V3LMA: Visual 3D-enhanced Language Model for Autonomous Driving –Supers...

  53. [63]

    But do not make too many changes to the core query

    Complexity Balance: The new query should be slightly more complex than the original core Query and require more steps to solve. But do not make too many changes to the core query

  54. [64]

    3.Answer Concealment: The new query must not reveal information about the golden answer

    Answer Uniqueness: The new query should point to the unique answer: golden answer, and should not point to other answers. 3.Answer Concealment: The new query must not reveal information about the golden answer

  55. [65]

    Do not modify the proper nouns appearing in it

    Natural Language Polish: After merging, polish the question to make it conform to human expression habits without changing the original meaning. Do not modify the proper nouns appearing in it. Valid Examples (20 in total): •Example 1: –Core Query:What is the 2nd positive integ...

    2025

  56. [2023]

    URLhttps://arxiv.org/abs/2309.07870

    arXiv

  57. [5711]

    doi: 10.18653/V1/2023.FINDINGS-EMNLP.378

    Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.FINDINGS-EMNLP.378. URL https://doi.org/10.18653/v1/2023.findings-emnlp.378

    2023 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.