Pith. sign in

REVIEW 4 major objections 5 minor 45 references

Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Auto-SLURP evaluates multi-agent assistants from intent to execution and finds even the best framework completes only 46% of requests.

desk verdict Useful benchmark artifact, but the headline success rates rest on 100 test queries with no error bars, so the rankings are provisional. read the letter →

arxiv 2504.18373 v1 pith:MGNCGSXZ submitted 2025-04-25 cs.CL

Lei Shen , Xiaoyu Shen This is my paper · ORCID
classification cs.CL
keywords Auto-SLURPmulti-agentframeworkssmartpersonalassistantend-to-endevaluationLLMagentsSLURPintentdetectionbenchmarkdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Auto-SLURP is a benchmark that turns the SLURP smart-home assistant dataset into an end-to-end test for LLM-based multi-agent frameworks. The authors relabel SLURP queries with the slot information needed to actually execute them and attach simulated backend servers and external APIs, so a framework must not only understand the user's request but also orchestrate specialized agents to perform it and return an answer. They report that four representative frameworks complete only 21% to 46% of requests successfully, with intent prediction the largest source of failure. The point of the paper is that multi-agent assistant frameworks should be evaluated on the whole pipeline, not just on natural language understanding, and that current frameworks are far from reliable enough for this task.

What carries the argument

The load-bearing mechanism is the simulated end-to-end execution pipeline. Each of 23 domains gets a dedicated simulated server, with external APIs for search, weather, and news, and every query is relabeled with the slots needed to run it. Frameworks must implement a fixed workflow: a program manager agent routes the request to intent, time, location, and URL agents, then a request agent calls the server and a response is generated. Success is scored by whether the entire pipeline completes correctly, which makes the benchmark measure coordination and tool use rather than classification alone.

What would settle it

Deploy the same four frameworks against the real smart-home services that the simulated servers approximate, such as real calendar, email, weather, and IoT APIs, and compare end-to-end success rates. If the ranking of frameworks changes materially, for example if AgentLite no longer leads, then scores on Auto-SLURP do not predict live assistant performance.

Watch

Extended reading notes

Core claim

The paper's central claim is that Auto-SLURP, built by extending SLURP's original intent-slot annotations and pairing them with simulated execution servers and external services, provides a valid end-to-end measure of multi-agent framework performance for smart personal assistants. On this measure, the best framework, AgentLite, succeeds on 46% of 100 test queries, while CamelAI, LangGraph, and AutoGen succeed on 21%, 32%, and 44% respectively. Error analysis attributes most failures to the intent agent. A finetuned Llama 3 8B intent agent raises AutoGen's end-to-end success from 40% to 62%, a 55% relative improvement, which the paper reads as evidence that component-level weaknesses, rather than framework-level orchestration alone, limit current assistants.

Load-bearing premise

The benchmark's scores transfer to real assistants only if the simulated servers and third-party APIs behave like production backends; the authors themselves flag that simulated services may not fully mimic real-world systems.

Editorial extensions

If this is right

  • If Auto-SLURP measures what it claims, framework design choices—how prompts are separated, how agents are orchestrated, whether reasoning steps are inserted—directly change end-to-end success rates, so the benchmark can compare frameworks on equal footing.
  • Because intent prediction is the dominant failure point, improving the intent component, for example by finetuning a smaller open model, is a concrete lever for whole-system gains; the paper's 55% relative improvement on AutoGen demonstrates this.
  • The benchmark's cost figures, from 0.14 to 0.80 USD per query across frameworks, make it possible to compare accuracy against operating cost for multi-agent assistant deployments.
  • With only 1,000 training and 100 test queries, Auto-SLURP is lightweight enough to be used for finetuning and rapid iteration in academic settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would inject realistic backend failures, such as timeouts, malformed responses, or authentication errors, into the simulated servers; if rankings change under injected noise, robustness to real-world server behavior is an additional axis the current metric does not capture.
  • The fixed workflow of one program manager plus specialized agents may favor frameworks that support role-specific prompts; a fairer comparison could let each framework define its own workflow and still measure only end-to-end success.
  • Because the data are English smart-home commands from SLURP, generalizing to other languages or enterprise domains would require rebuilding the simulated servers, so the current results should not be read as a statement about assistant performance outside that setting.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Auto-SLURP, a benchmark derived from the SLURP spoken-language-understanding dataset, in which user queries are re-labeled with enriched slot annotations and paired with simulated backend servers and external services. The goal is to support end-to-end evaluation of LLM-based multi-agent frameworks for personal-assistant scenarios, covering intent detection, task execution, and response generation. The authors evaluate four frameworks (CamelAI, LangGraph, AutoGen, AgentLite) using GPT-4 on a 100-query test subset and report successful execution rates between 21% and 46%, concluding that current state-of-the-art frameworks are substantially challenged by the benchmark. An ablation study replaces the intent agent with a fine-tuned Llama-3 8B model on the AutoGen framework and reports an improvement from an accuracy of 0.40 to 0.62. The dataset and code are publicly released on GitHub.

Significance. If the results are reliable, Auto-SLURP fills a genuine gap: existing benchmarks largely evaluate individual LLMs or single-agent tool use, whereas Auto-SLURP attempts to measure entire multi-agent orchestration pipelines end to end. The public release of the dataset, the deterministic temperature-zero evaluation protocol, and the explicit workflow definition are strengths that should facilitate reuse. However, the quantitative evidence base is thin: the headline success rates come from exactly 100 test queries with no confidence intervals, no stratification, and no power analysis, and the differences between frameworks are within binomial sampling error. The dataset-construction process also lacks documented quality assurance. Because the central claim that Auto-SLURP is a significant challenge for current frameworks rests on these numbers, the paper needs additional statistical grounding or more modest framing before the result can be considered established.

major comments (4)
  1. [Section 2 and Table 2] The central quantitative claim is based on only 100 test samples, selected uniformly at random from the SLURP test set, with no confidence intervals, no repeated seeds, and no power analysis. For n=100, the 95% Wilson interval for the reported success rate of 0.21 is approximately [0.14, 0.30] and for 0.46 is approximately [0.36, 0.56]; these intervals overlap substantially, so the ranking between AutoGen (0.44) and AgentLite (0.46) is well within sampling noise. The statement in Section 2 that this subset 'is considered sufficient' is not supported by any variance estimate or by a comparison of the subset's intent/domain distribution to the full SLURP test distribution. I recommend that the authors either enlarge the test set, report confidence intervals and ideally stratified sampling, or substantially soften the claim that the reported numbers establish a significant challenge and a meaningful framework ranking.
  2. [Section 3.3 and Appendix B] The automated evaluation tool is described only as 'consistent and efficient,' but its decision rules are not specified. In particular, the failure attribution criteria in Appendix B include subjective conditions such as 'if the URL agent receives an incorrect intent but is capable of correcting it and fails to do so,' with no operational definition of 'capable.' There is no inter-annotator agreement, no validation that the attribution matches human judgment, and no discussion of how partial failures or cascading errors are assigned to a single agent. This matters because Section 4.1 uses Table 3 to conclude that 'the main source of failure stems from the intent agent,' and Section 4.2 builds an ablation on that conclusion. Without a documented and validated attribution procedure, the failure analysis is not reproducible.
  3. [Section 2, Table 1] The dataset is the primary contribution, yet its construction is described with a single example and no annotation guidelines, no description of how the new slots were derived from the original SLURP annotations, no quality-control measures, and no report of annotator agreement. The enriched slots must be correct and complete for the end-to-end evaluation to be meaningful, because a wrong or missing slot annotation can cause a framework to fail through no fault of the framework. I recommend providing a detailed annotation protocol, example counts of slot types, and a reliability analysis, or at least a manual audit of a sample of the 1,000 training and 100 test annotations.
  4. [Section 6 and Section 3.1] The paper acknowledges that simulated servers may not fully mimic real systems, but it does not address the reproducibility risk of relying on live third-party APIs for domains such as weather, search, and news. Because these APIs return time-varying content and may change their interfaces, two runs of the benchmark at different times can yield different outcomes even with temperature set to zero. The authors should state whether API responses were cached or fixed, provide versioned snapshots of external service responses, or otherwise document how reproducibility is ensured. Without this, the 'automated evaluation tool' does not guarantee consistent comparison across framework versions or research groups.
minor comments (5)
  1. [Section 5] The sentence 'We assess the end-to-end execution tasks, not just the nature language understanding tasks' contains a typo: 'nature' should be 'natural.'
  2. [Table 5 caption] The caption says 'AutoGen original finetuned' but the column header is 'funetuning'; correct the typo and clarify that 'original' refers to the original Llama-3 8B model, not the GPT-4-based AutoGen configuration in Table 2.
  3. [Table 2 and Section 3.3] The metric is defined as 'successful execution rate' in Section 3.3, but Table 2 labels it 'acc' and Section 4.1 calls it 'accuracy score.' Use a single consistent term to avoid confusion.
  4. [Introduction and References] In the first paragraph of Section 1, 'SocialBench (Chen et al., 2024)' is cited, but the reference list entry 'Chen et al. 2024' is titled 'Roleinteract: Evaluating the social interaction of role-playing agents,' which does not match the cited benchmark name; please correct the reference. Also, 'ALFWorld (Shridhar et al.)' lacks a year and venue in the citation.
  5. [Appendix C, Tables 6 and 7] The domain-specific accuracy tables exclude domains with fewer than three samples, but they do not report the number of test samples per domain. For a 100-query test set, many domains likely contain very few samples, so percentages such as 0.0% versus 66.7% may represent one or two examples; reporting per-domain counts and confidence intervals would prevent overinterpretation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Auto-SLURP reports measured framework performance on a newly constructed benchmark, with no derivation step that reduces to its own inputs.

full rationale

Auto-SLURP is a dataset-construction and evaluation paper, not a predictive derivation. The central claim—that current multi-agent frameworks succeed on only 21% to 46% of requests—is an observed accuracy computed by running four external frameworks (CamelAI, LangGraph, AutoGen, AgentLite) on 100 held-out test queries. The ground-truth labels were produced by relabeling SLURP slots and simulating backend servers, but the frameworks' success rates are not defined by those labels in any circular way; the labels are the benchmark targets, and the measured rates are independent empirical outcomes. The paper performs no parameter fit to the test labels and then 'predicts' those same labels. The finetuned Llama-3 intent agent in Section 4.2 is trained on the 1,000-sample training set and evaluated on the 100-sample test set, which is standard supervised evaluation rather than circularity. Section 6's limitation that simulated servers 'may not fully mimic the behavior of real-world systems' is a validity threat, not a circular reduction: it questions whether benchmark scores transfer to deployment, not whether the scores are definitionally entailed by the benchmark construction. Similarly, the Section 2 statement that 100 test samples are 'considered sufficient' lacks a power analysis, but that is a statistical-robustness issue, not evidence that the reported accuracies are constructed from the evaluation criteria. Self-citations by one of the authors (e.g., Shen et al. 2022; Zhu et al. 2023) appear as background references for NLU and weakly supervised learning and are not load-bearing for the benchmark's design or conclusions. No uniqueness theorem, imported ansatz, or renaming of a known result is invoked. The failure attribution criteria in Appendix B are an author-defined scoring rubric; they determine how errors are categorized, but they do not define whether a query succeeds—success is determined by end-to-end task completion on the benchmark's simulated servers. Overall, the paper's derivation chain is self-contained: it builds a dataset and measures frameworks on it, so no circularity score above 0 is warranted.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. It relies on the SLURP dataset and on the fidelity of simulated servers, both of which are domain assumptions. There are no fitted free parameters in the benchmark construction itself.

assumptions (4)
  • domain assumption SLURP's original intent and query labels are a valid starting point for assistant task evaluation.
    Section 2 uses SLURP as the base and only relabels slots, trusting the original annotation quality and coverage.
  • domain assumption Simulated servers and third-party APIs emulate real backend services closely enough for benchmark scores to transfer to live use.
    Section 2 and the Limitations section; the authors acknowledge this may not hold for all domains.
  • ad hoc to paper Random selection of 1,000 training and 100 test samples is representative of the full SLURP distribution.
    Section 2 states random selection without stratification or analysis of domain coverage; Appendix C excludes small domains.
  • domain assumption Successful execution rate measured by the automated evaluation tool is a valid measure of assistant reliability.
    Section 3.3 defines the metric but gives no details of the automated tool's decision rules or validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant." pith.science (2026). https://pith.science/paper/MGNCGSXZ

@misc{pith2026250418373,
  author       = {Pith},
  title        = {Pith review of: Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MGNCGSXZ}},
  note         = {Machine review of arXiv:2504.18373}
}
read the original abstract

In recent years, multi-agent frameworks powered by large language models (LLMs) have advanced rapidly. Despite this progress, there is still a notable absence of benchmark datasets specifically tailored to evaluate their performance. To bridge this gap, we introduce Auto-SLURP, a benchmark dataset aimed at evaluating LLM-based multi-agent frameworks in the context of intelligent personal assistants. Auto-SLURP extends the original SLURP dataset -- initially developed for natural language understanding tasks -- by relabeling the data and integrating simulated servers and external services. This enhancement enables a comprehensive end-to-end evaluation pipeline, covering language understanding, task execution, and response generation. Our experiments demonstrate that Auto-SLURP presents a significant challenge for current state-of-the-art frameworks, highlighting that truly reliable and intelligent multi-agent personal assistants remain a work in progress. The dataset and related code are available at https://github.com/lorashen/Auto-SLURP/.

Figures

Figures reproduced from arXiv: 2504.18373 by the authors.

Figure 1
Figure 1. The workflow defined for the Auto-SLURP dataset. needs of evaluating multi-agent frameworks. This highlights a pressing need for a comprehensive and flexible benchmark that can rigorously and fairly assess the effectiveness of LLM-based multi-agent infrastructures across a range of scenarios. The vision of an intelligent personal assis￾tant—an AI system capable of understanding nat￾ural language and performing tasks… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 17 canonical work pages

  1. [1]

    Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schonherr, and Mario Fritz. 2023. https://api.semanticscholar.org/CorpusID:263310628 Cooperation, competition, and maliciousness: Llm-stakeholders interactive negotiation

    work page 2023

  2. [2]

    OpenAI Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. 2023. https://api.semanticscholar.org/CorpusID:257532815 Gpt-4 technical report

    work page 2023

  3. [3]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  4. [4]

    Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020. Slurp: A spoken language understanding resource package. arXiv preprint arXiv:2011.13205

    arXiv 2020

  5. [5]

    Karlsson, Jie Fu, and Yemin Shi

    Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, B \"o rje F. Karlsson, Jie Fu, and Yemin Shi. 2023 a . https://api.semanticscholar.org/CorpusID:263310605 Autoagents: A framework for automatic agent generation . In International Joint Conference on Artificial Intelligence

    work page 2023

  6. [6]

    Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Xing Gao, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang, et al. 2024. Roleinteract: Evaluating the social interaction of role-playing agents. arXiv preprint arXiv:2403.13679

    arXiv 2024

  7. [7]

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Ya-Ting Lu, Yi-Hsin Hung, Cheng Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2023 b . https://api.semanticscholar.org/CorpusID:263831900 Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors . In Internat...

    work page 2023

  8. [8]

    Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. 2023 c . T-eval: Evaluating the tool utilization capability of large language models step by step. arXiv preprint arXiv:2312.14033

    arXiv 2023

Show all 45 references
  1. [9]

    Marco Del Tredici, Gianni Barlacchi, Xiaoyu Shen, Weiwei Cheng, and Adri \`a de Gispert. 2021. Question rewriting for open-domain conversational qa: Best practices and limitations. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, p...

    2021

  2. [10]

    Jide S Edu, Jose M Such, and Guillermo Suarez-Tangil. 2020. Smart home personal assistants: a security and privacy review. ACM Computing Surveys (CSUR), 53(6):1--36

    2020

  3. [11]

    Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, Daniel LK Yamins, James J DiCarlo, Josh McDermott, Antonio Torralba, et al. 2021. The threedworld transport challenge: A visually guided task-and-motion planning benchmark for physically...

    2021 arXiv

  4. [12]

    Ran Gong, Qiuyuan Huang, Xiaojian Ma, Yusuke Noda, Zane Durante, Zilong Zheng, Demetri Terzopoulos, Li Fei-Fei, Jianfeng Gao, and Hoi Vo. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.200 M ind A gent: Emergent gaming interaction . In Findings of the Association for Co...

    2024 doi

  5. [13]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J \"u rgen Schmidhuber. 2024. https://openreview.net/forum?id=VtmBAGCN7o Meta GP...

    2024

  6. [14]

    Matthew B. Hoy. 2018. https://doi.org/10.1080/02763869.2018.1404391 Alexa, siri, cortana, and more: An introduction to voice assistants . Medical Reference Services Quarterly, 37(1):81--88. PMID: 29327988

    2018

  7. [15]

    Yunyun Huang, Xiaoyu Shen, Chuanyi Li, Jidong Ge, and Bin Luo. 2021. Dependency learning for legal judgment prediction with a unified text-to-text transformer. arXiv preprint arXiv:2112.06370

    2021 arXiv

  8. [16]

    LangChain. 2022. https://github.com/langchain-ai/langchain Langchain

    2022

  9. [17]

    LangGraph. 2023. https://github.com/langchain-ai/langgraph Langgraph

    2023

  10. [18]

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for "mind" exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  11. [19]

    Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, et al. 2024. Legalagentbench: Evaluating llm agents in legal domain. arXiv preprint arXiv:2412.17259

    2024 arXiv

  12. [20]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023. Agent...

    2023 arXiv

  13. [21]

    Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2021. Benchmarking natural language understanding services for building conversational agents. In Increasing naturalness and flexibility in spoken dialogue interaction: 10th international workshop on spoken dial...

    2021

  14. [22]

    Choubey, Tian Lan, Jason Wu, Huan Wang, Shelby Heinecke, Caiming Xiong, and Silvio Savarese

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Liangwei Yang, Zuxin Liu, Juntao Tan, Prafulla K. Choubey, Tian Lan, Jason Wu, Huan Wang, Shelby Heinecke, Caiming Xiong, and Silvio Savarese. 2024. https://arxiv.org/abs/2402.15538 Agentlite: A lightweight library for building and advanc...

    2024 arXiv

  15. [23]

    Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. Agentboard: An analytical evaluation board of multi-turn llm agents. arXiv preprint arXiv:2401.13178

    2024 arXiv

  16. [24]

    Xinyi Mou, Jingcong Liang, Jiayu Lin, Xinnong Zhang, Xiawei Liu, Shiyue Yang, Rong Ye, Lei Chen, Haoyu Kuang, Xuanjing Huang, et al. 2024. Agentsense: Benchmarking social intelligence of language agents through interactive scenarios. arXiv preprint arXiv:2410.19346

    2024 arXiv

  17. [25]

    Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. 2020. Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.09890

    2020 arXiv

  18. [26]

    Siyuan Qi, Shuo Chen, Yexin Li, Xiangyu Kong, Junqi Wang, Bangcheng Yang, Pring Wong, Yifan Zhong, Xiaoyuan Zhang, Zhaowei Zhang, et al. 2024. Civrealm: A learning and reasoning odyssey in civilization for decision-making agents. arXiv preprint arXiv:2401.10568

    2024 arXiv

  19. [27]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789

    2023 arXiv

  20. [28]

    Xiaoyu Shen, Youssef Oualil, Clayton Greenberg, Mittul Singh, and Dietrich Klakow. 2017. Estimation of gap between current language models and human performance

    2017

  21. [29]

    Xiaoyu Shen, Svitlana Vakulenko, Marco Del Tredici, Gianni Barlacchi, Bill Byrne, and Adri \`a de Gispert. 2022. Low-resource dense retrieval for open-domain question answering: A comprehensive survey. arXiv preprint arXiv:2208.03197

    2022 arXiv

  22. [30]

    Alfworld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations 2021

    2021

  23. [31]

    Hui Su, Xiaoyu Shen, Pengwei Hu, Wenjie Li, and Yun Chen. 2018. Dialogue generation with gan. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32

    2018

  24. [32]

    Hui Su, Zhi Tian, Xiaoyu Shen, and Xunliang Cai. 2024. Unraveling the mystery of scaling laws: Part i. arXiv preprint arXiv:2403.06563

    2024 arXiv

  25. [33]

    Wei Wang, Dan Zhang, Tao Feng, Boyan Wang, and Jie Tang. 2024. Battleagentbench: A benchmark for evaluating cooperation and competition capabilities of language models in multi-agent systems. arXiv preprint arXiv:2408.15971

    2024 arXiv

  26. [34]

    Henry Weld, Xiaoqi Huang, Siqu Long, Josiah Poon, and Soyeon Caren Han. 2022. A survey of joint intent detection and slot filling models in natural language understanding. ACM Computing Surveys, 55(8):1--38

    2022

  27. [35]

    White, Doug Burger, and Chi Wang

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. 2023. https://api.semanticscholar.org/CorpusID:263611068 Autogen: Enabling next-gen llm ...

    2023

  28. [36]

    Binxia Xu, Siyuan Qiu, Jie Zhang, Yafang Wang, Xiaoyu Shen, and Gerard De Melo. 2020. Data augmentation for multiclass utterance classification--a systematic study. In Proceedings of the 28th international conference on computational linguistics, pages 5494--5506

    2020

  29. [37]

    Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See Kiong Ng, and Jiashi Feng. 2023. Magic: Benchmarking large language model powered multi-agent in cognition, adaptability, rationality and collaboration. arXiv preprint arXiv: 2311.08562

    2023 arXiv

  30. [38]

    Xuesong Yang, Yun-Nung Chen, Dilek Hakkani-T \"u r, Paul Crook, Xiujun Li, Jianfeng Gao, and Li Deng. 2017. End-to-end joint learning of natural language understanding and dialogue manager. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASS...

    2017

  31. [39]

    Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, et al. 2024. Tooleyes: Fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios. arXiv preprint arXiv:2401.00741

    2024 arXiv

  32. [40]

    Rongzhi Zhang, Yulong Gu, Xiaoyu Shen, and Hui Su. 2021. Knowledge-enhanced session-based recommendation with temporal transformer. arXiv preprint arXiv:2112.08745

    2021 arXiv

  33. [41]

    Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. 2024. https://openreview.net/forum?id=mM7VurbA4r SOTOPIA : Interactive evaluation for social intelligence in language ...

    2024

  34. [42]

    Dawei Zhu, Xiaoyu Shen, Marius Mosbach, Andreas Stephan, and Dietrich Klakow. 2023. Weaker than you think: A critical look at weakly supervised learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1...

    2023

  35. [43]

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2024. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems, 36

    2024

  36. [44]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  37. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.