Pith. sign in

REVIEW 5 major objections 5 minor 28 references

MagicSelector claims that training agent task decomposition with a counterfactual reward—the NDCG gain a decomposed query produces over the raw query—removes spurious shortcuts and yields state-of-the-art tool retrieval across three agent b

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:04 UTC pith:NQ2ORUCS

load-bearing objection Solid industrial pipeline with a new benchmark; the 'counterfactual' is just a NDCG difference and MTDTool's gold labels lack human validation, but the ToolBench/ToolRet results stand on their own. the 5 major comments →

arxiv 2607.17751 v2 pith:NQ2ORUCS submitted 2026-07-20 cs.IR cs.AIcs.CL

MagicSelector: Joint Optimization for Agent Tool Selection via Counterfactual Decomposition and Progressive Reranking

HONOR Agentic Search Team: Zhengzong Chen , Lei Tang , Lijun Liu , Chuandi Jiang , Fan Yang , Keyun Chu , Chu Zhao , Shihao Liu
show 21 more authors
Minghang Li Bo Liang Can Wen Hailong Wu Jingnan Ju Mian Liu Nengbin Zhang Peiqiang Wang Penghe Nie Qinhui Gu Sijia Lv Siqi Chen Wei Zhang Yang Xu Yuhao Qian Yuxiang Zhang Zeng Cheng Zhen Wang Zuan Chen Yuanyuan Zhao Fei Huang
This is my paper
classification cs.IR cs.AIcs.CL
keywords tool retrievaltask decompositioncounterfactual rewardGRPOhard negative miningrerankingdynamic top-kmulti-turn mobile agent
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that teaching an agent to decompose ambiguous user requests into atomic subtasks before retrieving tools only pays off if the reward measures the causal effect of decomposition on retrieval quality. It introduces a counterfactual reward that compares retrieval NDCG with and without decomposition, blocking shortcuts such as repetitive rewriting. On top of that, it adds a progressive reranker trained with self-mined hard negatives and an adaptive truncation rule that stops at score and semantic cliffs. The reported results show large gains over prior methods on three benchmarks, including a new multi-turn mobile dataset. If the central claim holds, it gives a training signal that improves tool retrieval accuracy and out-of-domain generalization in agent systems.

Core claim

The paper's central claim is that the full MagicSelector pipeline—counterfactual task decomposition, progressive reranking with self-distillation hard negatives, and dual semantic boundary-aware dynamic Top-K—significantly outperforms state-of-the-art tool retrieval methods, with N@10 of 59.90 on ToolRet, 96.28 on MTDTool, and 96.6 average NDCG on ToolBench (in-domain). The load-bearing novelty is the counterfactual reward: instead of rewarding final retrieval quality directly, the model is trained to maximize the positive NDCG gain caused by decomposition over the raw query and to enforce full tool coverage. This is claimed to cut spurious correlations and improve out-of-domain generalizati

What carries the argument

The central object is the counterfactual reward R_cf = λ_ndcg·max(0, N_atom − N_raw) + λ_full·(I[G⊆T_atom] − I[G⊆T_raw]), computed from tool retrieval NDCG before and after query decomposition. It is fused with a preference reward from a process reward model and optimized via GRPO, so the policy learns decompositions that causally improve retrieval rather than mimic surface patterns. The other two mechanisms are a progressive cross-encoder reranker trained point-wise then list-wise with self-distilled hard negatives, and a dynamic Top-K that truncates at the max reranking-score gap and the max inter-tool semantic-similarity gap.

Load-bearing premise

The MTDTool ground truth—atomic task sequences and target tool sets—is generated entirely by an automated state-machine-plus-LLM pipeline and treated as correct labels, with no human verification or inter-annotator agreement reported, so the benchmark's scores may partly reflect the same LLM priors the agent is trained to imitate.

What would settle it

Randomly sample 200 MTDTool dialogues; have independent human annotators produce atomic task decompositions and tool labels; recompute N@10 and C@10 for MagicSelector and the best baseline. If the gap narrows drastically or the scores drop below the reported 96, the automated labels—not the method—are carrying the result. Alternatively, remove the counterfactual term and retrain; if out-of-domain N@10 on ToolRet does not degrade, the causal reward is not doing the claimed work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the counterfactual reward works as claimed, RL-based task decomposition in agents can be trained against downstream retrieval quality directly, reducing reward hacking and repetitive decompositions.
  • The self-distillation hard-negative loop can be applied to any retriever or reranker without extra human annotation, improving fine-grained discrimination among similar items.
  • The dual-boundary dynamic Top-K could be used as a drop-in replacement for fixed-k truncation, improving Completeness while using fewer context tokens.
  • MTDTool provides process-level (context, task, tool) annotations, enabling failure diagnosis at each stage rather than only endpoint accuracy.
  • On ToolBench, the decomposition module adds gains mainly in broader candidate pools, suggesting its benefit grows with tool library size.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the counterfactual reward is a specific instance of a general principle—rewarding an intervention by its downstream ranking gain—that could improve query rewriting and decomposition in other retrieval-augmented generation pipelines, not just tool retrieval.
  • Editorial inference: because MTDTool's labels come from the same generation pipeline that produces the dialogues, the absolute numbers may overstate real-world accuracy; the relative ordering of methods could still hold if label noise is uniform across models.
  • Editorial inference: the dynamic Top-K rule (max of score-gap and semantic-gap truncation) is parameter-free and O(n), so it could be transported to any ranked list with dense embeddings, e.g., document reranking or API recommendation.
  • Editorial inference: a direct extension would be to test the counterfactual reward with oracle decompositions generated by a stronger LLM or by humans, isolating label quality from the training signal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes MagicSelector, a tool-retrieval framework for LLM-based agents composed of three trainable/inference components: (1) a counterfactual task-decomposition reward that uses the difference between NDCG obtained with and without atomic decomposition as a causal signal; (2) a two-stage retriever + reranker trained with self-distillation hard-negative mining; and (3) a dynamic top-k strategy that truncates the reranked list at the maximum of two gap-based truncation points. The authors also construct MTDTool, a state-machine-generated multi-turn mobile tool-retrieval benchmark with atomic-task and tool-set annotations. The paper reports large improvements over baselines across ToolRet, MTDTool, and ToolBench, including an OOD C@10 of 96.69 on MTDTool, and argues that the joint framework yields higher retrieval accuracy and lower token consumption.

Significance. If the reported results are valid, the paper makes a useful practical contribution: it shows that a joint decomposition, retrieval, reranking, and adaptive truncation pipeline can improve end-to-end tool retrieval across heterogeneous benchmarks. The MTDTool construction pipeline is more transparent than many synthetic-benchmark papers, and the self-distillation hard-negative mining procedure and the O(n) dynamic truncation heuristic are simple and scalable enough to be reproduced. The paper's strongest asset is the breadth of evaluation: ToolRet, ToolBench, and MTDTool are covered with both in-domain and out-of-domain splits, and the decomposition component is ablated against retrieval-only and rerank-only variants. However, the central conceptual claim — that the counterfactual reward 'cuts off spurious correlations' — is not backed by a causal estimator, and the MTDTool evaluation rests on label validity that the paper does not document. The empirical gains on MTDTool are nevertheless large enough that the concerns are addressable with additional analysis, rather than fatal.

major comments (5)
  1. [§3.2.1–3.2.2, Eqs. (4)–(6)] The 'counterfactual reward' R_cf = λ_ndcg max(0, N_atom - N_raw) + λ_full(...) is described as quantifying the 'marginal causal gain of decomposition' via a 'do-intervention', but the definition contains no causal estimator. N_raw and N_atom are both computed with the same retrieval function R_K; no backdoor adjustment, instrumental variable, or randomized intervention is used. The abstract and Section 1 further claim this 'effectively cuts off spurious correlations'; this is a causal claim that the paper does not establish. The formula is a difference of retrieval metrics, not a causal effect. The authors should either rename the reward as 'retrieval-gain reward' and remove the causal interpretation, or provide an actual intervention design (e.g., randomized decomposition labels, propensity-based adjustment, or a controlled experiment).
  2. [§3.1 and §3.3, Eq. (5)] The preference reward in Eq. (5) is defined against 'y*, the human-annotated reference', but no human annotation step is described anywhere in the construction pipeline of Section 3.1 or Appendix A. The pipeline described in Figure 2 is a state-machine-driven dual-LLM generator plus an independent decomposition module; the appendix only gives LLM prompts. Since the same generated atomic-task/tool mappings are used as training labels for the reward model and as gold labels for all MTDTool metrics, the flagship MTDTool results — especially C@10 ≈ 96–97 in Table 2 — may measure self-consistency with an LLM-imposed schema rather than retrieval accuracy. This is an internal gap, not a disagreement with consensus. The authors must provide the annotation protocol, a description of the annotator pool, and at least an inter-annotator agreement measure, or stop calling y* human-annotated.
  3. [§5.1–5.2, Eqs. (26)–(29)] Two load-bearing claims about dynamic top-k are not supported. First, the Pareto-distribution justification is asserted from Figure 5 with no goodness-of-fit test, no fitted exponent, and no comparison to alternative distributions. Second, Eq. (29) states that taking the maximum of the two argmax truncation points 'maximizes recall' while preserving coherence, but this is not proven and is not generally true: the maximum of two gap positions can be larger than the true relevant-tool cutoff, introducing noise, or smaller if the semantic gap occurs early. The authors should fit the distribution formally, provide a recall/coherence trade-off curve over the two choices, and show an ablation where the two signals are used separately and in combination.
  4. [Tables 1–5] All reported results are single numbers with no error bars, no number of runs, and no significance tests. Several differences that are used to support substantive claims are small (e.g., Table 1 NTool with 'Ours-Decomp + Ours-Rerank' 59.90 vs 'Ours-Embedding-4B + Ours-Rerank-4B' 59.44 on N@10; Table 5 Customized N@10 60.30 vs Tool-DE-Rerank-4B 62.72, which the text acknowledges). Without variance estimates, the claim that the full pipeline 'significantly outperforms' baselines is not established. At minimum, run 3–5 independent seeds and report means with standard deviations, and use a paired test for the decomposition-vs-no-decomposition comparisons.
  5. [Table 2] The out-of-domain C@10 for the full system (96.69) is higher than the in-domain C@10 (95.33), while every other row in the table shows in-domain ≥ OOD or near parity. This inversion is never discussed. It either indicates that the OOD split is easier in terms of completeness than the in-domain split, or that C@10 is dominated by the fixed denominator and not comparable across splits. The authors should report the distribution of gold-set sizes in both splits and explain the inversion, or the reader cannot interpret the OOD claim.
minor comments (5)
  1. [§6.4, Table 6] The comparison between 'fixed-10' Completeness@10 and 'adaptive-k' Completeness@k uses different denominators; reporting a raw percentage without normalizing by the expected number of relevant tools can make the adaptive method appear better simply because it returns more tools. Normalize or report a recall-oriented metric with a token-cost axis.
  2. [Figure 7] The x-axis labels '0, 0.2, 0.5, 0.8, 1' are not tied explicitly to the preference-reward weight α from Eq. (6), and the panels do not state how the corresponding β is set. Make the mapping explicit and add error bars.
  3. [References] Several references are duplicated (ToolLLM appears as Qin et al. 2024a and 2024b; ReInvoke as Chen et al. 2024a and 2024b; AnyTool as Du et al. 2024a and 2024b). This is a formatting issue but makes the bibliography look inconsistent.
  4. [§3.4, Eq. (8)] The GRPO objective uses a KL penalty term βD_KL, but the same symbol β is also used for the counterfactual reward weight in Eq. (6). Rename one to avoid confusion.
  5. [Abstract] The abstract calls MTDTool 'the first task decomposition benchmark' for mobile scenarios. Given that HammerBench and other recent benchmarks include decomposition-like annotations, either qualify the claim or cite the relevant comparators.

Circularity Check

1 steps flagged

MTDTool's synthetic LLM labels double as the reward reference and the evaluation gold; core comparisons on ToolBench/ToolRet remain external.

specific steps
  1. self definitional [Section 3.1.4 (Eq. 3), Section 3.3 (Eq. 5); MTDTool results in Tables 1–2]
    "At the output end of the pipeline, an independent task decomposition module maps user requests containing multiple intents into an atomic task sequence A_t and establishes semantic associations with the candidate tool set T_cand,t: qt,Ht → A_t → T_cand,t. ... Let sφ(x,y) denote the PRM score for a generated result y given context x, and y⋆ be the human-annotated reference. The preference reward quantifies the relative advantage of the candidate over the reference: Rpref = σ(sφ(x,yi) − sφ(x,y⋆))."

    On the paper's own account, the only described source of the MTDTool atomic-task/tool labels is an automated LLM module (Appendix A shows the prompt); no human-annotation step, annotator pool, or agreement statistic appears anywhere. Thus y* in Eq. (5) and the gold labels used for the MTDTool rows of Tables 1–2 are traceable to the same synthetic labeling pipeline. The preference reward then trains the decomposition policy to match the benchmark's own gold, and the later N@10/C@10 report on that same benchmark measures self-consistency with the label generator rather than independently established retrieval correctness. The claim that decomposition improves MTDTool retrieval is therefore partly built into the training signal, not an independent confirmation.

full rationale

MagicSelector's results on ToolBench and ToolRet are external benchmarks with pre-existing labels, so the framework's retrieval, reranking, and dynamic top-k components retain independent empirical content. The main circularity concern is confined to the self-constructed MTDTool benchmark: its gold atomic-task/tool labels are generated by the same automated LLM pipeline that supplies the reference y* in Eq. (5), and the counterfactual reward Rcf is itself defined as the NDCG gain measured by the same retriever/labels later used as the headline metric. This makes the MTDTool claim of 'process-level annotations' and the full pipeline's 96+ N@10/C@10 partially self-referential. However, the paper also shows strong gains on ToolBench and ToolRet where the labels are not generated by this paper's pipeline, so the central claim does not reduce entirely to a fit or a self-citation chain. The author-overlap citation to Zhao et al. 2026a is attribution for the PCTD mechanism, but the reward equations are re-derived in this paper, so that self-citation is not load-bearing.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

The framework depends on several hand-set weights and process parameters, and its main new benchmark is generated by an automated pipeline without human verification. The causal-sounding counterfactual reward is a non-causal before/after difference, and the dynamic top-k relies on unproven distributional and gap-detection assumptions.

free parameters (7)
  • Preference reward weight α = not reported; swept in Figure 7
    Eq. 6 fuses rewards as R = α R_pref + β R_cf, but final α value is not stated; Figure 7 shows the system is sensitive to α.
  • Counterfactual reward weight β = not reported
    Needed for Eq. 6; no tuning details given.
  • λ_ndcg (counterfactual rank-gain weight) = not reported
    Appears in R_cf definition in Section 3.2.2; no value or tuning procedure given.
  • λ_full (completeness constraint weight) = not reported
    Appears in R_cf definition in Section 3.2.2; no value or tuning procedure given.
  • MTDTool topic-switching parameters p0, α, p_min = p0=0.10, α=0.85, p_min=0.01
    Eq. 2 hand-sets the dialogue topic-switching probability; these shape the entire benchmark.
  • Hard-negative count k and distillation iterations R = not reported
    Algorithm 1 requires k and R; the paper reports neither.
  • InfoNCE temperature τ = not reported
    Appears in Eq. 13 for retriever training; no value given.
axioms (6)
  • domain assumption Reranking scores follow a Pareto-like distribution and the largest adjacent gap marks the optimal truncation boundary.
    Section 5.1 asserts the Pareto form without fitting or testing; Section 5.2 uses argmax gaps to define Top_K_score and Top_K_sim, and the max of the two as the final cut.
  • domain assumption The LLM-generated state-machine pipeline yields correct ground-truth annotations for MTDTool.
    Section 3.1 describes automated generation with no human validation or inter-annotator agreement; Section 3.3 nonetheless refers to a "human-annotated reference" y*.
  • ad hoc to paper N_atom − N_raw equals the causal effect of decomposition on retrieval quality.
    Section 3.2.2 defines this as a counterfactual reward using do-notation, but no confounding control or causal estimator is provided; it is simply a before/after retrieval-score difference.
  • domain assumption The Process Reward Model s_φ gives valid preference scores for decomposition quality.
    Section 3.3 defines a five-dimension PRM but gives no training data, no validation, and no agreement with human judgments.
  • ad hoc to paper Taking max of the two argmax gap truncation points maximizes recall while preserving semantic coherence.
    Eq. 29 asserts K_final = max(Top_K_score, Top_K_sim) with no comparison against alternative fusion rules or an oracle.
  • domain assumption The reranked list forms a one-dimensional semantic space in which adjacent-tool similarity drops mark functional boundaries.
    Section 5 motivates this with TextTiling and ToolLLM references, but the assumption is not independently validated for tool lists.

pith-pipeline@v1.3.0-alltime-deepseek · 30460 in / 17276 out tokens · 191975 ms · 2026-08-01T17:04:06.107818+00:00 · methodology

0 comments
read the original abstract

We present MagicSelector, a joint optimization framework integrating Counterfactual task decomposition, Progressive reranking, and Dynamic Top-K, designed to address the fundamental challenges of tool retrieval in agents. MagicSelector is a specialized framework capable of translating ambiguous user instructions into executable atomic subtasks and guiding high-precision tool retrieval, effectively mitigating redundant noise and severe context distraction in out-of-domain (OOD) scenarios. We empower MagicSelector with these capabilities through three key contributions: (1) a preference-guided counterfactual task decomposition mechanism that utilizes a counterfactual reward to quantify the marginal causal gain of decomposition on retrieval ranking, effectively imposing fine-grained structural supervision on logical coherence; (2) a progressive tool reranking method driven by self-distillation hard negative mining, which optimizes both point-wise and list-wise relevance to enhance fine-grained discrimination among highly similar tools; and (3) a dual semantic boundary-aware dynamic Top-K strategy that adaptively monitors reranking score cliffs and inter-tool semantic shifts to dynamically truncate the candidate list, maximizing relevant tool recall while filtering long-tail noise. Evaluated on MTDTool, the first task decomposition benchmark we constructed tailored for mobile multi-turn interactions with process-level annotations, MagicSelector yields promising performance. Extensive experiments demonstrate that MagicSelector significantly outperforms state-of-the-art methods in terms of tool retrieval accuracy, OOD generalization capability, and overall token efficiency, thereby demonstrating the effectiveness of our proposed framework.

Figures

Figures reproduced from arXiv: 2607.17751 by Bo Liang, Can Wen, Chuandi Jiang, Chu Zhao, Fan Yang, Fei Huang, Hailong Wu, HONOR Agentic Search Team: Zhengzong Chen, Jingnan Ju, Keyun Chu, Lei Tang, Lijun Liu, Mian Liu, Minghang Li, Nengbin Zhang, Peiqiang Wang, Penghe Nie, Qinhui Gu, Shihao Liu, Sijia Lv, Siqi Chen, Wei Zhang, Yang Xu, Yuanyuan Zhao, Yuhao Qian, Yuxiang Zhang, Zeng Cheng, Zhen Wang, Zuan Chen.

Figure 1
Figure 1. Figure 1: The overall framework of MagicSelector. It presents a joint optimization pipeline integrat￾ing counterfactual task decomposition, progressive reranking, and dynamic Top-K for complex tool-use scenarios. The framework consists of three key phases: (1) Task Decomposition via Counterfactual RL, which introduces a counterfactual reward mechanism based on relative ranking gains to provide precise causal attribu… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the MTDTool construction pipeline. The data generation follows a four-stage process. (1) Initialization: Tools and scenarios are sampled to establish initial constraints, and two LLM generators produce the initial user query q 0 and assistant output o 0 . (2) State Evolution & Context Decision: At each turn t ≥ 1, the system decides whether to inherit the previous context or switch to a new top… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the preference-guided counterfactual task decomposition mechanism in MagicSelector. It is worth noting that Sφ denotes the preference reward model, R(·) represents the downstream tool retrieval model, and the scoring function M(·) is implemented using NDCG as the evaluation metric. to downstream tool retrieval performance and the semantic quality of the decomposition structure itself. To this e… view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the training workflow of the retrieval and progressive reranking module in MagicSelector. The module first trains an embedding retriever and a reranker with random negatives, then mines high-scoring incorrect tools from the training tool pool as hard negatives. The hard-negative dataset is used to retrain both models and obtain enhanced retrieval and reranking models. 4.1 Embedding-based Retrie… view at source ↗
Figure 5
Figure 5. Figure 5: Reranking score distribution of retrieved tools [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance comparison with dynamic Top-K across datasets. [PITH_FULL_IMAGE:figures/full_fig_p024_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Hyperparameter analysis under different α settings. 7 CONCLUSION In this work, we present MagicSelector, a joint optimization framework specifically designed to address the fundamental challenges of tool retrieval and task planning in Large Language Model (LLM) agents. A counterfactual task decomposition mechanism, a progressive reranking method, and a dynamic Top-K strategy collectively contribute to the … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 13 linked inside Pith

  1. [4]

    Planning and editing what you retrieve for enhanced tool learning

    27 HONOR Tenghao Huang, Dongwon Jung, Vaibhav Kumar, Mohammad Kachuee, Xiang Li, Puyang Xu, and Muhao Chen. Planning and editing what you retrieve for enhanced tool learning. InFindings of the Association for Computational Linguistics: NAACL 2024, pp. 975–988, 2024b. Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Rui...

    Pith/arXiv arXiv 2024

  2. [6]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. InInternational Conference on Learning Representations, volume 2024, pp. 52989–53046, 2024b. Xuan Lu, Haohang Huang, Rui Meng, Yaohui Jin, Wenjun Zeng, and Xiaoyu Shen. Tools are under- documented:...

    2024

  3. [11]

    Document ranking with a pretrained sequence-to-sequence model

    Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequence model. InFindings of the Association for Computational Linguistics: EMNLP 2020,

    2020

  4. [12]

    Ronak Pradeep, Rodrigo Nogueira, and Jimmy J. Lin. The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models.arXiv preprint arXiv:2101.05667,

    Pith/arXiv arXiv

  5. [14]

    CREATOR: Tool creation for disentangling abstract and concrete reasoning of large language models

    Cheng Qian, Chi Han, Yi Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. CREATOR: Tool creation for disentangling abstract and concrete reasoning of large language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, pp. 6922–6939,

    2023

  6. [15]

    Toolllm: Facilitating large language models to master 16000+ real-world apis

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. InInternational Conference on Learning Representat...

    2024

  7. [16]

    Sentence-BERT: Sentence embeddings using siamese BERT- networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing, pp. 3982–3992,

    2019

  8. [18]

    Retrieval models aren’t tool-savvy: Benchmarking tool retrieval for large language models

    Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, and Zhaochun Ren. Retrieval models aren’t tool-savvy: Benchmarking tool retrieval for large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 24497–24524,

    2025

  9. [19]

    Is chatgpt good at search? investigating large language models as re-ranking agent.arXiv preprint arXiv:2304.09542,

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agent.arXiv preprint arXiv:2304.09542,

    Pith/arXiv arXiv

  10. [20]

    Efficient context selection for long-context QA: No tuning, no iteration, just adaptive-k

    Chihiro Taguchi, Seiji Maekawa, and Nikita Bhutani. Efficient context selection for long-context QA: No tuning, no iteration, just adaptive-k. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 20105–20130,

    2025

  11. [21]

    Hammerbench: Fine-grained function-calling evalu- ation in real mobile assistant scenarios

    Jun Wang, Jiamu Zhou, Xihuai Wang, Xiaoyun Mo, Haoyu Zhang, Qiqiang Lin, Jincheng Jincheng, Muning Wen, Weinan Zhang, and Qiuying Peng. Hammerbench: Fine-grained function-calling evalu- ation in real mobile assistant scenarios. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 3350–3376, 2025a. Liang Wang, Nan Yang, and Furu Wei. ...

    2025

  12. [22]

    Improving text embeddings with large language models

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024a. Renxi Wang, Xudong Han, Lei Ji, Shu Wang, Timothy Baldwin, and Haonan Li. Toolgen: Unified tool ret...

    2024

  13. [23]

    Enhancing tool retrieval with iterative feedback from large language models

    Qiancheng Xu, Yongqi Li, Heming Xia, and Wenjie Li. Enhancing tool retrieval with iterative feedback from large language models. InFindings of the Association for Computational Linguistics: EMNLP 2024,

    2024

  14. [24]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629,

    30 HONOR Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629,

    Pith/arXiv arXiv

  15. [26]

    Survey on evaluation of llm-based agents.arXiv preprint arXiv:2503.16416,

    Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. Survey on evaluation of llm-based agents.arXiv preprint arXiv:2503.16416,

    Pith/arXiv arXiv

  16. [27]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176,

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176,

    Pith/arXiv arXiv

  17. [28]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Representations, volume 2024, pp. 15585–15606,

    2024

  18. [1997]

    Planning, creation, usage: Benchmarking llms for comprehensive tool utilization in real-world complex scenarios

    Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, et al. Planning, creation, usage: Benchmarking llms for comprehensive tool utilization in real-world complex scenarios. InFindings of the Association for Computational Linguistics: ACL 2024, pp. 4363–4400, 2024a. Shouzheng Huang, ...

    Pith/arXiv arXiv 2024

  19. [2005]

    Passage re-ranking with bert.arXiv preprint arXiv:1901.04085,

    Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert.arXiv preprint arXiv:1901.04085,

    Pith/arXiv arXiv 1901

  20. [2009]

    29 HONOR Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joëlle Pineau, and Luke Zettlemoyer

    doi: 10.1561/1500000019. 29 HONOR Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joëlle Pineau, and Luke Zettlemoyer. Improving passage retrieval with zero-shot question generation. InConference on Empirical Methods in Natural Language Processing,

    doi:10.1561/1500000019

  21. [2019]

    Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy J. Lin. Multi-stage document ranking with bert.arXiv preprint arXiv:1910.14424,

    Pith/arXiv arXiv 1910

  22. [2020]

    Api-bank: A comprehensive benchmark for tool-augmented llms

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. InProceedings of the 2023 conference on empirical methods in natural language processing, pp. 3102–3116, 2023a. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meisha...

    Pith/arXiv arXiv 2023

  23. [2021]

    Rankvicuna: Zero-shot listwise document reranking with open-source large language models.ArXiv, abs/2309.15088,

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankvicuna: Zero-shot listwise document reranking with open-source large language models.ArXiv, abs/2309.15088,

    Pith/arXiv arXiv

  24. [2022]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.tau-bench: A benchmark for tool- agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045,

    Pith/arXiv arXiv

  25. [2023]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InInternational Conference on Learning Representations, volume 2024, pp. 9025–9049,

    2024

  26. [2024]

    Re-invoke: Tool invocation rewriting for zero-shot tool retrieval

    Yanfei Chen, Jinsung Yoon, Devendra Singh Sachan, Qingze Wang, Vincent Cohen-Addad, Mohammad- hossein Bateni, Chen-Yu Lee, and Tomas Pfister. Re-invoke: Tool invocation rewriting for zero-shot tool retrieval. InFindings of the Association for Computational Linguistics: EMNLP 2024, pp. 4705– 4726, 2024a. Yanfei Chen, Jinsung Yoon, Devendra Singh Sachan, Qi...

    arXiv 2024

  27. [2025]

    Xueguang Ma, Xinyu Crystina Zhang, Ronak Pradeep, and Jimmy J. Lin. Zero-shot listwise document reranking with a large language model.ArXiv, abs/2305.02156,

    Pith/arXiv arXiv

  28. [2026]

    FIRST: Faster improved listwise reranking with single token decoding

    Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. FIRST: Faster improved listwise reranking with single token decoding. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8642–8652,

    2024