Pith. sign in

REVIEW 3 major objections 5 minor 57 references

This paper claims that the binding constraint on tool-using LLM agents is retrieval, not planning, and that evolving natural-language pseudo-tool descriptions at test time lifts pooled pass rate from 57.6% to 84.3% on StableToolBench.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 05:17 UTC pith:KF6EAGED

load-bearing objection FitText's core idea—dynamic, evolutionary retrieval via pseudo-tool descriptions—is genuinely worth engaging with, but its own tables undercut the headline mechanism: Memetic's +26.7-point gain is not explained by the recovery story the authors tell. the 3 major comments →

arxiv 2605.02411 v3 pith:KF6EAGED submitted 2026-05-04 cs.AI cs.IRcs.LGcs.MA

FitText: Evolving Agent Tool Ecologies via Memetic Retrieval

Kyle Zheng , Han Zhang , Renliang Sun , Chenchen Ye , Wei Wang This is my paper
classification cs.AI cs.IRcs.LGcs.MA
keywords tool retrievalLLM agentstest-time retrievalevolutionary searchpseudo-tool descriptionsmemetic algorithmstool useretrieval bottleneck
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

FitText argues that when an agent faces a large API catalog, the real bottleneck is not how well it plans but whether the right tools are ever put in front of it. Static retrieval from the user's original query can fail before planning begins, and the paper's remedy is to treat tool retrieval as a test-time search: the agent writes natural-language pseudo-tool descriptions, treats each retrieval against the fixed catalog as a cheap revisable test of that guess, and then refines, diversifies, and evolves those guesses. The evolved version, Memetic Retrieval, adds selection, crossover, mutation, and a memory penalty that discourages redundant searches. On StableToolBench's 16,464 APIs it reaches an 84.3% pooled pass rate, 26.7 points above static query retrieval, with gains growing alongside base-model capability and concentrating on ambiguous multi-tool tasks. The sympathetic reading: improving what the agent retrieves is a training-free, model-agnostic lever that matters more than improving the planner.

Core claim

The central claim is that the retrieval interface, not planning, binds end-to-end tool-agent performance, and that this constraint can be relaxed at inference time by embedding retrieval inside reasoning. The agent writes pseudo-tool descriptions—revisable guesses about the tool it needs—and uses retrieval against the fixed corpus as a cheap test of each guess. Multi-turn refinement, scattershot diversity, and memetic retrieval are positions in one design space; the memetic loop evolves a population under a fitness that rewards retrieval mass on existing tools and penalizes revisiting covered ground. Empirically, it lifts pooled pass rate on StableToolBench (16,464 APIs) from 57.6% to 84.3%

What carries the argument

The key object is the pseudo-tool description: a natural-language sentence hypothesizing what a needed tool should do, used as the retrieval probe instead of the raw user query. The engine is a memetic loop—seed a population of descriptions, score each by retrieval fitness (softmax similarity mass concentrated on top tools, minus a divergence penalty that suppresses overcrowding of already-explored belief regions), select the top half, crossover or mutate, locally refine offspring against retrieved tool exemplars, and repeat over generations. Retrieval is thus a fitness oracle over a fixed corpus, making the search training-free and model-agnostic.

Load-bearing premise

Everything rests on treating embedding cosine similarity as a faithful oracle for which tools can actually execute the needed function; the paper's own recovery numbers show converting retrieved availability into solved tasks is solver-dependent (Memetic's recovery is negative on the main solver), so if the embedding rewards paraphrasic similarity rather than functional utility, evolved descriptions may retrieve well without solving tasks.

What would settle it

Run Memetic Retrieval with a deliberately decorrelated retriever—e.g., randomly permuted or paraphrase-trained embeddings—and see whether the 26.7-point gain survives; if retrieval fitness is the operative signal, the gain should collapse when the oracle no longer tracks executable utility. Alternatively, test within-generation fitness against per-task solve outcomes: if higher-fitness descriptions do not yield higher solve rates on the same tasks, the fitness function is not measuring what the headline gain claims.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If retrieval is the binding constraint, end-to-end tool-agent performance should improve more from retrieval-side scaling—stronger analysis models, better probes—than from stronger planners; the paper's ablations support this asymmetry.
  • Dynamic re-retrieval converts a wrong first tool pick into recoverable search: among wrong-first-tool G2 tasks, a correct tool appears in 44–83% of dynamic retrievals versus 7–23% for static retrieval.
  • The gains concentrate on ambiguous multi-tool tasks (G2-category +36.5, G2-instruction +35.1 for the strongest tested model), the regime where static retrieval is weakest.
  • The strategy is deployment-bounded: under a weak legacy base model, memetic search amplifies noise and falls below the static baseline (32.2 vs 34.8 average), so it presupposes an LLM competent enough to act as a semantic operator.
  • Because the framework is training-free and orthogonal to planning, it composes with future stronger base models and planning advances rather than replacing them.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable implication of the paper's bottleneck claim: end-to-end tool-agent benchmarks that hand agents the gold tool set (schema injection) systematically mask retrieval failures, so realistic evaluation should withhold oracle tools.
  • The paper's own recovery data show fitness-to-solve conversion is solver-dependent—Memetic's recovery-proper is negative on the main solver while Scattershot is positive—suggesting an extension where retrieval fitness is augmented with a small downstream solve signal (e.g., tool-call success rates) to align evolution with end-task success.
  • The generate-retrieve-revise loop is plausibly general: any discrete grounding corpus with a cheap retrieval oracle—code snippets, database schemas, hardware documentation—could benefit from the same hypothesis-evolution mechanism, an untested conjecture the paper itself floats.
  • The near-wall-clock-neutral parallel cost (1.01x at 40-way concurrency) suggests the added evolutionary search is inexpensive enough to run as a default retrieval layer, not just a fallback for hard tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes FitText, a training-free framework for dynamic tool retrieval in LLM agents. Instead of retrieving tools once from the user query, the agent emits natural-language pseudo-tool descriptions during reasoning and uses them as retrieval probes. Strategies are organized in a design space from Single-Pass to Multi-Turn (DBD) to Scattershot to Memetic, which adds evolutionary selection, crossover/mutation, local refinement, and a tool memory. On StableToolBench (16,464 APIs) with GPT-5.4-mini, Memetic reaches an 84.3% pooled pass rate, +26.7 points over static query retrieval, with gains increasing with base-model capability. The paper also claims that retrieval, not planning, is the binding constraint on end-to-end agent performance, and reports a weak-model failure mode for the evolutionary search. The experiments span two benchmarks, three base models, and several honestly reported negative controls.

Significance. If the headline results hold, this is a practically significant contribution: a model-agnostic, training-free wrapper around the retrieval interface that yields large end-to-end gains on a difficult multi-tool benchmark. The paper is commendably honest in several places: it reports the Memetic V2 negative result, the weak-legacy-model reversal, the solver-dependent recovery direction, and the absence of a cost Pareto for the headline solver. The fitness function is a declared scoring rule rather than a fitted-to-benchmark objective, so the fit-to-target circularity common in some agent papers is avoided. However, the mechanism underlying the headline +26.7 pp gain is not established. The paper's own recovery analysis on the main solver shows that Memetic's recovery-proper is negative while Scattershot's is positive, which conflicts with the causal story in §5.5. The credibility of the empirical claim depends on resolving this inconsistency and reporting the missing planner-swap experiment.

major comments (3)
  1. [§5.5, Table 17 vs Table 1] The paper's causal story is that dynamic re-retrieval works by widening the candidate frame after a wrong first pick. However, Table 17 reports that on the main GPT-5.4-mini solver, Memetic's recovery-proper is -1.5 pp while Scattershot's is +7.5 pp. Table 1 shows Memetic's total gain over static is +26.7 pp, larger than Scattershot's +20.0 pp. Since the first-tool off-gold rate is flat across methods (§5.5), the entire effect cannot live after the wrong first pick, and the proposed recovery mechanism does not explain the headline result. No decomposition of the +26.7 pp by first-tool gold status, later-tool recovery, task subset, or tool-call volume is given, and the small-n explanation for the negative recovery-proper is offered without confidence intervals. The source of Memetic's advantage must be identified before the claim that retrieval-fitness selection converts into solves can b
  2. [§5.7, Appendix A.2] The central thesis 'retrieval, not planning, is the binding constraint' is supported by two pieces of evidence: scaling the analysis model improves retrieval (Fig. 4, legacy configuration) and swapping in a stronger planner 'did not improve pass rates.' The planner-swap result is asserted in prose only; Appendix A.2 repeats the assertion without a comparison table, model names, or numbers. Because this thesis is used to justify the entire approach, the experiment should be reported (or the claim downgraded). The analysis-model ablation is also legacy-only; no current-model analysis-model comparison is shown.
  3. [§4.5, Eq. (6), Table 17] The fitness oracle in Eq. (6) scores pseudo-tool descriptions by cosine-similarity mass concentration minus a KL memory penalty, but the paper does not validate that this retrieval fitness predicts executable utility. On the headline solver, the selected hypotheses convert into solves below the root-only frame ceiling (Memetic recovery-proper -1.5 pp), while Scattershot, which uses no fitness-based selection, converts at +7.5 pp. This is direct evidence that the embedding-space fitness and downstream task success come apart. A validation study (e.g., solve rates conditioned on fitness quartile, or Memetic vs. Scattershot at matched retrieval budget) is needed to support the claim that selection by Eq. (6) drives end-to-end gains.
minor comments (5)
  1. [Abstract, Table 5] The Abstract claims that FitText's reformulation strategies improve NDCG@5 by 2.7–10.6 points over static retrieval 'across all base models.' Table 5 does not support this: Gemma4-31B Single-Pass on Code has NDCG@5 25.95 vs. 26.85 for Query (negative), and Qwen3.6-35B Single-Pass on Web is only +0.86. Please correct the numerical claim or the range.
  2. [Table 10 vs. Table 3] Table 10 lists population size N=6, while the legacy Memetic row in Table 3 uses P=5. It should be stated explicitly which population size was used for the main Table 1 results.
  3. [Throughout, especially Table 17] Pass-rate differences between strategies are reported without confidence intervals or significance tests. Table 17 attributes Memetic's negative recovery-proper to small-n noise (203 hard query IDs), but without intervals the reader cannot assess whether the headline 26.7-point gain or the 6-9 point Memetic-vs-Scattershot gaps are stable.
  4. [§4.5, Table 10] The fitness hyperparameters λ, σ, m, θ are declared fixed after development. Please state explicitly whether these values were selected on the StableToolBench development set or on ToolRet, and report the controlled head-to-head of the four penalty formulations with enough detail to be reproduced.
  5. [Appendix D.3] The note that 'Memetic also loses on 2 of 5 splits (G1-Inst, G1-Tool)' is easy to misread as a pass-rate loss given Table 2 shows Memetic gaining on all splits. Clarify that these are recovery-proper split-level losses, not overall pass-rate losses.

Circularity Check

0 steps flagged

No significant circularity: the headline result is an external measurement, and the internal fitness objective is a declared, unlabeled scoring rule whose conversion into solves is explicitly shown by the paper to be imperfect.

full rationale

The paper's derivation chain is not circular. The central end-to-end claim (84.3% vs. 57.6% on StableToolBench; +26.7 pp) is an empirical measurement against an external judge and gold solve labels, not a consequence of the definitions. The internal scoring loop is Eq. 6, whose observation-likelihood term is a softmax over the same cosine similarities used by the retriever (Eq. 4.2: score(d, ti) = cos(e(d), e(ti))); this is a declared retrieval-fitness objective, not a parameter fitted to the benchmark labels. The ToolRet retrieval metrics use held-out gold relevance labels, so the reported NDCG improvements are not true by construction. The paper itself discloses that the fitness-to-solve link is loose: Table 17 reports Memetic recovery-proper of -1.5 pp on the main GPT-5.4-mini solver and +3.7 pp only on the legacy solver, so the fitness definition does not force the headline pass-rate gain. There are no self-citations, no imported uniqueness theorems, and no fitted parameter renamed as a prediction. The skeptical concerns raised in the reader's take—same model family as judge, the unexplained source of the +26.7 pp gain, and the weak support for 'retrieval, not planning'—are external-validity or mechanism-support issues, not circularity under the specified definitions.

Axiom & Free-Parameter Ledger

13 free parameters · 8 axioms · 3 invented entities

The method's contributions rest on a small set of domain assumptions about embedding-similarity-as-fitness, pseudo-tool semantic alignment, judge validity, and prompt compliance, plus declared-but-unfitted free parameters in the fitness function (Eq. 6). The search budgets (N, G, T, S) are the only parameters the paper admits to selecting on development data, which is appropriate. No new physical entities are introduced; 'pseudo-tool descriptions' and 'tool memory' are in-system constructs with no externally falsifiable handle.

free parameters (13)
  • KL-drift penalty weight λ = 1.0
    Eq. 6; declared constant, but its penalty family was selected via a head-to-head on hard queries (§4.5), so 'never tuned' overstates the case.
  • KDE kernel bandwidth σ = 0.5
    Eq. 6, Table 10; controls the tool-memory penalty's particle approximation.
  • Observation size m = 3
    Eq. 6; top-m retrieval mass in the observation-likelihood term.
  • Fitness threshold θ = 0.95
    Algorithm 1 early-stopping threshold (Table 10); stops evolution when the best fitness exceeds it.
  • Population size N (P) = 5/6
    Table 3 uses P=5, Table 10 reports N=6; internal inconsistency. Search-budget knob selected on development data.
  • Generations G = 3
    Algorithm 1, Table 10; search-budget knob.
  • Refinement turns T (DBD) = 3 or 5
    T=3 for GPT-5.4-mini/Gemma4-31B, T=5 for Qwen3.6-35B (Table 5 note, §D.4); selected per model.
  • Scattershot fan-out S = 5 or 10
    S=5 reported in headline; S=10 underperforms on the legacy grid (Table 3).
  • Temperatures τ_scatter / τ_memetic / τ_refine = 1.5 / 1.5 / ≤0.7
    Table 10; generation and refinement temperatures.
  • Crossover probability p_cross = 0.5
    Algorithm 1; Table 10.
  • Duplicate-detection threshold = 0.82
    Eq. 8, Appendix B.1; SequenceMatcher similarity gates whether retrieval fires.
  • Top-k and voting budget k, B = 5, 5
    Table 10; retrieval and output budget.
  • Embedding retriever choice = SimCSE-RoBERTa-large
    Fixed retriever throughout; the entire fitness signal depends on this choice, a modeling decision not an independent benchmark.
axioms (8)
  • domain assumption Cosine similarity in SimCSE space is a valid fitness oracle for whether a pseudo-tool description corresponds to an executable, useful tool.
    Eqs. 4–6 and Algorithm 1 lines 4–6: the entire evolutionary selection and tool-memory penalty rest on this premise. If the retriever ranks by surface similarity rather than functional utility, evolution optimizes the wrong objective.
  • domain assumption LLM-generated pseudo-tool descriptions are semantically closer to tool documentation than the raw user query.
    §4.2, Eqs. 1–4: the method's core value proposition (bridging the semantic gap). Partially evidenced by Single-Pass > Query in Tables 1 and 5.
  • domain assumption GPT-5.4-mini provides valid pass/fail judgments on StableToolBench and is unaffected by being the same model family as the solver.
    Table 1 header (judge gpt-5.4-mini); self-judging creates a risk of leniency toward own-style outputs, and no judge-model sensitivity test is reported.
  • domain assumption StableToolBench gold labels and the 765-task solvable subset are correct and complete.
    §5.1, Table 1 caption; all headline pass rates are computed on this subset.
  • domain assumption The base LLM will emit correctly delimited {BEGIN}...{END} pseudo-tool blocks during reasoning when needed.
    §4.2, Algorithm 2 lines 15–22: the dynamic trigger depends on prompt compliance; a model that fails to format blocks disables retrieval.
  • domain assumption The DFSDT reasoning loop (Qin et al., 2024b) is a sound solver substrate.
    Appendix B.1; taken from prior literature and used as the execution environment for all strategies.
  • standard math Softmax normalization of retrieval scores and the KDE particle approximation of KL divergence (Eqs. 5–6) are valid probability computations.
    Appendix C.2 gives the closed-form particle computation; no derivation issues identified.
  • domain assumption Token-cost model ($0.75/M input, $4.50/M output) supports the cost claims.
    §5.6; cost brackets are estimates, and the paper itself notes the $/pp Pareto exists only on the legacy solver.
invented entities (3)
  • Pseudo-tool description as doxastic belief state no independent evidence
    purpose: A revisable natural-language hypothesis of the needed tool's functionality, used as the retrieval probe.
    A conceptual construct; its only handle on reality is the paper's own benchmark results. No externally falsifiable signature beyond the reported pass rates.
  • Tool memory as KDE particle approximation of the posterior over tool-intent no independent evidence
    purpose: Suppresses redundant search via the KL-drift penalty in Eq. 6.
    An internal search mechanism; no outside-the-paper falsifiable handle.
  • Meme / memetic framing no independent evidence
    purpose: Rhetorical framing of evolutionary operators (crossover, mutation, selection) over descriptions.
    Terminology borrowed from cultural evolution; not a new physical or algorithmic entity.

pith-pipeline@v1.3.0-alltime-deepseek · 28017 in / 26022 out tokens · 253905 ms · 2026-08-04T05:17:54.389021+00:00 · methodology

0 comments
read the original abstract

Efficient reasoning is not only a matter of shortening an answer trace; for tool-using agents, it also depends on whether the agent is reasoning over the right action space. As API ecosystems scale to tens of thousands of endpoints, the semantic gap between user requests and tool documentation makes this problem concrete: static retrieval from the initial query can fail before planning begins, and stronger planning alone cannot recover a missing tool. We study this problem as budgeted test-time retrieval and introduce FitText, a training-free framework that makes the tool interface revisable during execution by generating, refining, and evolving natural-language pseudo-tool descriptions as retrieval probes. FitText supports serial refinement, parallel exploration, and Memetic Retrieval, which adds evolutionary selection, local refinement, and tool memory to avoid redundant search. On StableToolBench (16,464 APIs), Memetic FitText reaches an 84.3% pooled pass rate, improving +26.7 points over static retrieval, +22.2 over Single-Pass, +23.2 over Re-Invoke, and +27.5 over Xu-style root refinement. It leads on every evaluated current model, with gains growing alongside model capability, and produces the largest improvements on ambiguous multi-tool tasks where dynamic re-retrieval restores correct candidates after early mistakes. At 40-way concurrency, parallel population execution keeps batched wall-clock at 1.01x Single-Pass despite the added search work.

Figures

Figures reproduced from arXiv: 2605.02411 by Chenchen Ye, Han Zhang, Kyle Zheng, Renliang Sun, Wei Wang.

Figure 1
Figure 1. Figure 1: Comparison between Previous Query-based Retrieval and Dynamic Retrieval. The view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of our proposed dynamic tool retrieval framework. The figure com view at source ↗
Figure 2
Figure 2. Figure 2: The boundary cases of our design space. Each strategy generates [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of Memetic Retrieval. The framework progresses through five phases: view at source ↗
Figure 3
Figure 3. Figure 3: Memetic Retrieval, the apex of the design space. Five phases: [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of query analysis model on retrieval performance. Using GPT-4.1 as the view at source ↗
Figure 5
Figure 5. Figure 5: Effect of refinement turns on retrieval performance. The largest improvement view at source ↗

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 28 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J

    Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXi...

    Pith/arXiv arXiv 2025

  3. [3]

    Task-aware retrieval with instructions

    Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. arXiv preprint arXiv:2211.09260, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Advancing tool-augmented large language models: Integrating insights from errors in inference trees

    Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang. Advancing tool-augmented large language models: Integrating insights from errors in inference trees. arXiv preprint arXiv:2406.07115, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Re-invoke: Tool invocation rewriting for zero-shot tool retrieval

    Yanfei Chen, Jinsung Yoon, Devendra Sachan, Qingze Wang, Vincent Cohen-Addad, Mohammadhossein Bateni, Chen-Yu Lee, and Tomas Pfister. Re-invoke: Tool invocation rewriting for zero-shot tool retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 4705--4726, 2024

    2024

  6. [6]

    Gonzalez

    Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, Nicholas Thumiger, Aditya Desai, Ion Stoica, Ana Klimovic, Graham Neubig, and Joseph E. Gonzalez. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    See: Strategic exploration and exploitation for cohesive in-context prompt optimization

    Wendi Cui, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley Malin, Sricharan Kumar, and Jiaxin Zhang. See: Strategic exploration and exploitation for cohesive in-context prompt optimization. arXiv preprint arXiv:2402.11347, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Anytool: Self-reflective, hierarchical agents for large-scale api calls

    Yu Du, Fangyun Wei, and Hongyang Zhang. Anytool: Self-reflective, hierarchical agents for large-scale api calls. arXiv preprint arXiv:2402.04253, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Mcp-zero: Proactive toolchain construction for llm agents from scratch

    Xiang Fei, Xiawu Zheng, and Hao Feng. Mcp-zero: Proactive toolchain construction for llm agents from scratch. arXiv preprint arXiv:2506.01056, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Precise zero-shot dense retrieval without relevance labels

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels. arXiv preprint arXiv:2212.10496, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    Simcse: Simple contrastive learning of sentence embeddings

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pp.\ 6894--6910. Association for Computational Linguistics (ACL), 2021

    2021

  12. [12]

    Evoprompt: Connecting llms with evolutionary algorithms yields powerful prompt optimizers

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Evoprompt: Connecting llms with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models

    Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models. In ACL (Findings), 2024

    2024

  14. [14]

    Understanding the planning of llm agents: A survey

    Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of llm agents: A survey. arXiv preprint arXiv:2402.02716, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Metatool benchmark for large language models: Deciding whether to use tools and which to use

    Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Bruce Croft

    Victor Lavrenko and W. Bruce Croft. Relevance based language models. Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.\ 120--127, 2001

    2001

  17. [17]

    Corpus-steered query expansion with large language models

    Yibin Lei, Yu Cao, Tianyi Zhou, Tao Shen, and Andrew Yates. Corpus-steered query expansion with large language models. arXiv preprint arXiv:2402.18031, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33: 0 9459--9474, 2020

    2020

  19. [19]

    Exploring solution divergence and its effect on large language model problem solving

    Hang Li, Kaiqi Yang, Yucheng Chu, Hui Liu, and Jiliang Tang. Exploring solution divergence and its effect on large language model problem solving. arXiv preprint arXiv:2509.22480, 2025 a

    arXiv 2025

  20. [20]

    A survey of automatic prompt engineering: An optimization perspective

    Wenwu Li, Xiangfeng Wang, Wenhao Li, and Bo Jin. A survey of automatic prompt engineering: An optimization perspective. arXiv preprint arXiv:2502.11560, 2025 b

    Pith/arXiv arXiv 2025

  21. [21]

    Deepagent: A general reasoning agent with scalable toolsets

    Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Guanting Dong, Jiajie Jin, Yinuo Wang, Hao Wang, Yutao Zhu, Ji-Rong Wen, Yuan Lu, and Zhicheng Dou. Deepagent: A general reasoning agent with scalable toolsets. arXiv preprint arXiv:2510.21618, 2025 c

    arXiv 2025

  22. [22]

    Dmqr-rag: Diverse multi-query rewriting for rag

    Zhicong Li, Jiahao Wang, Zhishu Jiang, Hangyu Mao, Zhongxia Chen, Jiazhen Du, Yuanxing Zhang, Fuzheng Zhang, Di Zhang, and Yong Liu. Dmqr-rag: Diverse multi-query rewriting for rag. arXiv preprint arXiv:2411.13154, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Query rewriting via large language models

    Jie Liu and Barzan Mozafari. Query rewriting via large language models. arXiv preprint arXiv:2403.09060, 2024

    arXiv 2024

  24. [24]

    Tool-planner: Task planning with clusters across multiple tools

    Yanming Liu, Xinyue Peng, Jiannan Cao, Shi Bo, Yuwei Zhang, Xuhong Zhang, Sheng Cheng, Xun Wang, Jianwei Yin, and Tianyu Du. Tool-planner: Task planning with clusters across multiple tools. arXiv preprint arXiv:2406.03807, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Large language models: A survey

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Efficient and scalable estimation of tool representations in vector space

    Suhong Moon, Siddharth Jha, Lutfi Eren Erdogan, Sehoon Kim, Woosang Lim, Kurt Keutzer, and Amir Gholami. Efficient and scalable estimation of tool representations in vector space. arXiv preprint arXiv:2409.02141, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    Pith/arXiv arXiv 2021

  29. [29]

    Alexander Novikov, Ng \^a n V \ u , Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. Alphaevolve: A coding agent for scientific an...

    Pith/arXiv arXiv 2025

  30. [30]

    Art: Automatic multi-step reasoning and tool-use for large language models

    Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    Gorilla: Large language model connected with massive apis

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37: 0 126544--126565, 2024

    2024

  32. [32]

    Tool learning with foundation models

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang, Chaojun Xiao, et al. Tool learning with foundation models. ACM Computing Surveys, 57 0 (4): 0 1--40, 2024 a

    2024

  33. [33]

    Toolllm: Facilitating large language models to master 16000+ real-world apis

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, 2024 b

    2024

  34. [34]

    Colt: Towards completeness-oriented tool retrieval for large language models

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Colt: Towards completeness-oriented tool retrieval for large language models. arXiv e-prints, pp.\ arXiv--2405, 2024 a

    2024

  35. [35]

    Towards completeness-oriented tool retrieval for large language models

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Towards completeness-oriented tool retrieval for large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp.\ 1930--1940, 2024 b

    1930

  36. [36]

    Tool learning with large language models: A survey

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19 0 (8): 0 198343, 2025

    2025

  37. [37]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3 0 (4): 0 333--389, 2009

    2009

  38. [38]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: 0 68539--68551, 2023

    2023

  39. [39]

    Tooldreamer: Instilling llm reasoning into tool retrievers

    Saptarshi Sengupta, Zhengyu Zhou, Jun Araki, Xingbo Wang, Bingqing Wang, Suhang Wang, and Zhe Feng. Tooldreamer: Instilling llm reasoning into tool retrievers. arXiv preprint arXiv:2510.19791, 2025

    arXiv 2025

  40. [40]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36: 0 38154--38180, 2023

    2023

  41. [41]

    Retrieval models aren't tool-savvy: Benchmarking tool retrieval for large language models

    Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, and Zhaochun Ren. Retrieval models aren't tool-savvy: Benchmarking tool retrieval for large language models. arXiv preprint arXiv:2503.01763, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    The illusion of diminishing returns: Measuring long horizon execution in llms

    Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping. The illusion of diminishing returns: Measuring long horizon execution in llms. arXiv preprint arXiv:2509.09677, 2026

    arXiv 2026

  43. [43]

    Large language model reasoning failures

    Peiyang Song, Pengrui Han, and Noah Goodman. Large language model reasoning failures. arXiv preprint, 2026

    2026

  44. [44]

    Restgpt: Connecting large language models with real-world restful apis

    Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, et al. Restgpt: Connecting large language models with real-world restful apis. arXiv preprint arXiv:2306.06624, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    A statistical interpretation of term specificity and its application in retrieval

    Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28 0 (1): 0 11--21, 1972

    1972

  46. [46]

    Dynamic cheatsheet: Test-time learning with adaptive memory

    Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test-time learning with adaptive memory. arXiv preprint arXiv:2504.07952, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Query2doc: Query expansion with large language models

    Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    Toolgen: Unified tool retrieval and calling via generation

    Renxi Wang, Xudong Han, Lei Ji, Shu Wang, Timothy Baldwin, and Haonan Li. Toolgen: Unified tool retrieval and calling via generation. arXiv preprint arXiv:2410.03439, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Executable code actions elicit better llm agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning, 2024

    2024

  50. [50]

    Enhancing tool retrieval with iterative feedback from large language models

    Qiancheng Xu, Yongqi Li, Heming Xia, and Wenjie Li. Enhancing tool retrieval with iterative feedback from large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 9609--9619, 2024

    2024

  51. [51]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022

    2022

  53. [53]

    Toolrerank: Adaptive and hierarchy-aware reranking for tool retrieval

    Yuanhang Zheng, Peng Li, Wei Liu, Yang Liu, Jian Luan, and Bin Wang. Toolrerank: Adaptive and hierarchy-aware reranking for tool retrieval. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp.\ 16263--16273, 2024

    2024

  54. [54]

    Rossi, Somdeb Sarkhel, and Chao Zhang

    Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A. Rossi, Somdeb Sarkhel, and Chao Zhang. Toolchain*: Efficient action space navigation in large language models with a* search. arXiv preprint arXiv:2310.13227, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  56. [56]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  57. [57]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...