Pith.

Computation and Language · arXiv:2609.00549 v1 · submitted 1 Sep 2026 · 36 references

Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents

Seonghyeon Cho, Chanjun Park

The paper says

“Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps.”

more of the abstract

Aggregate metrics often compare retrieved versus non-retrieved tasks, introducing severe selection bias and failing to isolate the true effect of skill use. To measure this actual-use capability-which we formalize as Skill Following (SF)-we introduce the Retrieval-Invoked Actual-Use Effect (RAE). RAE computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned exclusively on tasks where the agent actively retrieved a skill. Evaluating 17 LLMs across coding and mathematical domains, we uncover a stark evaluation paradox: models frequently show positive aggregate retrieval lift but negative RAE. On MBPP+, multiple models that appear to benefit system-wide actually harm their own performance on the exact tasks where retrieval occurred. These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 9 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.

The paper convincingly shows that aggregate retrieval lift can mislead, and RAE is a useful corrective, but the headline reversal cells rely on very small non-retrieved subsets, which weakens the strength of the central paradox claim. This line is wrong

1 major objection · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 36 references say.

The paper says, at §5.1, Table 3, Appendix A

The review did not quote this passage.

Pith says: Objection 1 of 1 · major

The central claim of pervasive sign disagreement relies on comparing RAE against aggregate retrieval lift, yet the aggregate lift values for several headline reversal cells are computed over extremely small non-retrieved subsets (n_skip). For example, DeepSeek-V3.2 on MBPP+ seed 42 has n_skip = 3 (Appendix I, Table 19), and Gemini-2.5-Flash-lite has n_skip ≈ 16.

more of objection 1

The paper reports no confidence intervals or bootstrap variability for aggregate retrieval lift, and does not impose a minimum n_skip threshold for sign-disagreement counting. As a result, it is unclear whether the '+31.6 pp' and '+6.2 pp' values are statistically distinguishable from zero, and whether the 5/13 sign-disagreement count reflects a systematic paradox or noise from tiny denominators. The authors should (a) report n_skip for every cell used in sign-disagreement counts, (b) compute and report 95% confidence intervals for aggregate retrieval lift (e.g., via bootstrap or exact binomial methods), and (c) show how the sign-disagreement count changes when cells with n_skip below a threshold (e.g., 10 or 15) are excluded. Without this, the paper's strongest claim is insufficiently supported.

4 minor objections

The paper says, at Table 3 and Appendix B

Pith says: minor objection 1 of 4

Several cells report aggregate lift or RAE as '+0.0' or '-0.0'. This suggests rounding; please report the precise values or indicate that the metric is exactly zero (e.g., when b = c in RAE).

The paper says, at Appendix A, Equation (15)

Pith says: minor objection 2 of 4

Aggregate retrieval lift is defined as a difference of proportions. For cells with very small N_skip or N_ret, the denominator should be clearly stated. Consider reporting the raw numbers (correct_ret, N_ret, correct_skip, N_skip) in a supplementary table for all reportable cells.

show 2 more

The paper says, at §5.5, Table 12

Pith says: minor objection 3 of 4

The harmful-transition annotation uses GPT-5.5, which is not a publicly documented model at the time of review. The human audit (Appendix G.1) shows moderate agreement (α=0.377). The authors should explicitly note that the annotation is a post-hoc diagnostic and that the fine-grained label taxonomy is interpretive. The current language is sufficiently cautious; no action required.

The paper says, at General

Pith says: minor objection 4 of 4

The paper repeatedly uses 'pervasive paradox' (Abstract, §1). Given the concern about small-denominator aggregate lift, consider tempering this language to 'frequent sign disagreement' until the robustness of the aggregate side is established.

What would settle it

Recompute every reportable cell in Tables 3, 6, and 7 with (i) 20,000-resample bootstrap confidence intervals for both RAE and aggregate retrieval lift, and (ii) a reporting filter requiring n_skip >= 10, plus a sensitivity table for n_skip >= 5/10/20. Then re-count sign disagreements using only cells where both metrics are defined, both denominators meet the filter, and both 95% intervals exclude zero. If DeepSeek-V3.2 s42 or Gemini-2.5-Flash-lite s42 aggregate-lift intervals include zero, or if the sign-disagreement fraction drops materially below 5/13, the 'pervasive paradox' conclusion should be weakened.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 9 Sep 2026

  1. Reader holds up confidence high
  2. Skeptic disagrees: says it holds up only with conditions 1 min 1 s · 22,418 tokens read, 6,056 written Headline sign-disagreement cells rely on aggregate retrieval lift computed over tiny non-retrieved subsets, with no confidence intervals reported for either side.
  3. Editor wrote the verdict above 11 s · 23,092 tokens read, 1,131 written
  4. Referee major revision: 1 major objection, 4 minor 19 s · 22,610 tokens read, 2,039 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The core idea here is solid and overdue. RAE—conditioning the same-task comparison on the actual retrieval-invoked subset—is a clear methodological improvement over aggregate retrieval lift. The paper does a thorough job across 17 models, multiple benchmarks, partitioned seeds, and diagnostic controls. The empirical finding that positive aggregate lift and negative RAE co-occur in several models is genuinely interesting and should make people think twice about how they report tool-use benefits.

That said, the stress-test concern about the small non-retrieved denominator in the headline reversal cells is real. For DeepSeek-V3.2 on MBPP+ seed 42, aggregate lift is computed over only 3 non-retrieved tasks; moving one task flips the lift by 33 pp. The paper does not report confidence intervals for aggregate lift, so we have no idea whether +31.6 pp is statistically distinguishable from zero or from the negative RAE. This doesn't invalidate the RAE construction, but it does weaken the 'pervasive paradox' narrative—several of the sign-disagreement cells may be noise rather than a systematic effect. The authors do show that the count of sign-disagreement cells is stable under alternative ncond thresholds (Appendix A, Table 5), but that doesn't address the n_skip issue. They should add bootstrap intervals for aggregate lift and explicitly discuss the instability arising from near-empty denominator sets.

The paper acknowledges its own limitations clearly (Section 6, Limitations), which is good. The fixed BM25 retriever and fixed skill libraries also mean the results are not immediately generalizable. But as a methodological critique, the paper stands up. I would accept it with a request to address the aggregate lift uncertainty and to tone down the 'pervasive' language where it rests on fragile cells.

For peer review: yes, this deserves a serious referee. The contribution is real, the experiments are carefully done, and the main weakness is fixable. I would bring it to a reading group.

The strongest objection

The central paradox claim compares RAE against aggregate retrieval lift, but the aggregate lift in the two headline reversal cells is computed over very small non-retrieved task subsets. On MBPP+ seed 42, DeepSeek-V3.2 has 96.2% retrieval coverage (ncond=77 of 80, Appendix I, Table 19), so the non-retrieved denominator is only 3 tasks; moving one task flips aggregate lift by roughly 33 pp. Gemini-2.5-Flash-lite has roughly 80% coverage, leaving about 16 non-retrieved tasks. The reporting filter in Appendix A requires only that both denominators be nonzero; there is no minimum n_skip and no confidence interval for aggregate retrieval lift. Consequently, the reported '+31.6 pp' and '+6.2 pp' may be statistically indistinguishable from zero, and the 5/13 sign-disagreement count may partly reflect noise in a nearly empty denominator rather than a systematic evaluation paradox. The paper reports exact McNemar tests and bootstrap intervals for RAE, but not for the aggregate lift it is contrasted against. This asymmetry is the weakest load-bearing point: the RAE construction itself is sound, but the headline 'pervasive paradox' depends on the reliability of the comparison metric on the other side of the sign disagreement.

The reader's weakest assumption concerns the paired skill-disabled execution as a counterfactual control. The paper explicitly acknowledges this limitation in Section 6 ('RAE is not an unbiased causal effect over a pre-treatment task population') and bounds it with diagnostic controls. My concern is different and, I think, more directly load-bearing for the headline claim: the aggregate retrieval lift used in the sign-disagreement comparisons has no reported uncertainty, and in the two signature reversal cells its non-retrieved denominator is extremely small. The paper's own Appendix I shows DeepSeek-V3.2 s42 coverage of 96.2%, meaning the aggregate lift contrasts 77 retrieved tasks against 3 non-retrieved tasks. A single extra correct or incorrect answer in that 3-task group changes the lift by about a third of the reported magnitude. The same issue applies, less severely, to Gemini-2.5-Flash-lite. The reporting filter in Appendix A only checks that both denominators are nonzero, which is a weak condition for a headline quantitative paradox. The paper is otherwise careful: it gives exact McNemar p-values and bootstrap CIs for RAE, reports pooled counts, and includes retrieval-depth sensitivity. But none of that addresses the instability of the comparison metric. Since the abstract and introduction make strong claims that 'multiple models that appear to benefit system-wide actually harm their own performance,' and those claims hinge on cells like DeepSeek-V3.2 +31.6/-9.1, the lack of uncertainty quantification for the aggregate-lift side is a concrete, testable gap. I am not claiming the paradox is false; I am claiming the current evidence does not establish it as robust. A CONDITIONAL verdict asking for the aggregate-lift confidence intervals and n_skip sensitivity would settle whether the paradox survives. This is not an attack on RAE's validity; it is a request to make the contrast it is measured against as well-calibrated as the RAE estimates themselves.

The referee's report

If the empirical claims hold, RAE provides a principled correction to a blind spot in agent evaluation. The paired-design methodology is technically sound and addresses a genuine confound in retrieved-vs-skipped comparisons. The finding that several models show negative RAE despite positive aggregate lift—across multiple benchmarks and seeds—would be practically important for auditing tool-augmented systems. The diagnostic analyses (skill-content controls, adherence annotation) add mechanistic depth. However, the headline paradox claim depends on the reliability of the aggregate retrieval lift estimates it is contrasted against, and this is where the paper's reporting is incomplete.

Pith's reading of the paper (a machine's summary, not the authors')

The paper claims that standard evaluations of skill-using LLM agents answer the wrong question: they compare the tasks where a model chose to retrieve against the tasks where it did not, and those two sets differ in difficulty, so the comparison cannot say whether the retrieved skill actually helped. It formalizes Skill Following as the whole chain from invoking retrieval to integrating returned content into a correct answer, and proposes the Retrieval-Invoked Actual-Use Effect (RAE): the pass/fail difference between matched skill-enabled and skill-disabled runs of the same task, restricted to tasks where retrieval actually returned content. On 17 LLMs across MBPP+, HumanEval+, and Math500, the two views frequently disagree in sign — Gemini-2.5-Flash-lite and DeepSeek-V3.2, for example, show positive aggregate retrieval lift but negative RAE on MBPP+, and Llama-3.3-70B shows +14.2 pp aggregate lift with a $-39.4$ pp RAE on Math500. If the claim holds, aggregate retrieval lift is not evidence of tool-use proficiency, and evaluations should report same-task paired effects.

What carries the argument. The load-bearing object is the paired execution design behind RAE. Every task is run twice: a skill-enabled run has a search_skills tool and a fixed procedural skill library in its environment, while a skill-disabled run uses an identical prompt with the tool definition removed, sharing the same task text, generation seed, and decoding settings. Each task's two outcomes place it in one of four transition cells — concordant pass, helpful flip, harmful flip, concordant fail — and RAE is $(b-c)/n_{\mathrm{cond}}$ on the retrieval-invoked subset, OAE is $(b-c)/n_{\mathrm{pairs}}$ on all tasks, and the identity $\mathrm{OAE} = (n_{\mathrm{cond}}/n_{\mathrm{pairs}})\mathrm{RAE} + (n_{\mathrm{skip}}/n_{\mathrm{pairs}})\Delta_{\mathrm{skip}}$ connects them. The skill-disabled selection gap $\Delta_{\mathrm{select}}$ then measures the baseline difficulty difference between tasks where retrieval fired and tasks where it did not, which is exactly the confound that aggregate retrieval lift overlooks.

Core claim. On the paper's own terms, the central discovery is an evaluation paradox: an agent can look more capable with retrieval enabled while the retrieval actually damages the very tasks it is applied to. RAE is computed as the difference between helpful flips (skill-disabled run fails, skill-enabled run passes) and harmful flips (skill-disabled run passes, skill-enabled run fails), divided by the number of retrieval-invoked tasks, with OAE being the same paired quantity over the whole benchmark and the decomposition $\mathrm{OAE} = \frac{n_{\mathrm{cond}}}{n_{\mathrm{pairs}}}\mathrm{RAE} + \frac{n_{\mathrm{skip}}}{n_{\mathrm{pairs}}}\Delta_{\mathrm{skip}}$ showing how the two relate. Applying this protocol to 17 LLMs with fixed Markdown skill libraries and BM25 retrieval, the paper reports sign disagreement between aggregate retrieval lift and RAE in 5 of 13 reportable cells on MBPP+ seed 42 and in 3 of 13 on HumanEval+, with reversal patterns also on Math500. The paper traces the disagreement to retrieval-induced selection bias, quantified by the skill-disabled selection gap $\Delta_{\mathrm{select}}$, and to mechanistic failures — most harmful transitions are cases where the model retrieved a skill and then ignored it, solved independently, or leaked tool traces into the output. Skill-content controls keep RAE negative for the two anchor models even when retrieved text is random, filler, or corrupted, which the paper reads as evidence that the retrieval-to-answer chain itself, not the quality of the skill text, is the bottleneck.

The premise it rests on. The paired comparison assumes that the skill-disabled run is a clean counterfactual for the same task without retrieval — that removing the tool definition from the prompt changes nothing else about how the model behaves, even though the absence of the tool could subtly alter task interpretation or confidence, and even though the tasks being measured are exactly the ones the model itself chose to retrieve on.

If the paper is right (Pith's inferences, not the authors')

Benchmark reports should carry same-task paired metrics like RAE alongside aggregate lift, since the two frequently disagree in sign and aggregate lift alone can endorse a deployment that actively hurts the tasks using the tool.

High retrieval coverage should not be marketed as tool proficiency: the paper shows coverage and RAE decouple, with several high-coverage models showing negative actual-use effects.

Aggregate gains driven by selective retrieval on easy tasks will not survive regimes where retrieval is mandatory or difficulty-matched, so system comparisons based on aggregate lift need revalidation on paired outcomes.

Because RAE stays negative under random, filler, and corrupted skill content for the anchor models, improving the skill library's content alone will not rescue the pipeline; the integration and adherence stage is where the loss is incurred.

Editorial: the pair-and-condition protocol transfers to other tool types — code execution, web search, memory — wherever a parallel no-tool run is feasible; nothing in RAE requires skills to be Markdown documents.

Editorial: splitting RAE into an oracle-retrieval variant (retrieval forced on every task) and the free-choice variant would separate 'knowing when to fetch' from 'knowing how to use,' which the paper itself lists as the ablation needed for finer causal decomposition.

Editorial: given the dominant ignored/independent failure mode, a testable intervention is to require the model to explicitly transform or cite the retrieved skill before answering; if negative RAE softens, the bottleneck is attention to injected context rather than content relevance.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps. Aggregate metrics often compare retrieved versus non-retrieved tasks, introducing severe selection bias and failing to isolate the true effect of skill use. To measure this actual-use capability-which we formalize as Skill Following (SF)-we introduce the Retrieval-Invoked Actual-Use Effect (RAE). RAE computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned exclusively on tasks where the agent actively retrieved a skill. Evaluating 17 LLMs across coding and mathematical domains, we uncover a stark evaluation paradox: models frequently show positive aggregate retrieval lift but negative RAE. On MBPP+, multiple models that appear to benefit system-wide actually harm their own performance on the exact tasks where retrieval occurred. These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.81

“These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.13 · human participants 0.04 · controlled comparison 0.92 · randomized assignment 0.03 · states quantitative result 0.57 · compares with prior results 0.81 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps. background 0.86 0.06 0.04 asserted 1.00
Aggregate metrics often compare retrieved versus non-retrieved tasks, introducing severe selection bias and failing to isolate the true effect of skill use. gap 0.68 0.09 0.04 asserted 1.00
To measure this actual-use capability-which we formalize as Skill Following (SF)-we introduce the Retrieval-Invoked Actual-Use Effect (RAE). contribution 0.95 0.89 0.07 asserted 0.99
RAE computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned exclusively on tasks where the agent actively retrieved a skill. method 0.97 0.83 0.06 asserted 1.00
Evaluating 17 LLMs across coding and mathematical domains, we uncover a stark evaluation paradox: models frequently show positive aggregate retrieval lift but negative RAE. result 1.00 0.96 0.60 asserted 1.00
On MBPP+, multiple models that appear to benefit system-wide actually harm their own performance on the exact tasks where retrieval occurred. result 1.00 0.93 0.16 asserted 1.00
These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms. interpretation 1.00 0.85 0.84 asserted 1.00

What kind of paper the classifier reports this is: method or tool 0.68 · evidence from observation or experiment 0.96

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,586 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. Aggregate retrieval lift and RAE on MBPP+. The x-axis compares skill-enabled accuracy be￾tween retrieval-returned and retrieval-skipped task sets, whereas the y-axis compares paired skill-enabled and skill-disabled outcomes on the same retrieval-returned tasks. Shaded off-diagonal quadrants indicate sign dis￾agreement; labels (a)–(c) mark the cases discussed in the text. evaluation protocols typically answer one of two broad, system-level questions: whether providing an agent with access to a skill environment im￾proves average performance across an entire bench￾mark, or whether…
Figure 2 of the paper
Figure 2. Autonomous retrieval induces severe task
Figure 4 of the paper
Figure 4. Retrieval coverage versus RAE on MBPP+ seed 42. The horizontal line marks zero RAE. Multiple high-retrieval models exhibit negative RAE, proving that successful retrieval invocation (exposure) does not reliably translate to successful SF (utilization). several models with high retrieval rates neverthe￾less yield negative actual-use effects. To unpack this discrepancy, the corresponding stage-level di￾agnostic definitions and values are detailed in Ap￾pendix E. Retrieval is not Following If mere exposure to external skills were sufficient, models with higher retrieval coverage would…
Figure 5 of the paper
Figure 5. Adherence-conditioned diagnosis of retrieval

Showing 4 of the paper's 5 figures. Figures are the authors', reproduced from the paper.

References (36)
  1. [1] SkillsBench: Benchmarking how well agent skills work across diverse tasks , author=. arXiv preprint arXiv:2602.12670 , year= on PithPith's reviewarXiv
  2. [2] SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? , author=. arXiv preprint arXiv:2603.15401 , year= on PithPith's reviewarXiv
  3. [3] SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks , author=. arXiv preprint arXiv:2604.20087 , year= on PithPith's reviewarXiv
  4. [4] Program Synthesis with Large Language Models , author =. arXiv preprint arXiv:2108.07732 , year = on PithPith's reviewarXiv
  5. [5] Measuring Mathematical Problem Solving With the MATH Dataset , author=. NeurIPS , year= on Pith
  6. [6] Let's verify step by step , author=. International Conference on Learning Representations , volume= on Pith
  7. [7] Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year= on PithPith's reviewarXiv
  8. [8] Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , booktitle =. Is Your Code Generated by Chat. 2023 , url = on Pith
  9. [9] Agent kb: Leveraging cross-domain experience for agentic problem solving , author=. arXiv preprint arXiv:2507.06229 , year= on PitharXiv
  10. [10] Voyager: An open-ended embodied agent with large language models , author=. arXiv preprint arXiv:2305.16291 , year= on PithPith's reviewarXiv
  11. [11] Expel: Llm agents are experiential learners , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume= on Pith
  12. [12] Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems , volume= on Pith
  13. [13] Critic: Large language models can self-correct with tool-interactive critiquing , author=. International Conference on Learning Representations , volume= on Pith
  14. [14] The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models , author=. Forty-second International Conference on Machine Learning , year= on Pith
  15. [15] Instruction-following evaluation for large language models , author=. arXiv preprint arXiv:2311.07911 , year= on PithPith's reviewarXiv
  16. [16] Ragas: Automated evaluation of retrieval augmented generation , author=. Proceedings of the 18th conference of the european chapter of the association for computational linguistics: system demonstrations , pages= on Pith
  17. [17] The probabilistic relevance framework: BM25 and beyond , author=. 2009 , publisher= on Pith
  18. [18] Agentbench: Evaluating llms as agents , author=. International Conference on Learning Representations , volume= on Pith
  19. [19] Agentboard: An analytical evaluation board of multi-turn llm agents , author=. Advances in neural information processing systems , volume= on Pith
  20. [20] Webarena: A realistic web environment for building autonomous agents , author=. International Conference on Learning Representations , volume= on Pith
show 16 more
  1. [21] Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments , author=. Advances in Neural Information Processing Systems , volume= on Pith
  2. [22] Theagentcompany: benchmarking llm agents on consequential real world tasks , author=. Advances in Neural Information Processing Systems , volume= on Pith
  3. [23] -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. arXiv preprint arXiv:2406.12045 , year= on PithPith's reviewarXiv
  4. [24] Api-bank: A comprehensive benchmark for tool-augmented llms , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages= on Pith
  5. [25] Toolformer: Language Models Can Teach Themselves to Use Tools , author=. 2023 , eprint= on Pith
  6. [26] AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning , author=. 2024 , eprint= on Pith
  7. [27] Small LLMs Are Weak Tool Learners: A Multi-LLM Agent , author=. 2024 , eprint= on Pith
  8. [28] Llm agents making agent tools , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
  9. [29] Beyond Under-Alignment: Atomic Preference Enhanced Factuality Tuning for Large Language Models , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages= on Pith
  10. [30] Learning to use tools via cooperative and interactive agents , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages= on Pith
  11. [31] Deepseek-v3. 2: Pushing the frontier of open large language models , author=. arXiv preprint arXiv:2512.02556 , year= on PithPith's reviewarXiv
  12. [32] Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year= on PithPith's reviewarXiv
  13. [33] The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year= on PithPith's reviewarXiv
  14. [34] Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year= on PithPith's reviewarXiv
  15. [35] Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year= on PithPith's reviewarXiv
  16. [36] Qwen Team , month =. Qwen2.5: A Party of Foundation Models , url = on Pith
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment