Computation and Language · arXiv:2610.21098 v1 · submitted 15 Oct 2026 · 103 references
ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets
The paper says
“The success score of a tool agent reflects both its model and the budgets of the harness that runs it. We present ToolBottleneckBench, a benchmark that measures tool-agent performance under controlled pressures and fixed harness budgets.”
more of the abstract
Synthetic enterprise tasks stay fixed while one pressure varies: the registered tool catalog, the visible tool menu, the certified workflow, or the encoding of returned records. An exact grader records the first unmet obligation of each failed episode and the harness limit, if any, that ended it. Across 21 model and harness configurations, the visible menu is the only one of the three pressures that act on the model with a large penalty, and this penalty tracks truncation at the completion cap. Raising the cap sixteenfold removes most of the penalty in the truncation-heavy configurations. Most failures end at a harness limit, and the failure record identifies the assigned pressure no more reliably than a count of tool calls. The repairs that help change which tools reach the model or who issues the calls, and only two repairs, neither improving model behavior, beat every configuration switch. We recommend reporting pressure effects with the harness budgets under which they were measured. We release the benchmark at https://github.com/zichuanxu/tool-bottleneck-bench.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A careful benchmark paper whose headline menu-truncation finding survives the low-cap sweep anomaly, though the 41 pp interaction is likely inflated by run-to-run noise in the 1,024-token arm. This line is wrong
2 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 103 references say.
The paper says, at Appendix E.1, Table 9
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The load-bearing sixteenfold-cap sweep has an incompletely explained low-cap-arm discrepancy. For all six stratum-B configurations the sweep's 1,024-token contrast is 5.2–11.8 pp more negative than the released panel estimate (Rel. column), and the paper's explanation—a higher truncation share in the sweep—is contradicted by Gemini 3.8 Flash (the paper itself concedes 'all but one' configuration fits) and is quantitatively inadequate for Qwen3.8 Flash.
more of objection 1
For Gemini 3.8 Flash, sweep truncation is 64.9% versus 67.4% in the panel, yet the sweep contrast is 10.7 pp more negative. For Qwen3.8 Flash, a +2.8 pp truncation increase is paired with a −9.7 pp penalty increase, a slope of roughly −3.5 pp per truncation point versus the ≈−0.8 pp per point relation across the panel in Figure 6; the crossover in Table 7 measured −14.9 pp at 28.8% truncation for this configuration, against the sweep's −26.4 pp at 30.6%, and showed no improvement at 4,096 tokens. Since the paper states the 'primary reading uses only the within-sweep interaction', the headline 41.0 pp [36.2, 45.5] inherits whatever moves the low-cap arm. To be clear, the qualitative conclusion is not in doubt: recomputing the pooled interaction from the per-configuration rows of Table 9 after dropping both anomalous configurations still gives approximately 40 pp, and the high-cap arm is well validated. What is missing is a validation of the low-cap arm itself: (i) a sensitivity analysis substituting panel or crossover low-cap estimates for the sweep's, (ii) a within-sweep stratification of the low-cap contrast by truncation status, and (iii) the reasoning-token composition of the sweep's truncated responses for the two anomalous configurations. The paper's own 17.5% repeat-disagreement figure at the low cap shows run-to-run variation is large enough that this analysis is necessary, and the Figure 6 'spread' defense is weak because that relation is fan-shaped.
This objection may describe the copy Pith gave the reviewer, not the paper.
The paper says, at Appendix G, Section 5.4
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The claim that the connector-first router is success-superior 'by construction' is stronger than the argument supports. ITT success is the product A·S of availability and selection; the retriever's model-independence fixes A across models, but selection S is model-dependent, so the inequality A_router·S_incumbent > A_flat·S_switch does not follow from the retriever's being frozen unless one also bounds S (for instance, S_incumbent > (A_flat/A_router)·S_switch ≈ 0.25·S_switch).
more of objection 2
The empirical 21/21 result is credible and reproducible, and the separate observation that the router's mean success exceeds flat retrieval for only 1 of 21 configurations on the intersection cases supports the intended reading, but the 'by construction' phrasing should be replaced by an explicit bound on selection rates or by a statement that the superiority is empirical. This matters because the fourth headline finding ('only two repairs, neither improving model behavior, beat every configuration switch on success') leans on this claim.
This review says the paper's mathematics is wrong. Pith did not check that against the paper's source.
6 minor objections
The paper says, at Section 3.1
Pith says: minor objection 1 of 6
There is a typo: 'producematched variants' should be 'produce matched variants'.
The paper says, at Abstract, Section 1
Pith says: minor objection 2 of 6
The sentence 'The repairs that help change which tools reach the model or who issues the calls, and only two repairs, neither improving model behavior, beat every configuration switch' is grammatically incomplete; the first clause needs a predicate (for example, 'activate'), or the two clauses should be separated.
show 4 more
The paper says, at Section 5.2, Figure 6
Pith says: minor objection 3 of 6
The reported r=−0.95 is strongly influenced by the eight zero-truncation configurations clustered at one end of the relation; reporting the correlation restricted to the thirteen truncating configurations, or a regression slope with its interval, would make the descriptive claim easier to calibrate, given that the paper already correctly notes this correlation cannot isolate the cap.
The paper says, at Table 4, Figure 7
Pith says: minor objection 4 of 6
Per-configuration effects are reported without intervals; given the extreme heterogeneity in W effects (−30.6 to +34.7 pp), readers cannot tell which per-configuration contrasts are distinguishable from zero, and adding intervals for at least the extreme configurations would help.
The paper says, at Title
Pith says: minor objection 5 of 6
The title's 'Fixed Harness Budgets' sits in tension with the cap sweep, which deliberately varies one of the four budgets; the text is explicit that the sweep is a separate mechanism study, so consider qualifying the title (for example, 'Frozen-Budget Benchmark' or similar).
The paper says, at Appendix E.1
Pith says: minor objection 6 of 6
The 5 pp interaction margin and the stratum rules are described as 'pre-specified', but the paper does not describe the pre-specification mechanism or its timing; since the strata are functions of panel results, a sentence stating when and how the margin and strata were fixed would resolve this.
What would settle it
Rerun the 1,024-token arm of the sixteenfold-cap sweep (Appendix E.1) on the six stratum-B configurations with at least four repeats, concurrent with a fresh 16,384-token arm, and check whether the pooled low-cap contrast reproduces -43.7 pp or regresses toward the released panel values (pooled about -35 pp for these six; e.g., Qwen3.8 Flash: panel -16.7, crossover -14.9, sweep -26.4). If it regresses, revise the reported interaction (41.0 pp) downward and widen its interval to include repeat-level variation; if it reproduces -43.7, the discrepancy from the panel is a stable feature and the interaction stands. As a cheaper analytic check, recompute the pooled interaction with Table 4's released routed V effects as the low-cap baseline; if the result still clears the 5 pp margin, the central conclusion does not depend on which low-cap measurement is used.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 16 Oct 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Two things to know. This is a carefully instrumented benchmark paper: fixed task families, one pressure varied at a time, an exact nine-obligation grader, and a terminal flag that records which harness limit ended each failed episode. The headline result—most of the visible-menu penalty is completion-cap truncation—holds up in qualitative form, but the headline 41 pp interaction is probably inflated by a noisy low-cap arm.
What is genuinely new: the failure record itself, the sixteenfold cap sweep, and the repair protocol that compares every repair against every configuration switch. The statistical practice is above the norm: cluster bootstraps over three resampling units, Holm correction, a pre-specified 5 pp margin, a manipulation check that truncation drops to at most 1.4% at the high cap, and principal-stratification bounds. The ToolSandbox cross-check is honest external evidence that the menu penalty is environment-specific, and the paper does not try to hide that. The release includes the generator, grader, scored rows, and analysis code; that is real reproducible work, even though the text gives no commit hash to verify the exact released state.
Soft spots, in proportion. The stress-test's low-cap anomaly is real. For all six truncation-heavy configurations, the sweep's low-cap contrast is 5.2–11.8 pp more negative than the same configuration's released V effect. Gemini 3.8 Flash truncates less in the sweep (64.9% vs 67.4%) yet shows a 10.7 pp more negative contrast, and Qwen3.8 Flash pairs 2.8 pp more truncation with 9.7 pp more penalty. The paper's note that a higher truncation share explains the larger penalty does not cover those two cases. With 17.5% condition-level disagreement between the two low-cap repeats, run-to-run noise is the most plausible explanation. But substituting the released panel effects as the low-cap baseline leaves roughly a 32 pp pooled interaction, and every configuration's own interaction still clears the 5 pp margin. So the qualitative conclusion survives; only the exact magnitude is shaky.
The W and D contrasts are acknowledged composites—W also grows the required sources array, and compact is byte-larger than raw—so V is the clean comparison, which is fortunate since V is the headline pressure. The negative prompt-repair claims are weaker than the surrounding tone: the repairs were tested on cells with no detectable loss, so "does not activate" is close to a null result on a null cell. The failure-localization audit conditions on packets with exactly one failure and recovers almost no returned-state cases; that part is descriptive, not load-bearing.
Who gets value: anyone building or comparing tool-agent harnesses, and anyone writing leaderboards that should report budgets. I would send this to a serious referee and ask for a robustness pass on the low-cap sweep before publication.
The strongest objection
The central claim's load-bearing measurement is the sixteenfold-cap sweep (Appendix E.1): pooled over the six truncation-heavy configurations, the routed-lane visible-menu contrast is -43.7 pp at 1,024 tokens and -2.7 pp at 16,384, an interaction of 41.0 pp that clears the pre-specified 5 pp margin. The high-cap arm is well validated by the manipulation check (truncation falls to at most 1.4%). The least secure condition is the low-cap arm, which must reliably measure the frozen-budget menu penalty.
The low-cap arm is anomalous. For all six stratum-B configurations, its contrast is 5.2-11.8 pp more negative than the released panel V effect for the same estimand (Table 9, Rel. column, versus Table 4). The crossover provides a second independent measurement: for Qwen3.8 Flash, the panel gives -16.7 pp at 27.8% truncation, the crossover -14.9 pp at 28.8%, and the sweep -26.4 pp at 30.6%. At essentially equal truncation shares, the sweep is about 10-11.5 pp more negative. The paper's explanation, that the larger penalty comes with a higher truncation share at the same cap (Table 9 note), covers five of six configurations but is quantitatively inadequate: Gemini 3.8 Flash truncates less in the sweep (64.9% vs 67.4%) yet shows a 10.7 pp more negative contrast, and Qwen3.8 Flash pairs 2.8 pp more truncation with 9.7 pp more penalty, far steeper than the within-configuration slope the crossover implies. With only 2 repeats and 17.5% condition-level disagreement at the low cap (vs 8.1% at the high cap), run-to-run noise in the low-cap arm is the most plausible residual explanation, and the headline interaction inherits it.
Calibration: substituting the released panel effects as the low-cap baseline leaves roughly a 32 pp pooled interaction, and every configuration's own interaction clears the 5 pp margin under either baseline.
ToolBottleneckBench is careful where it matters: the routed-lane V contrast is the cleanest of the four pressures, the sweep arms are concurrent, the manipulation check is outcome-based rather than parameter-based, and the directly observed no-truncation principal stratum shows only -1.5 pp. Converging evidence - the r=-0.95 effect-truncation correlation, the never-truncating controls staying within 2.8 pp of zero, the crossover (penalty persists where truncation persists), the reasoning-only forensics, and the sweep - supports the central claim that in truncation-heavy configurations most of the visible-menu penalty is completion-cap truncation. The reader's weakest-assumption items (W's answer-length confound, compact's byte-size confound, nominal-versus-effective caps) are acknowledged by the authors, and none challenges the routed-lane V measurement that carries the headline; the nominal-cap concern is largely answered by the manipulation check, which measures truncation rather than assuming parameter equivalence.
The soft spot is the sweep's low-cap arm: its contrasts run 5.2-11.8 pp more negative than the released panel effects for the identical estimand in all six configurations, the truncation-share explanation fails for Gemini 3.8 Flash outright and is quantitatively implausible for Qwen3.8 Flash, and for Qwen3.8 Flash the crossover's independent measurement agrees with the panel, not the sweep. The most plausible cause is run-to-run variation (17.5% low-cap repeat disagreement), which would inflate the headline interaction. The concern is calibrated: re-baselining with the released effects leaves about a 32 pp interaction, so the qualitative claim survives any plausible correction.
A smaller scoping gap: the truncation-tracking and sweep results are demonstrated on the routed-menu lane only, while Table 4's fixed-menu lane shows zero-truncation configurations with 11-17 pp fixed-menu penalties (GPT-5.6 Luna -16.7, GPT-5.6 Terra -11.1), so the practical recommendation to report truncation rates generalizes less cleanly to fully visible menus; the Limitations disclose the routed-lane scope but do not draw this implication.
None of this moves the reader's verdict. The CONDITIONAL judgment (release not yet verifiable from the text, underpowered negative prompt-repair tests, outcome-conditioned decoder audit) stands, and my concern adds a quantitative-precision caveat on the sweep headline without changing the conclusion. Hence UNCHANGED.
The referee's report
Conditional on the cap-sweep estimate, this is a substantial and unusually rigorous contribution to tool-agent evaluation. The controlled matched-variant design, the exact grader with terminal flags, cluster bootstraps over three resampling units with Holm correction, pre-specified interaction and equivalence margins, and the manipulation checks (high-cap truncation falls to at most 1.4%) are all to the field's benefit. The falsifiable embedded controls—eight never-truncating configurations with V effects within 2.8 pp of zero, and sweep control strata M and N moving only 2.8 and 1.0 pp—strengthen the causal reading. The reproducibility infrastructure is a genuine strength: deterministic re-analysis reproducing every typed field, certificates for eight configurations, 469 test cases, and fully released scored rows at a disclosed total cost of USD 768.38. If correct, the claim that a success leaderboard under a large menu partly ranks configurations by how they spend the completion cap is an important, actionable caution, and the observation that all 849 'selection'-labeled failures made no call is a valuable lesson about obligation-order grading.
Pith's reading of the paper (a machine's summary, not the authors')
ToolBottleneckBench asks when a failing tool-using agent is at fault and when its execution harness is. The paper keeps 36 synthetic enterprise task families fixed, varies one of four pressures at a time — the registered tool catalog, the visible tool menu, the certified workflow, and the encoding of returned records — and has an exact grader record the first unmet obligation of each failed episode together with the harness limit that ended it. Its central result is that of the three pressures that act on the model, only the visible menu carries a large penalty on verified task success, and most of that penalty is the 1,024-token completion cap: growing the menu from 8 to 128 tool schemas costs −12.3 percentage points overall, while in the six truncation-heavy configurations a sixteenfold cap shrinks the pooled penalty from −43.7 to −2.7 pp. If the paper is right, a success leaderboard under a large menu partly ranks configurations by how they spend the completion cap, and a benchmark report that omits harness budgets does not measure the model alone.
What carries the argument. The load-bearing device is a matched-variant construction under a frozen harness contract. A deterministic compiler rewrites each task family into variants that differ in exactly one assigned pressure — registered catalog (R), visible menu (V), certified workflow (W), or returned-state encoding (D) — while the request, facts, answer, evidence rule, and operational tools stay fixed, so a paired within-family difference in VTS is attributed to the varied pressure. An exact grader checks nine obligations in a fixed order (availability through completion integrity) and emits a failure record pairing the first unmet obligation with a terminal flag naming the harness limit that ended the episode: completion cap, episode budget, call limit, turn limit, infeasibility stop, wrong conclusion, or another contract break. The decisive experiment is the sixteenfold completion-cap sweep, which reruns the 8-versus-128 menu contrast at 1,024 and 16,384 tokens on six truncation-heavy configurations, with truncation verified to fall from 30.6–69.8% of 128-schema episodes to at most 1.4%. A separate activation test, using an intersection-union inference with wrong-layer controls, decides whether a repair helps specifically under its own pressure rather than rescuing failures generically.
Core claim. The paper claims that a tool agent's verified task success (VTS) is a joint property of the model and of the harness that runs it, and that its design can separate the two by construction. Growing the visible menu from 8 to 128 tool schemas changes VTS by −12.3 pp under a routed menu and −17.5 pp when the full menu stays visible, while the certified-workflow pressure shows no pooled effect (+0.8 pp) and returned-state encodings show small effects. The menu penalty tracks completion-cap truncation across configurations ($r=-0.95$), and the sixteenfold cap sweep isolates the cap as the mechanism: pooled over the six truncation-heavy configurations, the visible-menu contrast is −43.7 pp at 1,024 tokens and −2.7 pp at 16,384 tokens, an interaction of 41.0 pp that clears the pre-specified 5 pp margin. Of all exactly graded failures, 91.5% end at a harness limit, the failure record identifies the assigned pressure no more reliably than a count of tool calls, and only two repairs — the connector-first router and the public-handoff controller, neither of which improves model behavior — beat every configuration switch on success.
The premise it rests on. Each pressure effect is read off a paired difference between two task variants that are meant to differ only in that pressure, yet by the paper's own account the workflow variant also lengthens the required answer from one to three sources, the compact encoding is byte-larger than raw, and one nominal completion cap is not one effective budget across providers.
If the paper is right (Pith's inferences, not the authors')
A success leaderboard under a large menu ranks configurations partly by how they spend the completion cap, so benchmark reports should state harness budgets and each configuration's truncation rate.
A first-failure label such as 'tool selection' should be read together with the terminal flag: all 849 'selection' failures came from episodes that made no call, 838 of them cut off at the completion cap, so the label reflects the grader's check order, not a wrong choice among tools.
Raising the completion cap is an effective repair for the menu penalty in truncation-heavy configurations, but a small pooled penalty of −2.7 pp survives even at sixteen times the cap.
The workflow pressure has no detectable pooled effect but opposite-signed per-configuration effects, from −30.6 to +34.7 pp, so a pooled average can hide configuration-level interactions with the harness.
Configurations that never truncate show menu effects between −2.1 and +2.8 pp, while truncation is not the only route to loss — DeepSeek V4.1 Flash loses 18.1 pp at 0.7% truncation.
A testable extension follows from the paper's observation that 70.2% of truncated first responses hold reasoning tokens only: count only visible text and arguments against the cap, and the prediction is that most of the menu penalty disappears without raising the cap at all.
The paper varied only the completion cap; its own limitation note invites varying the episode budget and the call limit the same way. If those sweeps behave like the cap sweep, 'pressure effects' reported by other benchmarks may be harness-budget effects in disguise.
The decoder result suggests a cheap diagnostic improvement: reporting truncation rate and call counts alongside the first-failure label would give practitioners most of the signal that a full failure record provides.
The negative rank correlation between ToolSandbox success and VTS across five configurations hints that harness effects can even reverse model rankings between environments; rerunning that external stratum at a sixteenfold cap would show whether the reversal is also budget-driven.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
The success score of a tool agent reflects both its model and the budgets of the harness that runs it. We present ToolBottleneckBench, a benchmark that measures tool-agent performance under controlled pressures and fixed harness budgets. Synthetic enterprise tasks stay fixed while one pressure varies: the registered tool catalog, the visible tool menu, the certified workflow, or the encoding of returned records. An exact grader records the first unmet obligation of each failed episode and the harness limit, if any, that ended it. Across 21 model and harness configurations, the visible menu is the only one of the three pressures that act on the model with a large penalty, and this penalty tracks truncation at the completion cap. Raising the cap sixteenfold removes most of the penalty in the truncation-heavy configurations. Most failures end at a harness limit, and the failure record identifies the assigned pressure no more reliably than a count of tool calls. The repairs that help change which tools reach the model or who issues the calls, and only two repairs, neither improving model behavior, beat every configuration switch. We recommend reporting pressure effects with the harness budgets under which they were measured. We release the benchmark at https://github.com/zichuanxu/tool-bottleneck-bench.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-16
The authors' main claim, as the classifier picked it out 0.76
“Across 21 model and harness configurations, the visible menu is the only one of the three pressures that act on the model with a large penalty, and this penalty tracks truncation at the completion cap.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.88 · human participants 0.04 · controlled comparison 0.44 · randomized assignment 0.03 · states quantitative result 0.95 · compares with prior results 0.32 · mentions code or data availability 0.96
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| The success score of a tool agent reflects both its model and the budgets of the harness that runs it. | background 0.98 | 0.17 | 0.07 | asserted 1.00 |
| We present ToolBottleneckBench, a benchmark that measures tool-agent performance under controlled pressures and fixed harness budgets. | contribution 1.00 | 0.92 | 0.07 | asserted 1.00 |
| Synthetic enterprise tasks stay fixed while one pressure varies: the registered tool catalog, the visible tool menu, the certified workflow, or the encoding of returned records. | method 0.98 | 0.84 | 0.04 | asserted 0.99 |
| An exact grader records the first unmet obligation of each failed episode and the harness limit, if any, that ended it. | method 1.00 | 0.92 | 0.03 | asserted 1.00 |
| Across 21 model and harness configurations, the visible menu is the only one of the three pressures that act on the model with a large penalty, and this penalty tracks truncation at the completion cap. | result 0.99 | 0.94 | 0.51 | asserted 1.00 |
| Raising the cap sixteenfold removes most of the penalty in the truncation-heavy configurations. | result 0.97 | 0.93 | 0.10 | asserted 1.00 |
| Most failures end at a harness limit, and the failure record identifies the assigned pressure no more reliably than a count of tool calls. | result 1.00 | 0.95 | 0.13 | asserted 1.00 |
| The repairs that help change which tools reach the model or who issues the calls, and only two repairs, neither improving model behavior, beat every configuration switch. | result 0.98 | 0.89 | 0.13 | asserted 0.99 |
| We recommend reporting pressure effects with the harness budgets under which they were measured. | future work 0.94 | 0.30 | 0.28 | asserted 0.80 |
| We release the benchmark at https://github.com/zichuanxu/tool-bottleneck-bench. | contribution 0.99 | 0.84 | 0.02 | asserted 0.92 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 7,457 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (12)
Figures are the authors', reproduced from the paper.
References (103)
- [1] Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models , author =. Findings of the Association for Computational Linguistics: ACL 2025 , pages =. 2025 , doi = on Pith
- [2] Yao, Shunyu and Shinn, Noah and Razavi, Pedram and Narasimhan, Karthik , booktitle =. 2025 , url = on Pith
- [3] Liu, Marianne Menglin and Garcia, Daniel and Parllaku, Fjona and Upadhyay, Vikas and Shah, Syed Fahad Allam and Roth, Dan , booktitle =. 2026 , doi = on Pith
- [4] Trivedi, Harsh and Khot, Tushar and Hartmann, Mareike and Manku, Ruskin and Dong, Vinty and Li, Edward and Gupta, Shashank and Sabharwal, Ashish and Balasubramanian, Niranjan , booktitle =. 2024 , doi = on Pith
- [5] Lu, Jiarui and Holleis, Thomas and Zhang, Yizhe and Aumayer, Bernhard and Nan, Feng and Bai, Haoping and Ma, Shuang and Ma, Shen and Li, Mengyu and Yin, Guoli and Wang, Zirui and Pang, Ruoming , booktitle =. 2025 , doi = on Pith
- [6] Babu, Rahul Suresh and Iyer, Laxmipriya Ganesh , journal =. 2026 , url = on Pith
- [7] Repantis, Vyzantinos and Gawde, Ameya and Singh, Harshvardhan and Blackwell, II, Joey , journal =. How Many Tools Should an. 2026 , url = on Pith
- [8] Kate, Kiran and Rizk, Yara and Ghosh, Poulami and Gulati, Ashu and Chakraborti, Tathagata and Wright, Zidane and Agarwal, Mayank , booktitle =. How Good Are. 2026 , doi = on Pith
- [9] Yao, Yilun and Tan, Xinyu and Liu, Chao-Hsuan and Li, Yaoming and Wang, Zhengyang and Yu, Wenhan and Tan, Zhewen and Tian, Yuxuan and Zhao, Guangxiang and Sun, Lin and Zhang, Xiangzheng and Yang, Tong , journal =. 2026 , url = on Pith
- [10] Kapoor, Sayash and Stroebl, Benedikt and Kirgis, Peter and Nadgir, Nitya and Siegel, Zachary S. and Wei, Boyi and Xue, Tianci and Chen, Ziru and Chen, Felix and Utpala, Saiteja and Ndzomga, Franck and Oruganty, Dheeraj and Luskin, Sophie and Liu, Kangheng and Yu, Botao and Aro... on Pith
- [21] Seiji Maekawa and Jackson Hassell and Pouya Pezeshkpour and Tom Mitchell and Estevam Hruschka , booktitle =. Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step. 2026 , doi = on Pith
- [22] Harsh Raj and Vipul Gupta and Anas Mahmoud and Razvan-Gabriel Dumitru and Darvin Yi and Aakash Sabharwal and Yunzhong He , journal =. Model or Harness?. 2026 , doi = on Pith
- [23] Musa Shams , journal =. 2026 , doi = on Pith
- [24] Shiyang Chen , journal =. Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in. 2026 , doi = on Pith
- [25] Kiran Kate and Tejaswini Pedapati and Kinjal Basu and Yara Rizk and Vijil Chenthamarakshan and Subhajit Chaudhury and Mayank Agarwal and Ibrahim Abdelaziz , journal =. 2025 , doi = on Pith
- [27] Weihao Zeng and Yuzhen Huang and Junxian He , booktitle =. 2026 , url = on Pith
- [28] Rahul Suresh Babu and Laxmipriya Ganesh Iyer , journal =. 2026 , doi = on Pith
- [29] Xiang Fei and Xiawu Zheng and Hao Feng , journal =. 2025 , doi = on Pith
- [30] Tiantian Gan and Qiyao Sun , journal =. 2025 , doi = on Pith
- [31] Jason Starace , journal =. Scaffold Effects on. 2026 , doi = on Pith
show 80 more
- [33] Yoon, Jeonghun and Kim, Dongchan , journal =. Diagnosis Is Not Prescription: Linguistic Co-Adaptation Explains Patching Hazards in. 2026 , doi = on Pith
- [34] Zhongwen Luan and Xiaoyu Zhang and Ming Hu and Yue Yang and Jiongchi Yu and Xiaohong Chen , journal =. Repair or Resample?. 2026 , doi = on Pith
- [35] Yunbei Zhang and Janet Wang and Yingqiang Ge and Weijie Xu and Jihun Hamm and Chandan K. Reddy , journal =. Stop Comparing. 2026 , doi = on Pith
- [36] Xiaofeng Lin and Yingxu Wang and Tung Sum Thomas Kwok and Daniel Guo and Sahil Arun Nale and Charles Fleming and Guang Cheng , journal =. 2026 , doi = on Pith
- [37] Cho, Yong-eun , journal =. It's Not the Capability: Harness Sensitivity Is Non-Monotone Across. 2026 , doi = on Pith
- [38] Holm, Sture , title =. Scandinavian Journal of Statistics , volume =. 1979 , url = on Pith
- [39] Berger, Roger L. , title =. Technometrics , volume =. 1982 , doi = on Pith
- [40] DerSimonian, Rebecca and Laird, Nan , title =. Controlled Clinical Trials , volume =. 1986 , doi = on Pith
- [41] Schuirmann, Donald J. , title =. Journal of Pharmacokinetics and Biopharmaceutics , volume =. 1987 , doi = on Pith
- [44] Yangze Liu and Zhongyi Han , journal =. What Does a Harness Buy?. 2026 , doi = on Pith
- [45] Xiaoyang Yuan and Qi Liu and Yubin Ruan and Xinyi Mou and Zhuomeng Zhang and Wenjin Wang and Hanying Jiao and Di Wu and Mingye Xu and Yi Bin and Ke Feng and Zixun Sun , journal =. 2026 , doi = on Pith
- [46] Jiaxuan Dai and Tianyi Huang , journal =. 2026 , doi = on Pith
- [47] Sayash Kapoor and Benedikt Stroebl and Zachary S. Siegel and Nitya Nadgir and Arvind Narayanan , journal =. 2024 , doi = on Pith
- [48] John Yang and Carlos E. Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik Narasimhan and Ofir Press , journal =. 2024 , doi = on Pith
- [49] Lost in the Middle: How Language Models Use Long Contexts , author =. Transactions of the Association for Computational Linguistics , volume =. 2024 , doi = on Pith
- [50] Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects , author =. Review of Economic Studies , volume =. 2009 , doi = on Pith
- [51] Principal Stratification in Causal Inference , author =. Biometrics , volume =. 2002 , doi = on Pith
- [52] Li, Minghao and Zhao, Yingxiu and Yu, Bowen and Song, Feifan and Li, Hangyu and Yu, Haiyang and Li, Zhoujun and Huang, Fei and Li, Yongbin , booktitle =. 2023 , url = on Pith
- [53] Qin, Yujia and Liang, Shihao and Ye, Yining and Zhu, Kunlun and Yan, Lan and Lu, Yaxi and Lin, Yankai and Cong, Xin and Tang, Xiangru and Qian, Bill and Zhao, Sihan and Hong, Lauren and Tian, Runchu and Xie, Ruobing and Zhou, Jie and Gerstein, Mark and Li, Dahai and Liu, Zhiyu... on Pith
- [54] Patil, Shishir G. and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E. , booktitle =. The. 2025 , url = on Pith
- [56] Du, Yufeng and Tian, Minyang and others , journal =. Context Length Alone Hurts. 2025 , url = on Pith
- [57] Wang, Zhenting and others , journal =. 2025 , url = on Pith
- [58] Luo, Ziyang and others , journal =. 2025 , url = on Pith
- [59] Guo, Zhicheng and Cheng, Sijie and Wang, Hao and Liang, Shihao and Qin, Yujia and Li, Peng and Liu, Zhiyuan and Sun, Maosong and Liu, Yang , booktitle =. 2024 , doi = on Pith
- [60] Rahul Suresh Babu and Laxmipriya Ganesh Iyer. 2026 a . https://doi.org/10.48550/arXiv.2606.06284 ToolChoiceConfusion : Causal minimal tool filtering for reliable LLM agents . arXiv preprint arXiv:2606.06284 on PithPith's reviewarXivdoi:10.48550/arxiv.2606.06284
- [61] Rahul Suresh Babu and Laxmipriya Ganesh Iyer. 2026 b . https://arxiv.org/abs/2606.15508 ToolMenuBench : Benchmarking tool-menu filtering strategies for reliable and efficient LLM agents . arXiv preprint arXiv:2606.15508 on PitharXiv
- [62] Roger L. Berger. 1982. https://doi.org/10.1080/00401706.1982.10487790 Multiparameter hypothesis testing and acceptance sampling . Technometrics, 24(4):295--300 on Pithdoi:10.1080/00401706.1982.10487790
- [63] Vishvesh Bhat, Jay Vaghasiya, Muhammad Ahmed Mohsin, and Asad Aali. 2026. https://doi.org/10.48550/arXiv.2607.02577 Benchmarking the benchmarks: A validity audit of tool-calling evaluation . arXiv preprint arXiv:2607.02577 on PithPith's reviewarXivdoi:10.48550/arxiv.2607.02577
- [64] Le Chen, Zishen Wan, Baixi Sun, Xiaolong Ma, Chih-Hsuan Yang, Feng Yan, Sheng Di, Franck Cappello, and Rajeev Thakur. 2026 a . https://doi.org/10.48550/arXiv.2608.31057 Measure before you manage: Evaluating agent working memory in coding agents . arXiv preprint arXiv:2608.31057 on PithPith's reviewarXivdoi:10.48550/arxiv.2608.31057
- [65] Mengzhuo Chen, Junjie Wang, Zhe Liu, Yawen Wang, Haiming Zheng, and Qing Wang. 2026 b . https://arxiv.org/abs/2606.06324 From failed trajectories to reliable LLM agents: Diagnosing and repairing harness flaws . arXiv preprint arXiv:2606.06324 on PithPith's reviewarXiv
- [66] Shiyang Chen. 2026. https://doi.org/10.48550/arXiv.2606.16364 Looking is not picking: An attention-segment account of tool-selection failures in LLM agents . arXiv preprint arXiv:2606.16364 on PithPith's reviewarXivdoi:10.48550/arxiv.2606.16364
- [67] Yong-eun Cho. 2026. https://doi.org/10.48550/arXiv.2605.26731 It's not the capability: Harness sensitivity is non-monotone across LLM agent tiers . arXiv preprint arXiv:2605.26731 on PithPith's reviewarXivdoi:10.48550/arxiv.2605.26731
- [68] Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, Nicholas Thumiger, Aditya Desai, Ion Stoica, Ana Klimovic, Graham Neubig, and Joseph E. Gonzalez. 2025. https://arxiv.org/abs/2502.08235 ... on PithPith's reviewarXiv
- [69] Jiaxuan Dai and Tianyi Huang. 2026. https://doi.org/10.48550/arXiv.2609.26911 TwinCheck : Evidence-grounded negative-twin verification for stateful tool agents . arXiv preprint arXiv:2609.26911 on PithPith's reviewarXivdoi:10.48550/arxiv.2609.26911
- [70] Rebecca DerSimonian and Nan Laird. 1986. https://doi.org/10.1016/0197-2456(86)90046-2 Meta-analysis in clinical trials . Controlled Clinical Trials, 7(3):177--188 on Pithdoi:10.1016/0197-2456(86)90046-2
- [71] Yufeng Du, Minyang Tian, et al. 2025. https://arxiv.org/abs/2510.05381 Context length alone hurts LLM performance despite perfect retrieval . arXiv preprint arXiv:2510.05381 on PitharXiv
- [72] Xiang Fei, Xiawu Zheng, and Hao Feng. 2025. https://doi.org/10.48550/arXiv.2506.01056 MCP-Zero : Active tool discovery for autonomous LLM agents . arXiv preprint arXiv:2506.01056 on PithPith's reviewarXivdoi:10.48550/arxiv.2506.01056
- [73] Constantine E. Frangakis and Donald B. Rubin. 2002. https://doi.org/10.1111/j.0006-341X.2002.00021.x Principal stratification in causal inference . Biometrics, 58(1):21--29 on PithPith's reviewdoi:10.1111/j.0006-341x.2002.00021.x
- [74] Tiantian Gan and Qiyao Sun. 2025. https://doi.org/10.48550/arXiv.2505.03275 RAG-MCP : Mitigating prompt bloat in LLM tool selection via retrieval-augmented generation . arXiv preprint arXiv:2505.03275 on PithPith's reviewarXivdoi:10.48550/arxiv.2505.03275
- [75] Ashritha Gonuguntla. 2026. https://doi.org/10.48550/arXiv.2608.08239 The replay gap: Static evaluation of model switching in LLM agents scores the wrong world . arXiv preprint arXiv:2608.08239. Efficient Reasoning Workshop at COLM 2026 on PithPith's reviewarXivdoi:10.48550/arxiv.2608.08239
- [76] Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.664 StableToolBench : Towards stable large-scale benchmarking on tool learning of large language models . In Findi... on Pithdoi:10.18653/v1/2024.findings-acl.664
- [77] Bhaskar Gurram. 2026. https://doi.org/10.48550/arXiv.2604.16706 Auditing automated evaluation, error propagation, and runtime mitigation in tool-using language agents . arXiv preprint arXiv:2604.16706 on PithPith's reviewarXivdoi:10.48550/arxiv.2604.16706
- [78] Sture Holm. 1979. https://www.jstor.org/stable/4615733 A simple sequentially rejective multiple test procedure . Scandinavian Journal of Statistics, 6(2):65--70 on Pith
- [79] Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S. Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, and 12 othe... on Pith
- [80] Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. 2024. https://doi.org/10.48550/arXiv.2407.01502 AI agents that matter . arXiv preprint arXiv:2407.01502 on PithPith's reviewarXivdoi:10.48550/arxiv.2407.01502
- [81] Kiran Kate, Tejaswini Pedapati, Kinjal Basu, Yara Rizk, Vijil Chenthamarakshan, Subhajit Chaudhury, Mayank Agarwal, and Ibrahim Abdelaziz. 2025. https://doi.org/10.48550/arXiv.2505.10570 LongFuncEval : Measuring the effectiveness of long context models for function calling . a... on PithPith's reviewarXivdoi:10.48550/arxiv.2505.10570
- [82] Kiran Kate, Yara Rizk, Poulami Ghosh, Ashu Gulati, Tathagata Chakraborti, Zidane Wright, and Mayank Agarwal. 2026. https://doi.org/10.18653/v1/2026.eacl-long.134 How good are LLM s at processing tool outputs? In Proceedings of the 19th Conference of the European Chapter of the... on Pithdoi:10.18653/v1/2026.eacl-long.134
- [83] David S. Lee. 2009. https://doi.org/10.1111/j.1467-937X.2009.00536.x Training, wages, and sample selection: Estimating sharp bounds on treatment effects . Review of Economic Studies, 76(3):1071--1102 on PithPith's reviewdoi:10.1111/j.1467-937x.2009.00536.x
- [84] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. https://aclanthology.org/2023.emnlp-main.187 API-Bank : A comprehensive benchmark for tool-augmented LLM s . In Proceedings of the 2023 Conference on Empirical ... on Pith
- [85] Yixuan Li, Yiyun Zhou, Yao Long Teng, Fuchao Yang, Yanchen Deng, Zhiyi Lyu, Xuyu Dong, Feng Chen, and Bo An. 2026. https://doi.org/10.48550/arXiv.2610.00917 Finding the right fit: Model-harness interactions across agent tasks . arXiv preprint arXiv:2610.00917 on PithPith's reviewarXivdoi:10.48550/arxiv.2610.00917
- [86] Xiaofeng Lin, Yingxu Wang, Tung Sum Thomas Kwok, Daniel Guo, Sahil Arun Nale, Charles Fleming, and Guang Cheng. 2026. https://doi.org/10.48550/arXiv.2606.09071 REFLECT : Intervention-supported error attribution for silent failures in LLM agent traces . arXiv preprint arXiv:2606.09071 on PithPith's reviewarXivdoi:10.48550/arxiv.2606.09071
- [87] Marianne Menglin Liu, Daniel Garcia, Fjona Parllaku, Vikas Upadhyay, Syed Fahad Allam Shah, and Dan Roth. 2026. https://doi.org/10.18653/v1/2026.acl-long.1573 ToolScope : Enhancing LLM agent tool use through tool merging and context-aware filtering . In Proceedings of the 64th... on Pithdoi:10.18653/v1/2026.acl-long.1573
- [88] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics, 12... on Pithdoi:10.1162/tacl_a_00638
- [89] Yangze Liu and Zhongyi Han. 2026. https://doi.org/10.48550/arXiv.2610.04433 What does a harness buy? Tokens , mostly . arXiv preprint arXiv:2610.04433 on PithPith's reviewarXivdoi:10.48550/arxiv.2610.04433
- [90] Abhilasha Lodha, Mahsa Pahlavikhah Varnosfaderani, Abir Chakraborty, and Abhinav Mithal. 2026. https://arxiv.org/abs/2606.10209 Less context, better agents: Efficient context engineering for long-horizon tool-using LLM agents . arXiv preprint arXiv:2606.10209 on PithPith's reviewarXiv
- [91] Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. 2025. https://doi.org/10.18653/v1/2025.findings-naacl.65 ToolSandbox : A stateful, conversational, interactive evaluation be... on Pithdoi:10.18653/v1/2025.findings-naacl.65
- [92] Zhongwen Luan, Xiaoyu Zhang, Ming Hu, Yue Yang, Jiongchi Yu, and Xiaohong Chen. 2026. https://doi.org/10.48550/arXiv.2608.25920 Repair or resample? R ethinking failure debugging in LLM multi-agent systems . arXiv preprint arXiv:2608.25920 on PithPith's reviewarXivdoi:10.48550/arxiv.2608.25920
- [93] Fabian Lukassen, Christoph Weisser, Thomas Kneib, and Alexander Silbersdorff. 2026. https://arxiv.org/abs/2607.10380 CAFE : A compound- AI factorial evaluation framework . arXiv preprint arXiv:2607.10380 on PithPith's reviewarXiv
- [94] Ziyang Luo et al. 2025. https://arxiv.org/abs/2508.14704 MCP-Universe : Benchmarking large language models with real-world Model Context Protocol servers . arXiv preprint arXiv:2508.14704 on PithPith's reviewarXiv
- [95] Ming Ma, Jue Zhang, Fangkai Yang, Yu Kang, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2026. https://doi.org/10.48550/arXiv.2512.06749 DoVer : Intervention-driven auto debugging for LLM multi-agent systems . In International Conference on Learning Representations on Pithdoi:10.48550/arxiv.2512.06749
- [96] Seiji Maekawa, Jackson Hassell, Pouya Pezeshkpour, Tom Mitchell, and Estevam Hruschka. 2026. https://doi.org/10.48550/arXiv.2509.26553 Towards reliable benchmarking: A contamination free, controllable evaluation framework for multi-step LLM function calling . In International ... on Pithdoi:10.48550/arxiv.2509.26553
- [97] Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. https://proceedings.mlr.press/v267/patil25a.html The Berkeley Function Calling Leaderboard ( BFCL ): From tool use to agentic evaluation of large language ... on Pith
- [98] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM : Facilitating large language mode... on Pith
- [99] Harsh Raj, Vipul Gupta, Anas Mahmoud, Razvan-Gabriel Dumitru, Darvin Yi, Aakash Sabharwal, and Yunzhong He. 2026. https://doi.org/10.48550/arXiv.2607.28802 Model or harness? A n interaction-centric taxonomy for localizing agent failures . arXiv preprint arXiv:2607.28802 on PithPith's reviewarXivdoi:10.48550/arxiv.2607.28802
- [100] Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh, and Joey Blackwell, II. 2026. https://arxiv.org/abs/2605.24660 How many tools should an LLM agent see? A chance-corrected answer . arXiv preprint arXiv:2605.24660 on PithPith's reviewarXiv
- [101] Donald J. Schuirmann. 1987. https://doi.org/10.1007/BF01068419 A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability . Journal of Pharmacokinetics and Biopharmaceutics, 15(6):657--680 on Pithdoi:10.1007/bf01068419
- [102] Musa Shams. 2026. https://doi.org/10.48550/arXiv.2607.27353 LayerRAG-Bench : A cross-layer reliability benchmark for agentic retrieval-augmented generation . arXiv preprint arXiv:2607.27353 on PithPith's reviewarXivdoi:10.48550/arxiv.2607.27353
- [103] Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, and Zhaochun Ren. 2025. https://doi.org/10.18653/v1/2025.findings-acl.1258 Retrieval models aren't tool-savvy: Benchmarking tool retrieval for large language models . In Findings of the Associat... on Pithdoi:10.18653/v1/2025.findings-acl.1258
- [104] Jason Starace. 2026. https://doi.org/10.48550/arXiv.2606.08529 Scaffold effects on GAIA : A controlled comparison . arXiv preprint arXiv:2606.08529 on PithPith's reviewarXivdoi:10.48550/arxiv.2606.08529
- [105] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. 2024. https://doi.org/10.18653/v1/2024.acl-long.850 AppWorld : A controllable world of apps and people for benchmarking interactiv... on Pithdoi:10.18653/v1/2024.acl-long.850
- [106] Zelin Wan, Arash Nourian, Xiaoxiao Li, Nihar Nandan, and Kamalakannan Nandagopal. 2026. https://doi.org/10.48550/arXiv.2608.29128 APIFlow-Bench : Measuring whether agents survive long, dependent API workflows . arXiv preprint arXiv:2608.29128 on PithPith's reviewarXivdoi:10.48550/arxiv.2608.29128
- [107] Wenbo Wang. 2026 a . https://doi.org/10.48550/arXiv.2609.03966 Interface-induced trajectory censoring . arXiv preprint arXiv:2609.03966 on PithPith's reviewarXivdoi:10.48550/arxiv.2609.03966
- [108] Zhenting Wang et al. 2025. https://arxiv.org/abs/2508.20453 MCP-Bench : Benchmarking tool-using LLM agents with complex real-world tasks via MCP servers . arXiv preprint arXiv:2508.20453 on PithPith's reviewarXiv
- [109] Zihao Wang. 2026 b . https://doi.org/10.48550/arXiv.2608.23628 Callability is not operability: Controlled interface interventions for LLM agents . arXiv preprint arXiv:2608.23628 on PithPith's reviewarXivdoi:10.48550/arxiv.2608.23628
- [110] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. https://doi.org/10.48550/arXiv.2405.15793 SWE -agent: Agent-computer interfaces enable automated software engineering . arXiv preprint arXiv:2405.15793 on PithPith's reviewarXivdoi:10.48550/arxiv.2405.15793
- [111] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/1b126cc38b8638e07bef37e7b2bb72bf-Abstract-Conference.html - B ench: A benchmark for tool-agent-user interaction in real-world domains . In International... on Pith
- [112] Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, and Tong Yang. 2026. https://arxiv.org/abs/2605.27922 Harness-Bench : Measuring harness effects across models in realistic agent wo... on PithPith's reviewarXiv
- [113] Jeonghun Yoon and Dongchan Kim. 2026. https://doi.org/10.48550/arXiv.2605.21958 Diagnosis is not prescription: Linguistic co-adaptation explains patching hazards in LLM pipelines . arXiv preprint arXiv:2605.21958 on PithPith's reviewarXivdoi:10.48550/arxiv.2605.21958
- [114] Xiaoyang Yuan, Qi Liu, Yubin Ruan, Xinyi Mou, Zhuomeng Zhang, Wenjin Wang, Hanying Jiao, Di Wu, Mingye Xu, Yi Bin, Ke Feng, and Zixun Sun. 2026. https://doi.org/10.48550/arXiv.2609.32498 DAAF : From failure localization to editable system assets in LLM agents . arXiv preprint ... on PithPith's reviewarXivdoi:10.48550/arxiv.2609.32498
- [115] Weihao Zeng, Yuzhen Huang, and Junxian He. 2026. https://proceedings.mlr.press/v306/zeng26g.html LOCA-bench : Benchmarking language agents under controllable and extreme context growth . In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Pro... on Pith
The first 100 of 103 references are listed here. All of them are in this paper's citation record.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment