The Bitter Lesson of Tool Calling

Elias Lumer; Ishan Patel; Sahil Sen; Vamse Kumar Subbiah

REVIEW 4 major objections 5 minor 12 references

The Bitter Lesson of Tool Calling

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Programmatic tool calling — models write Python to invoke tools — matches or beats native JSON tool calling in 11 of 14 models on BFCL v4, with the largest gains on long chains and high fan-out, while the three older OpenAI models fail on…

desk verdict A useful empirical comparison undermined by a brittle harness: the 11-of-14 headline and generation-tracking conclusion depend on a newline-escaping artifact that would likely vanish with a more robust interface. read the letter →

arxiv 2608.06370 v1 pith:2NOZGPEQ submitted 2026-08-06 cs.CL

Ishan Patel , Sahil Sen , Elias Lumer , Vamse Kumar Subbiah This is my paper

classification cs.CL

keywords programmatictoolcallingJSONfunctionuseLLMagentsBFCLv4parallelfan-outcontextrot

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that letting a language model call tools by writing a Python script, rather than emitting JSON objects, is a broadly viable interface rather than a niche trick. On a 309-entry subset of the BFCL v4 benchmark spanning 14 models released over 20 months, programmatic tool calling matches or exceeds native JSON tool calling in 11 of 14 models, with the GPT-5.6 family improving by roughly 10.6 percentage points over its own JSON baseline. The gains appear exactly where JSON tool calling is weakest: sequential multi-hop chains and high parallel fan-out, and programmatic tool calling stays stable under context flooding while JSON degrades by 2.3% on average. The paper argues that the remaining gap tracks model generation rather than model family, with the three older OpenAI models failing on a consistent literal newline-escaping issue.

What carries the argument

The object that carries the argument is the programmatic tool-calling harness: each benchmark function schema is compiled into a typed Python stub that captures its arguments and prints them as JSON, and the model writes a single script that imports the stubs, calls the relevant functions, and prints results to stdout. A subprocess executes the script, and a stop middleware terminates the agent loop after one turn, so the whole entry consumes the same number of LLM calls as the JSON baseline. This single-turn code-execution design is what converts multi-turn JSON tool calls into one script evaluation, and it is the mechanism behind the chaining, fan-out, and context-robustness results.

What would settle it

Run the same 309-entry BFCL v4 subset with live-return stubs — tools that actually compute and return values that can disagree with the model's parametric guesses — and compare the two paradigms; a reversal in chaining or fan-out accuracy under those conditions would refute the viability claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that programmatic tool calling — exposing tools as typed Python stubs that the model imports and calls from one script executed in a subprocess — matches or exceeds native JSON tool calling across most models and task structures tested. On the BFCL v4 subset, 11 of 14 models match or beat their JSON baseline, and the three newest GPT variants gain 4.2% to 10.6% absolute. The advantage concentrates in the chaining ablation, where programmatic tool calling reaches an 18.8% absolute gap over JSON at chain lengths of 12 or more, and in the parallelism ablation, where 13 of 14 models match or exceed baseline while JSON tool calling drops calls entirely above a model-specific fan-out threshold. Under a 128-schema flood condition, programmatic tool calling improves 5.5% on average while JSON degrades 2.3%. The observed failures are consistent: three older OpenAI models emit literal backslash-n escape sequences instead of real newlines in multiline scripts, causing syntax errors, and per-category means show a 14.1% average deficit in parallel categories that is largely attributable to those same models.

Load-bearing premise

The whole comparison rests on BFCL v4's echo-return stubs being representative of real tool-calling deployments, since those stubs return arguments verbatim and let the model compute intermediate values from its own parametric knowledge inside Python; if real tool returns were non-echo and changed downstream calls, the chaining and fan-out gains might not transfer.

Editorial extensions

If this is right

If programmatic tool calling is adopted, tool-calling agents can resolve long sequential chains in one inference turn rather than one turn per link, with latency roughly half of the JSON baseline for 13 of 14 models.
High fan-out stops being a structural limit: programmatic tool calling maintains 100% enumeration accuracy at N=100 where JSON tool calling drops to 0% for at least one frontier model.
Token cost favors programmatic tool calling above fan-out N≈26, so the paradigm becomes cheaper exactly where JSON output enumeration is most expensive.
The generation-line pattern implies that newer code-capable models inherit the programmatic interface naturally, so the viable-alternative conclusion is expected to strengthen as older failing models are retired.
Context flooding does not hurt programmatic tool calling, suggesting robustness to adversarial tool-schema overload in deployed agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the echo-stub setup generalizes to live APIs, tool interfaces may converge on code as the native agent action space, with JSON reserved for cross-system boundaries; the direct test is a live-return variant of BFCL v4.
The backslash-n failure in three older OpenAI models is a format-brittleness risk that is not monotonic with model size or release date, so harness design must probe for such brittleness instead of assuming generation parity.
The chaining gain depends on the model computing intermediate values from parametric knowledge rather than from tool returns; if a tool's real return disagrees with the model's guess, in-script substitution could produce wrong downstream calls while still scoring correct on echo stubs.
The 14.1% average deficit in parallel categories warns that programmatic tool calling's aggregate win is category-dependent, so deployments dominated by parallel calls should check per-category deltas before switching.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The systematic 14-model comparison on BFCL v4 is the real contribution—no one else has done this across this many models. The fan-out threshold for Claude Sonnet 5 (100% at N=70, 75% at N=72, 0% at N=100) is a clean, falsifiable observation, and the context-rot result (PTC stable while JSON degrades ~2%) is legitimate despite the small n.

But the headline is not trustworthy. The 11-of-14 and the §5.1 'generation predicts viability' story rest on three OpenAI models that fail because they emit literal \n instead of real newlines in the `python3 -c` command. That's a harness artifact, not a capability gap. The paper's counter—that GPT-5-nano succeeds with the same prompt—only shows model-specific instruction following; it doesn't rule out prompt sensitivity. A more robust interface (write to file, or base64) would likely recover those models, and the generation-tracking conclusion would collapse.

Other soft spots: the 18.8% chain-length gap at ≥12 is asserted but never shown in a table; the fan-out probes have tiny N per point; no code or data artifacts are provided despite promises of 'released with the evaluation harness.' The echo-return stub limitation is honestly acknowledged.

Net: this is a serious empirical paper with transparent limitations, but the central claim is contingent on a brittle quoting mechanism. The fan-out and context-rot findings are worth retaining. I'd send it to review because the question matters and the fix is straightforward—but I'd ask the authors to redo the harness, show the chain-length breakdown, and qualify the generation-tracking conclusion.

For me: not citable in its current form. I'd look at the revision.

Referee Report

4 major / 5 minor

Summary. The paper compares programmatic tool calling (PTC), in which a model writes a Python script against typed stubs that is executed in a shell subprocess, with native JSON tool calling on a 309-entry subset of BFCL v4 across 14 models. It reports that PTC matches or exceeds JSON tool calling in 11 of 14 models on the main evaluation, in 13 of 14 under parallel fan-out, and is stable under context flooding. The paper interprets the main result as showing that PTC viability tracks model generation rather than family, with the three older OpenAI models failing because they emit literal backslash-n sequences in multiline scripts. Additional claims are that PTC has an 18.8% absolute accuracy advantage over JSON at chain lengths at least 12, that JSON tool calling drops calls above fan-out 70-72 for Claude Sonnet 5 while PTC does not, and that PTC is a viable and robust alternative to JSON tool calling.

Significance. If the findings survive a more robust execution harness, the paper would be a useful contribution: it uses an external benchmark, a deterministic scorer, no fitted parameters, a 20-month model release span, and it reports confidence intervals and explicit limitations. The authors also state that entry IDs and the evaluation harness are released, which aids reproducibility. The main generalization is currently contingent on a fragile quoting mechanism, so the significance is real but conditional on correcting that issue.

major comments (4)

[§4.1, §5.1, §3.2] The three-model failure that drives the headline 11-of-14 result is a harness artifact, not a demonstrated capability gap. When a model emits a literal backslash-n inside the python3 -c '...' command required by Section 3.2, the subprocess fails with a syntax error on any multiline script. The rebuttal in Section 5.1 that GPT-5-nano succeeds with the same prompt and therefore prompt configuration is ruled out is not valid: model-specific variation in instruction following is exactly what prompt sensitivity means, and a trivial harness change (unescaping the backslash-n sequences, writing the script to a file, or passing base64-encoded code) could recover GPT-4o, GPT-4.1, and GPT-5.4-mini. The paper provides no evidence that those models would still fail under a more robust interface. The authors should rerun with such an interface and report whether those three models still fall below baseline; until then, the 11-of-14 claim and the generation-tracking conclusion are not established.
[Contributions and §4.2] The contribution bullet claiming an 18.8% absolute accuracy gap at chain lengths at least 12 is not supported by any reported analysis. Section 4.2 reports only per-model overall accuracy on the chaining subset in Table 3; there is no chain-length-stratified table, regression, or figure relating the accuracy gap to chain length. The authors should either add the missing stratified analysis or remove the claim from the contributions section.
[§5.1 and Table 1] The statement that 'the fix entered training data between GPT-5.4-mini and GPT-5' is chronologically impossible: GPT-5.4-mini has a release date of 2026-03-17, after GPT-5 and GPT-5-nano, both dated 2025-08-07. This internal inconsistency further weakens the capability-gap interpretation in Section 5.1 and should be corrected or replaced with a defensible temporal claim.
[§3.4, §7, Abstract] Because BFCL v4 uses echo-return stubs, the chaining advantage is obtained by having the model compute intermediate values from parametric knowledge rather than from tool outputs; the abstract and conclusion's 'viable and robust alternative' claim therefore goes beyond what the benchmark measures. The authors acknowledge this in Limitation 1, but the conclusion should be explicitly scoped to argument-serialization accuracy in echo-return settings, or the paper should add an experiment with real or simulated non-echo tool returns before claiming robustness in deployment.

minor comments (5)

[Table 2 and Abstract] The abstract says 'the GPT-5.6 family achieving a 10.6% improvement,' but only GPT-5.6-Sol and GPT-5.6-Terra show a 10.6% improvement; GPT-5.6-Luna improves by 4.2%. Please say 'two GPT-5.6 models' or give the range.
[Appendix C] The condensed programmatic prompt explicitly instructs the model to use real newlines, but the paper does not report how often the backslash-n failure occurred across models or whether any non-failing model also produced escaped newlines that the harness corrected. Reporting this would help distinguish harness brittleness from model capability.
[Table 5 and Abstract] The abstract describes PTC as 'stable' under context rot, but Table 5 shows a +5.5% average improvement from filtered to flood; please use a consistent description, such as 'stable or improved.'
[§5.3] The latency claim is based on wall-clock times, but no measurement details are provided about hardware, API variance, or number of runs; at minimum, state that these are single-run measurements or add error bars.
[References] BFCL v4 is cited only through the audit by Vaghasiya et al.; please cite the original benchmark release as well, or clarify that the audit is the source of the subset used here.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity: the headline BFCL v4 comparison is external and parameter-free, with only a minor disclosed design-level asymmetry in the chaining ablation.

other [Section 3.4 (Chaining) and Section 4.2 / Contribution 2]
"In programmatic tool calling, both calls appear in a single script. The model computes the intermediate value using parametric knowledge and passes it directly. ... Programmatic tool calling’s accuracy advantage on sequential tasks scales with chain length, reaching an 18.8% absolute gap over JSON tool calling at lengths ≥12, an effect absent at short chains and driven by the extra inference turn JSON tool calling incurs per link."

The chaining ablation defines PTC's chaining path as computing intermediate values from parametric knowledge inside a single script, while JSON tool calling must issue f1, await the tool-returned value, and then issue f2 in a separate turn. The stated driver of the measured advantage — one fewer inference turn per link and the permission to bypass tool outputs — is a property of the interface as defined, so the qualitative chain-length effect is largely fixed by the experimental design rather than independently discovered. The magnitude of the gap (18.8%) is still empirical, and the main 11-of-14 BFCL v4 result is a separate external-benchmark comparison, so this is a limited, supporting-claim asymmetry rather than a collapse of the paper's central argument.

full rationale

The paper's central claim is an empirical comparison against the external BFCL v4 benchmark with a deterministic scorer and no fitted parameters: 11 of 14 models match or exceed the JSON tool-calling baseline under programmatic tool calling. That result does not reduce to the paper's definitions or to any fitted input, and the three older OpenAI failures are reported transparently as newline-encoding artifacts rather than hidden by metric redefinition. The chaining ablation does contain a design-level asymmetry: PTC is allowed to compute intermediate values from parametric knowledge in one script while JSON must wait for tool returns across turns, which makes the qualitative chain-length advantage partly a consequence of the interface design. However, this is disclosed in Sections 3.4 and 7, and the paper separately reports enumeration versus aggregation accuracy to avoid overclaiming tool execution. Self-citations (e.g., Lumer et al. 2026 and Sen et al. 2026a) appear in related work and interpretation but are not load-bearing: the present experiments independently produce the fan-out, context-rot, and filesystem numbers. Taking all this into account, the derivation chain is not circular in any load-bearing sense; the minor chaining-design favoritism warrants only a low nonzero score.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted; the central result rests on the validity of the BFCL v4 subset, the echo-return stub design, and the interpretation of the backslash-n encoding failure as a capability gap. These are domain assumptions rather than adjustable constants.

assumptions (4)

domain assumption BFCL v4 ground-truth labels and deterministic normalized string comparison are an adequate measure of tool-calling correctness.
The entire evaluation scores against these labels; Section 3.3 invokes the benchmark's normalized string comparison without independent verification of label quality, and the paper cites an audit finding 20% evaluator-human misalignment in the LLM-judge mode.
domain assumption Echo-return stubs are a valid proxy for real tool execution when assessing paradigm viability.
Section 3.3 and Section 7 state each function returns its arguments verbatim, so the study measures argument serialization, not end-to-end tool use; the viable alternative conclusion assumes this transfers.
ad hoc to paper The literal backslash-n output failure of three OpenAI models is a model capability gap rather than an artifact of the specific PTC prompt or parser.
Section 5.1 interprets the failure as a capability gap based on one counterexample, GPT-5-nano, but no ablation varies the prompt's newline instructions; the aggregate 11-of-14 result depends on this interpretation.
domain assumption The 309-entry subset is representative of BFCL v4.
Section 3.3 says entries were sampled proportionally with per-category minimums, but the sampling procedure and entry list are not shown in the paper, only said to be released.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Bitter Lesson of Tool Calling." pith.science (2026). https://pith.science/paper/2NOZGPEQ

@misc{pith2026260806370,
  author       = {Pith},
  title        = {Pith review of: The Bitter Lesson of Tool Calling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2NOZGPEQ}},
  note         = {Machine review of arXiv:2608.06370}
}

read the original abstract

Tool use transforms LLMs into agents that act beyond their training data, and for code-capable models, programmatic tool calling extends this further by replacing rigid JSON calls with scripts that chain and parallelize naturally. However, a systematic evaluation of tools as code on an established benchmark across current and prior model generations under real-world task conditions has not been conducted. In this work, we empirically compare programmatic tool calling (PTC) to native JSON tool calling across 14 language models on BFCL v4. In the programmatic tool calling paradigm, tools are exposed as typed Python stubs that the model invokes through code, with execution and results handled in a single agent turn. Programmatic tool calling matches or exceeds native JSON tool calling in 11 of 14 models on BFCL v4, with the GPT-5.6 family achieving a 10.6% improvement over the JSON tool calling baseline. Further, it matches or outperforms baseline in 13 of 14 models under parallel fan-out, and holds stable under context rot conditions where baseline degrades 2.3% on average. Our results demonstrate that programmatic tool calling is a viable and robust alternative to JSON tool calling, with performance tracking model capability across release generations.

Figures

Figures reproduced from arXiv: 2608.06370 by the authors.

**Figure 2.** Accuracy (%) on the BFCL v4 subset for JSON tool calling and programmatic tool calling (PTC) by [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

[3]

In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics, ACL 2024, pages 9510–9529

T-Eval: Evaluating the tool utilization capability of large language models step by step. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics, ACL 2024, pages 9510–9529. Cloudflare

work page 2024
[5]

InFindings of the Association for Computational Linguistics: ACL 2024, pages 4363–4400

Planning, creation, usage: Benchmarking LLMs for comprehensive tool utiliza- tion in real-world complex scenarios. InFindings of the Association for Computational Linguistics: ACL 2024, pages 4363–4400. Hugging Face

work page 2024
[6]

https://www

Prime Agent: A self-improving RLM agent. https://www. primeintellect.ai/blog/prime-agent. Ac- cessed: 2026-08-06. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li

work page 2026
[8]

Recursive agent harnesses.arXiv preprint arXiv:2606.13643. 9 Xuying Ning, Katherine Tieu, Dongqi Fu, Tianxin Wei, Zihao Li, Yuanchen Bei, Jiaru Zou, Mengting Ai, Zhining Liu, Ting-Wei Li, Lingjie Chen, Yanjun Zhao, Ke Yang, Bingxuan Li, Cheng Qian, Gaotang Li, Xiao Lin, Zhichen Zeng, Ruizhong Qiu, and 23 others

work page Pith review arXiv
[9]

Sahil Sen, Akhil Kasturi, Elias Lumer, Anmol Gulati, and Vamse Kumar Subbiah

Code as agent harness.arXiv preprint arXiv:2605.18747. Sahil Sen, Akhil Kasturi, Elias Lumer, Anmol Gulati, and Vamse Kumar Subbiah. 2026a. Is grep all you need? How agent harnesses reshape agentic search. arXiv preprint arXiv:2605.15184. Sahil Sen, Elias Lumer, Anmol Gulati, and Vamse Ku- mar Subbiah. 2026b. Chronos: Temporal-aware conversational agents ...

arXiv
[10]

Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation

Benchmarking the benchmarks: A validity audit of tool-calling evalua- tion.arXiv preprint arXiv:2607.02577. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji

work page Pith review arXiv
[11]

InProceedings of the KDD 2026 Workshop on Software Engineering 3.0

When does restricting a coding agent to execute_code help? A regime × agent-design ablation. InProceedings of the KDD 2026 Workshop on Software Engineering 3.0. Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuan- jing Huang, and Jiecao Chen

work page 2026
[12]

Find the area of a trian- gle with a base of 10 units and height of 5 units

ToolHop: A query-driven benchmark for multi-hop tool use. In Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics, ACL 2025, pages 2995–3021. A Per-Category Accuracy on BFCL v4 CategorynJSON±PTC± simple_python40 86.1 10.789.59.7 multiple2095.411.1 93.9 11.9 parallel4085.510.9 71.4 13.5 par. multiple4085.510.9 71.4 13.5...

work page 2025

Show all 12 references

[2023]

InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pages 3102–3116

API-Bank: A comprehensive benchmark for tool-augmented LLMs. InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pages 3102–3116. Tong Liu, Cheng Qian, Matej Cief, Yuan He, Daniele Dan, Nikolaos Aletras, and Gabriella Kazai

2023
[2024]

InProceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics, ACL 2024, pages 12859–12870

API-BLEND: A comprehensive corpora for training and benchmark- ing API LLMs. InProceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics, ACL 2024, pages 12859–12870. Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Ji...

2024
[2025]

InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025, pages 7993–8006

CONFETTI: Conversational function-calling evaluation through turn-level interac- tions. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025, pages 7993–8006. Anthropic

2025
[2026]

Dongxin Guo, Jikun Wu, and Siu Ming Yiu

HyperTool: Beyond step-wise tool calls for tool-augmented agents.arXiv preprint arXiv:2606.13663. Dongxin Guo, Jikun Wu, and Siu Ming Yiu

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics, ACL 2024, pages 9510–9529

T-Eval: Evaluating the tool utilization capability of large language models step by step. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics, ACL 2024, pages 9510–9529. Cloudflare

work page 2024

[2] [5]

InFindings of the Association for Computational Linguistics: ACL 2024, pages 4363–4400

Planning, creation, usage: Benchmarking LLMs for comprehensive tool utiliza- tion in real-world complex scenarios. InFindings of the Association for Computational Linguistics: ACL 2024, pages 4363–4400. Hugging Face

work page 2024

[3] [6]

https://www

Prime Agent: A self-improving RLM agent. https://www. primeintellect.ai/blog/prime-agent. Ac- cessed: 2026-08-06. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li

work page 2026

[4] [8]

Recursive agent harnesses.arXiv preprint arXiv:2606.13643. 9 Xuying Ning, Katherine Tieu, Dongqi Fu, Tianxin Wei, Zihao Li, Yuanchen Bei, Jiaru Zou, Mengting Ai, Zhining Liu, Ting-Wei Li, Lingjie Chen, Yanjun Zhao, Ke Yang, Bingxuan Li, Cheng Qian, Gaotang Li, Xiao Lin, Zhichen Zeng, Ruizhong Qiu, and 23 others

work page Pith review arXiv

[5] [9]

Sahil Sen, Akhil Kasturi, Elias Lumer, Anmol Gulati, and Vamse Kumar Subbiah

Code as agent harness.arXiv preprint arXiv:2605.18747. Sahil Sen, Akhil Kasturi, Elias Lumer, Anmol Gulati, and Vamse Kumar Subbiah. 2026a. Is grep all you need? How agent harnesses reshape agentic search. arXiv preprint arXiv:2605.15184. Sahil Sen, Elias Lumer, Anmol Gulati, and Vamse Ku- mar Subbiah. 2026b. Chronos: Temporal-aware conversational agents ...

arXiv

[6] [10]

Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation

Benchmarking the benchmarks: A validity audit of tool-calling evalua- tion.arXiv preprint arXiv:2607.02577. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji

work page Pith review arXiv

[7] [11]

InProceedings of the KDD 2026 Workshop on Software Engineering 3.0

When does restricting a coding agent to execute_code help? A regime × agent-design ablation. InProceedings of the KDD 2026 Workshop on Software Engineering 3.0. Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuan- jing Huang, and Jiecao Chen

work page 2026

[8] [12]

Find the area of a trian- gle with a base of 10 units and height of 5 units

ToolHop: A query-driven benchmark for multi-hop tool use. In Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics, ACL 2025, pages 2995–3021. A Per-Category Accuracy on BFCL v4 CategorynJSON±PTC± simple_python40 86.1 10.789.59.7 multiple2095.411.1 93.9 11.9 parallel4085.510.9 71.4 13.5 par. multiple4085.510.9 71.4 13.5...

work page 2025