Pith.

Machine Learning · arXiv:2611.16304 v1 · submitted 12 Nov 2026 · 56 references

StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning

Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu

The paper says

“Reinforcement learning has improved the tool-use capabilities of large language models. However, better task performance can coincide with increased tool hallucination. Reliable tool use requires decisions to follow the current task state: available information, tool capabilities, and task progress.”

more of the abstract

Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information. We introduce StateAlign, combining task-state construction with state-contrastive reinforcement learning. The data cover five task conditions and pair states with different action requirements. Outcome rewards guide task completion; token-level contrasts compare the same generated reasoning and actions under paired states to train outputs that fit the current state. Experiments across five base models from three families show improvements on BFCL v4 multi-turn, ToolSandbox, and When2Call. Ablations show that state construction improves performance and reliability under different reward designs. The complete method further reduces tool hallucination across three Qwen3 sizes. On Qwen3-8B, StateAlign improves BFCL v4 multi-turn accuracy from 45.5% to 55.2% and reduces no-tool hallucination on SimpleToolHalluBench from 62.2% under outcome-only training to 56.1%. Behavioral analyses show more consistent decisions, more complete multi-call execution, and stronger recovery after tool errors.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. StateAlign is a solid, well-documented paper that introduces a genuinely new training signal—paired-state contrastive RL—for reducing tool hallucination in LLM agents. The main soft spot is the 70.4% raw agreement on condition labels, which the stress test correctly flags but the paper itself acknowledges only obliquely. This line is wrong

4 major objections · 8 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 56 references say.

The paper says, at §3.3, Eq. (2)-(3)

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The state-contrastive objective uses the base model π0 as a fixed scorer to compute token-level log-ratio weights r_{i,t}. This design assumes that π0's likelihoods under the paired contexts (h_i, z_i) and (h̃_i, z̃_i) provide a reliable signal for state compatibility.

more of objection 1

However, π0 is the same model that the paper diagnoses as failing to act appropriately in many cases (Section 2). If π0 itself has poor state awareness, the contrastive signal may be noisy or biased. The authors should provide analysis of the quality of π0's scoring, e.g., correlation with human judgments or with outcome rewards, to justify this design choice.

The paper says, at §4.2, Table 1

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

Most headline results are reported as single means without variance. Only BFCL in Table 2 includes standard deviations (from six evaluations). For ToolSandbox, When2Call, and SimpleToolHalluBench, the paper reports means over three evaluations but no variance. Given the modest improvements in some cases (e.g., Qwen3-14B BFCL 54.8 vs AWM-14B 55.5, within the standard deviation of ~1.5), it is unclear whether the claimed improvements are statistically significant.

more of objection 2

The authors should report variance or confidence intervals for all benchmarks, or at minimum state which differences exceed the measurement noise.

The paper says, at §4.3, Table 2B

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The ablation comparing 'Outcome only' vs 'StateAlign' on constructed states shows that adding the contrastive objective reduces hallucination. However, the 'Outcome only' baseline already uses the constructed states and the same outcome reward (Eq. 1). The improvement from StateAlign could come from the contrastive objective itself or from the fact that StateAlign uses a different training configuration (e.g., the 'successful-group share' of 20% reserved for paired successful trajectories, as noted in Appendix B).

more of objection 3

The authors should clarify whether the 'Outcome only' baseline also uses this group reservation or whether it is a pure GRPO without any pairing. If the latter, the comparison conflates the effect of the contrastive objective with the effect of the group selection strategy.

The paper says, at §5.1, Figure 4

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The qualitative example in Figure 4 is illustrative but the paper does not provide a quantitative analysis of how often the contrastive signal actually provides differentiated supervision when outcome rewards are tied. The authors mention that 'Mean first-response scores differ within every group despite identical outcome rewards' based on 50 groups, but no aggregate statistics (e.g., mean absolute deviation, proportion of tokens with non-zero scores) are reported.

more of objection 4

A more systematic analysis would strengthen the claim that the contrastive objective provides useful signal beyond outcome rewards.

8 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 8

The abstract states 'no-tool hallucination on SimpleToolHalluBench falls from 62.2% under outcome-only training to 56.1%'. It would be clearer to specify that this is the NTA (no-tool-available) setting, as the benchmark also has a DT (distractor-tool) setting.

The paper says, at §3.1, Table 5

Pith says: minor objection 2 of 8

Table 5 reports the distribution of task states in the training data. The 'Other' column for StateAlign is 0.0%, which is expected given the construction process. However, the caption says '— indicates that no such links are provided' for the Paired column; this is clear but could be stated more explicitly.

show 6 more

The paper says, at §4.1, Implementation Details

Pith says: minor objection 3 of 8

The paper states 'We train with LoRA on eight NVIDIA H100 80GB GPUs with λ=0.1'. The choice of λ=0.1 is not justified; a sensitivity analysis on this hyperparameter would be informative.

The paper says, at Appendix B, Table 6

Pith says: minor objection 4 of 8

The hyperparameter 'Successful-group share' is set to 20%. The paper should explain how this value was chosen and whether results are sensitive to it.

The paper says, at §5.2, Table 3

Pith says: minor objection 5 of 8

Table 3 reports 'Both correct' as the mean product of per-state action accuracies within each pair. This metric is not standard; the authors should clarify its interpretation and why it is preferred over simpler metrics like joint accuracy.

The paper says, at Appendix D.2

Pith says: minor objection 6 of 8

The annotation validation reports 70.4% raw agreement for condition judgments. The authors should discuss whether this level of agreement is sufficient for the claims made, especially given that the construction process uses a single model (Qwen3.8-27B) without human verification.

The paper says, at References

Pith says: minor objection 7 of 8

Several references are to 'Preprint' or 'arXiv' without indication of publication status. While this is acceptable for recent work, the authors should update references where possible (e.g., [Zhong et al., 2026] is marked as arXiv; [Zhao et al., 2026] is marked as ICML 2025 but the year in the citation is 2026).

The paper says, at Figure 1

Pith says: minor objection 8 of 8

Figure 1 plots BFCL v4 multi-turn accuracy against hallucination rates. The axes are not labeled with units (though percentages are implied). The figure would benefit from error bars or confidence regions.

What would settle it

Release the 30,433 constructed states with their condition and decision labels, then have three independent annotators re-label a stratified sample (at least 500 instances per condition, without re-weighting) using the Appendix F codebooks, and report raw per-condition agreement for INFEASIBLE and STOP. If agreement is high, retrain Qwen3-8B outcome-only and StateAlign on the subset of states whose labels are unanimously confirmed and re-run BFCL v4 multi-turn and SimpleToolHalluBench NTA/DT. If the NTA gap (62.2 vs 56.1) and the BFCL difference (53.0 vs 55.2) survive, the concern is resolved; if they shrink to noise, the central claim is not separable from construction-label noise.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 42 s · 21,839 tokens read, 13,140 written
  3. Editor wrote the verdict above 6 s · 22,747 tokens read, 520 written
  4. Referee major revision: 4 major objections, 8 minor 16 s · 22,212 tokens read, 1,967 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The paper's central idea—constructing paired task states and using a token-level contrastive reward to score the same generated tokens under two states with different valid actions—is genuinely new relative to the cited baselines. The five-condition state construction pipeline applied to ToolMind and LoopTool is also a real contribution. The empirical evaluation is broad: five base models from three families, multiple benchmarks, and controlled ablations that isolate state construction and the contrastive objective. The behavioral analyses (multi-call execution, error recovery) are a nice touch and give the reader confidence that the method changes behavior in the intended direction.

The load-bearing premise is that the 30k constructed instances faithfully encode the five task conditions. The paper reports 70.4% raw agreement with Claude Opus on condition judgments, and the stress test correctly notes that per-condition reliability for INFEASIBLE (6.5% of the pool) and STOP (11.8%) is unreported despite driving the hallucination claim. The authors acknowledge limitations on trajectory coverage but not this per-condition label reliability issue. That said, the 97.2% population-weighted figure suggests the low raw agreement is concentrated in rare classes, and the paper does report an additional 83.3% agreement check on 60 trained-model judgments. This is a soft spot, not a fatal flaw.

Other concerns: headline tables lack variance estimates for most benchmarks (only BFCL has standard deviations in Table 2), and the code and data are linked but not pinned to a commit. These are reporting issues, not demonstrated flaws.

Who this is for: researchers working on tool-use reliability for LLM agents, especially those interested in RL-based training signals beyond outcome-only rewards. The paper deserves a serious referee. I would recommend acceptance with requests for per-condition label reliability analysis, variance estimates for all benchmarks, and a pinned code release.

The strongest objection

The load-bearing premise is that the 30,433 constructed instances in Section 3.1 faithfully encode the five task conditions. This premise feeds every training signal: Eq. (1) gives zero reward for premature execution in NEED_USER/NEED_FETCH and terminal checks in INFEASIBLE/STOP, and Eqs. (2)-(3) weight tokens by log-ratios against reference paths z, z̃ that Qwen3.8-27B synthesizes. A wrong condition label can therefore contaminate both R_out and J_pair. The only independent check, Appendix A/D.2, reports 70.4% raw agreement with Claude Opus on condition judgments; the 97.2% population-weighted figure is concentrated in frequent classes and does not reveal whether INFEASIBLE (6.5% of the pool) and STOP (11.8%) are reliably labeled. These are exactly the classes behind SimpleToolHalluBench's NTA/DT metrics and When2Call infeasibility, so the claimed 62.2 -> 56.1 NTA reduction and the large Llama NTA drop could in part be artifacts of generator-specific label noise rather than state awareness. The paper acknowledges limitations on trajectory coverage, but not this per-condition label reliability issue.

I read the paper in good faith as a method paper whose central claim is conditional on constructed states faithfully representing real task states. The evaluation is broad, the ablations are well designed, and the overlap check against BFCL and When2Call is a genuine effort to control contamination. The reader's weakest assumption already targets construction quality, and I agree with that choice. I sharpen it in one respect: the aggregate 70.4% raw / 97.2% weighted agreement gap is not decisive by itself, but the absence of per-condition agreement for the rare conditions is important because the hallucination benchmarks test exactly those rare conditions. If INFEASIBLE and STOP labels are noisy, both the outcome reward and the state-contrast reward inherit that noise, so the central reduction in no-tool hallucination could reflect fitting the generator's label distribution rather than learning state awareness. This is a robustness concern, not a demonstrated flaw, and it can be settled by a stratified annotation audit plus a retraining check on validated states. I therefore keep the reader's CONDITIONAL verdict unchanged.

The referee's report

If the results hold, the paper makes a useful contribution by demonstrating that explicitly training on constructed task states with a contrastive objective can improve both task performance and reliability in multi-turn tool use. The approach is relatively simple and the empirical evaluation is broad across models and benchmarks. The behavioral analyses (Section 5) provide some insight into how the method changes agent behavior. However, the significance is somewhat tempered by the reliance on a single generator (Qwen3.8-27B) for state construction and the modest absolute gains on some metrics.

Pith's reading of the paper (a machine's summary, not the authors')

The paper tries to establish that training a tool-using language model with constructed task states and a token-level state-contrastive reinforcement-learning objective reduces the gap between what the model reasons it should do and what it actually does when information is missing, tools are unavailable, or clarification is needed. The authors argue that outcome-only RL can improve task completion while simultaneously increasing tool hallucination — the model calls a tool with fabricated arguments even when its own reasoning identifies the missing information. They propose StateAlign, which first converts tool-use trajectories into paired training states that differ in one condition (e.g. a required value present vs. missing), then combines a standard outcome reward for task completion with a token-level contrastive reward that compares the same generated reasoning and actions under both states. The contrastive reward penalizes tokens that assume availability of information or tools that the current state does not provide. Experiments on five base models from three families show consistent improvements on BFCL v4 multi-turn accuracy, ToolSandbox similarity, When2Call accuracy, and reductions in no-tool hallucination on SimpleToolHalluBench. On Qwen3-8B, BFCL v4 multi-turn accuracy rises from 45.5% to 55.2% and no-tool hallucination falls from 62.2% under outcome-only training to 56.1%. Behavioral analyses show more consistent decisions across task states, more complete multi-call execution, and stronger recovery after tool errors. If the paper is right, then a relatively lightweight data-construction and training procedure can make tool-using LLMs substantially more reliable without sacrificing task performance — a finding that matters for deploying agents in settings where hallucinated tool calls waste resources or cause harm.

What carries the argument. The key machinery is the combination of (1) a task-state construction pipeline that converts tool-use trajectories into five types of states (EXEC, NEED_USER, NEED_FETCH, INFEASIBLE, STOP) with explicit pairs differing in one condition, and (2) a token-level state-contrastive reinforcement-learning objective J_pair that compares the same generated reasoning and actions under paired states using a fixed base model as scorer. The contrastive score for each token is the clipped log-probability ratio rt = clip(log π0(yt | h, z, p) - log π0(yt | h̃, z̃, p), -1, 1), where π0 is the frozen base model, h and h̃ are the paired contexts, z and z̃ are reference paths describing appropriate actions, and p is the shared prefix. Positive scores favor tokens that fit the current state; negative scores favor tokens that fit the paired state. J_pair weights the policy's token log-likelihoods by these scores, providing fine-grained supervision beyond outcome rewards. The outcome objective J_out is a token-normalized GRPO objective on interaction outcomes, with rewards that penalize extra calls and require appropriate continuation after state changes. The combined objective J = J_out + λ J_pair is maximized with λ = 0.1.

Core claim. The paper claims that a reinforcement-learning agent trained on tool-use tasks can simultaneously improve task completion and reduce tool hallucination if it is trained with explicit awareness of the current task state — what information is available, what tools are present, and what remains to be done. The central discovery is that outcome-only RL (rewarding only final task success) is insufficient because it can reward outputs that happen to succeed while containing reasoning or actions that assume information the state does not provide. StateAlign addresses this by constructing, from each source trajectory, five types of task states (EXEC, NEED_USER, NEED_FETCH, INFEASIBLE, STOP) that differ in a single condition, and pairing each non-EXEC state with an EXEC counterpart. The model is then trained with a combined objective J = J_out + λ J_pair, where J_out is a token-normalized GRPO objective based on interaction outcomes, and J_pair is a token-level contrastive objective that uses a fixed base model to score each generated token under both the current state and its paired state, giving positive weight to tokens that fit the current state and negative weight to tokens that fit the paired state. This contrastive signal provides supervision even when outcome rewards are tied (e.g. all rollouts in a group succeed or all fail). The paper reports that across five base models (Qwen3-4B/8B/14B, Granite-4.2-8B, Llama-3.1-8B-Instruct), StateAlign improves every BFCL v4 category and When2Call accuracy and reduces no-tool hallucination, with the largest gains on Llama-3.1-8B-Instruct (BFCL multi-turn from 8.0% to 26.1%, NTA hallucination from 72.6% to 15.7%).

The premise it rests on. The load-bearing premise is that the Qwen3.8-27B-generated state construction produces training states that faithfully represent real task states, with no leaked values, no unrealistic retrieval tools, and correct action labels — the paper acknowledges this by reporting validation checks (70.4% agreement with Claude Opus on condition judgments, 90.0% on decisions) but if construction quality is lower in distribution than the validation sample, the outcome and contrastive rewards would encode artifacts rather than state awareness, weakening all downstream benchmark claims.

If the paper is right (Pith's inferences, not the authors')

If StateAlign generalizes, then training tool-using LLMs with state-contrastive rewards could become a standard component of agentic RL pipelines, replacing or supplementing outcome-only rewards to reduce hallucination without sacrificing task performance.

The method's reliance on a fixed base model as scorer means that the contrastive signal is available from the start of training, potentially accelerating learning compared with methods that require a learned reward model or environment dynamics.

The explicit construction of five task states (EXEC, NEED_USER, NEED_FETCH, INFEASIBLE, STOP) provides a template for data augmentation in tool-use training that other researchers can adopt and extend to new conditions (e.g. ambiguous requests, multiple missing values, tool errors).

The behavioral improvements — more consistent decisions, more complete multi-call execution, and stronger recovery after tool errors — suggest that state-aware training may reduce the need for hand-crafted fallback logic in deployed agents.

The paper's finding that outcome-only RL can increase hallucination while improving task performance implies that benchmark scores alone are insufficient for evaluating tool-use agents; reliability metrics (e.g. hallucination rates, clarification rates) should be reported alongside accuracy.

StateAlign's approach of contrasting paired states could be extended to other agentic domains beyond tool use, such as web navigation or robotics, where the distinction between a state that permits execution and one that requires information-gathering or declining is equally critical.

The paper's use of a fixed base model as scorer (rather than a learned reward model) suggests a family of methods where a weaker or frozen model provides contrastive supervision to a stronger policy — a form of self-distillation that may be applicable to other RL-from-human-feedback settings.

The five task-state categories are defined by the paper's construction pipeline, but the underlying principle — that an agent should be trained to distinguish states that differ in a single relevant condition — could be formalized as a general inductive bias for reliable agentic behavior.

The paper's observation that models can reason correctly but act incorrectly (e.g. state in reasoning that a value is missing, then call a tool with a fabricated value) points to a decoupling between reasoning and action that state-contrastive training directly addresses; this decoupling may be a general phenomenon in LLM agents that warrants further study.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Reinforcement learning has improved the tool-use capabilities of large language models. However, better task performance can coincide with increased tool hallucination. Reliable tool use requires decisions to follow the current task state: available information, tool capabilities, and task progress. Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information. We introduce StateAlign, combining task-state construction with state-contrastive reinforcement learning. The data cover five task conditions and pair states with different action requirements. Outcome rewards guide task completion; token-level contrasts compare the same generated reasoning and actions under paired states to train outputs that fit the current state. Experiments across five base models from three families show improvements on BFCL v4 multi-turn, ToolSandbox, and When2Call. Ablations show that state construction improves performance and reliability under different reward designs. The complete method further reduces tool hallucination across three Qwen3 sizes. On Qwen3-8B, StateAlign improves BFCL v4 multi-turn accuracy from 45.5% to 55.2% and reduces no-tool hallucination on SimpleToolHalluBench from 62.2% under outcome-only training to 56.1%. Behavioral analyses show more consistent decisions, more complete multi-call execution, and stronger recovery after tool errors.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 0.34

“Experiments across five base models from three families show improvements on BFCL v4 multi-turn, ToolSandbox, and When2Call.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.17 · human participants 0.03 · controlled comparison 0.93 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.95 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Reinforcement learning has improved the tool-use capabilities of large language models. background 1.00 0.05 0.02 asserted 1.00
However, better task performance can coincide with increased tool hallucination. background 0.79 0.14 0.03 asserted 0.50
Reliable tool use requires decisions to follow the current task state: available information, tool capabilities, and task progress. background 0.99 0.07 0.08 asserted 1.00
Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information. result 0.63 0.91 0.07 asserted 0.97
We introduce StateAlign, combining task-state construction with state-contrastive reinforcement learning. contribution 0.83 0.87 0.09 asserted 0.98
The data cover five task conditions and pair states with different action requirements. method 0.99 0.84 0.02 asserted 0.99
Outcome rewards guide task completion; token-level contrasts compare the same generated reasoning and actions under paired states to train outputs that fit the current state. method 1.00 0.87 0.05 asserted 0.96
Experiments across five base models from three families show improvements on BFCL v4 multi-turn, ToolSandbox, and When2Call. result 1.00 0.95 0.20 asserted 1.00
Ablations show that state construction improves performance and reliability under different reward designs. result 0.99 0.95 0.08 asserted 1.00
The complete method further reduces tool hallucination across three Qwen3 sizes. result 1.00 0.94 0.25 asserted 1.00
On Qwen3-8B, StateAlign improves BFCL v4 multi-turn accuracy from 45.5% to 55.2% and reduces no-tool hallucination on SimpleToolHalluBench from 62.2% under outcome-only training to 56.1%. result 1.00 0.95 0.18 asserted 1.00
Behavioral analyses show more consistent decisions, more complete multi-call execution, and stronger recovery after tool errors. result 0.91 0.94 0.19 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 8,734 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (10)
Figure 1 of the paper
Figure 1. Comparison of tool-use performance and re
Figure 2 of the paper
Figure 2. Task-state diagnosis on Qwen3-8B. (a) Each group pairs a state in which calling the target function is warranted with one that differs in a single condition; bars give the share of responses. (b) Response excerpts when a required phone number is missing. Blue shading marks the decision to ask; red shading marks the unpro￾vided value in the emitted call. • Experiments across five base models show improved performance on BFCL v4 multi￾turn, ToolSandbox, and When2Call. Behav￾ioral analyses show more consistent deci￾sions across task states, more complete multi￾call execution, and…
Figure 3 of the paper
Figure 3. Overview of StateAlign on an example task. (1) Editing one condition of a step from a source trajectory
Figure 4 of the paper
Figure 4. State-contrast scores for two successful rollouts when the image ID is missing; the paired state provides it.
Figure 5 of the paper
Figure 5. Multi-step execution and recovery on Qwen3-8B.
Figure 6 of the paper
Figure 6. Excerpts of a recorded state-contrast scoring pair. The tool definition and policy-generated prefix are
Figure 7 of the paper
Figure 7. Training dynamics on Qwen3-8B. (a) Mean outcome reward for Outcome only and StateAlign on
Figure 8 of the paper
Figure 8. System prompt for filtering tool-use demonstrations. The input contains the available tools, conversation,
Figure 9 of the paper
Figure 9. System prompt for constructing missing-information states. The input specifies the user message, target
Figure 10 of the paper
Figure 10. System prompt for constructing retrieval states. The input supplies the existing tool names, user message,

Figures are the authors', reproduced from the paper.

References (56)
  1. [1] Zhou, Yifei and Zanette, Andrea and Pan, Jiayi and Levine, Sergey and Kumar, Aviral , booktitle =. 2024 , editor = on Pith
  2. [3] Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =. 2023 , url = on Pith
  3. [7] Qin, Yujia and Liang, Shihao and Ye, Yining and Zhu, Kunlun and Yan, Lan and Lu, Yaxi and Lin, Yankai and Cong, Xin and Tang, Xiangru and Qian, Bill and Zhao, Sihan and Hong, Lauren and Tian, Runchu and Xie, Ruobing and Zhou, Jie and Gerstein, Mark and Li, Dahai and Liu, Zhiyu... on Pith
  4. [9] Liu, Weiwen and Huang, Xu and Zeng, Xingshan and Hao, Xinlong and Yu, Shuai and Li, Dexun and Wang, Shuai and Gan, Weinan and Liu, Zhengying and Yu, Yuanqing and Wang, Zezhong and Wang, Yuxian and Ning, Wu and Hou, Yutai and Wang, Bin and Wu, Chuhan and Wang, Xinzhi and Liu, Y... on Pith
  5. [11] Shaokun Zhang and Yi Dong and Jieyu Zhang and Jan Kautz and Bryan Catanzaro and Andrew Tao and Qingyun Wu and Zhiding Yu and Guilin Liu , booktitle = on Pith
  6. [12] Feng, Jiazhan and Huang, Shijue and Qu, Xingwei and Zhang, Ge and Qin, Yujia and Zhong, Baoquan and Jiang, Chengquan and Chi, Jinxin and Zhong, Wanjun , booktitle = on Pith
  7. [19] Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following , author =. 2026 , eprint = on Pith
  8. [20] Kong, Fanqi and Zhang, Jiayi and Deng, Mingyi and Wu, Chenglin and Luo, Yuyu and Liu, Bang , booktitle =. 2026 , editor = on Pith
  9. [21] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models , author =. Proceedings of the 43rd International Conference on Machine Learning , pages =. 2026 , editor = on Pith
  10. [22] Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning , author=. Proceedings of the 43rd International Conference on Machine Learning , year= on Pith
  11. [23] Patil, Shishir G and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E. , booktitle=. The Berkeley Function Calling Leaderboard (. 2025 , volume= on Pith
  12. [26] Zeng, Xingshan and Liu, Weiwen and Wang, Lingzhi and Li, Liangyou and Mi, Fei and Wang, Yasheng and Shang, Lifeng and Jiang, Xin and Liu, Qun , booktitle =. 2026 , url = on Pith
  13. [27] Reducing Tool Hallucination via Reliability Alignment , author =. Proceedings of the 42nd International Conference on Machine Learning , pages =. 2025 , volume = on Pith
  14. [28] An Yang and Anfeng Li and Baosong Yang and Beichen Zhang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Gao and Chengen Huang and Chenxu Lv and Chujie Zheng and Dayiheng Liu and Fan Zhou and Fei Huang and Feng Hu and Hao Ge and Haoran Wei and Huan Lin and Jialong Tang and... on Pith
  15. [29] Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Yang, Amy and Fan, Angela and Goyal, Anirudh and Hartshorn, Anthony and Yang, Aobo and Mitra, Archi and Sravankumar, Ar... on Pith
  16. [30] Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning , author =. 2026 , eprint = on Pith
  17. [35] Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle =. 2022 , url = on Pith
  18. [36] Peng, Bowen and Quesnelle, Jeffrey and Fan, Honglu and Shippole, Enrico , booktitle =. 2024 , url = on Pith
  19. [38] Emre Can Acikgoz, Jeremiah Greer, Akul Datta, Ze Yang, William Zeng, Oussama Elachqar, Emmanouil Koukoumidis, Dilek Hakkani-T \"u r, and Gokhan Tur. 2025. https://doi.org/10.18653/v1/2025.acl-long.605 Can a single model master both multi-turn conversations and tool use? C o AL... on Pithdoi:10.18653/v1/2025.acl-long.605
  20. [39] Yun-Shiuan Chuang, Chaitanya Kulkarni, Alec M. Chiu, Avinash Thangali, Zijie Pan, Shivani Shekhar, Yirou Ge, Yixi Li, Uma Kona, Linsey Pang, and Prakhar Mehrotra. 2026. https://doi.org/10.18653/v1/2026.acl-industry.87 Toward scalable verifiable reward: Proxy state-based evalua... on Pithdoi:10.18653/v1/2026.acl-industry.87
show 36 more
  1. [40] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha... on PithPith's reviewarXiv
  2. [41] Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2026. https://proceedings.iclr.cc/paper_files/paper/2026/file/4038c9208dfc22644c60ad39c24e5c53-Paper-Conference.pdf ReTool : Reinforcement learning for st... on Pith
  3. [42] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. 2025. https://doi.org/10.52202/085713-1544 Group-in-group policy optimization for LLM agent training . In Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 46375--46408. Curran Associates, Inc on Pithdoi:10.52202/085713-1544
  4. [43] Rongfeng Guo, Yinxuan Huang, Yusen Wu, Maoqing Zhong, Yunlu Chen, Meng Tang, Teng Long, and Vincent Tao Hu. 2026. https://arxiv.org/abs/2608.24368 From state to action: OODA-Tool for reliable multi-turn tool use . Preprint, arXiv:2608.24368 on PithPith's reviewarXiv
  5. [44] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 LoRA : Low-rank adaptation of large language models . In International Conference on Learning Representations on Pith
  6. [45] IBM Research . 2026. https://huggingface.co/blog/ibm-granite/granite-4-2 Granite 4.2 language models . Accessed: 2026-10-18 on Pith
  7. [46] Fanqi Kong, Jiayi Zhang, Mingyi Deng, Chenglin Wu, Yuyu Luo, and Bang Liu. 2026. https://proceedings.mlr.press/v306/kong26g.html I nfo PO : Information-driven policy optimization for user-centric agents . In Proceedings of the 43rd International Conference on Machine Learning,... on Pith
  8. [47] Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong Wang, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang, Yong Liu, Yasheng Wang, and 8 others. 2025. https://proceedings.iclr.cc/pa... on Pith
  9. [48] Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. 2024. https://doi.org/10.52202/079017-1... on Pithdoi:10.52202/079017-1725
  10. [49] Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. 2025. https://doi.org/10.18653/v1/2025.findings-naacl.65 ToolSandbox : A stateful, conversational, interactive evaluation be... on Pithdoi:10.18653/v1/2025.findings-naacl.65
  11. [50] Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. https://proceedings.mlr.press/v267/patil25a.html The berkeley function calling leaderboard ( BFCL ): From tool use to agentic evaluation of large language m... on Pith
  12. [51] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. https://openreview.net/forum?id=wHBfxhZu1u YaRN : Efficient context window extension of large language models . In The Twelfth International Conference on Learning Representations on Pith
  13. [52] Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Manoj Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Weiran Yao, Huan Wang, Silvio Savarese, and Caiming Xiong. 2025. https://doi.org/10.52202/085713-2184 APIGen-... on Pithdoi:10.52202/085713-2184
  14. [53] Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tur, Gokhan Tur, and Heng Ji. 2025. https://doi.org/10.52202/085713-3524 ToolRL : Reward is all tool learning needs . In Advances in Neural Information Processing Systems, volume 38, Main Conference, p... on Pithdoi:10.52202/085713-3524
  15. [54] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. https://proceedings.iclr.cc/paper_files/pa... on Pith
  16. [55] Hayley Ross, Ameya Sunil Mahabaleshwarkar, and Yoshi Suhara. 2025. https://doi.org/10.18653/v1/2025.naacl-long.174 When2Call : When (not) to call tools . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisti... on Pithdoi:10.18653/v1/2025.naacl-long.174
  17. [56] Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. https://doi.org/10.52202/075280-2997 Toolformer: Language models can teach themselves to use tools . In Advances in Neural Inf... on Pithdoi:10.52202/075280-2997
  18. [57] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. https://arxiv.org/abs/2402.03300 DeepSeekMath : Pushing the limits of mathematical reasoning in open language models . Preprint, arXiv:2402.03300 on PithPith's reviewarXiv
  19. [58] Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. 2025. https://arxiv.org/abs/2505.01441 Agentic reasoning and tool integration for LLMs via reinforcement learning . Preprint, arXiv:2505.01441 on PithPith's reviewarXiv
  20. [59] Zhaoyang Wang, Canwen Xu, Boyi Liu, Yite Wang, Siwei Han, Zhewei Yao, Huaxiu Yao, and Yuxiong He. 2026 a . https://proceedings.mlr.press/v306/wang26jh.html Agent world model: Infinity synthetic environments for agentic reinforcement learning . In Proceedings of the 43rd Intern... on Pith
  21. [60] Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Pei Chen, Ziwei Dong, Jing Huang, Jiri Gesi, Xianfeng Tang, Chen Luo, Qun Liu, Yisi Sang, Hanqing Lu, Manling Li, Jin Lai, and Dakuo Wang. 2026 b . https://doi.org/10.18653/v1/2026.acl-long.2037 Trajectory2Task : Training robust tool-calling... on Pithdoi:10.18653/v1/2026.acl-long.2037
  22. [61] Hongshen Xu, Zichen Zhu, Lei Pan, Zihan Wang, Su Zhu, Da Ma, Ruisheng Cao, Lu Chen, and Kai Yu. 2025. https://proceedings.mlr.press/v267/xu25ap.html Reducing tool hallucination via reliability alignment . In Proceedings of the 42nd International Conference on Machine Learning,... on Pith
  23. [62] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025 a . https://arxiv.org/abs/2505.09388... on PithPith's reviewarXiv
  24. [63] Chen Yang, Ran Le, Yun Xing, Zhenwei An, Zongchao Chen, Wayne Xin Zhao, Yang Song, and Tao Zhang. 2025 b . https://arxiv.org/abs/2511.15718 ToolMind technical report: A large-scale, reasoning-enhanced tool-use dataset . Preprint, arXiv:2511.15718 on PitharXiv
  25. [64] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. https://openreview.net/forum?id=WE_vluYUL-X ReAct : Synergizing reasoning and acting in language models . In International Conference on Learning Representations on Pith
  26. [65] Chenlong Yin, Zeyang Sha, Shiwen Cui, Changhua Meng, and Zechao Li. 2026. https://doi.org/10.18653/v1/2026.acl-long.376 The reasoning trap: How enhancing LLM reasoning amplifies tool hallucination . In Proceedings of the 64th Annual Meeting of the Association for Computational... on Pithdoi:10.18653/v1/2026.acl-long.376
  27. [66] Zhenrui Yue, Huimin Zeng, Yueqi Wang, Yaokun Liu, Fengran Mo, Jinghan Zhang, Mung Yao Jia, Gyuseok Lee, Yang Zhang, Na Wei, and Dong Wang. 2026. https://arxiv.org/abs/2609.36742 SIPO : Unifying reinforcement learning with on-policy self-distillation . Preprint, arXiv:2609.36742 on PithPith's reviewarXiv
  28. [67] Xingshan Zeng, Weiwen Liu, Lingzhi Wang, Liangyou Li, Fei Mi, Yasheng Wang, Lifeng Shang, Xin Jiang, and Qun Liu. 2026. https://openreview.net/forum?id=KznJt9Fhjc ToolACE-MT : Non-autoregressive generation for agentic multi-turn interaction . In International Conference on Lea... on Pith
  29. [68] Kangning Zhang, Wenxiang Jiao, Kounianhua Du, Yong Yu, Yuan Lu, Weiwen Liu, and Weinan Zhang. 2026 a . https://doi.org/10.18653/v1/2026.acl-long.968 L oop T ool: Closing the data -- training loop for robust LLM tool calls . In Proceedings of the 64th Annual Meeting of the A ss... on Pithdoi:10.18653/v1/2026.acl-long.968
  30. [69] Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. 2026 b . https://arxiv.org/abs/2505.00024 Nemotron-Research-Tool-N1 : Exploring tool-using language models with reinforced reasoning . In International Conferen... on PithPith's reviewarXiv
  31. [70] Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. 2026. https://proceedings.mlr.press/v306/zhao26be.html Self-distilled reasoner: On-policy self-distillation for large language models . In Proceedings of the 43rd International Conferen... on Pith
  32. [71] Weikang Zhao, Xili Wang, Chengdi Ma, Lingbin Kong, Zhaohua Yang, Mingxiang Tuo, Xiaowei Shi, Yitao Zhai, and Xunliang Cai. 2025. https://arxiv.org/abs/2508.18669 MUA-RL : Multi-turn user-interacting agent reinforcement learning for agentic tool use . Preprint, arXiv:2508.18669 on PithPith's reviewarXiv
  33. [72] Yanzhao Zheng, Yuanqiang Yu, Tianze Xu, Chao Ma, Zhentao Zhang, Jihuai Zhu, Baohua Dong, Hangcheng Zhu, and Ruohui Huang. 2026. https://arxiv.org/abs/2609.27421 Counterfactual constraint-conditioned on-policy distillation for multi-constraint instruction following . Preprint, ... on PithPith's reviewarXiv
  34. [73] Haitian Zhong, Jixiu Zhai, Lei Song, Jiang Bian, Qiang Liu, and Tieniu Tan. 2026. https://arxiv.org/abs/2602.03025 RC-GRPO : Reward-conditioned group relative policy optimization for multi-turn tool calling agents . Preprint, arXiv:2602.03025 on PitharXiv
  35. [74] Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. 2024. https://proceedings.mlr.press/v235/zhou24t.html A r CH er: Training language model agents via hierarchical multi-turn RL . In Proceedings of the 41st International Conference on Machine Learning, vol... on Pith
  36. [75] Yijin Zhou, Linqian Zeng, Xiaoya Lu, Wenyuan Xie, Dongrui Liu, Junchi Yan, and Jing Shao. 2026. https://arxiv.org/abs/2606.06976 Exploring agentic tool-calling decisions via uncertainty-aligned reinforcement learning . Preprint, arXiv:2606.06976. Accepted to EMNLP 2026 on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment