{"id":"8bfbc3c4-ed2b-4688-8b45-b82ed880f8b8","arxiv_id":"2608.02110","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A belief-state context manager trained with reinforcement learning and three auxiliary losses improves LLM tool invocation under dynamic user intent changes, per a new synthetic benchmark and two external benchmarks.","lead":"This paper introduces IACM-RL, a training framework that gives an LLM an explicit context manager to track shifting user goals and avoid reusing outdated parameters in multi-turn tool calls. It also builds a synthetic benchmark of 13 intent-fluctuation scenarios and reports gains over seven baselines on internal and two external benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RL policy is trained and evaluated entirely against a forgiving LLM simulator; real-backend validation covers only SFT, so the central loop/stale-error gains may not transfer.","rationale":"The reader's weakest assumption is that simulated tool execution is a faithful proxy for real backends under RL, and this is exactly the most load-bearing concern. The method's stated motivation is reliable deployment in real-world environments, and the headline improvements (loop suppression, stale-flag tracking) are learned entirely against an LLM simulator whose failure behavior is more forgiving than real APIs. The DynamicMockTool prompt in Appendix G.1 concretely specifies that missing parameters, type mismatches, and invalid enum values still produce normal, realistic responses, so the policy can learn that malformed calls succeed. Table 11's real-versus-simulated validation applies only to SFT models, not to the RL policy that uses the CM auxiliary losses and hierarchical reward; thus it does not test the learned behavior that the paper claims. Other concerns, such as metric circularity between reward and evaluation, affect all baselines equally and are partially mitigated by external benchmarks, whereas the simulation gap directly threatens the validity of the central claim for actual tool invocation. The recommended verdict remains CONDITIONAL, matching the reader's assessment; the condition is that the RL policy must be validated against real tool execution before the central claim can be fully accepted.","tokens_in":33480,"tokens_out":4300,"duration_ms":41662,"concrete_test":"Take the Stage-2 RL checkpoint (IACM-RL) and evaluate it on a held-out set of DynamicIntent trajectories (or the tau2-Bench live environment) with real MCP tool execution instead of MockBackend, computing the five metrics (JGA, SCRR, RIR, DTCR, ISSR) and comparing to the simulated-return numbers in Table 4. Track the fraction of tool calls that would have errored under the real API but received plausible simulated returns; if RIR or SCRR degrades by more than a few points, the central claim is not established for real backends.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that IACM-RL reduces infinite loops and stale-context errors under dynamic intent fluctuations is only as strong as the environment in which the RL policy was trained and evaluated. Algorithm 1 runs every PPO rollout against the LLM MockBackend, and the DynamicMockTool fallback prompt (Appendix G.1) explicitly instructs the simulator to return a 'normal, realistic tool call result' for missing required parameters, type mismatches, format errors, and invalid enum values; only nonexistent tools and undefined parameters produce errors. This means the RL policy is optimized in a world where malformed calls do not fail. The only real-backend comparison, Table 11, is conducted with SFT-only models (no RL), so it does not validate the learned stale-flag and loop-suppression behavior. Appendix F concedes that transfer to non-text or non-simulated settings is open. If real APIs reject the kinds of calls that the simulator rewards, the claimed loop reductions and stale-flag gains could vanish or even invert.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes IACM-RL, a two-stage PPO framework for tool invocation under dynamic intent fluctuations. The contributions are: (i) the DynamicIntent dataset/benchmark with 13 synthetic fluctuation scenarios and human-annotated ID/OOD test sets; (ii) a five-metric diagnostic suite (JGA, SCRR, RIR, DTCR, ISSR); (iii) a self-generated BeliefState Context Manager with structural stale flags; and (iv) a hierarchical intent-driven reward with three auxiliary consistency losses. Experiments on DynamicIntent, BFCL-V3, and τ2-Bench report that IACM-RL attains the highest average score (64.0), with strongest gains on OOD Cognitive, τ2-Bench airline, and long-dialogue state tracking.","tokens_in":33701,"tokens_out":9787,"duration_ms":82639,"significance":"The contribution is potentially useful: the synthetic data pipeline and metric suite fill a real gap, and the ablations cover auxiliary losses, reward layers, CM update policies, backbone scale, and seed variance for the main model. The central idea of decoupling state tracking from action generation via a self-generated BeliefState is clearly presented and plausibly effective. However, three methodological gaps currently limit confidence in the headline claims: the RL reward is built from the same five metrics used for evaluation on DynamicIntent, the RL policy is trained and evaluated only against LLM-simulated tool responses, and the main tables report single runs without baseline error bars. These issues are fixable and do not, in my assessment, invalidate the core method.","major_comments":[{"comment":"The hierarchical reward R_total is a weighted aggregate of JGA, SCRR, RIR, DTCR, and ISSR, and the DynamicIntent evaluation composites Cognitive, Behavioral, and Outcome are exactly these same five metrics. As a result, the reported reductions in stale-context errors and loops on DynamicIntent are partly a consequence of optimizing the evaluation objective itself. I recommend evaluating on held-out metrics not used in the reward, or adding an explicit reward-ablated control that separates optimization target from diagnostic.","section":"Section 4.3, Eq. (7) vs Section 3.3, Eqs. (1)-(5)"},{"comment":"All PPO rollouts execute tool calls through the LLM MockBackend, and the DynamicMockTool fallback returns normal realistic results for missing required parameters, type mismatches, format errors, and invalid enum values; only nonexistent tools and undefined parameters produce errors. The only real-backend validation (Table 11) concerns SFT-only models, not the RL policy, and Appendix F concedes that transfer to non-simulated settings is open. The central loop/stale-error gains may therefore not transfer to real APIs; please add a real-backend evaluation of the final RL policy on at least a subset, or explicitly limit the claims to simulated environments.","section":"Algorithm 1 and Appendix G.1"},{"comment":"The RIR definition is ambiguous. If equality of a_t and a_{t−Δ} is at the level of tool name, then a correct modification that re-invokes the same tool with updated arguments (Figure 10, Turn 2) is counted as a redundant or infinite loop; if equality is at the level of the full call, then looping calls that vary arguments escape detection. A precise definition of 'repeated tool call' and a dedicated infinite-loop metric are needed, since RIR underlies the Behavioral score and the loop-reduction claim.","section":"Section 3.3, Eq. (3)"},{"comment":"Seed variance is reported only for IACM-RL, not for any baseline, and most tables show single runs. Several headline gaps are small (e.g., BFCL 63.3 vs 63.2; MEM1 Avg 62.6 vs 64.0), so single-run comparisons do not support the 'highest overall average' claim. Please provide multi-seed means with standard deviations, or paired bootstrap intervals across dialogs, for all methods.","section":"Tables 4-12 and Appendix B.1"},{"comment":"The 'Format penalty only' row reports a τ2-Bench Airline value of 65.0, which exceeds the full model's 38.0 and is inconsistent with the text's claim that removing all reward layers harms performance. If 65.0 is not a column misalignment (i.e., if BFCL is actually 28.1), the table layout should be corrected; either way, the reward-layer ablation conclusions need to be re-verified.","section":"Table 9"}],"minor_comments":[{"comment":"The acronym UTAL is used without expansion; please define it at first use.","section":"Section 3.1"},{"comment":"The 'Avg' column is an unweighted mean over nine heterogeneous values (composite scores, benchmark accuracies, and τ2-Bench rewards); please justify this aggregation, since the headline 'highest overall average' depends on the chosen weighting.","section":"Section 5.2"},{"comment":"The statement that LLM simulation is a 'faithful proxy' is stronger than the reported Retail gap (18.4 vs 14.9) supports; please add a significance test or soften the wording.","section":"Table 11"},{"comment":"The attention visualizations are qualitative; a quantitative measure of attention concentration on current-goal tokens would better support the attention-dilution mechanism.","section":"Figures 6 and 12"},{"comment":"The evaluation threshold τ for the ISSR judge is never given a numeric value, although the reward-side threshold is stated as 0.3 in Section 4.3; please specify the evaluation threshold.","section":"Section 3.3, Eq. (5)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's dynamic-intent claims rest on an in-house benchmark whose metrics are also the training reward; combined with the simulation-only RL rollouts, this makes the headline numbers hard to interpret as evidence about real-world tool invocation. I would ask for a real-backend RL spot-check and a non-circular evaluation before acceptance, but the issues are addressable within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a serious piece of engineering with a genuinely reusable recipe: a self-generated BeliefState context manager with stale flags, a hierarchical reward, and three auxiliary losses, all trained in two PPO stages. Second, its central evidence is entirely inside a simulated world. The RL policy never touches real tool execution, so the headline gains on loops and stale-context errors are not yet validated outside that world.\n\nThe genuinely new parts are the DynamicIntent data pipeline with 13 fluctuation scenarios and the five-metric diagnostic suite, plus the CM design that explicitly marks overwritten parameters. That is a real gap in the tool-calling literature, and the paper addresses it with a lot of care: the graph-constrained splicing, the two-tier human annotation for test sets, and the ablations in Tables 5, 6, and 13 are informative. The authors also report seed variance and auxiliary-weight sensitivity, which is more than many papers do. What the paper claims internally is supported by its own setup.\n\nThe soft spots are real but mostly fixable. The most important is the one flagged in the stress test: Algorithm 1 rolls out every PPO iteration against an LLM MockBackend, and the DynamicMockTool fallback explicitly instructs the simulator to return normal results for missing parameters, type mismatches, and format errors. That means the RL policy is optimized in a world where malformed calls rarely fail. The only real-vs-simulated check, Table 11, is SFT-only, so it does not validate the learned stale-flag or loop-suppression behavior. That is a load-bearing gap, because the key claim is about reducing infinite loops, and the simulator is unusually forgiving.\n\nAlso softer: the hierarchical reward is assembled from the same five metrics used for evaluation, so part of the benchmark gain is optimizing the evaluator itself. That is not disqualifying, because the reward uses different shaping variants, but it does weaken the diagnostic metrics as an independent check. The external BFCL gain is just 0.2 points, and no error bars are reported for baselines. The authors do not release artifacts, which limits independent verification.\n\nOn the citation pattern and internal consistency: the paper is honest with its own limitations, and the related work is broad. I see no circularity or internal contradiction beyond the reward-metric overlap I mentioned.\n\nWho is this for? Someone working on tool-calling agents or context management in LLM post-training will get a clear, well-specified recipe and a stress-test benchmark. It deserves a serious referee. My recommendation: send it to peer review, but require the authors to validate the RL policy on real backends, report baseline error bars, and ideally release the data or a subset. Without that, the core transfer claim remains unproven.","headline":"A well-specified RL recipe for intent-fluctuation robustness that deserves referee time, though its central transfer claim is only as strong as the LLM-simulated training environment.","tokens_in":34315,"tokens_out":1349,"would_cite":true,"duration_ms":12000,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that decoupling state tracking from action generation, via a self-generated BeliefState Context Manager with structural stale flags and a hierarchical reward plus auxiliary losses, is what makes tool-invoking agents…","keywords":["intent fluctuation","tool invocation","belief state","context management","reinforcement learning","stale flag","dynamic intent benchmark","multi-turn agents"],"falsifier":"Take the trained RL policy (not an SFT variant) and run it on the same DynamicIntent test scenarios with real tool backends instead of the LLM mock, then measure the cognitive score, stale-context residual rate, and infinite-loop rate against PPO-noCM. If the margin over raw-history RL shrinks substantially or reverses once tool returns are real, the claim that the learned stale-flag and loop-suppression behavior transfers is falsified.","tokens_in":33219,"feed_emoji":"🤖","tokens_out":7874,"duration_ms":63270,"temperature":0.7,"pith_summary":"The paper is trying to establish that the main failure mode of long-horizon tool-calling agents under changing user intent—stale parameters, missed goal switches, and infinite API loops—comes from forcing the policy to re-derive state from a long raw history, and that this can be removed by having the agent generate its own compact belief-state block before each action. To test this, the authors build the DynamicIntent pipeline, producing thousands of trajectories across 13 intent-fluctuation scenarios, and pair it with a five-metric diagnostic suite. They then train IACM-RL with a hierarchical reward and three auxiliary losses so that the state-tracking capability is internalized in the policy weights. If the claim holds, agents can maintain accurate tool invocation even when users overwrite parameters, interrupt tasks, or insert irrelevant chit-chat, and the benefit should grow with conversation length rather than fade.","feed_headline":"Learned state tracking beats raw-history agents on noisy tool calls","feed_subtitle":"A self-generated belief-state block lets an LLM beat raw-history agents on dynamic tool-calling benchmarks.","key_machinery":"The central object is the BeliefState-based Self-Generated Context Manager: a compact five-field state block $b_t = (C_{\\text{slots}}, G_{\\text{current}}, A_{\\text{last}}, Q_{\\text{pending}}, I_{\\text{signal}})$ rendered as nine XML sub-blocks, drafted by the policy at each turn, injected into the system prompt, and updated at every user/tool boundary. The stale flag on $C_{\\text{slots}}$ is the load-bearing detail: an overwritten value is kept but explicitly forbidden. The optimization machinery is the hierarchical reward $R_{\\text{total}} = \\alpha R_{\\text{belief}} + \\beta R_{\\text{action}} + \\gamma R_{\\text{outcome}} + R_{\\text{format}}$, which gives field-level credit assignment over the five state fields, plus three auxiliary losses—action calibration ($\\mathcal{L}_{\\text{cal}}$), CM extraction ($\\mathcal{L}_{\\text{ext}}$), and state distillation ($\\mathcal{L}_{\\text{dist}}$)—which shape the CM tokens and then distill the CM-conditioned distribution into a CM-free student.","core_discovery":"IACM-RL claims that an LLM policy can be trained to decode, at every turn, a self-generated BeliefState paragraph—tracking the current goal, confirmed slots, last executed tool, pending questions, and intent signal—and to condition its tool call on this block instead of the raw dialogue history; overwritten parameters are marked with a structural stale flag so the policy is explicitly forbidden from reverting to them. On DynamicIntent, BFCL-V3, and $\\tau^2$-Bench, the resulting policy attains the highest overall average (64.0), remains within 0.2 points of its in-domain cognitive score on a fully disjoint out-of-domain tool pool, and suppresses loops and stale-value reuse. The three auxiliary losses each contribute: removing CM extraction drops out-of-domain cognitive score from 36.3 to 32.7, removing state distillation drops it to 31.4, and removing all three reduces the average to 61.5.","pith_inferences":["The stale-flag mechanism suggests a testable diagnostic rule: in long-horizon agent failures, a large share of errors should be attributable to regression to overwritten slot values rather than to missing tool selection; this could be measured on other multi-turn agent benchmarks by tagging consumable slot values.","If the CM-block training transfers to real APIs, the same two-stage RL recipe could plausibly extend to multimodal tool invocation and to non-tool agent state management, since the CM operates on textual state summaries; the paper explicitly leaves that transfer open.","Because CM gains grow with dialogue length, the approach offers a practical alternative to ever-longer context windows: train agents to emit compact state updates into the prompt instead of relying on retrieval or compression, at least for long-horizon text-based tool tasks."],"forward_implications":["On DynamicIntent, IACM-RL posts the highest average (64.0), beating seven PPO-trained context-management baselines and the SFT base.","On an out-of-domain split with a disjoint 244-tool pool, the cognitive score stays at 36.3 versus 36.5 in-domain, while the base model drops to 33.0 and RL-STA to 33.8, indicating schema-independent state tracking.","Gains grow with conversation length: on $\\tau^2$-Bench very-long dialogs (over 16k tokens), PPO-noCM scores 0.0 while IACM-RL reaches 34.8; on dialogs above 30k characters the no-CM pipeline collapses to 0.0 while a CM-injected base model still achieves 0.202.","Under adversarial interference (three fake tasks plus five chit-chat messages), IACM-RL retains 36.8% tool-call accuracy versus 25.6% for PPO-noCM, an 11.2-point margin.","Removing the belief reward layer causes the sharpest drop (average 64.0 to 57.4, BFCL 63.3 to 47.5), showing the cognitive layer is what preserves general calling proficiency."],"supporting_citations":[{"why":"ACON baseline: optimizes compression guidelines via failure analysis; the method must beat it on long-horizon intent scenarios.","marker":"Kang et al., 2026"},{"why":"MEM1 baseline: jointly trained memory/compression policy with RL; the closest learnable-context competitor.","marker":"Zhou et al., 2025"},{"why":"RL-STA baseline: single-turn anchors to break contextual inertia; motivates the decoupled state-tracking design.","marker":"Chen et al., 2026"},{"why":"$\\tau^2$-Bench: the long-horizon dual-control benchmark used to show CM gains grow with dialogue length.","marker":"Barres et al., 2025"},{"why":"BFCL-V3: the function-calling benchmark used to show explicit context management does not hurt general tool calling.","marker":"Patil et al., 2025"},{"why":"Contextual belief management: supplies the belief-state motivation that the self-generated CM extends.","marker":"Xu et al., 2026"},{"why":"LLMLingua baseline: token-level prompt compression that lags on cognitive state tracking.","marker":"Jiang et al., 2023"},{"why":"Goal drift evaluation: documents the deviation and infinite-loop failure modes IACM-RL targets.","marker":"Arike et al., 2025"}],"fun_headline_variants":["IACM-RL: belief-state tracking tames noisy tool calls","Self-generated belief states beat raw history in tool calling","Belief-state policy cuts tool-call loops in dynamic intents","LLM tool calls stabilized by self-tracked belief states"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that LLM-simulated tool returns behave enough like real API responses that a policy trained against them transfers; the paper only validates the simulation on SFT models, not on the RL-trained policy, so a structural difference between mock and real returns could erase the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["IACM-RL: belief-state tracking tames noisy tool calls","Self-generated belief states beat raw history in tool calling","Belief-state policy cuts tool-call loops in dynamic intents","LLM tool calls stabilized by self-tracked belief states"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1221,"prompt_tokens":947,"completion_tokens":274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":204}},"tokens_in":563,"tokens_out":274,"duration_ms":3025,"temperature":1.0,"reasoning_tokens":204,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:02:04.477198+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained RL policy (not an SFT variant) and run it on the same DynamicIntent test scenarios with real tool backends instead of the LLM mock, then measure the cognitive score, stale-context residual rate, and infinite-loop rate against PPO-noCM. If the margin over raw-history RL shrinks substantially or reverses once tool returns are real, the claim that the learned stale-flag and loop-suppression behavior transfers is falsified.","supporting_citations":[],"review_version":2}