Pith. sign in

REVIEW 5 major objections 4 minor 41 references

This paper claims that Behavior Agentic Optimization (BAO) — a two-stage recipe of behavior-enhanced warm-start supervised fine-tuning followed by behavior-regularized reinforcement learning — pushes the Pareto frontier between task perform

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:09 UTC pith:KBGBR5OA

load-bearing objection A solid empirical training recipe that likely improves over UserRL on the benchmark, but the user-engagement Pareto claim is only as strong as the LLM user simulators. the 5 major comments →

arxiv 2602.11351 v2 pith:KBGBR5OA submitted 2026-02-11 cs.AI cs.LG

Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization

Yihang Yao , Zhepeng Cen , Haohong Lin , Shiqi Liu , Zuxin Liu , Jiacheng Zhu , Zhang-Wei Hong , Laixi Shi
show 1 more author
Ding Zhao
This is my paper
classification cs.AI cs.LG
keywords proactive agentsagentic reinforcement learningPareto frontiermulti-objective optimizationuser engagementbehavior regularizationreward hackingLLM agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Proactive LLM agents are caught in a trade-off: asking users too many questions lowers satisfaction, while asking too few can leave tasks incomplete. The paper treats this as a multi-objective optimization problem and introduces Behavior Agentic Optimization (BAO), a training framework that first teaches an agent two inter-turn behaviors — retrospective reasoning (maintaining and revising hypotheses from history) and prospective planning (budget-aware scheduling and strategic querying) — and then regularizes those behaviors with turn-level penalties during reinforcement learning. The central claim is that BAO substantially outperforms the prior agentic RL baseline in both task score and user-effort reduction across Function-Gym, Telepathy-Gym, and Turtle-Gym, while also reducing reward hacking against LLM judge models. If correct, this means small open models can be trained to interact with users more efficiently than frontier commercial agents, with fewer redundant questions and higher first-attempt answer accuracy.

Core claim

The paper's central discovery is that the Pareto frontier between task performance and user engagement in proactive agents can be pushed forward by explicitly shaping inter-turn behaviors, not merely by tuning the weight of a user-effort penalty. Simply penalizing user interaction (varying w in the objective R(τ) − w·U(τ)) fails to improve the trade-off; instead, BAO first uses a teacher model to synthesize demonstration trajectories that exhibit retrospective reasoning and prospective planning, fine-tunes the student on these traces, and then applies two regularizers during GRPO-style RL: a penalty for consecutive user-involved actions without environment information gain, and a penalty for

What carries the argument

The central machinery is the pair of multi-turn behavior classes plus their regularization: (1) Retrospective Reasoning, decomposed into Memory Management (maintaining and pruning hypotheses about the hidden user context) and Hypothesis Refinement (explicitly pivoting away from contradicted assumptions), and (2) Prospective Planning, decomposed into Dynamic Scheduling (adapting exploration vs. exploitation to remaining turn budget) and Strategic Querying (asking boundary-classifying yes/no questions). These behaviors are injected into the policy via behavior-enhanced SFT data generated by a teacher model, then kept in check by turn-level reward shaping that penalizes (a) consecutive answer s

Load-bearing premise

The evaluation assumes that LLM-simulated users (Qwen3-8B during training, GPT-4o at test time) respond to agent interactions the way real human users would, in terms of patience, feedback quality, and satisfaction.

What would settle it

A controlled human-subject study where participants interact with BAO-trained and baseline-trained agents on equivalent tasks, measuring task completion rate, number of clarification requests, self-reported satisfaction, and willingness to reuse the agent; if human users find BAO's lower question rate to be under-informative or its early answers less trustworthy, the Pareto-frontier claim would not transfer beyond the LLM-simulator benchmark.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If BAO's claims hold, proactive agents can be trained to gather hidden user context with fewer clarifying questions, reducing the cognitive load on human users in real-world assistance and customer-service domains.
  • The reward-hacking reduction (lower train/eval reward gap and higher reward translation rate on Turtle-Gym) suggests that behavior regularization is a practical countermeasure to LLM-judge exploitation during RL, not just an efficiency trick.
  • The result that a 1.7B or 4B open model can match or exceed large commercial models on these tasks indicates that training methodology, not raw scale, is the main lever for proactive interaction quality.
  • Complementarity between retrospective reasoning and prospective planning (retro boosts long-horizon score, prospect boosts first-turn accuracy) implies that both behaviors should be co-training targets for robust multi-turn agents.
  • The Pareto-frontier framing gives a concrete evaluation protocol: report Pass@U-k curves and user involvement rates, not just aggregate reward, when claiming progress in user-centric agent training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would test BAO with real human users instead of LLM simulators; if the reduced interaction rate is perceived as under-asking rather than efficient, the user-satisfaction objective would need recalibration.
  • The behavior-regularization mechanism is likely portable to other multi-turn agent domains (tool use, web navigation) where redundant interaction and over-thinking are failure modes, though the specific penalty coefficients may need task-specific tuning.
  • The paper's framing suggests that further gains may come from learning the behavior prompts or penalty weights automatically rather than hand-crafting them, potentially closing the gap to a fully end-to-end multi-objective agentic RL objective.
  • The reward-hacking analysis hints that behavior regularization could serve as a general-purpose debiasing device for RL with judge models, reducing incentive to game the judge independently of the specific task.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes Behavioral Agentic Optimization (BAO), a two-stage pipeline for training proactive LLM agents. It formulates proactive agent training as a multi-objective problem balancing task reward against user effort (Eq. 1), first distilling four inter-turn behaviors (memory management, hypothesis refinement, dynamic scheduling, strategic querying) via GPT-4o-generated SFT traces, then applying GRPO with two turn-level reward shaping penalties (Eqs. 2 and 3). Experiments on the UserRL benchmark (Function-Gym, Telepathy-Gym, Turtle-Gym) with Qwen3-1.7B/4B report higher Pass@U-k, higher Score, and lower user-involvement rate (UR) than UserRL, with ablations and a reward-hacking analysis. The central claim is that BAO pushes forward the Pareto frontier between task performance and user engagement.

Significance. If the empirical claims hold, BAO is a useful and clearly presented contribution: it gives a concrete, transferable recipe for improving multi-turn agentic RL, it includes honest reward-hacking diagnostics (Fig. 9), and its ablations isolate the contributions of behavior enhancement and regularization. The formulation of the trade-off as a contextual MDP with a structured action space is clean, and the method is reproducible at the level of hyperparameters and prompts. However, the strength of the contribution depends on two assumptions that are only partially supported: (i) that the LLM-simulated users and judges used in Telepathy-Gym and Turtle-Gym measure real task completion and user burden, and (ii) that the user-effort metric (a count of Answer actions) captures what humans experience as burden. The paper's most defensible evidence is on Function-Gym, where rewards are rule-based; on the two LLM-judged tasks the claims are benchmark-relative rather than directly about human users.

major comments (5)
  1. [§5.2, Table 1 and Figure 5] The central quantitative claim that BAO 'substantially outperforms' UserRL is presented without uncertainty quantification in Table 1, even though Figure 5 states results are 'averaged over three random seeds' and shows shaded standard deviations. Without per-seed standard deviations or significance tests in Table 1, the reader cannot tell whether gaps such as BAO-4B vs UserRL-4B on Telepathy-Gym (Pass@U-1 0.5123 vs 0.4878) or BAO-1.7B vs UserRL-1.7B on Turtle-Gym (Pass@U-1 0.0563 vs 0.0479) are reproducible or within seed noise. Please report mean±std over the three seeds for every metric in Table 1, or explicitly state which numbers are single-seed.
  2. [§5.5, Turtle-Gym reward hacking] The reward-hacking analysis undercuts the interpretation of the Turtle-Gym results as improvements in real task completion. UserRL's reward translation rate is 0.154, meaning its training reward and Pass@U-1 collapse under the train/eval judge shift, while BAO's is 0.575. That is evidence that BAO is more robust to judge distribution shift, not evidence that it completes more real tasks or reduces real user effort. The paper states this clearly, but the abstract and Section 5.2 nevertheless present Turtle-Gym as supporting the general Pareto-frontier claim. Please add a human validation study (or at least a rubric-level agreement study between GPT-4o and human raters) on a sample of Telepathy/Turtle answers, or explicitly restrict the task-performance claim to Function-Gym and frame the other two tasks as robustness-to-simulator-shift results.
  3. [§3, Eq. (1) and Appendix B.1] There is an internal inconsistency in the definition of the user-involved action space, which is load-bearing for U(τ). Section 5.1 states that Action and Search are environment-involved (A_e) and only Answer is user-involved (A_u). Appendix B.1, however, concludes: 'the user involved actions Au contain Action, and the environment involved actions Ae contain Search and Action,' placing Action in both spaces. Since UR and Pass@U-k are defined through A_u, the paper must state one consistent definition; the current text makes the central metric ambiguous.
  4. [§5.1, evaluation setup and §3, Eq. (1)] The 'user effort' objective is operationalized solely as U(τ), the count of Answer actions. This is never validated as a proxy for human patience, cognitive load, or satisfaction. An agent that asks fewer answers but many verbose, repetitive Action questions could have lower UR but be more burdensome to a human user; conversely, a well-timed answer with one follow-up could be less burdensome than two terse premature answers. The Pareto-frontier claims about 'user engagement' therefore require a validation of the metric itself. Please provide evidence (human ratings, or at least correlations with response-length-entropy measures) or soften the language to 'number of answer submissions' rather than 'user burden/satisfaction.'
  5. [§5.1 and Appendix B.2, Telepathy-Gym baseline] For Telepathy-Gym, the UserRL baseline is configured 'the same as our BAO, since it is treated as an untrained evaluation set in UserRL.' This means the baseline is not a tuned or best-known configuration for this task, and the comparison may understate UserRL's capability. Please either use a properly tuned UserRL baseline on Telepathy-Gym, cite a specific configuration supported by the UserRL paper, or state this as a limitation in the main text rather than only in the appendix.
minor comments (4)
  1. [§5.5] Typo in the sentence 'despite a high reward translation rate (RTR = 0.154)': 0.154 is a low rate and the surrounding text treats it as a failure mode. It should read 'low reward translation rate.'
  2. [Figure 5] The axes appear blank in the caption and figure; labels for the x-axis (number of allowed user-involved actions) and y-axis (pass rate) should be printed.
  3. [Eq. (3)] The over-thinking penalty is introduced as λ_think (T−T′)/T′ for trajectories with T′ < T, but T′ = 0 is not handled; specify the convention for empty trajectories.
  4. [Appendix B.1] The sentence describing action spaces is grammatically and semantically garbled ('the user involved actions A_u contain Action...'). The intended classification should be stated cleanly in one place and referenced from Eq. (1).

Circularity Check

0 steps flagged

No circularity: the Pareto-frontier claim is an empirical benchmark comparison, not a construction from fitted inputs or self-citations.

full rationale

BAO's contribution is a training method (behavior-enhanced SFT followed by behavior-regularized GRPO, Eqs. 2-5), not a derived prediction. The central claim—higher Pass@U-k and lower user-involvement rate than UserRL—is a measured comparison on the UserRL benchmark tasks, where Function-Gym is fully rule-based and the LLM-simulated tasks are explicitly described with train/eval distribution shift (Sec. 5.1, 5.5). The objective in Eq. 1 defines U(τ) as a user-effort metric and the regularizations in Eqs. 2-3 act on that same objective, but the evaluation separately reports unshaped cumulative reward and pass rates; the score improvements on Function-Gym are not forced by the penalty terms. The paper even includes a reward-hacking analysis showing large train/eval gaps for UserRL (RTR=0.154) and BAO (0.575), acknowledging rather than hiding judge dependence. The only self-citations (e.g., Cen et al. 2025, Yao et al. 2025, and the UserRL baseline with overlapping authors) are used as background or baseline references, not as load-bearing proofs; the benchmark results are empirical and repeatable. Potential contamination from using GPT-4o as both SFT teacher and evaluation user simulator is a validity issue, not a circular derivation, and does not make the reported equations equivalent to their inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central comparison is empirical and rests on LLM-simulated users and LLM-as-judge rewards rather than real human engagement. The method's core penalties (λ_ans, λ_think) and the MOO weight w are hand-set hyperparameters whose values are not reported. No new physical or conceptual entities are introduced beyond the behavior categories.

free parameters (3)
  • w (trade-off weight in Eq. 1) = not reported
    Controls balance between task reward and user-effort penalty; paper only shows qualitative w1<w2<w3 in Figure 2 and does not state the value used in BAO experiments.
  • λ_ans (information-seeking penalty) = not reported
    Penalty scale for consecutive user-involved actions without environment information gain (Eq. 2); value is chosen by hand and not disclosed.
  • λ_think (over-thinking penalty) = not reported
    Penalty coefficient for failed trajectories ending before budget (Eq. 3); value is chosen by hand and not disclosed.
axioms (5)
  • domain assumption LLM-simulated users (Qwen3-8B during training, GPT-4o at evaluation) faithfully represent real user engagement and preferences.
    Section 5.1 and Appendix B.1 describe all user feedback and rewards as generated by LLM simulators; the 'user effort' claims are measured only against these simulators, not humans.
  • domain assumption LLM-as-judge reward models provide valid task rewards.
    Section 5.5 acknowledges reward hacking and large train/eval gaps (RTR 0.154 for UserRL on Turtle-Gym), so reward validity is a load-bearing assumption for comparing methods.
  • ad hoc to paper The four behavior patterns (memory management, hypothesis refinement, dynamic scheduling, strategic querying) are the right inductive biases for proactive agents.
    Section 4.1 defines these behaviors; they are injected via a GPT-4o teacher prompt (Appendix C.2) without independent evidence that they are necessary or sufficient.
  • domain assumption The contextual MDP with a fixed hidden user context c is an adequate model of proactive interaction.
    Section 3 assumes a fixed hidden context per episode; this ignores evolving user preferences, which is a simplification for the benchmark tasks.
  • ad hoc to paper Tuning the scalar weight w in Eq. (1) cannot improve the trade-off (Figure 2).
    Section 3 asserts this empirical premise to motivate BAO, but the experimental details (weights, seeds, curves) are not reported.

pith-pipeline@v1.3.0-alltime-deepseek · 24800 in / 15065 out tokens · 143725 ms · 2026-08-03T00:09:17.697486+00:00 · methodology

0 comments
read the original abstract

Proactive large language model (LLM) agents aim to actively plan, query, and interact over multiple turns, enabling efficient task completion beyond passive instruction following and making them essential for real-world, user-centric applications. Agentic reinforcement learning (RL) has recently emerged as a promising solution for training such agents in multi-turn settings, allowing them to learn long-horizon decision-making strategies. However, existing pipelines face a critical challenge in balancing task performance with user engagement, as passive agents cannot efficiently adapt to users' intentions while overuse of human feedback increases the burden on users, which forms a Pareto Frontier between these two objectives. To push forward this frontier, we propose Behavior Agentic Optimization (BAO), an agentic RL framework that enhances and regularizes inter-turn behaviors to improve information-gathering capabilities and suppress inefficient or redundant interactions with users. We evaluate BAO on multiple tasks from the UserRL benchmark suite and demonstrate that it substantially outperforms proactive agentic RL baselines in terms of both higher task performance and lower user efforts, while achieving comparable or even superior performance to commercial LLM agents, highlighting its effectiveness for training proactive, user-centric LLM agents in complex multi-turn scenarios. Our website: https://proactive-agentic-rl.github.io/.

Figures

Figures reproduced from arXiv: 2602.11351 by Ding Zhao, Haohong Lin, Jiacheng Zhu, Laixi Shi, Shiqi Liu, Yihang Yao, Zhang-Wei Hong, Zhepeng Cen, Zuxin Liu.

Figure 1
Figure 1. Figure 1: The overview of BAO. (Left): Behavior Enhancement. (Middle): Behavior-Regularized RL. (Right): Pareto-Frontiers between user engagement efforts and task performance. two subspaces: 1) User involved action space Au: includes actions that require explicit user feedback, such as answer verifications. 2) Environment involved action space Ae: in￾cludes actions that interact with tools or external systems withou… view at source ↗
Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Behavior examples from Turtle-Gym. Hidden twist: This person is a programmer; in the computer industry, old, large, and difficult-to-maintain code is referred to as a pile of excrement. Red: Prospective Planning; Blue: Retrospective Reasoning. Memory Management. The memory is used to maintain and update hypotheses about the hidden context c. With the memory, the agent retrieves and links related informatio… view at source ↗
Figure 4
Figure 4. Figure 4: Function-Gym training curves. BAO keeps a higher exploration ratio, achieving higher task performance with even fewer generated tokens compared to UserRL. In contrast, our BAO achieves better performance, with gen￾erally higher Pass@U-1, 2 values and a significantly lower action rate. This indicates that the trained model has greater confidence in producing correct answers at the early answer submissions a… view at source ↗
Figure 6
Figure 6. Figure 6: Behavior analysis on Function-Gym with Qwen3-4B as base models. ↑, ↓: the higher/lower, the better. Full behaviors enable a more balanced performance in BAO. Section 4.1, we conduct an ablation study by controlling the enhanced behavior types included in the SFT dataset. The results are shown in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Pareto frontiers in Function-Gym. The results are av￾eraged over three random seeds. The shaded area represents the standard deviation. BAO is with better Pareto Frontiers. Main Experiment Takeaways • Prior RL baseline models struggle with proactive agent tasks, with low Pass@U-k and high UR. • BAO learns more efficient proactive behaviors. • BAO pushes forward the Pareto frontiers of agents and approaches… view at source ↗
Figure 9
Figure 9. Figure 9: Turtle-Gym reward hacking issue analysis with Qwen3- 4B as base models. BAO achieves a lower occurrence of reward hacking, leading to a higher evaluation score. Turtle-Gym, where the sim-to-real gap is most substantial: during training, we use Qwen3-8B as the user simulator and reward model, whereas testing is conducted with GPT-4o. A common form of reward hacking of this task involves repeatedly generatin… view at source ↗
Figure 8
Figure 8. Figure 8: Answer diversity evaluation in Turtle-Gym. BAO ex￾plores answer space better by generating more diverse answers. in Figures 7 and 8, respectively. For both metrics, lower sim￾ilarity scores indicate greater diversity within an interaction trajectory and stronger exploration capability. From [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: We observe that BAO substantially reduces reward hacking, as reflected by lower rewards during training and a larger proportion of reward preserved during evaluation. This trend is consistent with the findings discussed in Section 5.5. UserRL BAO 0 250 500 750 1000 1250 Value Answer Token Number Train Eval UserRL BAO 0.00 0.05 0.10 0.15 Score Train Eval UserRL BAO 0.00 0.02 0.04 0.06 0.08 Pass@U-1 Train E… view at source ↗
Figure 11
Figure 11. Figure 11: Function-gym reasoning traces of UserRL (Qian et al., 2025c). The goal of this task is to infer the ground-truth underlying mapping function: f(x1, x2, x3, x4) = (x2 + x3) 2 /x1 ∗ x4 and submit the correct answer for the test case. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Function-gym reasoning traces of BAO. The goal of this task is to infer the ground-truth underlying mapping function: f(x1, x2, x3, x4) = (x2 + x3) 2 /x1 ∗ x4 and submit the correct answer for the test case. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Turtle-gym reasoning traces. Part-1 (Continued by [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Turtle-gym reasoning traces. Part-2 (Following [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 30 linked inside Pith

  1. [1]

    Consistently simulating human personas with multi-turn reinforcement learning.arXiv preprint arXiv:2511.00222,

    Abdulhai, M., Cheng, R., Clay, D., Althoff, T., Levine, S., and Jaques, N. Consistently simulating human personas with multi-turn reinforcement learning.arXiv preprint arXiv:2511.00222,

    arXiv

  2. [3]

    Reinforce- ment learning for long-horizon interactive llm agents

    Chen, K., Cusumano-Towner, M., Huval, B., Petrenko, A., Hamburger, J., Koltun, V ., and Kr¨ahenb¨uhl, P. Reinforce- ment learning for long-horizon interactive llm agents. arXiv preprint arXiv:2502.01600,

    Pith/arXiv arXiv

  3. [4]

    Gandhi, K., Chakravarthy, A., Singh, A., Lile, N., and Goodman, N. D. Cognitive behaviors that enable self- improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307,

    Pith/arXiv arXiv

  4. [5]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  5. [6]

    Con- textual markov decision processes.arXiv preprint arXiv:1502.02259,

    Hallak, A., Di Castro, D., and Mannor, S. Con- textual markov decision processes.arXiv preprint arXiv:1502.02259,

    Pith/arXiv arXiv

  6. [9]

    R., He, J., Yu, H., et al

    Jiang, B., Yuan, Y ., Shen, M., Hao, Z., Xu, Z., Chen, Z., Liu, Z., Vijjini, A. R., He, J., Yu, H., et al. Personamem- v2: Towards personalized intelligence via learning im- plicit user personas and agentic memory.arXiv preprint arXiv:2512.06688, 2025a. Jiang, D., Lu, Y ., Li, Z., Lyu, Z., Nie, P., Wang, H., Su, A., Chen, H., Zou, K., Du, C., et al. Verlt...

    arXiv

  7. [10]

    Quagmires in sft-rl post- training: When high sft scores mislead and what to use instead.arXiv preprint arXiv:2510.01624,

    Kang, F., Kuchnik, M., Padthe, K., Vlastelica, M., Jia, R., Wu, C.-J., and Ardalani, N. Quagmires in sft-rl post- training: When high sft scores mislead and what to use instead.arXiv preprint arXiv:2510.01624,

    arXiv

  8. [11]

    Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., and Roux, N. L. Vineppo: Refining credit assignment in rl training of llms.arXiv preprint arXiv:2410.01679,

    Pith/arXiv arXiv

  9. [12]

    Nv-embed: Improved techniques for training llms as generalist embedding models.arXiv preprint arXiv:2405.17428,

    Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catan- zaro, B., and Ping, W. Nv-embed: Improved techniques for training llms as generalist embedding models.arXiv preprint arXiv:2405.17428,

    Pith/arXiv arXiv

  10. [13]

    A., Dernoncourt, F., Kveton, B., Wu, J., Yu, T., Song, L., Yang, T., Qin, Y ., et al

    Li, L., Cai, P., Rossi, R. A., Dernoncourt, F., Kveton, B., Wu, J., Yu, T., Song, L., Yang, T., Qin, Y ., et al. A personalized conversational benchmark: Towards simulating person- alized conversations.arXiv preprint arXiv:2505.14106,

    Pith/arXiv arXiv

  11. [14]

    Let’s try again: Eliciting multi-turn reasoning 9 Behavioral Agentic Optimization in language models via simplistic feedback

    Liu, L., Wang, Z., Li, L., Xu, C., Lu, Y ., Liu, H., Sil, A., and Li, M. Let’s try again: Eliciting multi-turn reasoning 9 Behavioral Agentic Optimization in language models via simplistic feedback. In2nd AI for Math Workshop@ ICML 2025, 2025a. Liu, L., Wang, Z., Li, L., Xu, C., Lu, Y ., Liu, H., Sil, A., and Li, M. A simple” try again” can elicit multi-t...

    Pith/arXiv arXiv 2025

  12. [15]

    Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning.arXiv preprint arXiv:2503.21620,

    Lu, Z., Chai, Y ., Guo, Y ., Yin, X., Liu, L., Wang, H., Xiao, H., Ren, S., Xiong, G., and Li, H. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning.arXiv preprint arXiv:2503.21620,

    Pith/arXiv arXiv

  13. [16]

    Gui- r1: A generalist r1-style vision-language action model for gui agents.arXiv preprint arXiv:2504.10458,

    Luo, R., Wang, L., He, W., Chen, L., Li, J., and Xia, X. Gui- r1: A generalist r1-style vision-language action model for gui agents.arXiv preprint arXiv:2504.10458,

    Pith/arXiv arXiv

  14. [17]

    Hiertod: A task-oriented dialogue system driven by hier- archical goals.arXiv preprint arXiv:2411.07152,

    Mo, L., Jiang, S., Maharaj, A., Hishamunda, B., and Li, Y . Hiertod: A task-oriented dialogue system driven by hier- archical goals.arXiv preprint arXiv:2411.07152,

    Pith/arXiv arXiv

  15. [18]

    Not all thoughts are generated equal: Efficient llm reasoning via multi-turn reinforcement learning.arXiv preprint arXiv:2505.11827,

    Ning, Y ., Li, W., Fang, J., Tan, N., and Liu, H. Not all thoughts are generated equal: Efficient llm reasoning via multi-turn reinforcement learning.arXiv preprint arXiv:2505.11827,

    Pith/arXiv arXiv

  16. [20]

    Balrog: Benchmarking agentic llm and vlm reasoning on games.arXiv preprint arXiv:2411.13543,

    Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wolczyk, M., Khan, A., Pignatelli, E., Kuci´nski, Ł., Pinto, L., Fer- gus, R., et al. Balrog: Benchmarking agentic llm and vlm reasoning on games.arXiv preprint arXiv:2411.13543,

    Pith/arXiv arXiv

  17. [21]

    C., et al

    Prabhakar, A., Liu, Z., Zhu, M., Zhang, J., Awalgaonkar, T., Wang, S., Liu, Z., Chen, H., Hoang, T., Niebles, J. C., et al. Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay.arXiv preprint arXiv:2504.03601,

    Pith/arXiv arXiv

  18. [22]

    C., He, Q., Wang, H., Chen, X., Hakkani-T¨ur, D., Tur, G., and Ji, H

    Qian, C., Acikgoz, E. C., He, Q., Wang, H., Chen, X., Hakkani-T¨ur, D., Tur, G., and Ji, H. Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025a. Qian, C., Liu, Z., Prabhakar, A., Liu, Z., Zhang, J., Chen, H., Ji, H., Yao, W., Heinecke, S., Savarese, S., et al. Userbench: An interactive gym environment for user- centric agents....

    Pith/arXiv arXiv

  19. [23]

    Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  20. [24]

    Satori: Reinforcement learning with chain-of-action-thought en- hances llm reasoning via autoregressive search.arXiv preprint arXiv:2502.02508,

    Shen, M., Zeng, G., Qi, Z., Hong, Z.-W., Chen, Z., Lu, W., Wornell, G., Das, S., Cox, D., and Gan, C. Satori: Reinforcement learning with chain-of-action-thought en- hances llm reasoning via autoregressive search.arXiv preprint arXiv:2502.02508,

    Pith/arXiv arXiv

  21. [25]

    Language model personalization via reward factorization

    Shenfeld, I., Faltings, F., Agrawal, P., and Pacchiano, A. Language model personalization via reward factorization. arXiv preprint arXiv:2503.06358, 2025a. Shenfeld, I., Pari, J., and Agrawal, P. Rl’s razor: Why online reinforcement learning forgets less.arXiv preprint arXiv:2509.04259, 2025b. Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Pe...

    Pith/arXiv arXiv

  22. [26]

    Agentic reasoning and tool integration for llms via reinforcement learning.arXiv preprint arXiv:2505.01441,

    Singh, J., Magazine, R., Pandya, Y ., and Nambi, A. Agentic reasoning and tool integration for llms via reinforcement learning.arXiv preprint arXiv:2505.01441,

    Pith/arXiv arXiv

  23. [27]

    Training proactive and per- sonalized llm agents.arXiv preprint arXiv:2511.02208,

    Sun, W., Zhou, X., Du, W., Wang, X., Welleck, S., Neubig, G., Sap, M., and Yang, Y . Training proactive and per- sonalized llm agents.arXiv preprint arXiv:2511.02208,

    arXiv

  24. [28]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    Pith/arXiv arXiv

  25. [30]

    En- hancing personalized multi-turn dialogue with curiosity reward.arXiv preprint arXiv:2504.03206,

    Wan, Y ., Wu, J., Abdulhai, M., Shani, L., and Jaques, N. En- hancing personalized multi-turn dialogue with curiosity reward.arXiv preprint arXiv:2504.03206,

    arXiv

  26. [31]

    Opera: A dataset of observation, persona, rationale, and action for evaluating llms on human online shopping behavior simulation.arXiv preprint arXiv:2506.05606, 2025a

    Wang, Z., Lu, Y ., Li, W., Amini, A., Sun, B., Bart, Y ., Lyu, W., Gesi, J., Wang, T., Huang, J., et al. Opera: A dataset of observation, persona, rationale, and action for evaluating llms on human online shopping behavior simulation.arXiv preprint arXiv:2506.05606, 2025a. Wang, Z., Lu, Y ., Zhang, Y ., Huang, J., and Wang, D. Customer-r1: Personalized si...

    Pith/arXiv arXiv

  27. [33]

    Boad: Discovering hierarchi- cal software engineering agents via bandit optimization

    Xu, I., Zeng, G., He, Z., Jin, C., Pareja, A., Gutfreund, D., Gan, C., and Hong, Z.-W. Boad: Discovering hierarchi- cal software engineering agents via bandit optimization. arXiv preprint arXiv:2512.23631,

    arXiv

  28. [34]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a. Yang, Y ., Li, D., Dai, Y ., Yang, Y ., Luo, Z., Zhao, Z., Hu, Z., Huang, J., Saha, A., Chen, Z., et al. Gta1: Gui test-time scaling agent.arXiv preprint arXiv:2507.05791, 2025b. Yao, Y ., Zen...

    Pith/arXiv arXiv

  29. [35]

    Demys- tifying long chain-of-thought reasoning in llms.arXiv preprint arXiv:2502.03373,

    Yeo, E., Tong, Y ., Niu, M., Neubig, G., and Yue, X. Demys- tifying long chain-of-thought reasoning in llms.arXiv preprint arXiv:2502.03373,

    Pith/arXiv arXiv

  30. [36]

    Demysti- fying reinforcement learning in agentic reasoning.arXiv preprint arXiv:2510.11701,

    Yu, Z., Yang, L., Zou, J., Yan, S., and Wang, M. Demysti- fying reinforcement learning in agentic reasoning.arXiv preprint arXiv:2510.11701,

    arXiv

  31. [37]

    Satori- swe: Evolutionary test-time scaling for sample-efficient software engineering.arXiv preprint arXiv:2505.23604, 2025a

    Zeng, G., Shen, M., Chen, D., Qi, Z., Das, S., Gutfreund, D., Cox, D., Wornell, G., Lu, W., Hong, Z.-W., et al. Satori- swe: Evolutionary test-time scaling for sample-efficient software engineering.arXiv preprint arXiv:2505.23604, 2025a. Zeng, S., Wei, Q., Brown, W., Frunza, O., Nevmyvaka, Y ., Zhao, Y . K., and Hong, M. Reinforcing multi-turn rea- soning...

    Pith/arXiv arXiv 2025

  32. [38]

    Teaching language models to evolve with users: Dynamic profile modeling for personalized alignment.arXiv preprint arXiv:2505.15456,

    11 Behavioral Agentic Optimization Zhao, W., Sui, X., Hu, Y ., Guo, J., Liu, H., Li, B., Zhao, Y ., Qin, B., and Liu, T. Teaching language models to evolve with users: Dynamic profile modeling for personalized alignment.arXiv preprint arXiv:2505.15456,

    arXiv

  33. [39]

    L., Huang, J., Yu, C

    Zheng, L., Yin, L., Xie, Z., Sun, C. L., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. Sglang: Efficient execution of structured language model programs.Advances in neural information processing systems, 37:62557–62583, 2024a. Zheng, Y ., Zhang, R., Zhang, J., Ye, Y ., Luo, Z., Feng, Z., and Ma, Y . Llamafactory: Unifie...

    Pith/arXiv arXiv

  34. [40]

    E., and Zhou, W

    Zhu, C., Tao, M., Wang, T., Ding, D., Jiang, Y . E., and Zhou, W. Towards faithful and controllable personalization via critique-post-edit reinforcement learning.arXiv preprint arXiv:2510.18849,

    arXiv

  35. [41]

    Yes”, “No

    The tasksFunction-Gym,Telepathy- Gym, andTurtle-Gymare from the userrl suite (Qian et al., 2025c) and the original paper contains introductions about environments and tasks. To make our paper self-contained, we list some details about the tasks. The userrl tasks follow the gymnasium (Towers et al., 2024)-style environments, and the environment transitions...

    2024

  36. [2015]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Pith/arXiv arXiv

  37. [2022]

    T., Daruki, S., Tang, X., et al

    Ouyang, S., Yan, J., Hsu, I., Chen, Y ., Jiang, K., Wang, Z., Han, R., Le, L. T., Daruki, S., Tang, X., et al. Rea- soningbank: Scaling agent self-evolving with reasoning memory.arXiv preprint arXiv:2509.25140,

    Pith/arXiv arXiv

  38. [2023]

    U., De Cola, G., Deleu, T., Goul˜ao, M., Kallinteris, A., Krimmel, M., KG, A., et al

    Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goul˜ao, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments.arXiv preprint arXiv:2407.17032,

    Pith/arXiv arXiv

  39. [2024]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Pith/arXiv arXiv

  40. [2025]

    Behavior injection: Preparing language models for reinforcement learning.arXiv preprint arXiv:2505.18917,

    Cen, Z., Yao, Y ., Han, W., Liu, Z., and Zhao, D. Behavior injection: Preparing language models for reinforcement learning.arXiv preprint arXiv:2505.18917,

    arXiv

  41. [2026]

    Wei, Y ., Duchenne, O., Copet, J., Carbonneaux, Q., Zhang, L., Fried, D., Synnaeve, G., Singh, R., and Wang, S. I. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution.arXiv preprint arXiv:2502.18449, 2025a. Wei, Z., Yao, W., Liu, Y ., Zhang, W., Lu, Q., Qiu, L., Yu, C., Xu, P., Zhang, C., Yin, B., et al. Webagent-r1: Trai...

    Pith/arXiv arXiv