{"id":"5d22efa0-0a5f-41e1-9c90-85ea9e0146f2","arxiv_id":"2508.12935","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An end-to-end RL framework that uses simulated future dialogue and a learned future-oriented reward model to fine-tune LLMs for open-ended emotional support, reporting improved success rates on ESConv and ExTES.","lead":"RLFF-ESC trains emotional support chatbots by simulating future conversations with an LLM and using the predicted long-term impact as a reward for reinforcement learning. On two public datasets it reports higher goal-completion rates than prior chatbots, though that goal-completion metric comes from an LLM judge rather than human users.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Goal-completion gains are measured entirely inside an LLM user-simulator/critic loop, and no human check is performed on model-generated dialogues; the reported SR/AT advantage may be reward overoptimization.","rationale":"The reader's weakest assumption is that the future-oriented reward, derived from LLM-simulated future dialogues and an LLM critic, is a valid proxy for a real user's enduring emotional improvement. My concern is the same load-bearing point, sharpened to the evaluation circularity: the reported SR/AT metric is computed with an LLM user simulator and LLM critic, and the policy is trained to maximize a reward model distilled from a similar LLM critic. The paper does include a human agreement check for the critic on human-human dialogues (Appendix D.3) and a human evaluation of response quality (Table 2), which are real supporting evidence. However, neither validates the simulated goal-completion loop on model-generated dialogues. Because the central claim's strongest quantitative evidence is SR/AT, this unvalidated proxy is the critical weak spot. A human re-judgment of the generated dialogues would directly settle whether the SR/AT gains reflect true goal completion or reward overoptimization. Given the paper's partial supporting checks and plausible framework, CONDITIONAL remains the right verdict; my stress-test does not move it.","tokens_in":21570,"tokens_out":4845,"duration_ms":51316,"concrete_test":"Run a blind human re-judgment of the actual model-generated evaluation dialogues: for each system in Table 1 (e.g., RLFF-ESC, ICL-AIF, GPT-4o), sample 100 evaluation dialogues and have three psychology-trained annotators, blind to system identity, independently mark resolved/not-resolved using the same criteria as the critic (Table 9) and rate pre/post emotional intensity. Compute per-system human-judged SR and average turns to resolution. If RLFF-ESC's SR advantage does not replicate under human labels, the reported goal-completion superiority is an artifact of the LLM evaluation loop; if it does, the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of superiority in goal completion rests on SR/AT measured in a fully simulated loop. In Algorithm 1, the training reward is produced by a user simulator U and critic Mcrt; during training these are Qwen-2.5-72B, and during evaluation the same protocol uses GPT-4o. Thus both the GRPO training signal (via the future-oriented reward model, Eq. 6) and the reported success metric are generated by LLM role-play, not by human users. Appendix D.3 checks GPT-4o critic agreement with humans only on 100 existing human-human ESConv dialogues, not on the model-generated evaluation dialogues. The user simulator's realism is never validated. Since GRPO maximizes Rfut, the policy may learn responses that push an LLM-simulated user to the critic's 'resolved' state without improving a real user's emotional state. The paper's Limitations explicitly acknowledge that end-user emotional intensity was not assessed. Therefore the headline quantitative evidence (41.5% vs 30.8% for GPT-4o) is conditional on the unvalidated assumption that the LLM user/critic loop is a faithful proxy for real goal completion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RLFF-ESC, a reinforcement-learning framework for open-ended emotional support conversations. The method has three stages: (1) a multi-agent LLM simulation samples future dialogue trajectories for candidate system responses and produces scalar future-oriented rewards; (2) a LLaMA-based reward model is trained on these simulated labels; (3) GRPO optimizes the policy with a reward composed of the future-oriented reward and a format reward that enforces <think>/<response> structure. Experiments on ESConv and ExTES report that RLFF-ESC improves Success Rate and Average Turn over baseline methods, including GPT-4o and LLaMA-3.1-405B on ESConv, and wins pairwise human comparisons on response quality. The paper also includes ablations of the reward model design and an analysis of the evaluation threshold.","tokens_in":21756,"tokens_out":9197,"duration_ms":85336,"significance":"The framework is timely and potentially impactful: if the reported numbers are trustworthy, a 7B policy optimized with a future-oriented reward can outperform much larger models on a conversational goal-completion metric, and the use of simulated future trajectories as a reward signal is a clean idea that could transfer to other long-horizon dialogue tasks. The paper's strengths include a clear three-stage pipeline, an ablation study (Table 3) isolating the contribution of the reward model type, and an explicit limitations section that acknowledges the lack of end-user emotional assessment. The main caveat is that the headline goal-completion gains are measured with an LLM-as-judge loop that is structurally similar to the training signal, so the external validity of the central claim is not yet established.","major_comments":[{"comment":"The SR/AT metrics are computed by GPT-4o role-playing both the user simulator and the critic, with 'goal completed' defined as the critic reward exceeding 0.5, while the training reward model in §3.2 was trained on trajectories simulated by Qwen-2.5-72B in the same role-play setup. Because the policy is optimized to maximize that reward model, the reported goal-completion advantage (e.g., 41.5% vs 30.8% for GPT-4o on ESConv) is measured inside the same LLM-simulation paradigm that generated the training signal. Appendix D.3 validates the GPT-4o critic against humans only on 100 existing human-human ESConv dialogues, not on the model-generated evaluation dialogues, and the limitations state that end-user emotional intensity was not assessed. This is a load-bearing validity gap for the 'goal completion' half of the abstract's claim. Please add a human evaluation of goal completion on model-generated dialogues, or demonstrate that the SR/AT gains persist under an independent evaluation protocol (e.g., a different LLM judge family or a held-out judge with a different prompt), and validate the user simulator's realism.","section":"§4.2.1, Appendix B, Figure 3"},{"comment":"The formula for the future-oriented reward is not correctly typeset: Algorithm 1 line 14 reads 'rsys t,j = rt+k,j + 1 AvgT 2', which is not a well-defined expression. Since this scalar is thresholded to create the binary training labels for the reward model, the training-data construction is not reproducible as printed. Please provide the explicit equation for rsys in terms of r_{t+k,j} and AvgT, and report the threshold δ used to binarize the labels.","section":"Algorithm 1, §3.2, §3.3.1"},{"comment":"Equation (3) describes the reward model as 'a frozen LLaMA model followed by a linear layer', but Appendix B states 'We fully fine-tune LLaMA-3.2-1B as an LLM-based classifier'. These statements are contradictory. Please clarify which components of the reward model are trainable, since this is essential for reproducing the reward model and for interpreting its capacity.","section":"§3.3.1 Eq. (3) vs. Appendix B"}],"minor_comments":[{"comment":"The prompt for the user simulator says it is 'the prompt proposed by' but the citation is missing; please add the reference.","section":"Appendix A.2"},{"comment":"The Limitations section contains a duplicated phrase: 'For automatic evaluation, we utilize LLMs, For automatic evaluation, we utilize LLMs because...' Please remove the duplication.","section":"Limitations"},{"comment":"Table 2 reports pairwise win/lose percentages for five quality dimensions, but no inter-annotator agreement is reported for these judgments; please add agreement statistics (e.g., Krippendorff's alpha) for the human evaluation of response quality.","section":"§4.2.1 Human Evaluation"},{"comment":"The SR/AT comparisons with larger-scale LLMs are reported as point estimates without confidence intervals or significance tests; please report variance across evaluation runs or a statistical test.","section":"§4.2.1, Figure 3"},{"comment":"There is a typo 'thiking format reward' in the paragraph introducing Rthk-fmt; it should be 'thinking format reward'.","section":"§3.3.1"},{"comment":"The axis legend in Figure 6 labels the proposed method as 'RLSF-ESC'; this should be 'RLFF-ESC'.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the proposed method is interesting, but the central quantitative claim currently rests on an LLM-based evaluation loop that also generated the training signal. I would ask the editor to treat the requested human goal-completion evaluation (or equivalent external validation) as a prerequisite for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: they combine multi-agent future simulation, a learned reward model, GRPO, and explicit reasoning traces into one end-to-end training recipe for emotional support conversations. That integration is new enough, and the paper is clear about what is inherited from prior work. The ablations are thoughtful — DPO vs GRPO, ranking vs classification reward models — and the results are consistent across two backbones and two datasets. I give them credit for the design and for the attempt to ground the LLM critic with a human agreement check.\n\nThe big caveat is the evaluation loop. Both the training signal (Qwen-72B role-playing user and critic) and the final SR/AT metric (GPT-4o role-playing the same roles) come from LLM simulation. The agreement analysis in the appendix is only on existing human-human ESConv dialogues, not on model-generated trajectories. The Limitations explicitly say they did not assess changes in end-user emotional intensity. So the headline 41.5% vs 30.8% over GPT-4o could be reward overoptimization on the simulator rather than genuine goal completion. That is not a refutation, but it is an unopened box, and it is the load-bearing part of the paper.\n\nSmaller issues: the reward formula in Algorithm 1 is garbled as printed, so I cannot tell exactly what is being computed. No code is released, several hyperparameters (threshold δ, reward composition weights) are underspecified, and the human evaluation has no significance testing. Also, the comparison with 70B/405B models is zero-shot versus a task-finetuned 7B, which weakens the \"compact model beats giant model\" point.\n\nAll that said, the central idea is sensible and the paper is honestly written. It would be a good focus for a referee who can push on the evaluation loop and demand a real user study or at least a human check on simulated dialogues. I would not cite the reported numbers until those are in place, but I would read a revised version with the missing details, and I would send this to review rather than desk-reject.","headline":"A plausible RL training recipe for emotional support that deserves a serious referee, but the headline goal-completion numbers are measured entirely inside an LLM user/critic loop and are not yet anchored to real users.","tokens_in":22311,"tokens_out":2580,"would_cite":false,"duration_ms":29097,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A compact 7B model trained on simulated future dialogue resolves emotional support goals more often than GPT-4o in the same evaluation.","keywords":["emotional support conversation","reinforcement learning","GRPO","future-oriented reward","multi-agent dialogue simulation","reward model","LLM alignment","open-ended dialogue generation"],"falsifier":"Run RLFF-ESC and a strong baseline against real users and measure their emotional state before and after each conversation with a standard self-report instrument. If users of the RLFF-ESC system show no greater improvement in emotional state than users of the baseline, even though the LLM critic reports higher resolution rates, then the central claim that the future-oriented reward tracks enduring emotional recovery is refuted.","tokens_in":21324,"feed_emoji":"💬","tokens_out":7113,"duration_ms":66197,"temperature":0.7,"pith_summary":"This paper attempts to show that an emotional-support chatbot can be trained to aim at the user's long-term emotional state rather than only at the quality of the next reply. It proposes RLFF-ESC, which uses LLM-simulated future conversations to score each response by whether the user's problem eventually gets resolved, then optimizes a small LLM with reinforcement learning against that score. On two public datasets, the trained 7B model resolves more support goals than much larger models in matched evaluation, and human annotators prefer its responses on empathy, identification, suggestion, and overall quality. The intended payoff is that compact models can deliver open-ended, adaptive support without being restricted to predefined strategy menus.","feed_headline":"Small 7B chatbot beats GPT-4o at resolving support goals","feed_subtitle":"Reinforcement learning on simulated future dialogues lifts success rate to 41.5%, and human judges prefer its replies.","key_machinery":"The load-bearing object is the future-oriented reward model, a binary classifier built from a frozen LLaMA-3.2-1B plus a linear head that predicts whether a response resolves the user's emotional problem. It is trained on $(\\text{context}, \\text{response}, r)$ tuples produced by a three-agent simulation: the policy model acts as supporter, a second LLM plays the user, and a critic LLM grades whether the emotional issue is resolved over simulated future turns; the scalar reward combines the terminal critic rating with the number of turns to resolution. During RL, GRPO samples groups of candidate outputs, normalizes the reward $R_{\\mathrm{rlff}} = R_{\\mathrm{fut}} + \\alpha R_{\\mathrm{thk\\text{-}fmt}}$, and updates the policy to favor responses whose simulated futures end better. The format reward $R_{\\mathrm{thk\\text{-}fmt}}$ forces output into $\\langle\\text{think}\\rangle \\ldots \\langle/\\text{think}\\rangle$ and $\\langle\\text{response}\\rangle \\ldots \\langle/\\text{response}\\rangle$, which is how the explicit reasoning process is elicited. This design steers the model from immediate-token likelihood toward outcome-level consequences.","core_discovery":"RLFF-ESC claims that reinforcement learning from future-oriented rewards converts a general instruction-tuned LLM into an emotional supporter that reasons explicitly and resolves distress in fewer turns. The paper's central result is the success-rate advantage: Qwen2.5-7B with RLFF-ESC reaches 41.5% on ESConv versus 30.8% for GPT-4o, 27.7% for Qwen2.5-72B, and 23.9% for LLaMA-3.1-405B under the same zero-shot evaluation, while also improving average turns to goal. Pairwise human evaluation on 100 dialogues reports wins over every baseline on fluency, empathy, identification, suggestion, and overall quality. The authors interpret this as evidence that optimizing for estimated enduring emotional impact, rather than for matching ground-truth next utterances or for selecting predefined strategies, yields responses that are both more effective and more human-preferred.","pith_inferences":["Beyond the paper: because the reward model and the evaluation success metric are both LLM-based, the reported success-rate gains may partly reflect the policy learning to satisfy the LLM critic rather than to change a real human's emotional state; a user study with direct emotional measures would separate those explanations.","Beyond the paper: the same simulation-reward-GRPO pipeline could be applied to other long-horizon dialogue goals, such as tutoring, behaviour change, or negotiation, wherever the outcome is defined by the downstream conversation state rather than by a single turn.","Beyond the paper: the reward formula's explicit averaging over turns penalizes long conversations, so a testable extension is whether this efficiency pressure ever trades away depth for premature closure in real users who need more time.","Beyond the paper: the threshold analysis in the paper suggests that the definition of 'success' is sensitive to critic calibration; a natural extension is to train the reward model against graded emotional-outcome data rather than a binary resolved/not-resolved label."],"forward_implications":["If RLFF-ESC is correct, a 7B-class model can outperform frontier API models on goal completion in emotional support under matched zero-shot prompting.","Training on simulated future trajectories can substitute for costly human annotation of long-term conversational outcomes, making outcome-driven RL scalable.","Explicit reasoning before responding improves downstream resolution, so interpretable intermediate reasoning can serve as a practical training target for dialogue systems.","The method transfers across backbones (LLaMA-3.1-8B and Qwen2.5-7B) and across datasets (ESConv and ExTES), suggesting that the future-oriented reward signal, not a single model, drives the gain.","A classification-based future reward beats ranking-based and random reward designs in the ablation, so a simple binary 'resolved' prediction is an effective training signal for GRPO in this setting."],"supporting_citations":[{"why":"Introduces the ESConv dataset and the emotional support conversation task, supplying the primary benchmark and official split.","marker":"Liu et al. 2021"},{"why":"Provides GRPO, the group-relative policy optimization algorithm used for all RL updates.","marker":"Shao et al. 2024"},{"why":"Inspires the thinking-format reward via DeepSeek-R1's rule-based reward design, which enforces the reasoning-response structure.","marker":"Guo et al. 2025"},{"why":"Defines the PPDP baseline and the evaluation protocol with user simulator and critic that RLFF-ESC follows and outperforms.","marker":"Deng et al. 2024"},{"why":"Releases the ExTES dataset and serves as the SFT baseline, giving the second evaluation benchmark.","marker":"Zheng et al. 2023"},{"why":"Provides ICL-AIF, the self-play with AI feedback baseline that is the strongest competitor in the main comparison.","marker":"Fu et al. 2023"},{"why":"Provides GPD-Zero, the Monte Carlo tree search planning baseline adapted to the same backbones for comparison.","marker":"Yu et al. 2023"},{"why":"Supplies the LLaMA-3.1 backbones and the frozen LLaMA-3.2-1B used to build the future-oriented reward model.","marker":"Grattafiori et al. 2024"},{"why":"Supplies Qwen2.5-7B-Instruct-1M, the backbone that achieves the headline 41.5% success rate.","marker":"Yang et al. 2025"},{"why":"Provides GPT-4o, which serves as the zero-shot comparison model and as the user simulator and critic during evaluation.","marker":"Achiam et al. 2023"}],"fun_headline_variants":["7B RL model beats GPT-4o in emotional support success","Future-oriented RL lifts 7B chatbot past GPT-4o on support goals","Small model wins at emotional support with future-focus RL","RLFF-ESC: 7B model outperforms GPT-4o on ESConv success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the score an LLM critic gives to simulated future conversations is a valid proxy for real improvement in a real user's emotional state; the paper explicitly did not measure end users' emotional intensity.","fun_headline_variants_meta":{"raw":{"variants":["7B RL model beats GPT-4o in emotional support success","Future-oriented RL lifts 7B chatbot past GPT-4o on support goals","Small model wins at emotional support with future-focus RL","RLFF-ESC: 7B model outperforms GPT-4o on ESConv success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000767,"raw_usage":{"total_tokens":3405,"prompt_tokens":955,"completion_tokens":2450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2368}},"tokens_in":571,"tokens_out":2450,"duration_ms":18238,"temperature":1.0,"reasoning_tokens":2368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:16:55.386335+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RLFF-ESC and a strong baseline against real users and measure their emotional state before and after each conversation with a standard self-report instrument. If users of the RLFF-ESC system show no greater improvement in emotional state than users of the baseline, even though the LLM critic reports higher resolution rates, then the central claim that the future-oriented reward tracks enduring emotional recovery is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the PPDP baseline and the evaluation protocol with user simulator and critic that RLFF-ESC follows and outperforms."}],"review_version":1}