{"id":"9bb2cab8-0fa1-4551-be0a-ab8f6b360219","arxiv_id":"2412.14584","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LDPP automatically discovers latent dialogue policies from raw records and uses offline hierarchical reinforcement learning to plan in that latent space, outperforming strong baselines on proactive dialogue benchmarks.","lead":"A new framework called LDPP learns proactive dialogue strategies by mining abstract policy vectors from real conversation records and training a plan generator offline, without predefined strategy lists. It reports higher success rates than existing methods, including a standard ChatGPT baseline, on emotion-support and persuasion benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Primary concern: training and headline evaluation share the same ChatGPT critic, so reported gains may reflect optimization to that critic rather than true dialogue quality; a critic-independent check is needed.","rationale":"I agree with the reader's weakest assumption: the coupling between the training reward and the evaluation metric is the most load-bearing vulnerability. I did not find an internal inconsistency in the RL derivations (Eqs. 3-7 are standard IQL/REINFORCE, modulo an apparent typo in Eq. 5 where log p_phi(ht|zt) should presumably be log p_phi(zt|ht), which does not change the argument if the code implements the intended objective). The paper gives useful independent support: code release, ablation studies, a second critic replication, and human comparison against two baselines. However, those elements do not resolve the closed-loop concern, because the human evaluation does not include the ChatGPT baseline and is too small to establish rank. A critic-independent evaluation is feasible and would settle whether the headline gains are real or an artifact of reward hacking. Therefore my recommendation is UNCHANGED: keep the reader's CONDITIONAL verdict, with the condition being a critic-independent or human-labeled re-evaluation.","tokens_in":24516,"tokens_out":4499,"duration_ms":37712,"concrete_test":"Re-run the full ExTES and P4G self-play evaluations with human expert annotators as the critic on all test-set dialogues (200 per task), while keeping the training rewards from gpt-3.5-turbo-0613 unchanged; annotators assign the same four states and success threshold, blinded to system identity. Compute SSR/SR for LDPP, Standard Prompt+ChatGPT, PPDPP, and LoRA, with inter-annotator agreement and bootstrap confidence intervals. If LDPP's advantage over the ChatGPT baseline disappears or falls within noise under human labels, the central claim is critic-specific and the headline conclusion should be weakened. A cheaper computational check, if human rating of the full test set is infeasible, is to replace only the evaluation critic with a held-out model such as gpt-4o or Claude and report the ranking; a material reordering would corroborate the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LDPP's automatically discovered latent policies and offline hierarchical RL produce superior proactive dialogue, with headline SSR/SR of 0.723/0.903 on ExTES, 0.651/0.781 zero-shot on ESConv, and 0.733/0.795 on P4G. The most load-bearing assumption is that the ChatGPT critic's four-state judgments, with hand-set rewards [-1,-0.5,0.1,1.0], are a valid measure of dialogue success. This assumption is load-bearing for two coupled reasons. First, the same critic (gpt-3.5-turbo-0613) supplies the reward rt used to label the offline dataset (Sections 'Preliminaries' and 'Evaluation Methods: Critic model') and then supplies the success signal used in self-play evaluation (SSR/SR). The whole pipeline is trained to maximize this critic's ratings: the policy-distillation filter in Eq. 2 uses rt, the IQL targets in Eq. 3 use rt, and the advantage weights in Eqs. 5-6 are exp(Q - V), so both the policy planner and the P-Former are optimized toward this critic's notion of progress. If that notion diverges from real user outcomes, the reported gains over baselines may not transfer. Second, the user simulator in self-play evaluation is also ChatGPT, so the system is optimized and assessed inside a closed loop of the same model family; there is no independent gold-standard outcome label. The human evaluation in Table 2 only compares LDPP against PPDPP and LoRA on 50 sampled dialogues per task, not against ChatGPT and not on the full test set, and no significance or variance is reported, so it does not break the coupling. The appendix's Table 6, using critic gpt-3.5-turbo-0125, shows LDPP-w/o-2nd-Stage achieving higher SR than full LDPP on ExTES (0.875 vs. 0.860), further indicating sensitivity to critic identity. The concern is not that the authors are dishonest; it is an evaluation-validity gap: the reported superiority may be an artifact of overfitting to the evaluation critic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LDPP, a three-stage framework for proactive dialogue. Stage 1 discovers fine-grained latent policies from raw system utterances with a VQ-VAE-style encoder and codebook, using a P-Former to inject the continuous policy into a frozen LLM. Stage 2 distills the encoder into a policy planner and pre-trains Q/V networks with IQL, filtering low-reward samples. Stage 3 applies offline hierarchical RL, using IQL-style advantage-weighted updates for the planner and REINFORCE with an exponential-advantage final reward for the generator. Experiments on ExTES, ESConv, and P4G report SSR/SR improvements over predefined-policy, prompt-based, and fine-tuning baselines, including a claim of surpassing ChatGPT with a 1.8B Qwen backbone, along with ablations, human evaluation, and a second-critic check in the appendix.","tokens_in":24892,"tokens_out":5648,"duration_ms":39151,"significance":"If the reported results hold, the paper makes a useful contribution: it removes the need for manually defined policy taxonomies and online interaction for policy planning, and it demonstrates that latent-policy planning can be learned offline and transferred zero-shot. The framework is accompanied by code, an ablation of each stage, sensitivity analyses, and a worked REINFORCE derivation for the token-level MDP. However, the central empirical claim currently rests on a self-play evaluation whose user and critic are from the same ChatGPT family that also supplies training rewards; the paper's own second-critic result does not break that coupling. The significance of the results for real deployment therefore remains to be established.","major_comments":[{"comment":"The same model, gpt-3.5-turbo-0613, is used to assign per-turn rewards r_t on the static dataset and to compute success judgments in the self-play evaluation, and the user simulator is also ChatGPT. Because the reward filter (Eq. 2), the IQL targets (Eq. 3), and the advantage weights (Eqs. 5-6) all maximize this critic's rating, the reported SSR/SR gains in Table 1 may partly reflect overfitting to this specific critic rather than general dialogue quality. Table 6 replaces the critic with gpt-3.5-turbo-0125, but that is still the same model family and the same self-play protocol; it does not provide an independent gold standard. I ask for an evaluation with a critic from a different family or with human-annotated turn-level outcomes, and for a report of the correlation between the critic and human judgments.","section":"Section 'Evaluation Methods', 'Critic model', and Eqs. (2), (3), (5), (6)"},{"comment":"The paper reports that main experiments were run 'at least twice' and averages, but no variance, standard errors, or significance tests are given anywhere. Given the stochasticity of LLM self-play and the fact that each critic score is an average of 10 classifications, the gaps in Table 1 (e.g., LDPP SSR 0.723 vs Standard+ChatGPT 0.650) need error bars and a paired significance test before 'outperforms all baselines significantly' is supportable. Please provide per-run results and bootstrap or similar confidence intervals.","section":"Section 'Main Results' and Section 'Evaluation Methods'"},{"comment":"The paper repeatedly claims to be 'simulation-free' and to avoid 'dynamic interactions with simulated environments,' yet the P4G training set is expanded from 817 to 5,579 dialogues by ChatGPT self-play that completes partial dialogues (first 2-8 turns). Since the P4G test evaluation also uses ChatGPT self-play, the reported P4G gains (Table 1) may partly reflect alignment with the simulation protocol rather than the offline algorithm. Either report P4G results without the synthetic augmentation, or revise the simulation-free claim and discuss the effect of the augmentation.","section":"Appendix, 'Data Augmentation for P4G', and Abstract"},{"comment":"The human study is limited to 50 dialogues per task and to two baselines (PPDPP and LoRA), with no comparison against the ChatGPT prompt baseline that is central to the abstract's 'surpassing ChatGPT' claim, and no significance testing or inter-annotator agreement reported. Moreover, on ExTES, LDPP is not preferred over PPDPP on Identification (8% vs 8%) and is only modestly preferred on Comforting, so the human evidence is weaker than the self-play numbers and should be presented more cautiously.","section":"Section 'Human Evaluation' and Table 2"}],"minor_comments":[{"comment":"The term p_phi(h_t | z_t) should almost certainly be p_phi(z_t | h_t); as written, the high-level objective uses a reversed conditional that is inconsistent with the planner's input. Please correct this notation.","section":"Eq. (5)"},{"comment":"The right panel of Figure 2 labels the method 'LDDP'; this should be 'LDPP'.","section":"Figure 2"},{"comment":"The sentence 'According to Section ,' contains an empty cross-reference; please fill in the intended section number.","section":"Appendix, 'Proof and Explanation of Token-level MDP'"},{"comment":"The phrase 'shorted as LoRA' should be 'shortened as LoRA'; the same typo appears in the Human Evaluation appendix.","section":"Section 'Baselines'"},{"comment":"The caption says ESConv results use the planner trained on ExTES; since the column header already says 'Generalization to ESConv', please make the zero-shot nature of this column explicit to avoid confusion.","section":"Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the algorithmic novelty is real and the code release is helpful, but the current evidence is not sufficient for the strong claims made in the abstract. The requested changes (independent critic or human outcomes, variance reporting, and clarification of the P4G augmentation) are within the scope of a major revision. No concerns about citation behavior or overlap beyond normal conference style."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core idea is real: learning latent dialogue policies from raw transcripts via a soft VQ-VAE and then planning over them with offline hierarchical RL is a sensible alternative to predefined policy taxonomies and online user simulators. The three-stage pipeline is more than a stitching of known parts; the P-Former adapter that lets a frozen 1.8B LLM consume continuous policy vectors is a neat piece of engineering, and the ablation shows stage 3 carries the performance. The visualization and representative utterances suggest the discovered codebook entries are semantically coherent. Credit where due: the code is released, and the experimental coverage across three benchmarks and several base LLMs is above average for this area.\n\nThe soft spots are real and they sit exactly where the stress-test note points. Training rewards and headline evaluation are the same critic model (gpt-3.5-turbo-0613) with the same four-state prompt. The whole pipeline—distillation filter, IQL targets, advantage weights—is optimized toward that critic's opinion, and then the same critic measures success. That is a closed loop. The appendix's Table 6 makes it concrete: with the -0125 critic, LDPP-w/o-2nd-Stage gets higher SR than full LDPP on ExTES (0.875 vs 0.860), so the ranking is critic-sensitive. That is enough to say the reported magnitude of improvement over ChatGPT is not yet established.\n\nOther weaknesses in proportion: no variance or significance tests anywhere, only two-run averages; the sensitivity analysis in Tables 3 and 4 is hard to reconcile with the main configuration (K=24 is not the best K; T=2 outperforms T=8 in SSR); and \"simulation-free\" is an overstatement for P4G because the training set was expanded with ChatGPT self-play generation. The human evaluation is also thin: 50 dialogues per task, only against PPDPP and LoRA, no agreement or significance reported. These are fixable, not fatal.\n\nWho is this for: researchers working on dialogue policy planning, offline RL for dialogue, and anyone who wants a concrete example of latent-action hierarchical RL with a frozen LLM. It deserves a serious referee, but the authors need to break the train/eval critic coupling or the headline claim will keep being questioned.","headline":"The latent-policy offline RL pipeline is a genuine contribution, but the headline numbers are all measured with the same ChatGPT critic used for training, so the size of the claimed win over ChatGPT is not yet established.","tokens_in":25521,"tokens_out":1763,"would_cite":true,"duration_ms":13016,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LDPP shows that proactive dialogue agents can discover fine-grained latent policies directly from raw, unlabeled dialogue records and learn to plan them offline, outperforming predefined-policy baselines and even ChatGPT when paired with…","keywords":["proactive dialogue","latent policy","offline reinforcement learning","hierarchical RL","VQ-VAE","emotion support","persuasion dialogue","P-Former"],"falsifier":"Run LDPP and the strongest baselines in a user study with human participants (or with an independently calibrated, non-ChatGPT reward model) and compare success rates; if LDPP's advantage over the standard prompt shrinks or reverses, the latent-policy planning is optimizing the ChatGPT critic rather than genuinely better proactive behavior.","tokens_in":24354,"feed_emoji":"💬","tokens_out":8605,"duration_ms":59552,"temperature":0.7,"pith_summary":"The paper argues that proactive dialogue systems—agents that must steer conversations toward goals such as emotional support or persuasion—do not need hand-crafted strategy taxonomies or simulated online training. It introduces LDPP, which mines fine-grained 'latent policies' directly from raw dialogue transcripts using a vector-quantized autoencoder, then learns to plan those policies with an offline hierarchical reinforcement learning algorithm. A frozen 1.8B-parameter LLM, guided by a small trainable adapter called P-Former, follows the latent policies to generate responses. On ExTES, ESConv, and P4G, LDPP reports success rates above all baselines, including a ChatGPT prompt baseline, suggesting that small models can surpass much larger ones when given a good policy planner.","feed_headline":"Learned latent policies beat ChatGPT on proactive dialogue","feed_subtitle":"A 1.8B-parameter LLM with a learned planner and adapter outdoes larger models using raw dialogue records.","key_machinery":"The load-bearing object is the learned latent policy codebook $\\mathcal{Z} = \\{Z_k\\}_{k=1}^K$: a set of $K$ continuous policy vectors that, combined through a soft weighted sum of the encoder's distribution, represent fine-grained and mixed dialogue strategies. The P-Former (a stack of transformer layers with learnable policy tokens and cross-attention) converts a latent policy vector into tokens the frozen LLM can read, giving the generator latent-policy-following ability. The optimization machinery is offline hierarchical RL: at the high level, implicit Q-learning (IQL) with expectile regression learns $Q_\\alpha$ and $V_\\beta$ from ChatGPT rewards and weights the planner's updates by the advantage $\\exp(Q_\\alpha(h,z)-V_\\beta(h))$; at the low level, REINFORCE applies the same advantage as the terminal reward for token-level generation with the LLM frozen.","core_discovery":"LDPP's central claim is that dialogue policies can be represented as continuous latent vectors and discovered automatically from raw, unlabeled system utterances. A VQ-VAE-style encoder compresses each utterance into a distribution over a codebook and forms a policy vector as a weighted sum, allowing mixtures of strategies; the generator must reconstruct the utterance from history plus this vector, which forces the latent space to capture policy content. These vectors annotate the training dialogues, a policy planner is initialized by distilling the encoder and pre-trained with implicit Q-learning on ChatGPT-assigned per-turn rewards, and offline hierarchical RL (IQL at the policy level, REINFORCE at the token level) jointly improves the planner and response generation. The paper reports SSR/SR of 0.723/0.903 on ExTES, 0.651/0.781 zero-shot on ESConv, and 0.733/0.795 on P4G, surpassing all baselines in Table 1, including a standard ChatGPT prompt, with human evaluation agreeing on the overall quality.","pith_inferences":["The ChatGPT critic is used both as the RL reward and as the evaluation judge, so the reported advantage may partly reflect overfitting to that critic's notion of 'solved'; an evaluation with human judges or an independent reward model would test whether the gains persist outside the LLM-simulated loop.","The latent policies are currently opaque; pairing each codebook entry with natural-language descriptions generated from its top utterances (as the paper's Table 12 begins to do) is a natural step toward explainable, auditable dialogue strategies.","The framework's recipe—autoencoded action space, offline hierarchical RL, frozen text generator plus adapter—is generic and could be applied to other sequential text tasks with incomplete policy taxonomies, such as tutoring or negotiation, which are also characterized by proactive goals.","One testable implication is that the planner should be able to trade off exploration of rare latent policies against exploitation, which could be checked by measuring success as the codebook size $K$ is varied and the reward threshold is moved."],"forward_implications":["Dialogue agents for emotional support and persuasion can be built from existing corpora rather than expert-defined strategy sets, removing a major bottleneck for new proactive tasks.","Small frozen LLMs with a lightweight planner can outperform much larger prompted models, cutting the cost of deployment.","The same planner trained on ExTES transfers zero-shot to ESConv, suggesting discovered latent policies capture task-general supportive behaviors.","Offline training from static transcripts avoids the sim-to-real gap and API expense of online self-play used by prior methods.","Because latent policies are combinations of codebook entries, the representation can express mixed strategies that coarse discrete taxonomies miss."],"supporting_citations":[{"why":"Supplies the VQ-VAE codebook mechanism that LDPP adapts to discover latent policies from utterances.","marker":"(van den Oord, Kalchbrenner, and Kavukcuoglu 2016)"},{"why":"Provides the offline IQL algorithm used for the policy-level critic and planner updates.","marker":"(Kostrikov, Nair, and Levine 2021)"},{"why":"Defines the PPDPP baseline, the self-play evaluation paradigm, and the ChatGPT critic mechanism that LDPP adopts and improves.","marker":"(Deng et al. 2023b)"},{"why":"Provides the REINFORCE policy gradient used to optimize token-level response generation with advantage rewards.","marker":"(Sutton et al. 1999)"},{"why":"Inspires the P-Former cross-attention adapter that maps latent vectors into LLM-readable tokens, following BLIP-2-style bridging.","marker":"(Li et al. 2023)"},{"why":"Supplies the RoBERTa-Large backbone used for the utterance encoder and policy planner.","marker":"(Liu et al. 2019)"},{"why":"Provides the ExTES emotional-support dataset used for primary training and evaluation.","marker":"(Zheng et al. 2023a)"},{"why":"Provides the P4G persuasion dataset on which LDPP is trained and evaluated.","marker":"(Wang et al. 2019b)"},{"why":"Serves as the zero-shot generalization test set for emotion support.","marker":"(Liu et al. 2021b)"}],"fun_headline_variants":["Auto-discovered latent policies beat ChatGPT in dialogues","LDPP: mining real dialogues for policies that beat ChatGPT","Offline RL on latent policies tops ChatGPT with 1.8B","Proactive dialogue planning without simulation beats ChatGPT","Small model, big win: latent policies from raw data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ChatGPT critic's four-state judgments with hand-set rewards (worse/same/better/solved mapped to -1, -0.5, 0.1, 1.0) are assumed to be a valid reward signal for RL training and a valid measure of dialogue success, and the same critic is used for both, so if this proxy diverges from real user outcomes the reported gains may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Auto-discovered latent policies beat ChatGPT in dialogues","LDPP: mining real dialogues for policies that beat ChatGPT","Offline RL on latent policies tops ChatGPT with 1.8B","Proactive dialogue planning without simulation beats ChatGPT","Small model, big win: latent policies from raw data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1501,"prompt_tokens":994,"completion_tokens":507,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":427}},"tokens_in":610,"tokens_out":507,"duration_ms":4878,"temperature":1.0,"reasoning_tokens":427,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:05:28.640488+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LDPP and the strongest baselines in a user study with human participants (or with an independently calibrated, non-ChatGPT reward model) and compare success rates; if LDPP's advantage over the standard prompt shrinks or reverses, the latent-policy planning is optimizing the ChatGPT critic rather than genuinely better proactive behavior.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the VQ-VAE codebook mechanism that LDPP adapts to discover latent policies from utterances."},{"cited_title":"S.; McAllester, D","cited_arxiv_id":null,"evidence_quote":"Provides the REINFORCE policy gradient used to optimize token-level response generation with advantage rewards."}],"review_version":1}