{"id":"29466c8c-1d57-4647-9f37-1672f4f0bec6","arxiv_id":"2506.02827","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"TO-GATE applies trajectory-level direct preference optimization with a weighted response loss to improve preference elicitation dialogues, claiming 83.15% win rate versus 73.83% for STaR-GATE.","lead":"This paper proposes TO-GATE, a training method that uses contrastive learning on dialogue trajectories to make LLM agents ask better clarifying questions and summarize final answers when eliciting user preferences. It claims a 9.32% win-rate improvement over the prior STaR-GATE approach on simulated preference elicitation tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 9.32% response win-rate improvement over STaR-GATE is not statistically substantiated: it rests on a single run over 50 test tasks with no significance testing, and the closest baseline (ETO) is not compared.","rationale":"I read the paper in good faith: TO-GATE proposes a plausible trajectory-level DPO training scheme with a clarification resolver and summarizer, and the training recipe is coherent. The reader's verdict of REJECT is justified, but I believe the single most load-bearing concern is not the circular clarification metric (though that is real and should be fixed) but rather the statistical fragility of the headline response win-rate result. The abstract's '9.32% improvement' is a response win-rate comparison from Table 1, which is generated by a single run on a small simulated benchmark with no measure of variance. Without confidence intervals or a significance test, 'significantly outperforms' is unsupported. The λ inconsistency (Section 5.3 says λ=0.33, Table 2 says λ=2 is best and yields a lower score) further undercuts the reproducibility of the reported configuration. The omission of ETO, the most closely related DPO-based method, makes the SOTA claim incomplete. If the concrete test shows the CI excludes zero and ETO does not match, then the central claim would be on much firmer ground despite the other flaws; as reported, however, the evidence is not trustworthy. I therefore keep the reader's REJECT verdict (UNCHANGED) but partially disagree on which weakness is most load-bearing.","tokens_in":10433,"tokens_out":7014,"duration_ms":75884,"concrete_test":"Re-run the full TO-GATE and STaR-GATE pipelines with 10 random seeds (or bootstrap the 500 test pairs) and compute a 95% confidence interval for the difference in average win rate against M0; also run ETO under identical settings. If the CI includes zero, or ETO achieves a statistically indistinguishable win rate, the 'significant SOTA improvement' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that TO-GATE 'significantly outperforms' baselines rests on Table 1: a single run of win rates against M0 on a simulated test set of 50 tasks × 10 personas = 500 pairs (Appendix B). No standard deviation, confidence interval, or significance test is reported; the difference of 9.32 percentage points (83.15 vs 73.83) could be within run-to-run noise. Additionally, Section 5.3 fixes λ=0.33, while Table 2 — the only λ tuning experiment, on M1 — shows λ=2 is best with 78.66%, far below the main result of 83.15%; this internal inconsistency makes the reported configuration and improvement hard to reproduce. The paper also describes ETO (Song et al., 2024) as the closest DPO-based multi-turn method but never compares to it, so the 'state-of-the-art' claim is incomplete. The circular clarification metric highlighted by the reader is a real flaw, but it does not directly undermine the response win-rate headline; the response evaluation's statistical fragility is the load-bearing issue for the paper's central assertion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TO-GATE, a framework for training an LLM questioner to elicit human preferences through multi-turn clarifying questions and then summarize them into a final personalized response. The method frames this as trajectory optimization: for each task, it samples candidate dialogues, selects the best and worst trajectories according to the base model's likelihood of the gold response, and then trains with a DPO-style contrastive loss that is split into a clarification loss and a response loss with weight lambda. Experiments on a synthetic GATE-style benchmark compare TO-GATE with STaR-GATE and DPO, reporting win rates against the untrained base model and a clarification-score curve. The paper claims state-of-the-art performance, with a 9.32% improvement over STaR-GATE.","tokens_in":10654,"tokens_out":6482,"duration_ms":67378,"significance":"If the empirical claims held, TO-GATE would be a useful extension of STaR-GATE: replacing supervised training on positive trajectories with contrastive training over dynamically filtered trajectories is a plausible way to penalize uninformative clarification dialogues. The deterministic dual-pass evaluation for win rates is a reasonable safeguard against position bias and is a methodological strength. The DPO derivation in Section 4.1 is standard and correctly adapted to the trajectory setting. However, the paper's central empirical claim is not currently substantiated: the headline result rests on a single run, the clarification metric is the same quantity used for training-data selection, the closest DPO-based baselines are described but not compared, and the lambda used in the main experiments is inconsistent with the tuning study. These issues are load-bearing because the paper's contribution is primarily empirical.","major_comments":[{"comment":"The headline response result is a single run on a test set of 10 personas × 50 tasks, with no standard deviation, confidence interval, or significance test. The 9.32-percentage-point gap between TO-GATE (83.15) and STaR-GATE (73.83) is therefore not established as real; it may be within run-to-run and GPT-4 judge variation. Moreover, each row of Table 1 is a win rate against M0, so the difference between rows is an indirect comparison, not a pairwise test. Please report multiple random seeds (at least 3) and paired significance tests over the 50 tasks.","section":"§5.4, Table 1, Appendix B"},{"comment":"The clarification metric in Eq. (11) is the log-probability the base model assigns to the gold response given the dialogue. Algorithm 1 line 4 selects the 'best' and 'worst' trajectories using the same quantity (with π0 in place of QBASE). Training therefore directly optimizes the quantity used to report clarification improvements, so the monotone improvement in Figure 4 is partly by construction and does not provide independent evidence of better preference elicitation. Please evaluate clarification quality with an independent measure, or at least discuss this coupling and show that response win rates improve without relying on the trained metric.","section":"§4.4 (Eq. 11) and Algorithm 1 (line 4)"},{"comment":"The main experiments set λ=0.33, but the only λ sweep reports λ∈{1,2,3,6} and finds λ=2 best at 78.66%, well below the 83.15% reported in Table 1. The paper does not explain why λ=0.33 was selected or how it relates to the sweep, and it does not report the λ=0.33 point. This inconsistency makes the reported configuration hard to reproduce and weakens the claim that the chosen hyperparameters are well motivated.","section":"§5.3 and §5.5 (Table 2)"},{"comment":"The related work identifies ETO (Song et al., 2024) as an extension of DPO to multi-turn dialogues and cites DMPO and SDPO, but none of these methods appears in the experiments. Because ETO is the closest comparison for trajectory-level DPO, omitting it leaves the 'state-of-the-art' claim incomplete. Please include ETO (and ideally DMPO) under the same evaluation protocol.","section":"§2 Related Work and §5.2 Baselines"}],"minor_comments":[{"comment":"The x-axis of Figure 4 shows M0–M4 while Section 5.2 says all models are trained for three epochs; Figure 5 uses M0–M3. Please reconcile the number of epochs displayed and discussed.","section":"Figure 4"},{"comment":"There are numerous typos: 'contrastive learing' in the Figure 1 caption, 'looses' after Eq. (10), 'Reloplayer' in Table 3, and 'tend of reults curves' in Section 5.4.","section":"Throughout"},{"comment":"The abstract contains a stray '}' after the STaR-GATE citation; please fix the LaTeX error.","section":"Abstract"},{"comment":"Equation (1) uses pQBASE before QBASE is defined; please define QBASE before first use and clarify its relationship to π0 in Algorithm 1.","section":"Eq. (1)"},{"comment":"The text says the evaluation 'averages scores across all possible response orderings', but only two orderings are possible for a pair; consider rephrasing to 'both orderings'.","section":"§4.4"},{"comment":"The column headers A-B and B-A are not defined in the caption; please specify which model is A and which is B in the dual-pass evaluation.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper's core idea is reasonable and the DPO derivation is standard, but the empirical support is thin. I recommend major revision with a request for multi-seed experiments, an independent clarification evaluation, and the ETO baseline. If the authors cannot supply these, the paper should be rejected. There is also a concern that the clarification metric is circular, which the authors should address head-on rather than by reinterpreting the existing curves."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: TO-GATE is a sensible incremental idea—apply trajectory-level DPO with separate clarification and response losses on top of STaR-GATE's setup, growing the preference dataset dynamically. That's not a huge leap, but it is a real extension, and the deterministic dual-pass win-rate evaluation is a nice fix for position bias. The paper does something useful there.\n\nWhat's new: the weighted separation of DPO losses for clarifying questions versus final responses, and the trajectory exploration loop that builds positive/negative pairs using the base model's gold-response likelihood. The derivations are standard DPO/Bradley-Terry; no red flags there. The ablation suggests both components matter, and the paper is honest in its Limitations that the whole setup is simulated.\n\nThe soft spots are real and cluster around the evidence. The response win-rate headline—83.15 vs 73.83, the 9.32% gain—comes from a single run on 10 personas and 50 test tasks, with no variance or significance test. That's not enough to support 'state-of-the-art.' The clarification metric in Eq. 11 is the same likelihood used to filter positive/negative trajectories in Algorithm 1, so the monotone clarification curves in Figure 4 are partly built into the training signal; that's circular for the clarification claim, even if it doesn't directly touch the response win-rate. Then there's the lambda inconsistency: the paper fixes lambda=0.33 for the main result, but its own tuning table on M1 shows lambda=2 is best, at 78.66%—well below the 83.15% headline. That mismatch needs an explanation. And ETO, described as the closest DPO-based multi-turn method, is never compared; that weakens the 'outperforms baselines' claim.\n\nNone of this means the method doesn't work. The flaws are fixable: run multiple seeds, report CIs, compare ETO, and re-derive or re-report the lambda choice. The core recipe is coherent and worth a serious referee, but this version doesn't establish the headline. I'd send it to review with a request for major revision, not desk-reject it, and I'd tell the authors to tighten the evaluation and resolve the lambda contradiction. Readers working on conversational alignment and preference elicitation will get something out of the framing; anyone relying on the numbers should wait for a revision.","headline":"A plausible incremental training recipe for preference elicitation, but the evidence as reported is too shaky to accept: single-run small evaluation, a circular clarification metric, and an internal lambda inconsistency.","tokens_in":11192,"tokens_out":1974,"would_cite":false,"duration_ms":21221,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TO-GATE trains LLMs to ask better clarifying questions by optimizing whole dialogue trajectories with a DPO-style contrastive loss, beating prior methods by 9.32%.","keywords":["preference elicitation","clarifying questions","trajectory optimization","Direct Preference Optimization","dialogue agents","personalized response generation","multi-turn dialogue","LLM alignment"],"falsifier":"If a different frozen base model or a different gold-response distribution changed which dialogues are selected as best and worst, and the reported win-rate advantage shrank or reversed, the filtering metric would be shown to be the active ingredient rather than the training method. A direct test would be to train TO-GATE with trajectories selected by human judges or by an oracle reward, and compare win rates against the likelihood-selected version.","tokens_in":10222,"feed_emoji":"🎯","tokens_out":3366,"duration_ms":31326,"temperature":0.7,"pith_summary":"The paper claims that learning to ask clarifying questions and to summarize a final personalized answer works best when the model optimizes whole dialogue trajectories, not when it merely copies high-scoring conversations. Its TO-GATE framework combines a clarification resolver, which uses contrastive preference learning on positive and negative question paths, with a summarizer that separately weights the loss on final responses. On standard preference-elicitation tasks, the trained questioner beats STaR-GATE, the supervised positive-trajectory baseline, with an average win rate of 83.15 versus 73.83. The authors argue that explicitly penalizing degenerate dialogues and separating the learning signal for questions from the signal for final answers is what drives the improvement.","feed_headline":"Trajectory optimization lifts preference elicitation by 9.32%","feed_subtitle":"Contrastive training on good vs bad questioning dialogues beats supervised tuning, reaching 83.15% average win rate.","key_machinery":"The load-bearing mechanism is trajectory optimization with two coupled modules. The clarification resolver repeatedly explores dialogues with a roleplayer, selects the best and worst conversations by the frozen base model's log-likelihood of the gold response, and applies the DPO contrastive loss to prefer the good trajectory over the bad one, with a dynamic dataset that grows through an exploration-collection-training loop. The summarizer adds a separate DPO term for the final response, combined with the clarification loss through a weight parameter λ. A deterministic dual-pass evaluation, which averages win rates over both response orderings, is introduced to remove position bias from the GPT-4 judgments.","core_discovery":"On the paper's own terms, the central discovery is that treating the entire clarifying dialogue as a trajectory and applying Direct Preference Optimization-style contrastive learning to distinguish successful from unsuccessful trajectories yields better questions and final responses than supervised fine-tuning on the best trajectories alone. The improvement comes from two coupled ingredients: penalizing bad conversation paths, and using a separate loss term for the final response with an adjustable weight. The paper reports a 9.32% improvement over the strongest baseline and state-of-the-art results on GATE-style preference elicitation tasks, with the clarification resolver contributing the larger share of the gain.","pith_inferences":["Inference: the evaluation metric and the training filter are the same quantity, log p_QBASE(gold response | dialogue), so the reported clarification gains may partly reflect the model learning to satisfy that particular likelihood signal rather than elicitation quality in general; a human study or a held-out judge would test this.","Inference: the trajectory-optimization loop is not specific to preference elicitation and could transfer to other multi-turn interaction tasks where a gold outcome or user-satisfaction signal is available.","Inference: because the authors note that M3 shows a divergence between clarification and response trends, an interesting extension would be an early-stopping or adaptive weighting schedule keyed to response quality rather than clarification quality alone."],"forward_implications":["Question-asking policies improve from contrastive trajectory learning: TO-GATE's 83.15 average win rate beats STaR-GATE's 73.83.","Penalizing bad trajectories matters: removing the clarification resolver lowers win rate by 5.65 percentage points, more than removing the summarizer, which costs 2.12 points.","Separate losses for questions and final responses improve alignment: a λ near 2, which balances the two terms, outperforms settings that over-weight responses.","Averaging over both response orderings in evaluation removes position bias and gives reproducible comparison scores.","The trend of the curves suggests that additional training epochs could further improve the model, since clarification performance keeps rising through M3."],"supporting_citations":[{"why":"STaR-GATE, the supervised positive-trajectory baseline that TO-GATE extends and compares against.","marker":"Andukuri et al., 2024"},{"why":"Introduces GATE, the preference-elicitation task setup and dataset that the experiments build on.","marker":"Li et al., 2025"},{"why":"Supplies the Direct Preference Optimization contrastive loss that the clarification resolver adapts to dialogue trajectories.","marker":"Rafailov et al., 2023"},{"why":"STaR self-improvement loop that motivates the exploration-collection-training cycle used to build the dynamic dataset.","marker":"Zelikman et al., 2022"},{"why":"Extended Turn-level Optimization, a prior extension of DPO to multi-turn settings that the paper positions against.","marker":"Song et al., 2024"}],"fun_headline_variants":["Optimizing dialogue trajectories yields 9.32% better preference elicitation","Contrastive trajectory learning improves preference elicitation by 9.32%","Penalizing bad question paths lifts preference elicitation 9.32%","9.32% better preference elicitation via trajectory optimization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the frozen base model's likelihood of producing the gold response after a dialogue is a faithful measure of how well that dialogue elicits human preferences, and the same measure both filters training data and scores the results.","fun_headline_variants_meta":{"raw":{"variants":["Optimizing dialogue trajectories yields 9.32% better preference elicitation","Contrastive trajectory learning improves preference elicitation by 9.32%","Penalizing bad question paths lifts preference elicitation 9.32%","9.32% better preference elicitation via trajectory optimization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000994,"raw_usage":{"total_tokens":4149,"prompt_tokens":822,"completion_tokens":3327,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":3249}},"tokens_in":438,"tokens_out":3327,"duration_ms":21317,"temperature":1.0,"reasoning_tokens":3249,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:14:36.291928+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a different frozen base model or a different gold-response distribution changed which dialogues are selected as best and worst, and the reported win-rate advantage shrank or reversed, the filtering metric would be shown to be the active ingredient rather than the training method. A direct test would be to train TO-GATE with trajectories selected by human judges or by an oracle reward, and compare win rates against the likelihood-selected version.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"STaR-GATE, the supervised positive-trajectory baseline that TO-GATE extends and compares against."}],"review_version":1}