{"id":"1b2bda39-56d4-4ee1-ab67-49b242c262b1","arxiv_id":"2411.16305","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SUIT uses counterfactual turn replacements to select high-quality subgoal training samples and iteratively retrains task-oriented dialog models, reaching a COMBINED score of 105.02 on MultiWOZ 2.2.","lead":"This paper introduces SUIT, a training method for task-oriented dialog systems that learns from successful dialogs by focusing on the turns that were necessary for success. It reports a new state-of-the-art combined score on the MultiWOZ 2.2 benchmark, though the gain over prior systems is small and reported without variance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The same Nekvinda-Dušek success evaluator drives both subgoal selection (§2.3) and final COMBINED scores (Table 1), so the reported SOTA margin may reflect metric overfitting rather than genuine goal completion; replacement counterfactuals also build incoherent turns.","rationale":"The reader's weakest assumption—that the Nekvinda-Dušek success function is a faithful oracle and that its reuse in final metrics creates a circular selection-evaluation loop—is the most load-bearing concern for the paper's central claim. The reported SOTA margin is small, and the method's core innovation is precisely the selection of training data based on this evaluator, so the risk of metric overfitting directly undermines the claim that SUIT improves task completion. I agree with the reader's conditional verdict: the paper should not be rejected, but it cannot be accepted at face value without addressing the evaluator coupling. My attack adds a concrete mechanism for why the circularity could be severe: the counterfactual replacements often create state/action-response mismatches, so the evaluator may flip due to local incoherence rather than the importance of the original subgoal. This strengthens the reader's concern rather than replacing it. The proposed concrete test—swapping the selection-level success function for a database-grounded checker—would disentangle genuine task relevance from metric-specific tuning and would settle whether the SOTA improvement is real. The reader's other points (no error bars, no code release, independence assumption) are valid but secondary; they affect the strength of evidence, while the evaluator circularity threatens the validity of the evidence itself. Therefore the verdict should remain CONDITIONAL, pending independent verification of the success criterion.","tokens_in":10459,"tokens_out":10061,"duration_ms":96205,"concrete_test":"Re-run the SUIT pipeline exactly as described, but replace the Nekvinda-Dušek success function used for subgoal selection with an independent, database-grounded goal-completion checker (e.g., verify that the final booked entity actually exists in the database and satisfies all user constraints and requested slots), while still reporting the final metrics with the standard evaluator. If the selected subgoals are genuinely task-relevant, the COMBINED score on the standard metric should remain competitive with the reported 105.02; if it drops substantially, the original gain was an artifact of optimizing the same checker used for evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—SUIT reaches a new SOTA COMBINED score of 105.02—rests on a pipeline that uses the identical success function (Nekvinda and Dušek, 2021) both to label dialogs as successful/unsuccessful during subgoal selection and to compute the reported INFORM/SUCCESS/COMBINED metrics. In Alg. 1 step (3), a subgoal is retained only if replacing it with a counterpart from an unsuccessful dialog flips this specific evaluator from success to failure; conversely, for DPO the dispreferred samples are exactly those replacements that make the evaluator fail. The final scores in Table 1 are then measured with the same function. This couples the data-selection criterion to the evaluation metric, so the training set is enriched for patterns that this particular rule-based checker treats as necessary. If the evaluator is lenient or gameable—for example, checking the predicted belief state rather than verifying actual database entries, or counting surface mentions of slots—the model can learn to satisfy it without genuinely completing user goals. The problem is amplified by the one-replacement-at-a-time design: replacing a state but keeping the original action/response (or vice versa) can create an incoherent turn that causes the evaluator to flip for mechanical reasons unrelated to the original subgoal's causal role. The reported improvement over DIACTTOD is only 0.62 COMBINED points, with no error bars or significance tests, making it plausible that the gap is an artifact of this evaluator-coupling. Without an independent check of task success, the SOTA claim is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SUIT, an iterative training method for end-to-end task-oriented dialog systems. Starting from a Flan-T5-large model trained with supervised fine-tuning, SUIT samples multiple dialog variants per user goal, evaluates them with the Nekvinda and Dušek success function, and identifies 'relevant subgoals' by turn-wise counterfactual replacement: a subgoal from a successful dialog is kept as training data only if replacing it with the corresponding part of an unsuccessful dialog flips the success evaluation. These selected subgoals are then used for SFT or DPO training, and the process is repeated for up to two iterations. The authors report a COMBINED score of 105.02 on MultiWOZ 2.2, which they claim is a new state of the art, surpassing DIACTTOD's 104.4.","tokens_in":10769,"tokens_out":6133,"duration_ms":59087,"significance":"If the reported gains are robust, the work is a useful contribution: it offers a simple, model-agnostic way to extract dense training signals from sparse dialog-level success, and the ablation in Table 1—where training on the filtered subgoal set (2166 samples) outperforms training on all successful subgoals (31586 samples)—supports the core filtering idea. The method uses only an off-the-shelf LLM and the standard MultiWOZ evaluator, which makes it easy to reproduce in principle, and the paper explicitly names its limitations, including reliance on ground-truth goals and turn-wise replacement. However, the significance is currently limited by the coupling between the evaluator used for data selection and the evaluator used for final scoring, and by the absence of any variance or significance reporting for the headline SOTA claim.","major_comments":[{"comment":"The distant-supervision rule in Algorithm 1 (step 3) selects a subgoal as relevant only if replacing it with a counterpart from an unsuccessful dialog flips the Nekvinda and Dušek success function from success to failure, and Table 1 then reports INFORM, SUCCESS, and COMBINED scores computed with the same function. This creates a selection-evaluation coupling: the training data are enriched for patterns that this particular rule-based checker treats as necessary, so the reported 0.62-point COMBINED improvement over DIACTTOD may reflect overfitting to the checker rather than improved goal completion. The manuscript should evaluate with a different success evaluator (for example, the MultiWOZ 2.1 or 2.4 evaluator, or a human evaluation on a sample) and should report results across at least three seeds to establish that the advantage is not an artifact of this coupling.","section":"§2.3, Algorithm 1, Table 1"},{"comment":"The stopping criterion is described as 'the COMBINED score is not increasing any further' (Section 3.1), and Appendix A.1 states 'As stopping criteria we use the COMBINED score,' but the paper never states whether this decision was made on a held-out development set or on the test set. If the test set was used to choose the number of iterations, the reported 105.02 is an optimistically selected number and the comparison with DIACTTOD, which was not selected in this way, is not apples-to-apples. Please clarify the stopping procedure and, if test-set information was used, rerun the comparison using a development set for model selection.","section":"§3.1 and Appendix A.1"},{"comment":"No error bars, number of seeds, or significance tests are reported for any of the SUIT variants or the baselines. The headline margin over DIACTTOD is 0.62 COMBINED points (105.02 vs. 104.4), and the margin of SUIT2 (DPO-SFT) over SUIT1 (SFT) is only 0.37 points; these gaps are within the range of typical seed variance for MultiWOZ experiments reported elsewhere in the literature. Without a multi-seed comparison and a significance test, the claim 'SUIT reaches new state-of-the-art performance' is not supported. At minimum, please report mean and standard deviation over three to five seeds for the main configurations.","section":"Table 1"},{"comment":"The one-replacement-at-a-time rule assumes that subgoal contributions are independent and additive. Replacing a belief state while keeping the original action/response, or replacing an action/response while keeping the state, can create an internally incoherent turn; for example, the negative state in Table 7 swaps the train departure and destination while the response is unchanged. The evaluator may then flip for mechanical reasons—because the dialog is no longer a coherent whole—rather than because the original subgoal is causally important. The paper should provide evidence that the selected subgoals are coherent and genuinely relevant, for example by human annotation of a sample of selected subgoals or by an ablation that performs joint replacements of state and action/response.","section":"§2.3 and Table 7"}],"minor_comments":[{"comment":"The notation in Algorithm 1 is confusing: line (3) uses 'Do' and 'S′ot' without explicitly defining Do as an unsuccessful dialog, and the set-builder notation in line (1) is hard to parse. Please clarify the definitions of Do, S′ot, and Dc.","section":"§2.2 and Algorithm 1"},{"comment":"The column headers of Table 3 are ambiguous: '# Dialogs' is followed by two numeric columns without clear labels, and the meaning of the six columns under '# Successful Dialogs per Goal' is not immediately clear from the caption. Please relabel the columns or explain them in the caption.","section":"Table 3"},{"comment":"The limitations section acknowledges reliance on ground-truth goals and the turn-wise replacement design, but it does not mention the coupling between the success evaluator used for subgoal selection and the final evaluation metric. This coupling is a central risk of the approach and should be discussed as a limitation.","section":"§6 (Limitations)"},{"comment":"The phrase 'We enforce that the samples contain greedy generations' is vague. Please specify how the greedy generation is combined with the k=2 stochastic samples to produce the stated k^2+1 dialogs per user goal.","section":"Appendix A.1, footnote 1"},{"comment":"In the Ethical Considerations section, 'GPT/LAMA models' should likely read 'GPT/LLaMA models'; as written it is unclear whether the authors mean the LAMA benchmark or something else.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the coupling between the success evaluator used for data selection and the one used for final scoring, together with the lack of any statistical significance evidence for a 0.62-point SOTA margin. These issues are fixable within the manuscript's scope if the authors add multi-seed experiments and an evaluation with a different success function or human judgment, and if they clarify the stopping criterion. I do not see a fatal internal inconsistency, but the SOTA claim is not yet supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a real new idea and one genuinely informative ablation, but the headline SOTA number is not backed by enough statistics to take at face value. I'd send it to review, with the expectation of a solid revision.\n\nThe new thing is SUIT's way of mining training data from a model's own samples: generate many continuations for each user goal, label dialogs success/failure with the Nekvinda-Dušek evaluator, then find subgoals that are causally load-bearing by replacing one turn at a time and seeing when success flips. That is more targeted than just mixing successful dialogs or using contrastive/RL losses. The key result, for me, is Table 1's comparison between selected subgoals (2166 samples) and all successful dialogs (31586 samples): the selected set wins. That supports the paper's central claim that not all successful turns are equally useful. The approach is also cheap and model-agnostic, which is a real virtue.\n\nThe soft spots are what the stress test says, though I wouldn't call them fatal. First, the same evaluator is used both to select subgoals and to compute the reported metrics. That is an honest concern: the training signal and the test metric are the same function, so some overfitting to its particular notion of success is built in. The paper's own numbers show the gain over DIACTTOD is 0.62 COMBINED points with no error bars, no seeds, no significance test. That margin could easily evaporate. Second, the one-replacement-at-a-time rule assumes subgoal contributions are independent, and it can generate incoherent turns like a taxi departure swapped in from a different conversation. The authors acknowledge the turn-wise design but wave away the incoherence. Third, no code is released, which makes the numbers hard to check.\n\nNone of this kills the paper. The subgoal-selection idea stands on its own, and the ablation is the kind of evidence that makes the method credible even if the SOTA claim is weak. What's missing is standard experimental hygiene: multiple seeds, error bars, and ideally an independent or human success judgment to make sure the evaluator isn't being gamed.\n\nWho is it for: people working on end-to-end ToD training, especially anyone using self-sampling or iterative data selection. I'd give it a serious referee rather than desk reject. If I were the reviewer, my main request would be variance reporting and code, plus a bit more discussion of when the counterfactual replacement creates genuinely negative examples.","headline":"The paper presents a genuinely new subgoal-selection method that appears to help, but the SOTA claim needs variance bars and a guard against the success evaluator doing double duty.","tokens_in":11372,"tokens_out":2901,"would_cite":true,"duration_ms":28222,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an iterative training loop which samples dialogs from the model and keeps only subgoals whose replacement flips a success evaluator can train a task-oriented dialog system to new state-of-the-art accuracy on…","keywords":["task-oriented dialogue","subgoal selection","iterative training","distant supervision","direct preference optimization","counterfactual replacement","MultiWOZ 2.2","end-to-end dialog systems"],"falsifier":"Run the same iterative loop with the selected subgoals replaced by an equal-sized random sample of subgoals from successful dialogs; if the random set reproduces the 105.02 combined score, relevance selection is not the cause. Alternatively, rerun subgoal selection with a different goal-completion evaluator: if the selected subgoals change and the reported gain disappears, the result is an artifact of the specific evaluation function.","tokens_in":10221,"feed_emoji":"💬","tokens_out":6830,"duration_ms":56801,"temperature":0.7,"pith_summary":"This paper proposes SUIT, an iterative training procedure for task-oriented dialog systems that learns from sparse end-of-dialog success signals by singling out the subgoals that actually matter. The central claim is that a counterfactual selection rule—sampling several dialogs per user goal, then keeping only the belief states or action/response pairs whose replacement with a counterpart from an unsuccessful dialog flips the success evaluation—produces a small, high-quality training set. On MultiWOZ 2.2, training on the 2,166 selected subgoal samples outperforms training on all 31,586 successful-dialog samples, and two iterations of the loop set a new combined score of 105.02. This matters because end-to-end dialog systems usually receive feedback only at the end of a conversation, and naive use of successful dialogs cannot distinguish turns that cause success from turns that merely co-occur with it.","feed_headline":"Subgoal-only training hits 105.02 on MultiWOZ 2.2","feed_subtitle":"SUIT keeps only the turns whose removal flips success, beating full-dialog training at a 105.02 combined score.","key_machinery":"The load-bearing mechanism is one-replacement-at-a-time counterfactual evaluation. A 'subgoal' is a turn-level piece of a dialog: either the predicted belief state or the joint action/response pair. For each successful sampled dialog, the method finds an unsuccessful dialog with the same user goal and swaps in the corresponding subgoal at the corresponding turn; if the standardized evaluation function flips the dialog from successful to unsuccessful, the original subgoal is kept as relevant training data. This gives a parameter-free credit assignment signal, and the outer loop alternates this selection with SFT or DPO training, sampling fresh dialogs from the newly trained model at each iteration.","core_discovery":"The discovery is that dialog-level success can be decomposed into turn-level credit without human labels or learned reward models. For each user goal, SUIT samples multiple dialogs from the current model, evaluates each with the reference MultiWOZ success function, and applies distant supervision: for every successful dialog, it replaces one turn's belief state or action/response pair with the corresponding turn from an unsuccessful dialog sharing the same goal. If the modified dialog is judged unsuccessful, the original subgoal is marked relevant and becomes a positive training sample, and the replacing subgoal becomes a negative sample for DPO. This selection is what lets a much smaller training set outperform a much larger one, and the procedure can be repeated by sampling from the updated model. The paper reports that the best two-iteration model reaches INFORM 90.0, SUCCESS 87.1, and COMBINED 105.02 on MultiWOZ 2.2, improving on the previous state of the art.","pith_inferences":["A direct extension is to use the selected subgoals as counterfactual explanations: the negative replacements found during selection are concrete examples of what would make the same dialog fail, which could support error analysis or user-facing justifications.","The one-turn-at-a-time rule likely underestimates joint dependencies; subgoals that only matter in combination would be missed, so a multi-turn replacement variant is a natural test.","The method's transfer to other datasets depends on having paired successful and unsuccessful dialogs for the same goal; on data without multiple rollouts, a user simulator or paraphrase-based goal perturbation would be needed to create the contrast."],"forward_implications":["Training on selected subgoals rather than all successful dialogs improves INFORM from 87.0 to 89.8 and SUCCESS from 79.4 to 84.0 in the first SFT iteration, despite using thousands fewer samples.","A second iteration can still help: SUCCESS rises from 84.0 to 87.1, showing that iterative resampling adds useful signal.","Both SFT and DPO benefit from the selected data, and the best configuration combines DPO-selected data with a final SFT pass.","Because the method uses the evaluation function and the model's own samples, it can be applied to any off-the-shelf language model without reward models or human preference annotations."],"supporting_citations":[{"why":"Supplies the success evaluation function used both as the distant-supervision oracle for subgoal relevance and as the final reported INFORM/SUCCESS metrics.","marker":"(Nekvinda and Dušek, 2021)"},{"why":"Provides MultiWOZ 2.2, the benchmark dataset all experiments and comparisons use.","marker":"(Zang et al., 2020)"},{"why":"Direct Preference Optimization, the preference-learning algorithm trained on selected positive/negative subgoal pairs.","marker":"(Rafailov et al., 2024)"},{"why":"DIACTTOD, the previous state-of-the-art end-to-end system that SUIT's combined score of 105.02 surpasses.","marker":"(Wu et al., 2023)"},{"why":"MARS, a contrastive-learning end-to-end baseline used in the leaderboard comparison.","marker":"(Sun et al., 2023)"},{"why":"KRLS, a reinforcement-learning baseline with keyword-sensitive reward, used in the comparison.","marker":"(Yu et al., 2023)"}],"fun_headline_variants":["SUIT trains on only goal-relevant turns, hits 105.02","Iterative subgoal selection reaches new MultiWOZ SOTA","Drop irrelevant turns: dialog training jumps to 105.02","Subgoal-aware iterative training tops MultiWOZ at 105.02"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The selection procedure assumes the automatic success evaluator is a faithful oracle for whether the user's goal was met, and that each subgoal's contribution to success can be assessed by swapping it in isolation.","fun_headline_variants_meta":{"raw":{"variants":["SUIT trains on only goal-relevant turns, hits 105.02","Iterative subgoal selection reaches new MultiWOZ SOTA","Drop irrelevant turns: dialog training jumps to 105.02","Subgoal-aware iterative training tops MultiWOZ at 105.02"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1505,"prompt_tokens":869,"completion_tokens":636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":560}},"tokens_in":485,"tokens_out":636,"duration_ms":6744,"temperature":1.0,"reasoning_tokens":560,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:15:13.651827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same iterative loop with the selected subgoals replaced by an equal-sized random sample of subgoals from successful dialogs; if the random set reproduces the 105.02 combined score, relevance selection is not the cause. Alternatively, rerun subgoal selection with a different goal-completion evaluator: if the selected subgoals change and the reported gain disappears, the result is an artifact of the specific evaluation function.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the success evaluation function used both as the distant-supervision oracle for subgoal relevance and as the final reported INFORM/SUCCESS metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MultiWOZ 2.2, the benchmark dataset all experiments and comparisons use."}],"review_version":1}