{"id":"65889fc6-bdab-42e3-802e-afdd56e1e288","arxiv_id":"2501.11292","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CMR improves multi-party response generation by contrastively learning speaker styles in a first stage and jointly training response generation with contrastive objectives in a second stage.","lead":"This paper introduces CMR, a two-stage contrastive learning framework for multi-party dialogue response generation. It aims to capture speaker styles and thematic transitions without graph annotations, and reports large gains over strong baselines on the FRIENDS and UBUNTU datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation fairness is not established: UBUNTU baselines use original data while CMR uses filtered data; FRIENDS baseline protocol unspecified.","rationale":"The reader identified the evaluation-fairness assumption as the weakest link, and the manuscript itself provides direct evidence for the concern in the UBUNTU comparison: the footnote to Table 1 states that GSN*, ASRG*, and ChatMDG* are reported from the original papers on the original UBUNTU dataset, while CMR uses the filtered subset described in Appendix B. This is an explicit data mismatch, not a mere ambiguity. On FRIENDS, the cleaning procedure is described but its application to baselines is never stated; given the manual removal of low-context samples and the bad-word filter, the test distribution may differ substantially from what baselines in prior work used. The paper reports averages over five runs without variances, so the reader cannot even check whether the large gains are stable. These issues directly threaten the central claim of empirical superiority. The method itself is coherent and the ablations are informative, so a conditional verdict is appropriate: the paper could be accepted after a full rerun of all baselines on the identical cleaned data and splits, with significance metrics reported. I agree with the reader's assessment and see no reason to change the conditional verdict.","tokens_in":13620,"tokens_out":5187,"duration_ms":51674,"concrete_test":"Re-run Table 1 on FRIENDS using the exact cleaned 5000/653 split and training protocol for all baselines (BART, T5, HeterMPC, MADNet, CONT) with identical hyperparameters and 5 seeds; if the F1 gap between CMR-T5 and CONT shrinks below the reported 8.14 vs 6.46 or overlaps within standard deviation, the headline claim fails. For UBUNTU, retrain GSN, ASRG, and ChatMDG on the cleaned subset instead of citing original-paper numbers; if CMR's advantage over same-scale baselines disappears, the generalization claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—CMR significantly outperforms state-of-the-art models—rests on comparisons that are not demonstrably on identical data. In Table 1, the UBUNTU rows for GSN*, ASRG*, and ChatMDG* are explicitly reported from the original papers on the original UBUNTU dataset, whereas CMR is evaluated on the filtered subset defined in Appendix B (dialogues with ≥6 utterances and >512 tokens, ~67k conversations). Comparing a model trained and evaluated on cleaned data against numbers from unfiltered data can produce spurious gains. On FRIENDS, Appendix B describes aggressive cleaning (bad-word list, manual removal of low-context samples, two-speaker removal) yielding 5000/653 train/test conversations, but Section 4 never states whether BART, T5, HeterMPC, MADNet, and CONT were retrained on this exact cleaned split. If they were not, the reported near-doubling of F1 (8.14 vs 6.46 for CONT) could be an artifact of the cleaning rather than of CMR. The absence of error bars or significance tests further prevents assessing whether the differences are robust. This directly undermines the paper's strongest empirical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CMR, a two-stage contrastive learning framework for multi-party dialogue response generation. Stage I trains utterance representations to distinguish speakers via InfoNCE losses; Stage II jointly optimizes response generation with a contrastive objective using gold responses, previous utterances, other speakers' utterances, in-batch samples, and beam-search outputs as negatives. The method is evaluated on FRIENDS and UBUNTU with T5 and LLaMA-3.1 backbones, reporting large gains over graph-based and transformer baselines on automatic metrics, plus LLM-judge and human evaluations. The paper also includes ablations, negative-sample sensitivity, rare-speaker generalization, per-speaker model comparison, and speaker-count robustness analyses.","tokens_in":13852,"tokens_out":2989,"duration_ms":31091,"significance":"If the reported results hold under identical evaluation conditions, CMR would be a useful and inexpensive addition to multi-party dialogue generation: it requires no graph annotations, adds no inference-time overhead, and appears to transfer to decoder-only LLMs. The paper's strength is its breadth of validation: automatic metrics, ablations, human evaluation, LLM preference, and several auxiliary analyses. However, the central comparative claim rests on cross-model comparisons whose data-processing parity is not established, and no variance or significance information is reported despite claims of 'averaged over five runs.'","major_comments":[{"comment":"The UBUNTU comparison is not demonstrably apples-to-apples. The footnote to Table 1 states that GSN*, ASRG*, and ChatMDG* are 'directly report[ed] ... from the original paper' on the original UBUNTU dataset, while CMR is evaluated on the filtered subset described in Appendix B (dialogues with at least 6 utterances and more than 512 tokens, about 67k conversations). A model trained and tested on a cleaned subset is not comparable to published numbers from the unfiltered dataset; the reported F1/BLEU gains may be artifacts of the cleaning. The authors must either retrain and evaluate all baselines on the same filtered data and split, or provide a careful justification that the published results are directly comparable.","section":"Section 4.4, Table 1, Appendix B"},{"comment":"For FRIENDS, the paper never states whether BART, T5, HeterMPC, MADNet, and CONT were retrained and evaluated on the exact cleaned split (5000 training / 653 test conversations after bad-word filtering and manual removal of low-context and two-speaker samples). Without this statement, the near-doubling of F1 over CONT (8.14 vs 6.46) and the gains over MADNet could stem from the aggressive cleaning rather than from CMR. Please clearly specify, for every row in Table 1, whether the model was run by the authors on the same preprocessed data and split, and if not, what data were used.","section":"Section 4.2 and Appendix B (FRIENDS)"},{"comment":"The text says results are 'averaged over five runs,' but no standard deviations, confidence intervals, or significance tests are reported anywhere in the paper. Given that several conclusions rest on small differences in the ablation table (e.g., F1 8.14 vs 8.17 vs 8.20 in Appendix D) and on comparative gains of less than one point, the absence of variance measures makes it impossible to judge whether the improvements are robust. Please report per-run variation and at least basic significance testing (e.g., paired bootstrap or paired t-test on the five runs).","section":"Section 4.4 and Table 1"}],"minor_comments":[{"comment":"The title contains a typo: 'Speaker-ware' should be 'Speaker-aware'; similarly, Section 5.5 contains the phrase 'training separate models for each speakers' and 'significantly poorer performance' that should be grammatically corrected.","section":"Title and Section 5.5"},{"comment":"The FRIENDS dataset is cited to Shmueli and Ku (2019), which is the SocialNLP EmotionX 2019 challenge paper; this appears to be the wrong reference for the FRIENDS multi-party dialogue benchmark. Please verify and use the correct dataset citation.","section":"References, Section 4.1"},{"comment":"The visualization section states 'As shown in Figure 2' when referring to the attention-weight plots; the plots are actually in Figure 4. Also, panels (a) and (b) would benefit from clear axis labels and a caption explaining how the attention weights are aggregated.","section":"Appendix E"},{"comment":"Please clarify which baselines were reimplemented or rerun by the authors versus taken from prior papers; currently only the UBUNTU asterisked entries are marked, leaving the provenance of the FRIENDS baseline numbers unclear.","section":"Section 4.2, Table 1"},{"comment":"The LLM-based preference evaluation would benefit from the exact judge prompt, the sampling protocol for the 500 instances, and an agreement or consistency measure for the GPT-4o judgments; without these details the preference percentages are hard to interpret.","section":"Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"This is a promising empirical paper, but the central claim of state-of-the-art performance is currently not supported because the evaluation protocol does not establish that all comparisons are on identical preprocessed data. The issue is fixable within the manuscript's scope by retraining baselines on the same split or clearly demonstrating comparability, and by adding variance/significance information. The manuscript is appropriate for the journal's scope if these concerns are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe new thing here is a two-stage contrastive training recipe for multi-party response generation: Stage I contrasts utterances to learn speaker-discrimination, Stage II adds contrastive negatives (previous utterance, other speakers, beam-search samples) on top of the generation loss. That specific combination is, as far as I know, not in the literature, and the paper does a thorough job of ablating each piece. The gains over CONT, HeterMPC, and MADNet are consistent across automatic metrics, human eval, and an LLM preference test, and the inference cost is zero beyond the backbone. That is a real plus.\n\nBut the evaluation is not set up to support the claim. The UBUNTU rows for GSN, ASRG, and ChatMDG are explicitly reported from the original papers on the original dataset, while CMR is evaluated on a filtered subset (≥6 utterances, >512 tokens, ~67k conversations). Comparing against numbers from a different, larger, noisier set is not a fair comparison. For FRIENDS, the paper describes aggressive cleaning (bad-word filtering, manual removal of low-context samples, two-speaker removal) ending with 5000 train/653 test conversations, but never states whether BART, T5, HeterMPC, MADNet, and CONT were retrained on this exact split. If they were not, the near-doubling of F1 against CONT could be an artifact of the cleaning. The absence of error bars or significance tests makes it harder to judge robustness, even though the numbers are averaged over five runs.\n\nThat said, the method itself is plausible and well-ablated. The visualization of attention weights is a nice sanity check, and the robustness analyses (speaker count, rare speakers, per-speaker models) are above the norm. The limitations section is honest about training cost and the setting where CMR is most useful.\n\nFor a serious referee, the paper needs a major revision that pins down the evaluation protocol: retrain all baselines on the same cleaned splits, or report results on the original splits, and add confidence intervals. Once that is done, I think it becomes a solid method paper. As it stands, the central claim of SOTA is not established.\n\nWho is this for? Researchers working on multi-party dialogue or contrastive generation. I would not cite it in its current form. I would, however, send it to a serious referee rather than desk-reject it, because the method is novel enough and the evidence, once cleaned up, could be valuable.","headline":"Promising two-stage contrastive framework for multi-party response generation, but the headline numbers are not trustworthy because the evaluation is not on identical data.","tokens_in":14355,"tokens_out":2527,"would_cite":false,"duration_ms":23426,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CMR trains a dialogue model in two contrastive stages—first to distinguish speakers, then to generate—and the paper reports it beats graph-based baselines on FRIENDS and UBUNTU with no extra inference cost.","keywords":["multi-party dialogue","response generation","contrastive learning","speaker awareness","InfoNCE loss","self-supervised learning","LLM fine-tuning","dialogue benchmarks"],"falsifier":"Retrain BART, T5, HeterMPC, MADNet, and CONT on the precise cleaned FRIENDS and UBUNTU splits used by CMR, with identical evaluation protocols; if the reported metric gaps (for instance, CMR-T5 F1 8.14 vs CONT 6.46) collapse to statistical noise, the claim that CMR's contrastive stages drive the gains is falsified.","tokens_in":13424,"feed_emoji":"💬","tokens_out":8231,"duration_ms":69640,"temperature":0.7,"pith_summary":"The paper sets out to show that multi-party response generation does not need annotated graph structures: a model can learn who is speaking and what the conversation is about from self-supervised contrastive training alone. It proposes CMR, which first trains the encoder to pull utterances by the same speaker together and push utterances by other speakers apart, and then fine-tunes generation jointly with a contrastive objective that uses the gold response as positive and several kinds of negatives. On FRIENDS and UBUNTU, CMR with T5 and LLaMA 3.1 backbones outscores graph-based and contrastive baselines on F1, BLEU, and ROUGE, and the gains come with no added inference cost because the contrastive machinery is dropped at test time. A careful reader would care because the recipe is annotation-free and plugs into ordinary pre-trained models, so if it holds it makes multi-party dialogue capability much cheaper to obtain.","feed_headline":"Two-stage contrastive training lifts multi-party dialogue responses","feed_subtitle":"Speaker-aware contrastive training beats graph models on FRIENDS and UBUNTU at no extra inference cost.","key_machinery":"The central object is a two-stage InfoNCE contrastive-learning training scheme. InfoNCE is a self-supervised loss that pulls a query's representation toward a positive example and pushes it away from negatives; here the query is an utterance (Stage I) or a dialogue context (Stage II). Stage I teaches the encoder to separate speaking styles by contrasting same-speaker utterances against other-speaker and in-batch utterances. Stage II couples generation with a contrastive objective whose negatives come from four sources—the target speaker's previous utterance, other speakers' responses, in-batch samples, and beam-search-generated responses—so that the model learns theme shifts and speaker-specific facts while learning to generate. The mechanism is what lets the paper avoid annotated graphs: no graph construction is needed at any point, and the contrastive machinery is removed at inference, leaving a standard backbone.","core_discovery":"CMR's central claim is that the structure and dynamics of a multi-party dialogue can be captured by two stages of InfoNCE contrastive learning instead of by hand-annotated graphs. Stage I, speaker discrimination, makes utterance representations from the same speaker cluster together while separating utterances from other speakers in the same conversation and from random speakers in the batch. Stage II trains response generation together with a contrastive objective whose positive is the gold response and whose negatives are the target speaker's own previous utterance, other speakers' replies, in-batch utterances, and beam-search-generated responses; the authors argue this combination teaches the model thematic transitions and speaker-specific facts. The paper reports that CMR-T5 and CMR-LLaMA beat graph-based models (HeterMPC, MADNet), a contrastive baseline (CONT), and the plain backbones on FRIENDS, with F1 rising from 6.46 for CONT-T5 to 8.14 for CMR-T5, and that the Stage-II-only variant outperforms same-scale baselines on UBUNTU. It also claims, to the best of the authors' knowledge, to be the first application of contrastive learning to multi-party dialogue generation, to generalize to LLaMA 3.1 through LoRA, and to leave inference exactly as expensive as the bare backbone.","pith_inferences":["One testable extension is to apply the same two-stage recipe to other role-structured text-generation tasks, such as meeting summarization or multi-agent collaborative writing, where consistent identity across contexts matters.","Stage I is essentially unsupervised speaker representation learning, so it could be reused as pretraining for diarization or persona-consistent generation even where no generation labels exist.","The magnitude of the FRIENDS gains (F1 nearly doubling over BART) makes the data-cleaning protocol a genuine confound: if the baselines were not retrained on the same cleaned split, part of the reported gap may come from the filtering, not from CMR.","The appendix's finding that CMR still works for rare and unseen speakers suggests the framework may transfer to open-domain games or episodic media, but that transfer is an extrapolation beyond what the paper demonstrates."],"forward_implications":["Removing either stage hurts performance: dropping Stage I lowers F1 by 0.97 points and dropping Stage II by 1.73 points on FRIENDS, so both contrastive stages carry weight.","Because the contrastive components are used only in training, a deployed CMR model is identical to its plain T5 or LLaMA backbone in speed and memory, unlike graph-based systems that build graphs during inference.","On datasets where speakers rarely repeat, like UBUNTU, the Stage-II-only variant still beats same-scale baselines, so the second stage alone transfers the benefit.","The paper's human and LLM-judge evaluations suggest the improvement is visible in speaker appropriateness and style consistency, matching the design goal of the two stages."],"supporting_citations":[{"why":"Supplies the InfoNCE contrastive loss used in both CMR training stages.","marker":"(Oord et al., 2018)"},{"why":"Defines the CONT contrastive text-generation baseline whose F1 and BLEU scores CMR must beat on FRIENDS.","marker":"(An et al., 2022)"},{"why":"Provides HeterMPC, the heterogeneous-graph baseline that CMR compares against to show no annotations are needed.","marker":"(Gu et al., 2022a)"},{"why":"Provides MADNet, the fully-connected-graph baseline with latent addressee inference that CMR claims to outperform.","marker":"(Gu et al., 2023)"},{"why":"Supplies the BART backbone and baseline; the architecture that many graph methods build on.","marker":"(Lewis et al., 2020)"},{"why":"Supplies the T5 backbone used by CMR-T5 and CONT-T5, central to the main experiments.","marker":"(Raffel et al., 2020)"},{"why":"Supplies the FRIENDS dataset used for the main F1/BLEU/ROUGE evaluations and the ablation studies.","marker":"(Shmueli and Ku, 2019)"},{"why":"Supplies the UBUNTU benchmarking corpus used to test the Stage-II-only generalization.","marker":"(Lowe et al., 2015)"}],"fun_headline_variants":["Speaker-aware contrastive learning replaces graphs in multi-party dialogue","First contrastive framework for multi-party dialogue generation","Two-stage contrastive learning beats graph models on dialogue","CMR: Contrastive learning for multi-party response generation","No graphs needed: contrastive learning advances multi-party dialogue"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every comparison model was trained and tested on exactly the same cleaned FRIENDS and UBUNTU splits that CMR used; the paper's appendix describes manual filtering and bad-word removal but the main text never states that all baselines received that same data.","fun_headline_variants_meta":{"raw":{"variants":["Speaker-aware contrastive learning replaces graphs in multi-party dialogue","First contrastive framework for multi-party dialogue generation","Two-stage contrastive learning beats graph models on dialogue","CMR: Contrastive learning for multi-party response generation","No graphs needed: contrastive learning advances multi-party dialogue"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000312,"raw_usage":{"total_tokens":1784,"prompt_tokens":966,"completion_tokens":818,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":740}},"tokens_in":582,"tokens_out":818,"duration_ms":6729,"temperature":1.0,"reasoning_tokens":740,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:25:09.452575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain BART, T5, HeterMPC, MADNet, and CONT on the precise cleaned FRIENDS and UBUNTU splits used by CMR, with identical evaluation protocols; if the reported metric gaps (for instance, CMR-T5 F1 8.14 vs CONT 6.46) collapse to statistical noise, the claim that CMR's contrastive stages drive the gains is falsified.","supporting_citations":[],"review_version":1}