{"id":"feb09331-471d-47ae-a72f-1344becec8c4","arxiv_id":"1908.09890","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A training procedure that samples negative responses by semantic distance to learn multi-granularity representations improves next-utterance retrieval and downstream transfer.","lead":"Dialog models usually learn one representation at a time. This paper trains several models, each forced to distinguish responses of different similarity, then combines them to improve retrieval and transfer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MGT-vs-ensemble comparison is asymmetric: MGT selects the best checkpoint from 5 separately trained runs, while the ensemble baseline uses 5 checkpoints from one run, so the claimed gains over ensembling may be an artifact.","rationale":"The paper's abstract and Section 5.1 claim that MGT outperforms ensembling, and the only direct evidence is the comparison against a checkpoint ensemble from a single run. Because MGT's components are separately trained models, the comparison is not apples-to-apples; the difference could be entirely due to ensembling methodology rather than multi-granularity. This is not an external-consensus disagreement but an internal control problem: the baseline is weaker than standard practice. The reader's weakest_assumption focused on the provenance of the response encoder used for the distance matrix, which is important for reproducibility but does not directly invalidate the headline comparison. The reader did note checkpoint asymmetry in the rationale, so agreement is partial. A concrete fix, retraining the ensemble with independent seeds, would settle whether the central claim survives. I therefore keep a conditional posture rather than moving to reject outright, but the condition must include a standard ensemble baseline.","tokens_in":10486,"tokens_out":8112,"duration_ms":90927,"concrete_test":"Retrain the \"Ensemble (5)\" baseline as five fully independent dual-encoder runs with different random initializations, using the same hyperparameters, number of epochs, and validation-based checkpoint selection as the five MGT component models. Evaluate the averaged probabilities of the five validation-best checkpoints on MultiWOZ and Ubuntu, reporting paired bootstrap confidence intervals. If this standard independent-seed ensemble matches or exceeds MGT's numbers in Tables 2, 3 and 5, the claimed advantage over ensembling is an artifact of the asymmetric setup; if MGT still wins, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.2 defines the comparison asymmetrically: \"For MGT, the highest performing checkpoint at each granularity is selected using the validation score. For the ensemble method, the top performing checkpoints are selected from a single run.\" MGT therefore trains five independent models, one per granularity, and selects the best validation checkpoint from each, while the \"Ensemble (5)\" baseline is a snapshot ensemble from one training trajectory. These five checkpoints are correlated and contain only one seed's worth of diversity, so the baseline is weaker than a standard ensemble of five independently initialized models. The reported gains of MGT over \"Ensemble (5)\" in Tables 2, 3 and 5 consequently conflate the multi-granularity manipulation with the number of training runs (five versus one) and with five times the checkpoint-selection freedom. This is the load-bearing comparison for the paper's central claim that MGT outperforms ensembling. The missing hard-negative-only ablation is a secondary concern: even with a fair baseline, that control would be needed to isolate \"multi-granularity\" from \"hard negatives.\" But the immediate flaw is that the baseline does not represent standard ensembling.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Multi-Granularity Training (MGT), a negative-sampling procedure for next-utterance retrieval. The method splits the pool of possible negative responses into distance-based segments using cosine similarity of response encodings (Eqs. 9-12), trains one model per segment, and ensembles the L models at inference. Experiments on MultiWOZ and Ubuntu use a dual-encoder baseline and the DAM architecture, reporting retrieval gains over a single model and over an ensemble of five checkpoints (Tables 2 and 3), probe experiments that associate granularity level with bag-of-words and dialog-act prediction (Table 4), and transfer experiments that show gains for linear probes and fine-tuning (Tables 5 and 6). The paper claims that MGT learns multiple distinct granularities of representation and that this yields stronger and more transferable representations.","tokens_in":10724,"tokens_out":3054,"duration_ms":36666,"significance":"If the results are robust, the contribution is worthwhile: MGT is architecture-agnostic, is evaluated on two corpora, and gives a concrete training-side mechanism for controlling representation granularity without architectural changes. The idea of replacing random negatives with distance-controlled negatives is simple and potentially reusable. The paper also provides a useful comparison against ensembling, although that comparison is currently not fair, and the granularity analysis is partially circular. With a corrected experimental protocol, a hard-negative ablation, and clearer statements about the distance encoder, the central claims would be much better supported. As written, the evidence is not yet sufficient for the paper's main conclusions.","major_comments":[{"comment":"The headline comparison between MGT and ensembling is asymmetric. Section 4.2 states: 'For MGT, the highest performing checkpoint at each granularity is selected using the validation score. For the ensemble method, the top performing checkpoints are selected from a single run.' This means MGT uses five independently initialized training runs (one per granularity) and selects the best checkpoint from each, while Ensemble (5) uses five checkpoints from one run and therefore has only one seed's worth of diversity. The gains of MGT over Ensemble (5) in Tables 2, 3, and 5 conflate the multi-granularity sampling with the number of independent runs and with five times the checkpoint-selection freedom. The authors should rerun the ensemble baseline with five independently seeded models, selecting the best validation checkpoint from each run, and use the same compute budget. Without this, the central claim that MGT outperforms ensembling is not supported.","section":"Section 4.2"},{"comment":"The granularity analysis in Table 4 is close to circular. Each of the L=5 models is trained exclusively on negatives drawn from one distance segment, so it is nearly guaranteed that the model trained on closest negatives does best at word-level bag-of-words prediction and that the model trained on far negatives does best at abstract dialog-act prediction. The observed trend largely confirms that the sampling procedure assigned the models to the expected buckets, not that the representations are organized by granularity in a way that would arise from the training objective itself. A stronger test would evaluate each model on held-out candidate sets stratified by the same distance segments, or probe all L models on both the close and far subsets; if the high-granularity model is better only on close-negative subsets and the low-granularity model only on far-negative subsets, that would be direct evidence of the claimed specialization.","section":"Section 5.2, Table 4"},{"comment":"The construction of the distance matrix D is underspecified and potentially self-referential. The paper defines d(R_i, R_k) via an LSTM response encoder f_r but does not state how f_r is trained, on which data, or whether it is frozen before any MGT model is trained. If f_r is trained on the same corpus and is one of the models whose representations are later evaluated, then the negative sampling is partly a function of the very model under test, and the reported transfer results may be inflated. The manuscript should state the provenance of f_r, clarify that it is held fixed during MGT training, and ideally report sensitivity to the choice of f_r.","section":"Section 3.3, Eq. (11)"},{"comment":"The paper repeatedly claims 'All bold-face results are statistically significant to p < 0.01' (Tables 2-6), but Section 4.2 reports no number of independent runs, no standard deviations, and no description of the significance test. With a single run or correlated runs, the p-value claim cannot be checked. The authors should report the number of seeds, the variance across runs, and the exact test used for each bolded comparison.","section":"All tables, Section 4.2"}],"minor_comments":[{"comment":"The definition of P_i^l uses b(D_i, l-1) as a lower bound, but b is defined only for l in [1,L] and no value is given for b(D_i, 0). This should be clarified, for example by defining b(D_i, 0) = -infinity or by stating that segment 1 has no lower bound.","section":"Section 3.3, Eq. (12)"},{"comment":"The table and caption contain typos: '3nd', 'suggest' (missing 't'), and 'Alo ng'. These should be corrected.","section":"Table 1"},{"comment":"The footnote about the DAM reimplementation being slightly below the published performance is useful, but the sentence 'We speculate that given a DAM implementation that matches their reported results, MGT would obtain a similarly-sized improvement' is speculative and should either be supported by an additional experiment or removed.","section":"Section 4.2.3 / footnote 1"},{"comment":"The phrase 'roughly double the improvement obtained by ensembling' refers to a difference of +0.76 versus +0.41 on R10@1; presenting the raw numbers and a confidence interval would be more informative than a ratio of small increments.","section":"Section 5.1.2"}],"recommendation":"major_revision","confidential_remarks":"The reviewer believes the core idea is promising, but the main empirical claim requires a fair ensemble baseline and better statistical reporting. The granularity analysis also needs a non-circular validation. These are addressable with additional experiments, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: MGT is a simple, architecture-agnostic training trick that does something genuinely new. Instead of sampling negatives uniformly, it buckets responses by cosine distance from the ground-truth response, trains one model per bucket, and ensembles them. On its face, it helps retrieval on MultiWOZ and Ubuntu, and the transfer results are plausible. The segmentation idea is not in the cited literature, and the paper explains it clearly. Credit where due: the granularity analysis in Table 4 is a nice check, and applying MGT to both a dual encoder and DAM shows the procedure has some generality.\n\nThe soft spots are real and they hit the paper's central claim. The comparison against ensembling is asymmetric. Section 4.2 says MGT selects the best checkpoint from each of five separately trained models (one per granularity), while the ensemble baseline selects top checkpoints from a single run. That gives MGT five times the run diversity and five times the checkpoint-selection freedom. A fair baseline would be five independently initialized models, each trained normally, with best checkpoints selected per run. Without that, the reported gains over \"Ensemble (5)\" conflate multi-granularity with just having a stronger ensemble.\n\nSecond, there is no hard-negative-only control. If you train one model on the closest bucket, you might capture most of the gain. The paper never isolates \"multi-granularity\" from \"hard negatives.\" Table 4's ranking is somewhat self-fulfilling: models trained on close negatives should be better at fine-grained tasks. The transfer results in Table 5 are more interesting, but they inherit the same baseline problem.\n\nThird, the distance matrix construction is underspecified. Which encoder computed the cosine similarities? If it's the baseline dual encoder trained on the same data, the negative sampling is partly self-referential, though not circular with respect to retrieval evaluation. Fourth, all tables claim p < 0.01 but report no error bars or run counts. That is a minor fix but should be done.\n\nWho is this for? People working on response selection and dialog representation learning. The idea is worth engaging with, and the paper is honest and clearly written. But the load-bearing comparison needs to be fixed. My recommendation: send it to peer review, because the idea deserves referee time. Expect heavy revision: add a proper independent-run ensemble baseline, a hard-negative ablation, and variance reporting. If the gains survive those controls, this is a solid contribution.","headline":"A clever negative-sampling idea that improves retrieval, but the key ensembling comparison is set up unfairly and needs a hard-negative control.","tokens_in":11212,"tokens_out":1946,"would_cite":true,"duration_ms":21185,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multi-Granularity Training samples negative responses from distance-based buckets, forcing a dialog model to represent language at multiple granularities and improving next-utterance retrieval, transfer, and fine-tuning.","keywords":["multi-granularity training","negative sampling","next utterance retrieval","dialog representation learning","dual encoder","deep attention matching","representation transfer","response selection"],"falsifier":"Repeat the MGT experiments with the distance matrix $D$ replaced by random bucket assignments (or by a frozen, untrained encoder), keeping all other settings fixed; if retrieval and transfer scores stay at MGT levels, the semantic-distance segmentation is not what carries the improvement.","tokens_in":10286,"feed_emoji":"💬","tokens_out":11226,"duration_ms":96037,"temperature":0.7,"pith_summary":"Multi-Granularity Training (MGT) is a training-time procedure that controls the granularity of latent dialog representations by changing how negative candidate responses are sampled. Instead of drawing negatives uniformly from all responses, MGT sorts every response by its cosine distance to the ground-truth response, splits the sorted list into $L$ segments, and trains one model per segment; near negatives force fine-grained word-level representations, far negatives force abstract topic-level ones. The paper reports that ensembling these $L$ models beats a same-size ordinary ensemble on next-utterance retrieval on both MultiWOZ (MRR 82.74 vs 81.53) and Ubuntu (R10@1 68.7 vs 66.9 for dual encoders; 75.30 vs 74.95 for DAM), and that frozen MGT representations transfer better to bag-of-words and dialog act prediction. If the claim holds, MGT offers a parameter-free, architecture-agnostic way to get stronger and more general dialog representations from any retrieval model.","feed_headline":"Distance-bucketed negatives improve dialog retrieval and transfer","feed_subtitle":"Sorting negative responses by semantic distance improves dialog models with no new parameters.","key_machinery":"The load-bearing object is the distance-based negative-sampling rule in Eq. 12: for each ground-truth response $R_i$, all responses in the training set are sorted by cosine distance $d(R_i, R_k)$ computed from the LSTM response encoder's output embeddings, the sorted list is split into $L$ distance segments, and the model assigned to granularity $l$ samples its $k-1$ negatives uniformly from the $l$-th segment. This turns the otherwise arbitrary choice of negatives into a dial over representation granularity: lexically close negatives require word-level discrimination, distant negatives require abstract, topic-level discrimination. The $L$ trained models are combined by averaging their softmax scores (Eq. 7), and the same distance matrix doubles as an analysis instrument when each model's frozen representations are probed with a linear layer on bag-of-words and dialog act prediction.","core_discovery":"The central claim is that explicitly training a set of models on negative candidates drawn from different distance buckets makes each model specialize at a different granularity, and combining their predictions yields better next-utterance retrieval than a standard ensemble of identical models. The paper shows this on two datasets and two base architectures, with the largest retrieval margins on MultiWOZ (82.74 MRR, 72.18% Hits@1 vs 81.53 and 69.47% for the ensemble) and on Ubuntu with dual encoders (68.7 vs 66.9 R10@1) and DAM (75.30 vs 74.95 R10@1). The granularity analysis is the load-bearing evidence for the mechanism: frozen models from the closest bucket score best on bag-of-words prediction and worst on dialog act prediction, while the farthest bucket inverts that ordering. MGT representations also transfer to downstream tasks better than the baselines, both frozen (BoW F1 67.51 vs 64.11; DA F1 22.85 vs 22.39) and after fine-tuning on dialog acts (33.46 vs 31.71).","pith_inferences":["MGT could be iterated: after the $L$ models are trained, their own encoders could rebuild the distance matrix and re-bucket the negatives, potentially sharpening granularity control beyond the single pass the paper tests.","The same distance-bucketed negative sampling may transfer beyond response selection, for example to contrastive representation learning or hard-negative mining in retrieval-augmented generation, where the bucket index acts as a difficulty dial.","The paper's granularity ordering suggests a practical recipe for task transfer that it does not explore: choose the bucket whose granularity matches the target task, or learn a per-task weighting over buckets, instead of always ensembling all $L$.","A cleaner test of the mechanism would compare MGT against non-semantic bucketings (e.g., random or lexical-only splits) to rule out the possibility that any structured negative sampling, not semantic distance, drives the gains."],"forward_implications":["On MultiWOZ, a five-model MGT ensemble reaches MRR 82.74 and Hits@1 72.18%, beating a same-size ordinary ensemble at 81.53 MRR and 69.47% Hits@1.","On Ubuntu, MGT raises R10@1 from 66.9 to 68.7 for dual encoders and from 74.95 to 75.30 for DAM, roughly doubling the improvement that plain ensembling gives.","Granularity is explicit, not just diversity: the highest-granularity model is best at bag-of-words prediction (59.43 F1) and worst at dialog act prediction (15.46 F1), with the ordering reversed for the lowest-granularity model.","MGT representations transfer better without fine-tuning (BoW F1 67.51 vs 64.11 for the ensemble; DA F1 22.85 vs 22.39) and after fine-tuning on dialog acts (33.46 vs 31.71).","Because the change is purely in training data construction, MGT can be dropped onto any retrieval architecture without adding parameters at inference time."],"supporting_citations":[{"why":"supplies the dual encoder architecture and the Ubuntu retrieval corpus used in the main experiments.","marker":"(Lowe et al., 2015)"},{"why":"provides the DAM architecture, including the open-source implementation used for the second architecture experiment.","marker":"(Zhou et al., 2018)"},{"why":"provides the MultiWOZ dataset used for retrieval training, validation, and the granularity and transfer analyses.","marker":"(Budzianowski et al., 2018)"},{"why":"defines the ensemble method that serves as the comparison baseline for MGT.","marker":"(Perrone and Cooper, 1992)"},{"why":"introduces the retrieval baselines and training setup on Ubuntu that the dual encoder experiments build on.","marker":"(Kadlec et al., 2015)"},{"why":"provides the pretrained GloVe embeddings used to initialize the Ubuntu encoders.","marker":"(Pennington et al., 2014)"}],"fun_headline_variants":["Distance-bucketed negatives refine dialog retrieval","Granularity-specific negatives lift dialog accuracy","Multi-granularity training outperforms ensembles","Distance-sorted negatives improve transfer and retrieval","Negative sampling by distance yields granular models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the cosine similarity between response-encoder outputs is a reliable measure of semantic distance, so that partitioning candidates by this score genuinely separates fine-grained from abstractly different responses; the paper does not specify how that response encoder is trained.","fun_headline_variants_meta":{"raw":{"variants":["Distance-bucketed negatives refine dialog retrieval","Granularity-specific negatives lift dialog accuracy","Multi-granularity training outperforms ensembles","Distance-sorted negatives improve transfer and retrieval","Negative sampling by distance yields granular models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1272,"prompt_tokens":853,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":469,"tokens_out":419,"duration_ms":4834,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:58:21.994771+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the MGT experiments with the distance matrix $D$ replaced by random bucket assignments (or by a frozen, untrained encoder), keeping all other settings fixed; if retrieval and transfer scores stay at MGT levels, the semantic-distance segmentation is not what carries the improvement.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the DAM architecture, including the open-source implementation used for the second architecture experiment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the ensemble method that serves as the comparison baseline for MGT."}],"review_version":1}