{"id":"5c891ae3-5243-4852-8316-f0bed3bd8cec","arxiv_id":"2508.19996","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ReSURE reduces the harm of noisy dialogue data during fine-tuning by grouping samples by dialogue depth and softly down-weighting high-loss examples, improving multi-turn benchmarks modestly.","lead":"A new fine-tuning method called ReSURE reweights dialogue training samples based on how many turns they have, down-weighting examples with unusually high losses. It aims to fix the common problem where adding more dialogue data degrades a chatbot's quality, and it reports small gains on several benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The positive-scaling claim rests on Spearman correlations that are not consistent with the three mixture levels shown; 0.211 and 0.800 cannot arise from three untied points, so the central evidence needs reconciliation.","rationale":"The reader correctly flags thin empirical support and the mismatch between the stated per-turn motivation and the actual whole-dialogue reweighting in Equations (5)-(6). My concern is adjacent but sharper: the abstract's central scaling evidence, the Spearman correlations, is internally inconsistent as presented. With only three mixture levels, values such as 0.211 and 0.800 cannot come from a standard Spearman calculation, so either the number of levels is under-reported or the statistic is wrong. This is not a question of taste or additional experiments; it is a defect in the reported evidence for the paper's most prominent quantitative claim. I also agree with the reader that single-seed results with no variance and a very small human evaluation leave the robustness claims under-supported. Credit is due for a simple, clearly implemented method, released code, and evaluations across multiple model families; however, those positives do not repair the broken scaling analysis. The manuscript should not be accepted or conditionally accepted as-is. It needs a corrected, transparent computation of the correlations and, ideally, multi-seed results before the central claim can be assessed. The unresolved 'Figure X' placeholder in Section 4.4.2 further prevents verification of the scaling curves. My verdict is UNVERDICTED rather than REJECT because the underlying method may survive a corrected analysis; the current manuscript simply does not support the headline claim.","tokens_in":18575,"tokens_out":7606,"duration_ms":75953,"concrete_test":"Recompute Table 4 from the raw per-level scores used to draw Figure 2. For each benchmark, list the mixture levels (x) and scores (y), then run scipy.stats.spearmanr and report n and the p-value. If n=3, valid coefficients are -1, -0.5, 0.5, or 1, so 0.211 and 0.800 are impossible as computed and must be corrected or replaced with a different analysis; if ties or additional levels were used, disclose them and recompute with significance. As a second check, run the H vs H+N+L comparison with at least 3-5 seeds and report mean ± standard deviation to confirm the positive trend is not a single-seed artifact.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The paper's headline claim is that ReSURE 'enjoys positive Spearman correlations (0.21, 1.00, 0.80)... regardless of data quality,' presented as evidence that it can leverage increasing mixed-quality data. The only direct support is Table 4, which reports correlations between dataset complexity and performance. The text and Figure 2 describe exactly three mixture levels: H, H+N, and H+N+L (Section 4.4.2). For three untied points, Spearman's rho can only take values in {-1, -0.5, 0.5, 1}; 0.211 and 0.800 cannot arise. Therefore either the table was computed over a different, undocumented set of points, or the values are miscalculated. The same section also refers to an unresolved 'Figure X' placeholder, so the underlying scaling curve cannot be inspected. Because this correlation is the paper's central evidence that more mixed-quality data helps rather than hurts, the abstract's scaling claim is not currently supported by the manuscript. In addition, the methodology described in Equations (5) and (6) reweights a single whole-dialogue loss rather than isolating per-turn errors, which is a separate design concern; however, the statistical defect alone is decisive for the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReSURE, an online loss-reweighting method for fine-tuning multi-turn dialogue models. The method groups training dialogues by their maximum turn depth, maintains online per-group loss statistics with Welford's algorithm, flags samples whose whole-dialogue loss exceeds a turn-group-specific threshold, and softly down-weights those samples. Experiments compare ReSURE against Vicuna-Tuning, Baize, and ChatGLM3 on in-domain, MT-Bench, and MT-Bench-Ext, using mixtures of datasets labeled high, normal, and low quality. The paper reports that ReSURE consistently improves or stabilizes performance as more mixed-quality data is added, and claims positive Spearman correlations (0.21, 1.00, 0.80) between dataset complexity and performance across the three benchmarks, interpreting this as evidence that ReSURE can leverage increasing amounts of mixed-quality data without degradation.","tokens_in":18778,"tokens_out":8115,"duration_ms":70782,"significance":"If the empirical claims hold, ReSURE is a simple, computationally light alternative to static data filtering that could easily be dropped into existing instruction-tuning pipelines, and the authors do provide public code. The turn-depth-grouped loss statistics are a reasonable heuristic, and the soft-reweighting design is more flexible than hard filtering. However, the headline scaling claim is central to the paper's contribution and is currently not supported by the reported statistics: the Spearman values in Table 4 are inconsistent with the three mixture levels described in the experiment. The method's motivation and implementation also diverge (whole-dialogue reweighting versus per-turn loss reweighting), and the empirical evaluation is thin (single seed, three-point scaling curves, a 20-dialogue human evaluation). These issues materially affect the paper's validity and require substantial revision rather than minor copy edits.","major_comments":[{"comment":"The reported Spearman correlations (ReSURE: 0.211, 1.000, 0.800) cannot be obtained from the three mixture levels (H, H+N, H+N+L) described in Section 4.4.2 and Figure 2. For three untied points, Spearman's rho can only take values in {-1, -0.5, 0.5, 1}; the values 0.211 and 0.800 are impossible without additional data points or a different definition of dataset complexity. Even allowing ties among the three performance values, no tie pattern reproduces these coefficients. This makes the abstract's scaling claim unsupported. Please specify exactly how many points were used, provide the underlying data, and recompute the correlations; if the correlations are over individual datasets rather than mixture levels, clarify that. The unresolved \"Figure X\" placeholder in this section also prevents verification of the scaling curves.","section":"Section 4.4.2 and Table 4"},{"comment":"The method computes turn-group statistics but applies a single weight to the whole-dialogue loss ℓ_s. The threshold in Equation (5) uses the maximum turn-group index b of the sample, and Equation (6) down-weights the entire sample loss. This is whole-dialogue reweighting, not per-turn loss reweighting. The paper's motivation emphasizes early-turn error propagation, but the implementation cannot isolate which turn caused the high loss, so the claimed mechanism is not directly realized. Either change the method to compute and reweight per-turn losses, or revise the framing to describe ReSURE as down-weighting entire dialogues whose aggregate loss is anomalous within a turn-depth group.","section":"Section 3.2 and 3.3, Equations (5)-(7)"},{"comment":"The sentence \"ReSURE consistently outperforms the base model by 6.11%, 9.82%, and 2.86% on the in-domain benchmark, MT-Bench, and MT-Bench-Ext, respectively\" is misleading. These percentages are the maximum gains, obtained on ShareGPT, not representative of all datasets. Table 1 shows negative relative changes for M2Lingual (-0.56% on In-Domain-Test), WildChat (+0.88% only; -0.56% on In-Domain? actually +0.88% on In-Domain, +1.12% on MT-Bench, +1.20% on Ext), and other cases where ReSURE is below the base model. Please report the mean and per-dataset relative changes, and avoid claiming uniform superiority without statistical support.","section":"Section 4.4.1 and Table 1"},{"comment":"The scaling conclusion rests on three points per curve, a single random seed (seed=42), and no error bars or confidence intervals. Small evaluation noise at any level could change the sign of the trend, and the 0.211 in-domain correlation is close to zero. The paper should include multiple seeds or bootstrap intervals, and the discussion should acknowledge the fragility of three-point trend claims. This is particularly important because the positive-scaling claim is the paper's main advertised contribution.","section":"Section 4.4.2 and Figure 2"}],"minor_comments":[{"comment":"Equation (4) uses \"M2(b)_s\" which is not defined; the text defines SSD(b)_s and calls it the Sum of Squared Deviations. Please unify the notation with Welford's standard M2 term, or consistently use SSD throughout.","section":"Section 3.1, Equation (4)"},{"comment":"The subscripts in Equation (2) are inconsistent: the left-hand side and update use µ(b)_s and µ(b)_s-1, while the right-hand side reference µ(b)_t-1 and t(b). Clarify the indexing so the recursion is unambiguous.","section":"Section 3.1, Equation (2)"},{"comment":"The reference \"Figure X\" is an unresolved placeholder; the associated claims about Vicuna-Tuning dropping by 0.75 on MT-Bench and 0.90 on MT-Bench-Ext cannot be checked. Include the figure or cite the correct existing figure.","section":"Section 4.4.2"},{"comment":"The human evaluation uses only 20 dialogues (10 each from MT-Bench and MT-Bench-Ext). With three annotators, the claim that ReSURE outperforms baselines \"across all dimensions\" needs inter-annotator agreement statistics and confidence intervals, or the claim should be softened.","section":"Appendix C"},{"comment":"The caption contains a typo: \"Vicuna-T uning\" should be \"Vicuna-Tuning\".","section":"Figure 3 caption"},{"comment":"The reference list contains duplicates: Wu et al. 2023a and 2023b are the same paper, and the same for other entries. Please deduplicate and use consistent citation labels.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The Spearman correlation issue is the most serious problem: the reported values are arithmetically impossible for the described experiment, and the scaling claim is the paper's headline. If the authors cannot supply a reproducible computation that yields 0.211 and 0.800, the scaling claim should either be removed or drastically qualified. The turn-level motivation versus whole-dialogue implementation gap also needs direct addressing, as reviewers may otherwise question the method's premise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Alex,\n\nQuick take: this is a modest but sensible loss-reweighting idea applied to multi-turn dialogue fine-tuning. It is not a conceptual breakthrough, and the empirical package is thinner than the abstract suggests. The one thing you should know: the central evidence for “positive scaling” is Table 4, where ReSURE reports Spearman correlations of 0.211, 1.000, and 0.800 across three mixture levels (H, H+N, H+N+L). With three untied points, Spearman can only take the values -1, -0.5, 0.5, or 1. The values 0.211 and 0.800 cannot come from three points. Either the table was computed over a different, undocumented set of points, or the numbers are simply wrong. The same section also refers to an unresolved “Figure X” placeholder, so the underlying curve cannot be inspected. This is load-bearing: the abstract’s claim that ReSURE “paves the way” for using large-scale mixed-quality data rests on those correlations.\n\nCredit where it is due. The core mechanism is easy to grasp and implement: group samples by maximum turn depth, maintain online mean and standard deviation via Welford, and softly downweight losses above mean + 1 standard deviation with an exponential weight and a percentile floor. This is an established family of robust loss reweighting, and the turn-depth grouping is a reasonable domain twist. The paper ships code, tests four model sizes across Qwen and LLaMA, includes a task-mixture experiment with GSM8K, and an ablation shows the reweighting term matters, even if the improvements are around one percent. Table 1 shows consistent small gains over the base model across most datasets and benchmarks. So there is a real, if modest, effect underneath the packaging.\n\nSoft spots, in proportion. First, the Spearman problem above. Second, the motivation says early-turn errors propagate, but the method reweights a single whole-dialogue loss per sample, grouped by max turn depth; it never isolates turn-level errors, so the story is weaker than the title. Third, single seed, no error bars, no significance tests, and a human evaluation of only 20 dialogues. Fourth, no comparison against the closest reweighting baselines, such as Learning to Reweight or generalized cross-entropy; Vicuna, Baize, and ChatGLM3 are fine-tuning baselines, not robust-loss baselines. Fifth, hyperparameters like alpha and the percentile floor are tuned on validation; that is normal but should be stated more plainly.\n\nWho is this for? People working on noisy instruction data for dialogue. The paper is salvageable: fix the correlations, add multiple seeds with variance, clarify the per-turn versus whole-dialogue mismatch, and include at least one robust-loss baseline. I would not desk reject it, but I would send it back for major revision. In its current form it is not citable as evidence for scaling large mixed-quality data.","headline":"A plausible, well-scoped loss-reweighting trick for multi-turn dialogue fine-tuning, but the headline scaling claim rests on Spearman correlations that cannot be computed from the three points shown.","tokens_in":19324,"tokens_out":3580,"would_cite":false,"duration_ms":34841,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReSURE claims that adaptively down-weighting high-loss dialogues by turn depth keeps multi-turn fine-tuning robust to noisy and mixed-quality data, with positive scaling where standard fine-tuning degrades.","keywords":["multi-turn dialogue","supervised fine-tuning","loss reweighting","noisy supervision","Welford online statistics","instruction tuning","data quality","LLM robustness"],"falsifier":"Train on a synthetic multi-turn corpus with noise injected only into specified later turns, and compare per-turn response accuracy of ReSURE versus ordinary fine-tuning; if final-turn accuracy is not higher, the claimed early-turn error propagation mechanism is not what drives the gains.","tokens_in":18343,"feed_emoji":"💬","tokens_out":7577,"duration_ms":67345,"temperature":0.7,"pith_summary":"This paper tries to establish that multi-turn dialogue fine-tuning can absorb low-quality and off-task training data without any pre-filtering step, by continuously estimating which dialogues are unreliable and softly damping their loss. The claim is that supervision mistakes in early turns propagate into later turns, so a turn-depth-aware reweighting scheme should reduce that cascade and keep optimization positive as data scales. The authors test this on high-, normal-, and low-quality dialogue sets and on mixtures that add math problems, reporting that their method maintains or improves response quality while standard tuning degrades. If the claim holds, large mixed-quality corpora become usable for dialogue instruction tuning without manual curation.","feed_headline":"As noisy data grows, adaptive loss weighting holds dialogue quality","feed_subtitle":"A turn-aware reweighting scheme keeps fine-tuning from falling as datasets mix high- and low-quality supervision.","key_machinery":"The central object is a per-dialogue-depth online estimate of the mean and standard deviation of whole-dialogue loss, maintained with Welford's algorithm. For each group of dialogues with the same maximum supervised turn index, the running statistics define a threshold $\\tau^{(b)}_s = \\mu^{(b)}_s + \\alpha \\sigma^{(b)}_s$; losses above it are softly scaled down by $\\exp(-(\\ell_s-\\tau^{(b)}_s)/\\tau^{(b)}_s)$, with a percentile floor. This mechanism is what lets the training loop detect unreliable supervision dynamically and regulate trust by dialogue depth without discarding samples.","core_discovery":"ReSURE reweights every training dialogue by comparing its whole-dialogue supervised cross-entropy loss against running statistics maintained separately for each dialogue-depth group. After a warm-up phase, a dialogue whose loss exceeds the group mean by more than one standard deviation is labeled unreliable and its gradient contribution is decayed exponentially, with a floor so that hard examples are not zeroed out; such dialogues are also excluded from the statistics update, so outliers do not contaminate the baseline. The paper reports that this procedure improves or preserves scores on in-domain held-out data, MT-Bench, and MT-Bench-Ext across six datasets, and yields positive Spearman correlations between dataset complexity and performance (0.21, 1.00, 0.80) while Vicuna-Tuning degrades. It also reports that mixing in GSM8K math data does not collapse dialogue quality, and that combining ReSURE with static pre-filtering improves over either alone.","pith_inferences":["Because the method reweights a whole dialogue on one loss value, it cannot by itself tell which turn inside that dialogue is unreliable; a natural extension is to run the same statistics per turn and test whether the gains grow.","The Spearman correlations are computed over a small number of mixture points, so the claim of positive scaling regardless of data quality would be stronger with many more dataset sizes and noise levels.","Down-weighting high-loss samples may also down-weight genuinely hard but valuable examples, so combining ReSURE with a difficulty-aware sampling policy is a plausible next test.","The same loss-calibration idea could apply to other supervised fine-tuning settings where noise arrives in structured groups, such as by task or domain, though the paper only tests multi-turn dialogue."],"forward_implications":["Adding larger or noisier multi-turn datasets to fine-tuning should no longer systematically hurt response quality; ReSURE's positive Spearman correlations (0.21, 1.00, 0.80) are the paper's evidence for this.","Off-task data mixed into dialogue training, such as GSM8K, can be absorbed without collapsing conversational performance, and the paper reports a small improvement in math accuracy as well.","Static pre-filtering and adaptive reweighting address different weaknesses, since ReSURE plus pre-filtering outperforms either approach alone.","The mechanism transfers across the four instruction-tuned models tested in the paper, so it is not tied to one architecture."],"supporting_citations":[{"why":"Supplies the online mean and variance update (Welford's algorithm) used to track per-dialogue-depth loss statistics.","marker":"(Welford, 1962)"},{"why":"Establishes the learning-to-reweight-examples approach that motivates soft reweighting and the anomaly factor setting.","marker":"(Ren et al., 2018)"},{"why":"Provides the noisy-label loss reweighting practice cited for the alpha = 1.0 threshold and soft suppression.","marker":"(Zhang and Sabuncu, 2020)"},{"why":"Defines the Vicuna-Tuning baseline whose degradation ReSURE is compared against.","marker":"(Chiang et al., 2023)"},{"why":"Provides MT-Bench and the LLM-as-judge evaluation protocol used for main scores and Spearman analysis.","marker":"(Zheng et al., 2023)"},{"why":"Supplies the MT-Bench-Ext benchmark used to test generalization to more complex multi-turn settings.","marker":"(Kwan et al., 2024b)"},{"why":"Supplies GSM8K, the off-task math dataset used to simulate task-level noise.","marker":"(Cobbe et al., 2021)"},{"why":"Provides DeBERTa, the model used in the static pre-filtering baseline that ReSURE is compared with and combined against.","marker":"(He et al., 2020, 2021)"}],"fun_headline_variants":["ReSURE: adaptive loss reweighting tames unreliable dialogue supervision","Turn-level loss reweighting keeps dialogue fine-tuning stable on noisy data","Down-weight bad turns on the fly to boost multi-turn dialogue response quality","Online stats spot unreliable turns, preserving dialogue coherence in training","Mixed-quality supervision? Adaptive reweighting holds dialogue fine-tuning steady"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a whole-dialogue loss more than one standard deviation above its turn-depth group mean marks unreliable supervision rather than a hard but useful example, and that down-weighting the whole dialogue can repair errors originating in early turns.","fun_headline_variants_meta":{"raw":{"variants":["ReSURE: adaptive loss reweighting tames unreliable dialogue supervision","Turn-level loss reweighting keeps dialogue fine-tuning stable on noisy data","Down-weight bad turns on the fly to boost multi-turn dialogue response quality","Online stats spot unreliable turns, preserving dialogue coherence in training","Mixed-quality supervision? Adaptive reweighting holds dialogue fine-tuning steady"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00053,"raw_usage":{"total_tokens":2544,"prompt_tokens":926,"completion_tokens":1618,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":1526}},"tokens_in":542,"tokens_out":1618,"duration_ms":12455,"temperature":1.0,"reasoning_tokens":1526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:48:19.971852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train on a synthetic multi-turn corpus with noise injected only into specified later turns, and compare per-turn response accuracy of ReSURE versus ordinary fine-tuning; if final-turn accuracy is not higher, the claimed early-turn error propagation mechanism is not what drives the gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the learning-to-reweight-examples approach that motivates soft reweighting and the anomaly factor setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the noisy-label loss reweighting practice cited for the alpha = 1.0 threshold and soft suppression."}],"review_version":1}