{"id":"01765e79-1183-4139-8d02-c475338ae766","arxiv_id":"2501.00747","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DIVE combines global sample pooling with diversity-aware data selection to counter output diversity collapse in iterative preference learning for LLMs.","lead":"A new training recipe called DIVE adds data selection and pooled sampling to iterative self-improvement so language models keep producing varied solution paths while learning. It reports 10-45% higher diversity on math benchmarks with roughly unchanged accuracy, which matters because self-training on a model's own outputs tends to narrow its output distribution.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Diversity gains may be inflated by best-iteration selection and missing error bars; the vanilla baseline's selection mechanism is unspecified, so the 10–45% claim is not yet statistically anchored.","rationale":"I read the paper in good faith. The method is clearly described, the ablations are sensible, and the alternative diversity metrics in §4.3 (NV-Embed, Stella, Distinct Equation Chains) directly address the reader's worry that the embedding signal is circular. For that reason, the reader's weakest assumption—Sentence-BERT may capture style rather than reasoning-path diversity—is not the most load-bearing issue; the paper already provides some evidence that the improvement generalizes across metric families. The more insecure condition is statistical and procedural. The abstract's 10–45% range is taken from tables that select the best of six iterations per method, with no error bars and no repeated seeds. A single run of a stochastic pipeline cannot support a quantitative effect claim, and the unexplained drop in vanilla diversity when K goes from 10 to 50 suggests the baseline may involve a hidden selection rule that is never stated. If the released code shows vanilla selects pairs uniformly at random, the concern reduces to variance; if it does not, the entire comparison may be apples-to-oranges. In either case, the right response is to keep the paper's verdict conditional: the central claim is plausible and worth publishing as a proposal, but it needs multi-seed, fixed-iteration results and a precise baseline protocol before the 10–45% effect can be taken as established. I see no internal inconsistency or fatal flaw, so I do not move the verdict to reject; I also do not see grounds to accept unconditionally.","tokens_in":12061,"tokens_out":7592,"duration_ms":78732,"concrete_test":"Re-run the GSM8k Sample-50 comparison (Vanilla vs Global+Selection) with at least 5 random seeds, fix the evaluation at iteration 6 (or a pre-registered iteration), and report mean ± std plus a bootstrap confidence interval for each diversity metric. Additionally, inspect the released code to confirm the vanilla baseline constructs P=5 preference pairs uniformly at random from the current pool; if it does not, rerun the comparison with an explicitly random-selection baseline. The claim stands only if the minimum DIVE-vs-vanilla gap across seeds exceeds the pooled standard deviation by a factor of 2, and the same ordering holds under the corrected baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical assertion—DIVE raises diversity by 10–45% while preserving accuracy—rests on Tables 1–2, which report the best-performing iteration out of six for each method, with no error bars and no repeated-seed runs. Since the pipeline is stochastic (nucleus sampling, DPO training), a single run cannot show that differences as small as 0.02 in SentBERT score are outside noise. More importantly, the vanilla baseline's preference-pair construction is never specified: §2.1 says only 'we select P responses,' and §3.1.3 gives K and P but not the selection rule. If vanilla selects pairs in a non-random way (e.g., the first P by generation order, or a fixed ordering by log-probability), the comparison is not a controlled test of DIVE's selection mechanism. The counter-intuitive result that vanilla diversity decreases when K increases from 10 to 50 (Table 1) is unexplained without this detail. The alternative metrics in §4.3 mitigate the embedding-circularity concern, but they are evaluated on the same best-iteration protocol and also lack variance estimates. Thus the load-bearing condition—that the reported gain is a real, reproducible effect of DIVE rather than selection noise or an unstated baseline artifact—is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DIVE (Diversified Iterative Self-Improvement), a framework that modifies iterative Direct Preference Optimization (DPO) training to counteract the loss of output diversity in self-improvement. Two components are introduced: Sample Pool Expansion, which increases the number of sampled responses per question and reuses responses from all previous iterations, and Data Selection, which filters responses with an Isolation Forest on Sentence-BERT embeddings and then greedily selects preference pairs to maximize diversity. Experiments on GSM8K and MATH with Mistral-7B compare Vanilla ISI, Global, Selection, and Global+Selection over six iterations. The paper reports that Global+Selection yields a 10–45% relative increase in diversity metrics (Distinct N-grams and Sentence-BERT cosine similarity) while keeping accuracy roughly unchanged. Ablations and additional diversity metrics (NV-Embed, Stella, equation chains, distinct answers) are provided, together with an analysis of diversity across question difficulty levels.","tokens_in":12319,"tokens_out":4787,"duration_ms":44302,"significance":"If the central empirical claim holds, DIVE is a useful and simple contribution to an important problem: diversity collapse in iterative preference learning for reasoning tasks. The paper is clearly written, ships code, includes ablations for both proposed components, and partially addresses the metric-circularity concern by evaluating with alternative embedding and task-specific diversity metrics. The difficulty-level analysis (Section 4.2) is a nice addition. However, the load-bearing quantitative claim is not yet statistically anchored: the headline tables report the best of six iterations per method, there are no error bars or repeated-seed runs, and the vanilla baseline's selection mechanism is left unspecified. These issues are fixable and do not invalidate the approach, but they must be resolved before the 10–45% claim can be accepted.","major_comments":[{"comment":"The headline diversity and accuracy results are taken from the best-performing of six iterations per method, with no standard deviations, confidence intervals, or repeated-seed runs. Because the pipeline is stochastic (nucleus sampling, DPO training, greedy selection with a random start), this protocol can systematically overstate the advantage of DIVE: a lucky iteration is reported as representative. This is load-bearing for the central '10–45% relative increase' claim. Please report per-iteration results with variance across seeds (or at least the full per-iteration distribution for every method) and assess the statistical significance of the differences.","section":"§3.2, Tables 1–2"},{"comment":"The vanilla baseline is under-specified. Section 2.1 Step 2 states only that 'we select P responses' from the correct and incorrect pools, and Section 3.1.3 gives K and P but not the selection rule used by Vanilla. If vanilla selects, for example, the first P responses in generation order or by a fixed score ordering, then the comparison against Global+Selection is not a controlled test of the proposed selection mechanism. This also makes the counter-intuitive result that vanilla diversity decreases when K grows from 10 to 50 (Table 1) impossible to interpret. The authors should specify the vanilla pair-construction rule and ensure that the only differences between conditions are the proposed components.","section":"§2.1 and §3.1.3"},{"comment":"The data-selection pipeline uses the same diversity signals that are the primary evaluation metrics: the greedy selection maximizes a diversity score on the candidate list, and the Isolation Forest filter uses Sentence-BERT embedding distances, while Section 3.1.2 evaluates diversity with Sentence-BERT similarity and distinct N-grams. Consequently, part of the reported gain is selected for by construction. The alternative metrics in Section 4.3 (NV-Embed, Stella, equation chains) are a useful partial response, but they are shown only for the best-accuracy iteration and without variance estimates, so the circularity concern is not fully resolved. Please either measure diversity with held-out metrics not used in selection, or report all alternative metrics with the same statistical rigor as the headline numbers.","section":"§2.2.2 and §3.1.2"},{"comment":"The quality-preservation claim is not statistically anchored. Several accuracy comparisons are close but in the wrong direction (e.g., MATH Sample 50 @50: 0.586 for Global+Selection vs 0.606 for Vanilla; GSM8k Sample 50 @50: 0.972 vs 0.975). Without error bars or a repeated-seed analysis, 'maintaining performance quality' cannot be distinguished from a small but real degradation, and the reader cannot judge whether the trade-off is acceptable. Report confidence intervals for @1 and @50 and discuss the accuracy/diversity trade-off explicitly.","section":"Tables 1–2 and §3.2"}],"minor_comments":[{"comment":"The greedy selection step does not specify which diversity measure is optimized in Step 2; please state whether it is the same Distinct-N/Sentence-BERT metric used in evaluation and how any hyperparameters (e.g., the n-gram range) are chosen.","section":"§2.2.2"},{"comment":"The text says models are trained 'on all the preference pairs constructed so far,' which appears to conflict with the per-iteration D_t_pref definition in Section 2.1; please clarify whether the DPO training set accumulates across iterations and whether the same holds for the vanilla baseline.","section":"§3.1.3"},{"comment":"The number of preference pairs per iteration varies between 10k and 30k depending on the setting, and questions without both a correct and an incorrect response are skipped; please report pair counts per method and iteration, since differences in dataset size alone could affect both accuracy and diversity.","section":"§3.1.3, footnote 1"},{"comment":"The Isolation Forest contamination threshold (or equivalent outlier-rate setting) is not reported; this parameter directly controls how aggressively the response pool is pruned and should be stated for reproducibility.","section":"§2.2.2"},{"comment":"The caption says 'Only the results from the iteration with the highest accuracy are shown' while Figure 6 reports all iterations; please use a consistent protocol and explain why the choice of iteration differs from that in Tables 1–2.","section":"Figure 5 caption"},{"comment":"The phrase 'the first study focused on this problem' is a strong claim; please either qualify it with the specific setting (diversity-preserving data selection in ISI for math reasoning) or cite concurrent work.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concerns in the reader's report are well founded: the paper's central claim is plausible but not yet statistically established. The most important fixes are per-seed/per-iteration reporting and a fully specified vanilla baseline. If the authors can supply these, the paper could become a solid contribution; without them, the headline 10–45% claim should be treated as preliminary. The paper is within the journal's scope and the code release is a positive factor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: DIVE is a useful, honest empirical paper on countering diversity collapse in iterative self-improvement. The two-part recipe (pool expansion plus isolation-forest-filtered greedy selection) is not present in the cited prior work, and the paper mostly delivers what it claims. I am fairly convinced the effect is real, but the headline 10-45% numbers are not statistically anchored as reported, so the paper needs revision rather than rejection.\n\nWhat is actually new: the combination of a global accumulated sample pool with outlier-filtered greedy diversity selection for DPO-style iterative training is a reasonable and underexplored design. The Q1 analysis comparing global pooling against merely sampling more per question is the most useful part: it shows you get more diversity per unit compute by reusing old data, which is a practical insight. The difficulty-level analysis in Q2 is also a nice addition, and the paper is clearly written. They released code, which is credit.\n\nThe soft spots are real but not fatal. Tables 1 and 2 report the best iteration out of six for each method, with no error bars or repeated runs. That makes small differences hard to interpret. The iteration plots in Figure 2 (and Appendix Figure 6 for alternative metrics) show the trend more honestly, and those plots do support the qualitative claim. Still, the paper ought to give fixed-iteration comparisons and variance estimates. The vanilla baseline's preference-pair construction is never specified — Section 2.1 just says \"we select P responses\" without saying how. Given the counter-intuitive result that vanilla diversity drops when K goes from 10 to 50, the selection rule matters a lot. The circularity concern (selecting on the same Sentence-BERT/distinct-n-gram signals used for evaluation) is partially mitigated by the alternative metrics in Section 4.3, but those are also shown without variance, so the mitigation is incomplete. The small accuracy drops on MATH (e.g., @50 from 0.606 to 0.586) are honestly reported, though \"maintaining performance\" is a bit generous for a 3% drop.\n\nWho this is for: people working on post-training diversity, DPO variants, and model collapse. It deserves a serious referee; I would send it out and ask for fixed-iteration tables, repeated seeds, and an explicit description of the vanilla baseline's selection mechanism. With those, the core claim would be credible enough for publication.","headline":"A plausible and useful anti-collapse recipe for iterative preference learning, with real empirical support but headline stats that need variance estimates and a spelled-out baseline.","tokens_in":12849,"tokens_out":1366,"would_cite":true,"duration_ms":15258,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By expanding the candidate pool to include every previous iteration's generations and then selecting diverse, high-quality preference pairs, DIVE raises the output diversity of iteratively self-improving language models by 10–45% on MATH…","keywords":["iterative self-improvement","output diversity","preference learning","direct preference optimization","mathematical reasoning","data selection","sample pool expansion","model collapse"],"falsifier":"Take the DIVE-selected preference pairs and score their reasoning-path diversity with a surface-independent measure—for example, structural equation-chain similarity on GSM8k solutions, or a human annotation of solution strategy—and compare with vanilla ISI selection at matched pool size; if the 10–45% diversity advantage shrinks or vanishes under that measure, the embedding signal was tracking style rather than reasoning diversity. A cheaper check: measure how much of DIVE's reported diversity gain survives when the evaluation metric is replaced by distinct-answers or distinct-equation-chains, which do not overlap with the Sentence-BERT selection signal.","tokens_in":11837,"feed_emoji":"🎲","tokens_out":12443,"duration_ms":94996,"temperature":0.7,"pith_summary":"Iterative self-improvement (ISI) lets a language model train on its own outputs round after round, but the model tends to converge on a narrow set of response styles, which is especially costly in math reasoning where many valid solution paths exist. DIVE adds two steps to that loop: it expands the pool of candidate responses (sampling more answers per question and reusing every previous iteration's generations) and then selects preference pairs that are both high-quality and diverse, using embedding-based outlier filtering followed by greedy diversity maximization. On GSM8k and MATH with Mistral-7B, the paper reports a 10–45% relative increase in four output-diversity metrics for both correct and incorrect responses, while top-1 and top-50 accuracy stay essentially level with vanilla ISI. The paper argues that the diversity of the selected training examples, not the raw size of the pool, is what determines whether the trained model stays diverse. If it holds, DIVE offers a plug-in remedy for diversity collapse in preference-based self-improvement that costs nothing in accuracy.","feed_headline":"DIVE lifts self-improving math models' diversity 10–45%","feed_subtitle":"Pooling earlier answers and selecting diverse, high-quality pairs keeps math reasoning varied at no accuracy cost.","key_machinery":"The load-bearing mechanism is the Data Selection pipeline acting on an expanded candidate pool. The pool is expanded two ways: sampling $K=10$ or $50$ responses per question instead of fewer, and accumulating generations across iterations into a global pool $\\mathcal{D}^t_{\\mathrm{pool}} = \\bigcup_{i=1}^{t} \\mathcal{D}^i_{\\mathrm{pool}}$. Selection then runs Isolation Forest (an outlier-detection method) on Sentence-BERT embedding distances to discard low-quality outliers, and a greedy algorithm that iteratively adds the remaining response whose inclusion maximizes the embedding-based diversity of the chosen set, building preference pairs from the correct and incorrect halves of that curated pool. The induced preference pairs feed the DPO objective (with an additional negative-log-likelihood term on chosen sequences), so diversity is injected at the training-data level rather than through decoding-time tricks.","core_discovery":"The paper's central claim is that diversity collapse in iterative preference learning can be reversed by controlling what goes into the preference pairs, without giving up accuracy. Specifically, DIVE combines Sample Pool Expansion—sampling $K$ responses per question and pooling generations from all previous iterations into $\\mathcal{D}^t_{\\mathrm{pool}} = \\bigcup_{i=1}^{t} \\mathcal{D}^i_{\\mathrm{pool}}$—with Data Selection that first removes semantic outliers via Isolation Forest on Sentence-BERT embeddings (vector representations of response meaning) and then greedily picks responses that maximize the embedding-distance diversity of the selected set. On the MATH and GSM8k datasets, with Mistral-7B as the base model and up to six iterations of direct preference optimization (DPO) plus a negative-log-likelihood term, DIVE yields a 10–45% relative improvement in four diversity metrics (distinct n-grams and the complement of Sentence-BERT cosine similarity) for both correct and incorrect responses, while @1 and @50 accuracy remain comparable to vanilla ISI. Ablations show that pooling alone does not consistently help, that selection is the component that reliably raises diversity, and that greedy selection without the Isolation Forest filter can hurt accuracy. The authors further show that the diversity gains generalize to seven additional metrics, including equation-chain diversity and distinct-answer counts, and that DIVE helps across all question difficulty levels, with the largest relative diversity loss in vanilla ISI occurring on the easiest questions.","pith_inferences":["If embedding distance truly captures solution-path diversity, the same selection recipe should transfer to other self-improvement tasks with multiple valid answers—code generation, open-ended instruction following, and multi-step planning—where the style-versus-strategy trade-off would need to be re-checked.","DIVE's habit of keeping every previous iteration's generations in the pool suggests a testable scheduling principle: an ISI loop should maintain a reservoir of older, more diverse samples and decide by selection, not recency.","One could adapt DIVE to online preference learning by using the greedy selector as a diversity-aware buffer sampler, turning the per-iteration selection cost into a streaming update rather than a full re-scan of the pool.","A harder test the paper does not run: whether the 10–45% diversity gains translate into accuracy gains under self-consistency decoding (majority vote over diverse paths); if they do, DIVE would improve not just sample-space breadth but downstream task performance."],"forward_implications":["Applying DIVE's two components—pool expansion and embedding-based selection—to any DPO-style iterative self-improvement loop should slow or stop the per-iteration decline in output diversity on mathematical reasoning tasks.","Diverse preference data can be chosen without a reward model or extra human labels: correctness is judged by the gold final answer and diversity by embedding distance, so the method is fully self-supervised once gold answers exist.","Larger sampling (K=50) combined with global pooling and selection gives the best diversity, whereas larger sampling without selection reduces diversity relative to K=10.","Because self-improvement on these benchmarks mostly re-ranks the sample space (stable @50 accuracy), DIVE preserves the breadth of that sample space rather than changing the model's ceiling.","The diversity gains are not confined to the metrics used during training: they transfer to distinct-equation-chain and distinct-answer metrics."],"supporting_citations":[{"why":"It supplies the DPO objective used to train on the curated correct/incorrect preference pairs.","marker":"Rafailov et al., 2024"},{"why":"It documents diversity loss under preference optimization and provides the distinct n-gram and Sentence-BERT diversity metrics used in evaluation.","marker":"Kirk et al., 2023"},{"why":"It establishes the diversity-reduction and re-ranking phenomena in iterative preference learning that DIVE targets.","marker":"Wu et al., 2024"},{"why":"It provides the Isolation Forest algorithm used to filter outlier responses before greedy selection.","marker":"Liu et al., 2008"},{"why":"It defines Sentence-BERT embeddings, used both for outlier filtering and for the greedy diversity-maximizing selection.","marker":"Reimers, 2019"},{"why":"It supplies the GSM8k benchmark, whose gold final answers define which generated responses are correct.","marker":"Cobbe et al., 2021"},{"why":"It supplies the MATH benchmark, whose gold final answers define correct and incorrect pools for preference pairs.","marker":"Hendrycks et al., 2021"},{"why":"It validates the distinct n-gram metric as aligning with human diversity judgments, used in evaluation.","marker":"Tevet and Berant, 2020"},{"why":"It provides Mistral-7B, the base model fine-tuned and iteratively trained in all experiments.","marker":"Jiang et al., 2023"}],"fun_headline_variants":["Diverse self-improvement: 10-45% more varied math solutions","DIVE boosts output diversity in self-improving LLMs up to 45%","DIVE: diverse self-improvement, 45% more varied math reasoning","Sample pooling plus smart selection keeps LLM reasoning diverse (45% gain)","DIVE's two tricks raise math output diversity up to 45%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption the whole result rests on is that Sentence-BERT embedding distance measures diversity of mathematical reasoning paths; if it mostly captures wording style, DIVE's selection inflates the very metrics used to evaluate it, and the reported gains would not reflect genuinely different solution strategies.","fun_headline_variants_meta":{"raw":{"variants":["Diverse self-improvement: 10-45% more varied math solutions","DIVE boosts output diversity in self-improving LLMs up to 45%","DIVE: diverse self-improvement, 45% more varied math reasoning","Sample pooling plus smart selection keeps LLM reasoning diverse (45% gain)","DIVE's two tricks raise math output diversity up to 45%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001122,"raw_usage":{"total_tokens":4684,"prompt_tokens":978,"completion_tokens":3706,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":3606}},"tokens_in":594,"tokens_out":3706,"duration_ms":26221,"temperature":1.0,"reasoning_tokens":3606,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:43:48.235548+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the DIVE-selected preference pairs and score their reasoning-path diversity with a surface-independent measure—for example, structural equation-chain similarity on GSM8k solutions, or a human annotation of solution strategy—and compare with vanilla ISI selection at matched pool size; if the 10–45% diversity advantage shrinks or vanishes under that measure, the embedding signal was tracking style rather than reasoning diversity. A cheaper check: measure how much of DIVE's reported diversity gain survives when the evaluation metric is replaced by distinct-answers or distinct-equation-chains, which do not overlap with the Sentence-BERT selection signal.","supporting_citations":[],"review_version":1}