{"id":"3027af30-cdd6-4c7a-b166-2650ca8f5df0","arxiv_id":"2507.02173","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"DTS, which generates diverse solution strategies before writing solutions, improves GSM8K by 7.1 points and MATH by 4.2 points over an untuned base model at 1.03x baseline compute.","lead":"Researchers compared ways to create training data for teaching large language models math reasoning, and found that first listing five solution approaches, then solving each one, produced the best benchmark gains. The method costs about 3% more compute than plain sampling and beats a nearly 5x more expensive search method.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DTS gains are not independently measured: the same GSM8K test set is used for reward-model selection (§4.2) and for hyperparameter/epoch selection (§5.2, Fig. 3), so the headline margins—especially 0.2 points on MATH—may be artifacts of test-set selection.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the evaluation is not independent of the final test benchmarks. Section 4.2 selects the reward model on the GSM8K test set, and Section 5.2/Figure 3 select hyperparameters and epochs on the same GSM8K test and MATH-500 used for the advertised numbers. Because the reported numbers are best-of-sweep, single-seed maxima, the 0.2-point DTS margin over baseline on MATH is especially fragile; selecting epoch 2 for DTS+DPO is what produces 52.4, while its epoch-5 score of 51.6 would lose to the baseline. The DTS method itself is a plausible and cheap idea, and the 1.03x compute accounting is a useful contribution, but the experimental protocol does not support the claimed margin. A corrected protocol with a held-out selection split and multiple seeds could plausibly recover a smaller DTS advantage on GSM8K, which is why the appropriate current verdict is REJECT (unchanged) rather than a permanent rejection of the method.","tokens_in":21051,"tokens_out":5864,"duration_ms":64776,"concrete_test":"Hold out a fixed 20% subset of each evaluation benchmark (GSM8K test and MATH-500) before any model selection. Use only those subsets for reward-model selection, hyperparameter sweeps, and epoch selection; freeze all choices; then evaluate the frozen configurations once on the untouched 80% of each benchmark, repeating with at least 5 random seeds. If the chosen DTS configuration does not beat the best tuned baseline by more than one standard error on both untouched splits, the reported margins are selection artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, the final GSM8K and MATH numbers must be measurements, not selections. The paper uses the GSM8K test set twice before reporting the headline results. Section 4.2 and Table 4 choose the reward model by minimizing incorrect_higher on the GSM8K test set, and that reward model labels every preference pair used by all strategies. Then Section 5.2 and Figure 3 select learning rates, beta, gamma, and the best epoch from sweeps evaluated on the same GSM8K test set and MATH-500. The reported DTS scores (83.2 GSM8K, 52.4 MATH) are therefore maxima over roughly 76 training runs and 5 epochs, not independent estimates. The DTS advantage over the strongest tuned baseline is 2.5 points on GSM8K (83.2 vs 80.7) but only 0.2 points on MATH (52.4 vs 52.2, both DPO-family), and that 0.2-point gap is within the noise of best-of-sweep selection with a single seed. On MATH, DTS+DPO reaches 52.4 only at epoch 2; at epoch 5 it is 51.6, below the baseline's 52.2. Reporting gains relative to the untuned base model (7.1%/4.2%) further inflates the apparent effect. This is a correctness risk in the evaluation protocol, not a disagreement with consensus: if the final test sets are used for selection, the ordering of strategies is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how the way preference data are generated affects math performance after preference optimization. It compares temperature sampling, chain-of-thought prompting, MCTS, and a new method, Diversified-ThinkSolve (DTS), which first generates five distinct solution approaches and then produces a solution for each approach. Using Llama-3.1-8B-Instruct, preference data from MetaMathQA, Nemotron-70B as the reward model, and SFT/DPO/ORPO/SimPO trainers, it reports DTS as the best strategy (83.2% GSM8K, 52.4% MATH), with a claimed 1.03x compute overhead over baseline. The appendices provide hyperparameter sweeps, epoch curves, code listings, and qualitative solution examples.","tokens_in":21446,"tokens_out":10080,"duration_ms":105047,"significance":"The research question is timely: whether structured diversity in preference pairs can improve mathematical reasoning is practically important, and the DTS pipeline is simple and described in enough detail to reproduce. The paper also reports a large hyperparameter sweep, which is useful documentation. However, the central empirical claim is not supported by the current evaluation. The reward model, hyperparameters, and best epoch are all selected using the same GSM8K test set and MATH-500 that are later reported as final results, and all experiments use a single seed. The headline comparison to the base model also obscures that the margin over the best tuned baseline is only 0.2 percentage points on MATH. The compute-efficiency claim appears to rest on an inconsistent token accounting. These are load-bearing issues, not presentation problems.","major_comments":[{"comment":"The final numbers are selected, not measured. The reward model is chosen by minimizing incorrect_higher on the GSM8K test set (§4.2, Table 4), and that reward model labels every preference pair. The reported DTS results are the best over a sweep of 76 training runs and 5 epochs, with hyperparameters and the best epoch selected on the same GSM8K test set and MATH-500 (§5.2, Table 2; §5.3, Figure 3). Thus 83.2% and 52.4% are maxima of a selection procedure. The 0.2-point MATH gap between DTS+DPO (52.4) and Baseline+DPO (52.2) is within the noise of best-of-sweep selection; Table 6 even shows DTS+DPO at epoch 5 (51.6) below Baseline+DPO at epoch 5 (52.2). A held-out validation split (or repeated splits) and pre-specified configurations are needed before any strategy ranking can be claimed.","section":"§4.2, §5.2, Table 2, Figure 3"},{"comment":"The experiments use a single random seed (42, §A.1) and no significance tests. On MATH-500, a 0.2 percentage point difference is one problem, so the reported DTS advantage over the best baseline on MATH is not statistically meaningful. Even the GSM8K margin (2.5 points over Baseline+SimPO) needs error bars across seeds, especially since the same test set was used to select the configuration. Reporting best-epoch accuracy in Table 1 and Table 6 compounds the selection problem; averages over epochs do not fix this because the hyperparameters were still chosen on the test set.","section":"§A.1, Table 1, Table 6"},{"comment":"The claimed 1.03x compute overhead for DTS is inconsistent with the described pipeline. Section 3.4 says DTS generates N=5 approaches and then 'for each generated approach' produces a complete solution, which is 1 ThoughtGenerator call plus 5 SolutionGenerator calls per problem. Appendix A.5.1, however, counts only one SolutionGenerator call (495 tokens total vs 482 for baseline). Using the authors' own token estimates, DTS should be 147 + 5*348 = 1,887 tokens per problem. Meanwhile, the baseline generates 5 completions per problem (§3.1), so baseline should be 5*482 = 2,410 tokens per problem. In either consistent accounting, the ratio is not 1.03x: it is 0.78x if baseline is 5 completions, or 3.9x if the 482 baseline figure is treated as per-problem. The efficiency comparison in Table 3 and the abstract's 'marginal overhead' claim therefore need to be redone.","section":"§3.4, §A.5.1, Table 3"}],"minor_comments":[{"comment":"The phrase 'illicit chain of thought' should be 'elicit chain of thought'.","section":"§3.2"},{"comment":"The word 'configratuions' should be 'configurations'.","section":"§5.2"},{"comment":"The CoT+SimPO average GSM8K value (53.0%) is much lower than the best value; since Figure 3 shows a collapse after epoch 1, consider reporting the median or noting the collapse in the table caption to avoid confusion.","section":"Table 1"},{"comment":"The reference list contains several entries that do not appear to be cited in the text (e.g., Chen et al. 2024a, Guo et al. 2024a, Gururangan et al. 2024, Khattab et al. 2024, Khot et al. 2022); please either cite them or remove them.","section":"References"},{"comment":"The main text says the reward model has an 'average inaccuracy rate of 3.11% on the GSM8K test set', but Table 4 reports different error rates for different generator models; clarify that 3.11% corresponds to Llama-3.2-3B-IT outputs.","section":"§4.2, Table 4"}],"recommendation":"reject","confidential_remarks":"The core problem is the evaluation protocol. The paper would need a full re-run with proper train/validation/test separation and multiple seeds, plus corrected compute accounting, before the claims can be assessed. As submitted, I do not see a path to acceptance without new experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the headline numbers are selected, not measured. The GSM8K test set is used twice before the final results are reported—once to choose the reward model and once to pick hyperparameters and best epochs—so the reported DTS advantage is partly an artifact of test-set selection. The MATH gap over the best baseline is 0.2 points, which on a single seed and best-of-sweep selection is no signal at all. On MATH at epoch 5, DTS+DPO is actually below Baseline+DPO (51.6 vs 52.2).\n\nThat said, the paper is not empty. DTS as a two-stage 'generate five approaches, then solve each' pipeline is a simple, cheap, and plausible way to create preference data. The comparison across four data strategies and four alignment methods, with a careful reward-model evaluation and a transparent token-budget analysis, is a reasonable piece of empirical work. The writing is clear and the appendix gives enough detail to reproduce the pipeline.\n\nThe problems are real and load-bearing. Section 4.2 selects the reward model by minimizing incorrect_higher on the GSM8K test set. Section 5.2 and Figure 3 then select learning rates, beta/gamma, and best epochs from sweeps evaluated on the same GSM8K test set and MATH-500. The reported 83.2 and 52.4 are maxima over roughly 76 training runs and five epochs, with no repeated seeds and no significance tests. The abstract's 7.1%/4.2% gains are relative to the untuned base model, not to the best tuned baseline, which makes the effect look much larger than the data support. The limitations section notes the narrow benchmark scope and reward-model dependence, which is honest, but it never flags the selection problem, and that is the central issue.\n\nWould I accept it as is? No. But it deserves a serious referee rather than a desk reject, because the question is meaningful and the flaw is fixable: rerun with a held-out validation split for reward-model and hyperparameter selection, report multiple seeds with variance, and compare against the tuned baseline. If the DTS margin survives that, the paper becomes a modest but useful contribution. As submitted, the ordering of strategies is not established.","headline":"The DTS pipeline is plausible and cheap, but the headline gains are not independently measured: the same test sets are used for reward-model selection, hyperparameter selection, and final reporting, so the claimed margins—especially the 0.2-point MATH gap—are not established.","tokens_in":21954,"tokens_out":2662,"would_cite":false,"duration_ms":31378,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structured diversity in preference data—first propose several solution approaches, then solve each one—raises an 8B model's GSM8K accuracy to 83.2% and MATH to 52.4%.","keywords":["preference optimization","mathematical reasoning","data diversification","Diversified-ThinkSolve","direct preference optimization","SimPO","GSM8K","MATH benchmark"],"falsifier":"Run the DTS pipeline again with the reward model and hyperparameters picked on a held-out validation set and report final scores on a GSM8K and MATH test split that was never touched during any selection step; if the DTS advantage over temperature sampling shrinks to within noise, the structured-diversity claim is falsified. A complementary control: replace the ThoughtGenerator's 'five approaches' prompt with a prompt that asks for five paraphrased versions of the same approach; if performance does not drop, diversity of method is not the active ingredient.","tokens_in":20845,"feed_emoji":"🧮","tokens_out":11136,"duration_ms":105016,"temperature":0.7,"pith_summary":"The paper claims that how preference data is generated matters more for math alignment than which preference optimizer is used. It compares four ways of building preference pairs—temperature sampling, chain-of-thought prompting, Monte Carlo tree search, and its own Diversified-ThinkSolve (DTS)—and finds that DTS, which first proposes several distinct solution approaches and then writes a full solution for each, delivers the largest gains on both GSM8K and MATH. The best DTS configurations reach 83.2% on GSM8K and 52.4% on MATH-500, improvements of 7.1 and 4.2 percentage points over the base 8B model. These gains cost only 1.03 times the compute of plain sampling, whereas MCTS costs 4.85 times and performs worse. If the result holds, structured diversity of reasoning strategies is a cheaper and more effective source of alignment signal than expensive search.","feed_headline":"Split-first reasoning data lifts LLM math to 83.2%","feed_subtitle":"A thought-then-solve preference pipeline beats temperature sampling, CoT, and MCTS at 1.03x baseline compute.","key_machinery":"The load-bearing mechanism is the DTS preference-pair pipeline: a ThoughtGenerator produces five deliberately different high-level approaches for each problem, and a SolutionGenerator turns each approach into a full solution, giving five solutions per problem. Pairs are then filtered by a 'mixed correctness' rule—only problems where exactly two or three of the five solutions are correct are kept—and a reward model picks the highest-scoring correct solution as the preferred completion and the highest-scoring incorrect one as the rejected completion. This explicit separation of thought from execution is what creates strategic diversity: solutions that follow different mathematical routes rather than paraphrases of the same route. The paper argues this richer contrast in the preference data, rather than the particular optimizer, is what lifts reasoning performance.","core_discovery":"On the paper's own terms, the central discovery is that two-phase generation makes better preference data. In DTS, a ThoughtGenerator module is prompted to list five distinct approaches for a math problem, mentioning relevant concepts, formulas, or techniques; a SolutionGenerator then produces one complete worked solution for each approach. This decouples the choice of a reasoning strategy from its execution, so the resulting preferred and rejected pairs differ in method rather than only in wording. Trained on these pairs, an 8B instruction-tuned model reaches 83.2% on GSM8K (with SimPO) and 52.4% on MATH-500 (with DPO), while the same base model without alignment scores 76.1% and 48.2%. DTS outperforms temperature sampling, chain-of-thought prompting, and MCTS across nearly every optimizer, at a compute overhead of only 1.03x over baseline—strong evidence for the paper's claim that data diversity and quality, not the optimization algorithm, drive math improvements.","pith_inferences":["A fair holdout evaluation could shrink the reported DTS advantage, because the reward model and the best epochs were selected using the same GSM8K test and MATH-500 numbers that appear as final results; this is a testable prediction, not a claim the paper makes.","The think-then-solve recipe likely transfers to other decomposable reasoning domains—code generation, planning, proof search—where enumerating strategies before solving produces informative preference pairs; the paper only evaluates math benchmarks.","Varying the 'mixed correctness' filter (2-3 correct out of 5) and the reward model would test whether DTS's advantage comes from strategic diversity or from the particular quality filter; the paper does not ablate these choices.","Controlling for token count and output format across strategies would test whether DTS gains come from diversity or simply from longer, more structured generations; the paper reports compute cost but not this format control."],"forward_implications":["DTS preference data with SimPO reaches 83.2% on GSM8K, and DTS with DPO reaches 52.4% on MATH-500—gains of 7.1 and 4.2 points over the base model.","Structured diversity beats random diversity: DTS outperforms temperature sampling while using only 1.03x its compute, so the improvement is nearly free.","Expensive search is not the answer for this setting: MCTS consumes 4.85x baseline compute and still trails DTS on both benchmarks.","Data generation strategy matters more than optimizer: across SFT, DPO, ORPO, and SimPO, the spread in results is larger between data strategies than between optimizers.","Untamed diversity can destabilize training: CoT and MCTS preference data cause severe epoch-to-epoch drops with SimPO (CoT falls to 53.0% average GSM8K; MCTS collapses to single digits on MATH), so diverse data must also be coherent."],"supporting_citations":[{"why":"Supplies the MetaMathQA problem set, deduplicated to 13,929 queries, from which every strategy generates preference data.","marker":"Yu et al., 2023"},{"why":"Defines DPO, the optimizer that yields the best MATH result (52.4%) with DTS data.","marker":"Rafailov et al., 2023"},{"why":"Defines SimPO, the reference-free optimizer that yields the best GSM8K result (83.2%) with DTS data.","marker":"Meng et al., 2024"},{"why":"Defines ORPO, the third compared optimizer, which performs best for MCTS data.","marker":"Hong et al., 2024"},{"why":"The modular language-model programming approach used to build the ThoughtGenerator and SolutionGenerator modules of DTS.","marker":"Khattab et al., 2023b"},{"why":"Provides the implementations used for the chain-of-thought and MCTS data-generation baselines.","marker":"codelion, 2024"},{"why":"The reward model chosen to score completions and select preferred and rejected pairs for all strategies.","marker":"NVIDIA NeMo Team, 2024"},{"why":"The tree-search method behind the MCTS strategy, anchoring both the comparison and the compute-cost analysis.","marker":"Feng et al., 2023"}],"fun_headline_variants":["Two-step reasoning data yields 7.1% GSM8K jump","Diverse thought-then-solve data beats MCTS in LLM math","Split-first preference data lifts math scores at 1.03x compute","Data diversity, not optimization, drives math gains in LLMs","Think diverse, then solve: preference data boosts math"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the final test numbers measure the methods fairly, which requires the reward model and the best training settings to be chosen without looking at the test benchmarks; in the paper, reward-model selection and the epoch and hyperparameter sweeps use the GSM8K test set and MATH-500, so the headline gains may be partly selected for rather than independently measured.","fun_headline_variants_meta":{"raw":{"variants":["Two-step reasoning data yields 7.1% GSM8K jump","Diverse thought-then-solve data beats MCTS in LLM math","Split-first preference data lifts math scores at 1.03x compute","Data diversity, not optimization, drives math gains in LLMs","Think diverse, then solve: preference data boosts math"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00051,"raw_usage":{"total_tokens":2476,"prompt_tokens":930,"completion_tokens":1546,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1469}},"tokens_in":546,"tokens_out":1546,"duration_ms":11071,"temperature":1.0,"reasoning_tokens":1469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:35:55.831356+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the DTS pipeline again with the reward model and hyperparameters picked on a held-out validation set and report final scores on a GSM8K and MATH test split that was never touched during any selection step; if the DTS advantage over temperature sampling shrinks to within noise, the structured-diversity claim is falsified. A complementary control: replace the ThoughtGenerator's 'five approaches' prompt with a prompt that asks for five paraphrased versions of the same approach; if performance does not drop, diversity of method is not the active ingredient.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The reward model chosen to score completions and select preferred and rejected pairs for all strategies."}],"review_version":1}