{"id":"22856819-dcb3-4e7d-b4e2-a2cca021e012","arxiv_id":"2506.15706","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MDPO applies a SimPO-style length-normalized reward to preference pairs built at solution, inference, and step granularities, yielding small accuracy gains on math benchmarks.","lead":"A new training method for math reasoning LLMs gives preference feedback at three levels: whole solutions, transitions between steps, and individual calculations. On GSM8K and MATH, it gains roughly one to two accuracy points over standard DPO variants.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central empirical claim rests on single-run accuracy differences of 0.9–2.3 percentage points with no error bars, decoding protocol, or code; the reported advantage over SimPO/Step-DPO may be within run-to-run noise.","rationale":"I focused on statistical reliability rather than the data-construction heuristic emphasized by the reader. The heuristic concern is real but less load-bearing: even if the error(wi) > error(wi−1) criterion is imperfect, noisy preference pairs could still produce accuracy gains, so the empirical claim does not strictly depend on it. In contrast, the measurement concern is prior: if the reported deltas are within run-to-run or decoding variance, then no mechanism-level interpretation, including the granularity story, can be concluded. The reader did flag missing error bars and code in the rationale, so there is partial agreement, but the reader's weakest_assumption focuses elsewhere. A rerun with multiple seeds and full release of code/data is the one check that would settle whether the advantage over SimPO and Step-DPO is genuine. Until then, the conditional verdict remains appropriate: the method is plausible and coherent, but the central empirical claim is not yet fully supported.","tokens_in":9136,"tokens_out":6275,"duration_ms":84475,"concrete_test":"Release the training/eval code and the constructed 30k preference dataset, then rerun Table 2 for Qwen2-7B-Instruct with at least 5 independent seeds (same data, same hyperparameters) under the same decoding protocol. Report mean ± standard deviation for base, DPO, SimPO, Step-DPO, and MDPO on GSM8K and MATH. The central claim stands only if the MDPO − best-baseline difference exceeds the pooled seed standard deviation (or a paired statistical test yields p < 0.05) on both datasets. If release is not possible, an independent reimplementation from the paper alone would itself test reproducibility, because Section 4.3 omits k, gamma, and the GPT-4 prompts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MDPO improves GSM8K/MATH accuracy over DPO, SimPO, and Step-DPO (Tables 1–2). For this to hold, the reported 0.9–2.3 percentage-point differences must reflect a stable, reproducible advantage. The paper reports no standard deviations, no number of seeds, no decoding protocol (greedy vs. sampling, temperature), and no code or data. On GSM8K (1319 test items), +0.9 points is roughly 12 questions; on MATH, +2.3 points is roughly 115 questions. Without any measure of evaluation variance, these differences—especially Llama3-8B's +0.9/+1.2 and Qwen2's +1.7 over base—could be within normal seed-to-seed or decoding variation. The baseline methods are listed without their training hyperparameters, data composition, or epochs, so the comparison is not auditable. Section 6.1 even extrapolates to large models based on 'trends reported by other scholars,' which cannot substitute for variance information. If these are single-seed results, the headline superiority claim is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Multi-Granularity Direct Preference Optimization (MDPO), a preference-optimization method for mathematical reasoning that applies a SimPO-style average-log-likelihood reward at three granularities: whole-solution pairs (Sol2Sol), inference-level pairs that replace an unreliable next-step transition (Infer2Infer), and step-level pairs that correct a faulty calculation (Step2Step). The training data are constructed automatically by sampling reasoning paths, detecting error-rate jumps across step windows, and using GPT-4 to correct the first calculation error. Experiments with Qwen2-7B-Instruct and Llama3-8B-Instruct report improvements over the base models and over DPO, SimPO, and Step-DPO on GSM8K and MATH, with ablations attributing gains to each granularity and a step2step-only experiment on GSM-HARD and MATH. The paper claims that the unified objective aligns training with the generation metric.","tokens_in":9381,"tokens_out":6798,"duration_ms":78239,"significance":"If the reported gains are real, MDPO makes a useful contribution: it shows that fine-grained preference supervision at multiple levels can be combined in one SimPO-style objective, and it provides an annotation-free pipeline for constructing such data. The inference-level supervision via error-rate jumps is a sensible and cheap heuristic, and the step2step focus on computation errors is a well-motivated extension. However, the current evidence is insufficient to establish the central claim: no variance, seeds, decoding protocol, or code are reported, the baseline implementations are not auditable, and the step2step data are not clearly mapped to the unified loss. The ablation structure is appropriate and the dataset sizes are standard, so a careful revision with proper statistics could make the case convincing.","major_comments":[{"comment":"The central claim that MDPO outperforms DPO, SimPO, and Step-DPO rests on single-run accuracy differences of 0.9 to 2.3 percentage points, with no standard deviations, number of seeds, decoding protocol, or evaluation code reported. On GSM8K the Llama3 improvement of +0.9 points corresponds to roughly 12 test questions, and on MATH the Qwen improvement of +2.3 points corresponds to roughly 115 questions, so these differences can easily fall within run-to-run or decoding variation. This is load-bearing for the main claim, and the concern raised in the stress-test note therefore lands.","section":"Tables 1-2, Section 6.1"},{"comment":"The baselines in Table 2 are not auditable because the paper does not report their training hyperparameters, epochs, data composition, or any implementation details. Additionally, the MDPO loss in Eq. (5) includes the margin gamma and Section 4.3 uses k sampled paths, but neither gamma nor k is reported anywhere; the only hyperparameter given is beta=0.4. Without these values, the comparison and the method cannot be reproduced.","section":"Section 5 and Eq. (5)"},{"comment":"The Infer2Infer mechanism depends on the heuristic that a step is unreliable when error(w_i) > error(w_{i-1}) over k sampled paths, and that GPT-4 corrections of the first calculation error yield valid step-level preferences. The paper reports neither k nor the sampling protocol, and it does not validate the heuristic against human-annotated error positions or any alternative error-localization baseline. If the error-rate jump does not pinpoint the actual error, the fine-grained supervision loses its intended signal, so this is a load-bearing gap in the data-construction pipeline.","section":"Section 4.3"},{"comment":"The unified objective defines y_w and y_l as sequences of steps from step_k to step_n, but the Step2Step data are constructed as single-step pairs (x||step0..k-1, stepwin_k, steplose_k). The paper does not specify how a single-step target is fed into the length-normalized average-log-likelihood reward, i.e., whether y is only the one step or whether the model is required to continue beyond it. This ambiguity makes the actual Step2Step training loss underspecified and weakens the claim that all three granularities share one objective.","section":"Eq. (5), Sections 4.2-4.3"},{"comment":"The comparison between 'Qwen2-7B-Instruct + step2step' and the DPO/Step-DPO baselines is confounded, because the step2step row is trained solely on step2step preference pairs while DPO and Step-DPO use full-solution preference data. The observed differences therefore cannot be attributed cleanly to the step2step supervision; a comparison with baselines trained on the same step-level pairs, or an ablation that adds step2step to a shared data mix, is needed.","section":"Table 4 and Section 6.4"}],"minor_comments":[{"comment":"The sentence about reward ranking contains a typo: 'pθ(yw, x) > pθ(yl, |x)' should read 'pθ(yw | x) > pθ(yl | x)'.","section":"Section 1"},{"comment":"The generator used to sample the k reasoning paths is not identified, and the prompt template for requiring '[Step i]' prefixes is not given; please specify the model, temperature, and prompt.","section":"Section 4.3"},{"comment":"Only beta is reported; please report gamma and k, and state the total number of preference pairs contributed by each granularity rather than only the combined 30,000.","section":"Section 5"},{"comment":"The extrapolation to larger models is based on 'trends reported by other scholars' rather than on data from this paper; this sentence should be removed or clearly labeled as speculation.","section":"Sections 6.1 and 7"},{"comment":"Qwen2 and Llama3 are cited as Bai et al. (2023) and Touvron et al. (2023), which are the Qwen and Llama-1 technical reports; please cite the Qwen2 and Llama-3 model releases.","section":"References"},{"comment":"The 'Win Rate' metric is described only in the caption; please define the exact test subset, the decoding method, and the number of examples used to compute this rate.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent but incomplete empirical study. The core idea is interesting and the data pipeline is potentially useful, but the experimental evidence does not yet meet the bar for a strong claim of superiority. I recommend major revision rather than rejection because the missing statistics and implementation details are within the scope of a revision. I would also ask the editor to verify that the related-work citations are appropriate, since some citations point to older model reports. No code or data release is mentioned, and for a methods paper in cs.LG this is a significant barrier to reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate incremental contribution, and the specific combination—three granularities of preference data in a single SimPO-style objective—is new. But the empirical evidence is thinner than the prose suggests, and the paper needs a reproducibility pass before the numbers should be trusted.\n\nWhat's genuinely good: the idea of treating solutions, inferences (transitions between steps), and individual steps as separate preference granularities is coherent, and unifying them under SimPO's reference-free length-normalized reward is a natural extension that hasn't been done in exactly this way. Step-DPO did step-level, SimPO did the objective; MDPO puts them together and adds an inference level. The data construction is fully automatic and requires no human labels, which is practically useful if it works. The ablation on Qwen2-7B-Instruct (82.5 → 83.2 → 83.4 on GSM8K) is monotonic and supports the claim that each granularity adds something. The Step2Step effect on GSM-HARD (+3.4) is the most interesting result in the paper—it suggests the computational-error correction really does target a weakness that solution-level DPO misses.\n\nThe soft spots are the usual ones, and they matter more here because the headline gains are small. No error bars, no number of seeds, no decoding protocol (greedy vs. temperature). On GSM8K, +0.9 points is about 12 questions; on MATH, +2.3 is about 115. Without variance information, the advantage over SimPO/Step-DPO could easily be run-to-run noise. The data construction details are under-specified: k (number of sampled paths), γ (reward margin), and the GPT-4 correction prompts are not reported. Baselines are listed without training hyperparameters, data composition, or epochs, so the comparison is not auditable. The error-rate jump heuristic (error(w_i) > error(w_{i-1})) is plausible but unvalidated—there's no check against human-annotated error positions. The scaling claim in Section 6.1, based on 'trends reported by other scholars,' should be cut or clearly labeled as speculation.\n\nOne more thing: the paper reads like a draft. There are typos and incomplete references (e.g., '?' in the Related Work). Not fatal, but worth cleaning.\n\nWho this is for: people working on preference optimization for math reasoning, especially those interested in data construction heuristics. The idea is worth engaging with; the empirical claims need confirmation.\n\nRecommendation: send it to peer review, but with a clear expectation of major revision. A referee should ask for code/data, error bars, seeds, decoding details, the missing hyperparameters, and a validation of the error-location heuristic. If those come, MDPO could be a solid applied contribution.","headline":"A coherent incremental contribution—multi-granularity preference data in a SimPO-style objective—but the reported gains are small and the empirical evidence is under-reported, so the numbers should not be trusted without a reproducibility pass.","tokens_in":9871,"tokens_out":2293,"would_cite":false,"duration_ms":29215,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training on whole solutions, reasoning transitions, and calculation steps yields higher math accuracy than DPO.","keywords":["direct preference optimization","mathematical reasoning","multi-granularity supervision","large language models","GSM8K","MATH","step-level preference","SimPO"],"falsifier":"Compare the steps flagged by the error-rate-jump heuristic against human-annotated error locations on a sample of wrong solutions; if agreement is near chance, the Inference2Inference supervision is not locating real errors. Alternatively, train an equally sized MDPO variant with randomly chosen 'unreliable' steps; if it matches the real heuristic's accuracy, the localization signal is not what drives the gains.","tokens_in":8945,"feed_emoji":"🧮","tokens_out":6502,"duration_ms":70042,"temperature":0.7,"pith_summary":"The paper argues that direct preference optimization (DPO) fails to improve long-chain mathematical reasoning because it treats an entire wrong solution as one rejected block, so it cannot tell the model which step went wrong. To fix this, MDPO builds preference pairs at three granularities: whole solutions, transitions between reasoning steps, and individual computational steps, and trains them with one unified objective that uses average log-likelihood as the reward, matching how the model generates at inference. The paper reports gains over base models and over DPO, SimPO, and Step-DPO: Qwen2-7B-Instruct improves by 1.7% on GSM8K and 2.3% on MATH, while Llama3-8B-Instruct improves by 0.9% and 1.2%. If the method works as claimed, fine-grained preference data can be built without manual annotation and still improve both logical reasoning and arithmetic reliability.","feed_headline":"Three-level preference training beats DPO on math reasoning","feed_subtitle":"MDPO corrects whole solutions, reasoning transitions, and calculation steps, lifting GSM8K and MATH scores","key_machinery":"The load-bearing mechanism is a family of continuation tasks trained with one unified SimPO-style objective: the model receives the problem plus the first $k-1$ steps and must generate the rest, with reward defined as the length-normalized average log-likelihood of the continuation and a target margin separating accepted from rejected continuations. Solution2Solution sets $k=0$; Inference2Inference sets $k=i-1$ and uses the unreliable inference as the rejected continuation; Step2Step sets $k=i-1$ and uses the corrected calculation as the accepted continuation. The unreliable-inference detector drives Inference2Inference: an inference is flagged when the error rate of sampled continuations rises from prefix window $w_{i-1}$ to $w_i$, i.e. $e(w_i) > e(w_{i-1})$. Step2Step uses a separate instruction-following model to locate the first arithmetic error and replace it with the correct computation.","core_discovery":"The central claim is that decomposing preference optimization into three nested granularities—Solution2Solution for complete chains, Inference2Inference for the transition between consecutive steps, and Step2Step for a single calculation—yields better mathematical reasoning than optimizing only at the solution level. Each granularity is recast as the same text-completion task: given the problem and the first k steps, continue to the answer. The reward is the length-normalized average log-likelihood of the continuation, the same quantity the model maximizes during generation, with a target reward margin, so training and inference objectives align. The paper also claims the preference pairs can be constructed automatically: sample multiple solutions, take correct-answer paths as winners and wrong-answer paths as losers, flag an inference as unreliable when the error rate among sampled continuations jumps after that step, and use an external instruction-following model to identify and correct the first calculation error for step-level pairs. The reported experiments show MDPO outperforming DPO, SimPO, and Step-DPO on GSM8K and MATH, with the largest margin on MATH, which the paper attributes to the step-level computation signal.","pith_inferences":["A testable extension is to replace the error-rate-jump heuristic with human-annotated error positions; if the two disagree often, the Inference2Inference gains likely come from added data volume rather than from pinpointing true errors.","The same three-granularity decomposition could transfer to other long-chain domains beyond arithmetic, such as code debugging or multi-hop fact-checking, where wrong prefixes and wrong transitions can be separated.","Reporting the number of sampled paths $k$ and the agreement between the heuristic and human labels would let readers estimate how much of the 1.7%-2.3% gain is attributable to fine-grained localization."],"forward_implications":["Training a model on all three granularities should generalize better than solution-level DPO alone, because the model learns which steps to distrust rather than only which answers are bad.","The unified objective should raise the win rate, the fraction of test cases where the accepted continuation has higher average log-likelihood than the rejected one, thereby aligning training with generation.","Automatic construction of the preference data means the method can be applied to new math datasets without human labels, as long as final-answer checking is available.","Because MDPO outperforms Step-DPO most clearly on MATH, the largest gains should appear on datasets with heavy computation, such as GSM-HARD or problems with fractions and multi-digit arithmetic."],"supporting_citations":[{"why":"Supplies the SimPO objective that MDPO adapts: a reference-free reward based on length-normalized average log-likelihood with a target margin.","marker":"Meng et al. [2024]"},{"why":"Defines DPO and the preference-optimization setup, and serves as the main baseline whose whole-solution rejection MDPO argues is too coarse.","marker":"Rafailov et al. [2023]"},{"why":"Step-DPO is the step-level preference baseline that MDPO compares against and extends with additional granularities.","marker":"Lai et al. [2024]"},{"why":"Provides the GSM8K training and evaluation set used to construct preference data and measure accuracy.","marker":"Cobbe et al. [2021]"},{"why":"Provides the MATH dataset with harder, computation-heavy problems where MDPO reports its largest gains.","marker":"Hendrycks et al. [2021]"},{"why":"Provides GSM-HARD, the computation-heavy variant used to test the step2step contribution.","marker":"Gao et al. [2023]"},{"why":"Qwen2 is the base model for the main experiments.","marker":"Bai et al. [2023]"},{"why":"Llama3 is the second base model for the main experiments.","marker":"Touvron et al. [2023]"}],"fun_headline_variants":["MDPO: three-level preference tuning for better math","Multi-granularity DPO improves GSM8K and MATH","Step-aware DPO sharpens mathematical reasoning","Aligning training and generation boosts math LLMs","MDPO outperforms DPO on long-chain math tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on the assumption that a jump in error rate between consecutive prefix windows reliably marks the actual bad step, so that correcting that step gives the model true supervision; if the flagged step is often not where the error occurs, the fine-grained signal loses its advantage.","fun_headline_variants_meta":{"raw":{"variants":["MDPO: three-level preference tuning for better math","Multi-granularity DPO improves GSM8K and MATH","Step-aware DPO sharpens mathematical reasoning","Aligning training and generation boosts math LLMs","MDPO outperforms DPO on long-chain math tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000321,"raw_usage":{"total_tokens":1878,"prompt_tokens":1087,"completion_tokens":791,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":703,"completion_tokens_details":{"reasoning_tokens":712}},"tokens_in":703,"tokens_out":791,"duration_ms":9624,"temperature":1.0,"reasoning_tokens":712,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:23:54.863863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the steps flagged by the error-rate-jump heuristic against human-annotated error locations on a sample of wrong solutions; if agreement is near chance, the Inference2Inference supervision is not locating real errors. Alternatively, train an equally sized MDPO variant with randomly chosen 'unreliable' steps; if it matches the real heuristic's accuracy, the localization signal is not what drives the gains.","supporting_citations":[],"review_version":1}