Pith. sign in

REVIEW 5 major objections 6 minor 13 references

MDPO: Multi-Granularity Direct Preference Optimization for Mathematical Reasoning

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training on whole solutions, reasoning transitions, and calculation steps yields higher math accuracy than DPO.

desk verdict A coherent incremental contribution—multi-granularity preference data in a SimPO-style objective—but the reported gains are small and the empirical evidence is under-reported, so the numbers should not be trusted without a reproducibility pass. read the letter →

arxiv 2506.15706 v1 pith:EZ2XCRJS submitted 2025-05-30 cs.LG cs.AI

Yunze Lin This is my paper
classification cs.LGcs.AI
keywords directpreferenceoptimizationmathematicalreasoningmulti-granularitysupervisionlargelanguagemodelsGSM8KMATHstep-levelSimPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that direct preference optimization (DPO) fails to improve long-chain mathematical reasoning because it treats an entire wrong solution as one rejected block, so it cannot tell the model which step went wrong. To fix this, MDPO builds preference pairs at three granularities: whole solutions, transitions between reasoning steps, and individual computational steps, and trains them with one unified objective that uses average log-likelihood as the reward, matching how the model generates at inference. The paper reports gains over base models and over DPO, SimPO, and Step-DPO: Qwen2-7B-Instruct improves by 1.7% on GSM8K and 2.3% on MATH, while Llama3-8B-Instruct improves by 0.9% and 1.2%. If the method works as claimed, fine-grained preference data can be built without manual annotation and still improve both logical reasoning and arithmetic reliability.

What carries the argument

The load-bearing mechanism is a family of continuation tasks trained with one unified SimPO-style objective: the model receives the problem plus the first $k-1$ steps and must generate the rest, with reward defined as the length-normalized average log-likelihood of the continuation and a target margin separating accepted from rejected continuations. Solution2Solution sets $k=0$; Inference2Inference sets $k=i-1$ and uses the unreliable inference as the rejected continuation; Step2Step sets $k=i-1$ and uses the corrected calculation as the accepted continuation. The unreliable-inference detector drives Inference2Inference: an inference is flagged when the error rate of sampled continuations rises from prefix window $w_{i-1}$ to $w_i$, i.e. $e(w_i) > e(w_{i-1})$. Step2Step uses a separate instruction-following model to locate the first arithmetic error and replace it with the correct computation.

What would settle it

Compare the steps flagged by the error-rate-jump heuristic against human-annotated error locations on a sample of wrong solutions; if agreement is near chance, the Inference2Inference supervision is not locating real errors. Alternatively, train an equally sized MDPO variant with randomly chosen 'unreliable' steps; if it matches the real heuristic's accuracy, the localization signal is not what drives the gains.

Watch

Extended reading notes

Core claim

The central claim is that decomposing preference optimization into three nested granularities—Solution2Solution for complete chains, Inference2Inference for the transition between consecutive steps, and Step2Step for a single calculation—yields better mathematical reasoning than optimizing only at the solution level. Each granularity is recast as the same text-completion task: given the problem and the first k steps, continue to the answer. The reward is the length-normalized average log-likelihood of the continuation, the same quantity the model maximizes during generation, with a target reward margin, so training and inference objectives align. The paper also claims the preference pairs can be constructed automatically: sample multiple solutions, take correct-answer paths as winners and wrong-answer paths as losers, flag an inference as unreliable when the error rate among sampled continuations jumps after that step, and use an external instruction-following model to identify and correct the first calculation error for step-level pairs. The reported experiments show MDPO outperforming DPO, SimPO, and Step-DPO on GSM8K and MATH, with the largest margin on MATH, which the paper attributes to the step-level computation signal.

Load-bearing premise

The method depends on the assumption that a jump in error rate between consecutive prefix windows reliably marks the actual bad step, so that correcting that step gives the model true supervision; if the flagged step is often not where the error occurs, the fine-grained signal loses its advantage.

Editorial extensions

If this is right

  • Training a model on all three granularities should generalize better than solution-level DPO alone, because the model learns which steps to distrust rather than only which answers are bad.
  • The unified objective should raise the win rate, the fraction of test cases where the accepted continuation has higher average log-likelihood than the rejected one, thereby aligning training with generation.
  • Automatic construction of the preference data means the method can be applied to new math datasets without human labels, as long as final-answer checking is available.
  • Because MDPO outperforms Step-DPO most clearly on MATH, the largest gains should appear on datasets with heavy computation, such as GSM-HARD or problems with fractions and multi-digit arithmetic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to replace the error-rate-jump heuristic with human-annotated error positions; if the two disagree often, the Inference2Inference gains likely come from added data volume rather than from pinpointing true errors.
  • The same three-granularity decomposition could transfer to other long-chain domains beyond arithmetic, such as code debugging or multi-hop fact-checking, where wrong prefixes and wrong transitions can be separated.
  • Reporting the number of sampled paths $k$ and the agreement between the heuristic and human labels would let readers estimate how much of the 1.7%-2.3% gain is attributable to fine-grained localization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper proposes Multi-Granularity Direct Preference Optimization (MDPO), a preference-optimization method for mathematical reasoning that applies a SimPO-style average-log-likelihood reward at three granularities: whole-solution pairs (Sol2Sol), inference-level pairs that replace an unreliable next-step transition (Infer2Infer), and step-level pairs that correct a faulty calculation (Step2Step). The training data are constructed automatically by sampling reasoning paths, detecting error-rate jumps across step windows, and using GPT-4 to correct the first calculation error. Experiments with Qwen2-7B-Instruct and Llama3-8B-Instruct report improvements over the base models and over DPO, SimPO, and Step-DPO on GSM8K and MATH, with ablations attributing gains to each granularity and a step2step-only experiment on GSM-HARD and MATH. The paper claims that the unified objective aligns training with the generation metric.

Significance. If the reported gains are real, MDPO makes a useful contribution: it shows that fine-grained preference supervision at multiple levels can be combined in one SimPO-style objective, and it provides an annotation-free pipeline for constructing such data. The inference-level supervision via error-rate jumps is a sensible and cheap heuristic, and the step2step focus on computation errors is a well-motivated extension. However, the current evidence is insufficient to establish the central claim: no variance, seeds, decoding protocol, or code are reported, the baseline implementations are not auditable, and the step2step data are not clearly mapped to the unified loss. The ablation structure is appropriate and the dataset sizes are standard, so a careful revision with proper statistics could make the case convincing.

major comments (5)
  1. [Tables 1-2, Section 6.1] The central claim that MDPO outperforms DPO, SimPO, and Step-DPO rests on single-run accuracy differences of 0.9 to 2.3 percentage points, with no standard deviations, number of seeds, decoding protocol, or evaluation code reported. On GSM8K the Llama3 improvement of +0.9 points corresponds to roughly 12 test questions, and on MATH the Qwen improvement of +2.3 points corresponds to roughly 115 questions, so these differences can easily fall within run-to-run or decoding variation. This is load-bearing for the main claim, and the concern raised in the stress-test note therefore lands.
  2. [Section 5 and Eq. (5)] The baselines in Table 2 are not auditable because the paper does not report their training hyperparameters, epochs, data composition, or any implementation details. Additionally, the MDPO loss in Eq. (5) includes the margin gamma and Section 4.3 uses k sampled paths, but neither gamma nor k is reported anywhere; the only hyperparameter given is beta=0.4. Without these values, the comparison and the method cannot be reproduced.
  3. [Section 4.3] The Infer2Infer mechanism depends on the heuristic that a step is unreliable when error(w_i) > error(w_{i-1}) over k sampled paths, and that GPT-4 corrections of the first calculation error yield valid step-level preferences. The paper reports neither k nor the sampling protocol, and it does not validate the heuristic against human-annotated error positions or any alternative error-localization baseline. If the error-rate jump does not pinpoint the actual error, the fine-grained supervision loses its intended signal, so this is a load-bearing gap in the data-construction pipeline.
  4. [Eq. (5), Sections 4.2-4.3] The unified objective defines y_w and y_l as sequences of steps from step_k to step_n, but the Step2Step data are constructed as single-step pairs (x||step0..k-1, stepwin_k, steplose_k). The paper does not specify how a single-step target is fed into the length-normalized average-log-likelihood reward, i.e., whether y is only the one step or whether the model is required to continue beyond it. This ambiguity makes the actual Step2Step training loss underspecified and weakens the claim that all three granularities share one objective.
  5. [Table 4 and Section 6.4] The comparison between 'Qwen2-7B-Instruct + step2step' and the DPO/Step-DPO baselines is confounded, because the step2step row is trained solely on step2step preference pairs while DPO and Step-DPO use full-solution preference data. The observed differences therefore cannot be attributed cleanly to the step2step supervision; a comparison with baselines trained on the same step-level pairs, or an ablation that adds step2step to a shared data mix, is needed.
minor comments (6)
  1. [Section 1] The sentence about reward ranking contains a typo: 'pθ(yw, x) > pθ(yl, |x)' should read 'pθ(yw | x) > pθ(yl | x)'.
  2. [Section 4.3] The generator used to sample the k reasoning paths is not identified, and the prompt template for requiring '[Step i]' prefixes is not given; please specify the model, temperature, and prompt.
  3. [Section 5] Only beta is reported; please report gamma and k, and state the total number of preference pairs contributed by each granularity rather than only the combined 30,000.
  4. [Sections 6.1 and 7] The extrapolation to larger models is based on 'trends reported by other scholars' rather than on data from this paper; this sentence should be removed or clearly labeled as speculation.
  5. [References] Qwen2 and Llama3 are cited as Bai et al. (2023) and Touvron et al. (2023), which are the Qwen and Llama-1 technical reports; please cite the Qwen2 and Llama-3 model releases.
  6. [Figure 2] The 'Win Rate' metric is described only in the caption; please define the exact test subset, the decoding method, and the number of examples used to compute this rate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central empirical claim rests on held-out test-set accuracy, and the objective is an explicit reuse of SimPO's reward rather than a self-derived prediction.

full rationale

MDPO's central claim is that training on preference data at three granularities improves GSM8K/MATH accuracy. The training objective (Eq. 5) is SimPO's loss (Eq. 4) applied to conditional continuations; the paper explicitly credits SimPO ('Inspired by this research, we propose MDPO'), so reusing that objective is attribution, not a hidden circular derivation. The three granularities are distinguished by how preference pairs are constructed: Sol2Sol uses complete correct/incorrect solutions, Infer2Infer uses an error-rate-jump heuristic to locate unreliable inferences, and Step2Step uses GPT-4 to correct the first calculation error. None of these constructions defines the evaluation metric (final-answer accuracy on held-out test sets) in terms of the training signal; the test sets are independent of the 30,000 training pairs built from GSM8K/MATH training splits. The paper reports no self-citations by the author, so no load-bearing self-citation chain exists. The win-rate figure in Section 6.5 directly reflects the quantity optimized by the SimPO-style loss, but it is presented as a sanity check of the objective, not as the evidence for the headline accuracy gains. Concerns about single-run comparisons and missing variance are legitimate correctness/robustness risks, not circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method rests on data construction heuristics (error-rate jump, GPT-4 correction) and on the SimPO objective. No new physical or mathematical entities are introduced. The hyperparameters beta, gamma, and the sampling count k are chosen by hand and are not fully reported.

free parameters (3)
  • beta (reward scaling) = 0.4
    Set by hand; the margin gamma is mentioned in the loss but its value is not reported.
  • gamma (reward margin) = not reported
    Appears in the loss function, but the paper gives no value, limiting reproducibility.
  • k (number of sampled reasoning paths) = not reported
    Used in the Infer2Infer data construction to compute error rates; the paper never specifies k.
assumptions (3)
  • domain assumption A step is unreliable when error(wi) > error(wi-1) over sampled paths
    This heuristic is the basis for selecting inf erlose in Infer2Infer. It is stated without validation against human judgment.
  • domain assumption GPT-4 can correctly identify and correct the first computational error in a step
    The Step2Step data construction relies on GPT-4 to produce stepwin; the paper only verifies the final answer, not the step correction itself.
  • domain assumption SimPO's average log-likelihood reward aligns with generation metrics
    The objective is inherited from SimPO (Eq. 1-4). MDPO does not derive this alignment but assumes it as a design choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MDPO: Multi-Granularity Direct Preference Optimization for Mathematical Reasoning." pith.science (2026). https://pith.science/paper/EZ2XCRJS

@misc{pith2026250615706,
  author       = {Pith},
  title        = {Pith review of: MDPO: Multi-Granularity Direct Preference Optimization for Mathematical Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EZ2XCRJS}},
  note         = {Machine review of arXiv:2506.15706}
}
read the original abstract

Mathematical reasoning presents a significant challenge for Large Language Models (LLMs) as it requires ensuring the correctness of each reasoning step. Researchers have been strengthening the mathematical reasoning abilities of LLMs through supervised fine-tuning, but due to the inability to suppress incorrect outputs, illusions can easily arise. Recently, Direct Preference Optimization (DPO) has been widely adopted for aligning human intent by using preference data to prevent LLMs from generating incorrect outputs. However, it has shown limited benefits in long-chain mathematical reasoning, mainly because DPO struggles to effectively capture the differences between accepted and rejected answers from preferences in long-chain data. The inconsistency between DPO training and LLMs' generation metrics also affects the effectiveness of suppressing incorrect outputs. We propose the Multi-Granularity Direct Preference Optimization (MDPO) method, optimizing the mathematical reasoning of LLMs at three granularities: Solution2Solution, Inference2Inference, and Step2Step. Solution2Solution focuses on the correctness of entire long-chain reasoning; Inference2Inference concentrates on logical reasoning between steps; Step2Step corrects computational errors in steps, enhancing the computational capabilities of LLMs. Additionally, we unify the training objectives of the three granularities to align with the generation metrics. We conducted experiments on the open-source models Qwen2 and Llama3, achieving improvements of 1.7% and 0.9% on the GSM8K dataset, and 2.3% and 1.2% on the MATH dataset, outperforming DPO and other DPO variant methods. Furthermore, we also provide a pipeline for constructing MDPO training data that is simple and does not require manual annotation costs.

Figures

Figures reproduced from arXiv: 2506.15706 by the authors.

Figure 1
Figure 1. (Left) Given a math word problem, which, when fully reasoned through the solution chain, can be broken [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. MDPO has different training objectives from DPO and Step-DPO, mainly in the formulation of rewards and [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 13 linked inside Pith

  1. [1]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609,

    arXiv

  2. [5]

    Orpo: Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691,

    arXiv

  3. [7]

    Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms.arXiv preprint arXiv:2402.16352,

    9 Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms.arXiv preprint arXiv:2402.16352,

    arXiv

  4. [9]

    Orca-math: Unlocking the potential of slms in grade school math

    Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830,

    arXiv

  5. [11]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    arXiv

  6. [12]

    Generative ai for math: Part i–mathpile: A billion-token-scale pretraining corpus for math

    Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part i–mathpile: A billion-token-scale pretraining corpus for math. arXiv preprint arXiv:2312.17120,

    arXiv

  7. [13]

    Automatic chain of thought prompting in large language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493,

    arXiv

  8. [2017]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

Show all 13 references
  1. [2020]

    Mathscale: Scaling instruction tuning for mathematical reasoning

    Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02884,

    arXiv

  2. [2021]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306,

    arXiv

  3. [2022]

    Step-dpo: Step-wise preference optimization for long-chain reasoning of llms

    Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629,

    arXiv

  4. [2023]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,

    arXiv

  5. [2024]

    Improve mathematical reasoning in language models by automated process supervision

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.