{"id":"7f8f041d-c024-4c6e-86c5-8ca7c9769bab","arxiv_id":"2412.15244","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MPPO trains LLMs without a reference model by treating the geometric mean of response-token probabilities as the reward and jointly suppressing multiple negative responses; the Pair-MNM variant reports the best MT-Bench score in the paper.","lead":"A team from Alibaba proposes MPPO, a training method that aligns large language models with human preferences by using the model's own average response likelihood as a reward and allowing many low-quality answers to be used at once. It reports gains on MT-Bench over DPO and SimPO, with simpler training that needs no reference model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward proxy in Eq. (4) is unvalidated: geometric-mean likelihood can improve MPPO's loss by favoring short, generic, high-probability text rather than human preference, so the reported SOTA gains do not yet establish alignment.","rationale":"The reader's weakest assumption is exactly the reward-proxy claim in Eq. (4), and my analysis agrees: the entire method rests on the assertion that the geometric mean of the policy's per-token probabilities is a valid reward signal. The paper does not validate this monotonicity, and the loss is constructed so that optimizing it can increase average likelihood independently of true quality. The gradient scaling (p/T) gives a concrete mechanism by which the objective favors short and generic responses, making the concern more than a vague worry about reward misspecification. This is a correctness risk, not merely a disagreement with current consensus, because it threatens the internal claim that higher MPPO loss reduction corresponds to better alignment. The paper's own Figure 3 discussion for Point-CE demonstrates that average likelihood can rise uniformly across all responses, and Section 5.2 concedes that the likelihood-reward relationship is only a loose positive correlation. Since the same reward function underlies the Pair-wise variants, the observation applies to the headline results. The recommended verdict remains CONDITIONAL (unchanged from the reader): the method is plausible and the experiments are suggestive, but the central reward assumption and the numerical claims need validation before acceptance. A direct correlation test on held-out preference data would settle whether Eq. (4) is a reliable reward proxy, which is the single most load-bearing uncertainty in the paper.","tokens_in":11235,"tokens_out":7305,"duration_ms":85961,"concrete_test":"On a held-out split of UltraFeedback (or any preference dataset with multiple scored responses per prompt), compute p from Eq. (5) under the initial SFT model and measure the per-prompt Spearman/Kendall rank correlation between p and the GPT-4 scores. Repeat for SimPO's length-normalized log reward and DPO's implicit reward. If p's correlation is not significantly positive, or is substantially below the other rewards, the central Eq. (4) assumption fails and the MPPO benchmark gains cannot be attributed to preference alignment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All benchmark claims depend on Eq. (4) treating the geometric mean of per-token likelihoods, r_MPPO(x,y)=p, as a proxy for human preference. The paper provides no correlation or ablation showing that p tracks GPT-4 preference scores; Section 5.2 admits only 'a positive correlation, not an exact correspondence' between average likelihood and reward values, without quantifying it. This matters because the Pair-Single/Pair-MNM losses in Eqs. (8) and (10) use raw p, not log p or a reference-normalized ratio. The gradient for each token scales as (p/T) times the sigmoid factor, so longer or lower-likelihood responses receive systematically smaller updates. Thus MPPO can reduce its loss by increasing average token probability, favoring short, generic, high-fluency text, while ignoring correctness, detail, and other preference-relevant qualities. The reported MT-Bench 6.16 and Arena-Hard 21.6 gains could reflect this bias. No code, exact hyperparameters, response-length statistics, or error bars are provided to rule out this failure mode.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MPPO (Multi Pair-wise Preference Optimization), a reference-model-free preference optimization method that defines the reward of a response as the geometric mean of the policy's per-token likelihoods. It introduces Point-wise, Pair-wise, and List-wise implementations, with an emphasis on Pair-wise variants that use one chosen response and one or more rejected responses from the same prompt. Using Llama3-8B trained on UltraFeedback, the authors report that the Pair-MNM variant achieves MT-Bench 6.16 and Arena-Hard 21.6, claiming state-of-the-art performance relative to DPO, KTO, ORPO, and SimPO. The paper's central claim is that directly fitting the average likelihood of model responses to reward values, combined with multiple negative samples, improves alignment quality.","tokens_in":11461,"tokens_out":5889,"duration_ms":61588,"significance":"If the central claim holds, the paper would offer a simple, reference-free objective that can consume multiple negative responses per prompt, which is practically attractive for sparse preference-data settings and for reducing GPU memory overhead. The algorithm is easy to state and, aside from a concrete inconsistency in Eqs. (11)-(12), internally coherent. The paper also makes falsifiable benchmark predictions on public datasets with GPT-4-based judges. However, the significance is currently tempered by two unaddressed issues: the reward proxy in Eq. (4) is not validated against human or GPT-4 preference scores, and all benchmark conclusions rest on single runs without error bars or complete training details. The claimed 'latest SOTA' status is therefore not yet established.","major_comments":[{"comment":"The reward proxy is unvalidated, and the paper itself concedes in Section 5.2 only 'a positive correlation, not an exact correspondence' between average likelihood and reward values, without quantifying the correlation. Because all MPPO losses in Eqs. (6)-(13) use p directly, a decrease in the training loss can be driven by raising average token likelihood—favoring short, generic, high-probability text—rather than by improving human preference. I request a quantitative validation of p against the GPT-4 scores in UltraFeedback (e.g., per-prompt Spearman rank correlation or pairwise classification accuracy) and an ablation with a length-controlled or reference-normalized reward to rule out this failure mode. Without such evidence, the benchmark gains in Table 2 do not yet establish that Eq. (4) is a valid reward proxy.","section":"§3.2, Eq. (4); §5.2"},{"comment":"All experimental claims are based on a single run per method, with no standard errors, confidence intervals, significance tests, or response-length statistics. The Pair-MNM MT-Bench score of 6.16 versus SimPO's 5.97 is a 0.19-point gap on an 80-prompt benchmark, and on Arena-Hard Pair-MNM (21.6) trails SimPO (23.4), so the text's characterization of Pair-MNM as 'the latest SOTA algorithm' is not supported by the reported evidence. Section 4.1.1 says only the learning rate was adjusted, but no learning rate, number of epochs, batch size, warmup, or other training details are provided, and no code is released. Multiple seeds, variance reporting, and a full training configuration are required before the comparisons can be evaluated.","section":"Table 2; §4.1.1; §5.1"},{"comment":"The displayed equations for Pair-MCS and Pair-MCM do not match their textual description. The text says the method randomly selects two responses from N+1 and compares them by score, which should produce C(N+1,2) pair terms. As printed, Eq. (11) is identical to Pair-MNS (sum over i=1..N of log sigma(p_w - p_li)) and Eq. (12) is identical to Pair-MNM (log sigma(N p_w - sum_i p_li)), with only the underbrace changed to 'total of C2 N+1 items'. Either the loss formulas are incorrect or the description is incorrect. Because the RQ4 comparison between MNM and MCM depends on this distinction, the correct equations and their corresponding experimental results must be provided.","section":"§3.3.2, Eqs. (11)-(12)"},{"comment":"The title and Section 3.3.2 claim support for arbitrary numbers of negative samples, but all experiments use exactly N=3 because UltraFeedback provides four responses per prompt. No sensitivity analysis over N is reported, and the N multiplier in Eq. (10) changes the scale of the sigmoid argument and therefore the effective optimization temperature. Without varying N or at least reporting how performance behaves as N grows, the paper's central generality claim is not demonstrated. I recommend adding an N-sweep (e.g., N=1,2,3,5,7) or explicitly restricting the claim to the N=3 setting.","section":"§3.3.2, Eq. (10); §4.1.2"}],"minor_comments":[{"comment":"The abstract states that MPPO 'outperforms DPO, ORPO, and SimPO,' but Table 2 shows SimPO ahead on Arena-Hard; this claim should be qualified.","section":"Abstract"},{"comment":"The Point-wise losses use p, a geometric mean of per-token probabilities, as if it were a score in [0,1]. The scores are normalized to 0.1-1, while p for long responses is typically much smaller, creating a target mismatch. This may partly explain the Point-wise underperformance in Table 2 and should be discussed explicitly.","section":"§3.3.1, Eqs. (6)-(7)"},{"comment":"The statement that MPPO has 'no hyperparameters' is overstated: the learning rate is a hyperparameter, and Eq. (10)'s N multiplier and the sigmoid temperature are implicit design choices.","section":"§4.1.1 and Figure 2"},{"comment":"The limitations section says the work covers 'others like logistic ranking loss and ListNet,' but these methods do not appear in the paper; either remove them or clarify that they are future work.","section":"Limitations and Ethics"},{"comment":"Some references are incomplete or informal (e.g., 'Abhimanyu, Dubey, and et al.' and 'OpenAI Josh, Achiam, and et al.'), and Figures 3 and 4 do not label the x-axis in interpretable units or state the total number of training steps; these should be cleaned up.","section":"References and Figures"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern lands: the unvalidated reward proxy and the absence of any variance or hyperparameter reporting are the main risks to the central claim. I do not recommend rejection because the algorithm is simple, the equations are mostly clear, and the identified problems are addressable with additional experiments and corrections. However, the 'SOTA' characterization should be softened, and the MCS/MCM inconsistency must be resolved before the paper can be considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nMPPO is a reasonable reference-free preference optimization recipe, but the headline results are not yet solid enough to call SOTA. The genuinely new part is the systematic comparison of point-wise, pair-wise, and list-wise losses with multiple negatives, and the Pair-MNM variant's merged objective is clean. The equations are straightforward, and the paper honestly reports that SimPO still beats it on Arena-Hard.\n\nThe biggest soft spot is the reward proxy in Eq. (4): geometric mean of per-token likelihoods. The loss directly pushes p_w above p_l, so the model can satisfy the objective by increasing average token probability—possibly favoring shorter, more generic text—without actually improving preference alignment. The paper never validates that p tracks human/GPT-4 preference scores; Section 5.2 only says there is 'a positive correlation, not an exact correspondence' and doesn't quantify it. The stress-test note about length bias is not far-fetched, and the paper provides no response-length statistics or ablations to rule it out.\n\nThe experimental reporting is thin: one run, no error bars, no code, no hyperparameters. The claimed 'significant enhancement' over SimPO on MT-Bench is 0.19 points, while Arena-Hard is 1.8 points behind. That is not a robust basis for the abstract's 'remarkable advantages.' Pair-Single is effectively SimPO's length-normalized reward with beta=1 and gamma=0 under an exp transform; the paper doesn't acknowledge this close kinship, which is a minor but real oversight. The circularity concern (reward defined as a function of the policy) is real but not fatal—it's a self-contained objective, not a claim about an external reward model.\n\nWhat the paper does well: it is simple, clearly written, and the multi-negative variants are a useful addition to the toolbox. The analysis of why Pair-MNM outperforms Pair-MNS and Pair-MCM is plausible, even if only supported by training curves from a single run. The limitations section is candid about the data-rich/scarce boundary, which is a plus.\n\nWho is this for? Researchers working on reference-free preference optimization who want a compact comparison of multi-negative objectives. It deserves a serious referee, but the authors should be asked to validate the reward proxy, report variance, provide code, and temper the SOTA claim.\n\nRecommendation: send to peer review, with major revision expectations.","headline":"MPPO is a clean reference-free preference optimization variant, but unvalidated reward proxy and single-run results mean the reported gains are suggestive, not SOTA.","tokens_in":12012,"tokens_out":2820,"would_cite":false,"duration_ms":28370,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MPPO claims that the geometric mean of a model's per-token likelihoods is a sufficient reward signal for reference-free preference optimization with any number of negative responses.","keywords":["MPPO","preference optimization","reference-free alignment","average likelihood reward","multi-response preference data","pairwise ranking","sparse preference data","LLM alignment"],"falsifier":"Compute the rank correlation between $r_{\\mathrm{MPPO}}(x,y) = \\left(\\prod_i \\pi_\\theta(y_i \\mid x, y_{<i})\\right)^{1/|y|}$ and held-out human or GPT-4 preference scores over a fixed pool of responses; if the correlation is not positive and monotone, or if a length-confounded training set makes the MPPO loss drop while win rates fall, the reward proxy has failed to track preference.","tokens_in":11034,"feed_emoji":"🤖","tokens_out":10986,"duration_ms":94508,"temperature":0.7,"pith_summary":"The paper introduces MPPO, a preference-optimization algorithm for large language models that does away with reference models and learned reward models. Its central proposal is to score each response by the geometric mean of the policy's per-token probabilities and to optimize that score directly through a Bradley-Terry-style pairwise objective. This makes it possible to train on one prompt with one chosen response and arbitrarily many rejected responses, so realistic sparse preference data is used more fully. Across pointwise, pairwise, and listwise variants, the pairwise variant that merges all rejected responses against the single best response (Pair-MNM) performs best, reaching an MT-Bench score of 6.16 with Llama3-8B, above DPO, ORPO, and SimPO, and an Arena-Hard win rate of 21.6, above DPO and ORPO. A reader should care because the method removes a major source of memory cost and hyperparameter sensitivity while reporting competitive or better alignment quality.","feed_headline":"MPPO hits MT-Bench 6.16 with a reference-free likelihood reward","feed_subtitle":"Pairing the best response against all rejects lifts Llama3-8B past DPO and SimPO on MT-Bench without extra GPU memory.","key_machinery":"The load-bearing object is the geometric-mean reward $r_{\\mathrm{MPPO}}(x,y) = \\left(\\prod_i \\pi_\\theta(y_i \\mid x, y_{<i})\\right)^{1/|y|}$, the per-token likelihood of the response averaged over its length. This single scalar replaces both the learned reward model of RLHF and the $\\log(\\pi_\\theta/\\pi_{\\mathrm{ref}})$ ratio of DPO, so the policy's own output probabilities are the reward. For the best-performing variant, Pair-MNM, the reward is used in the objective $\\mathcal{L}_{\\mathrm{Pair\\text{-}MNM}} = -\\mathbb{E}\\left[\\log\\sigma\\left(N p_w - \\sum_{i=1}^{N} p_{l_i}\\right)\\right]$, where one chosen response is compared jointly against all rejected responses. The mechanism that carries the argument is that widening the gap between the chosen response's average likelihood and the merged rejected likelihoods is what improves downstream human-preference benchmarks.","core_discovery":"The paper's claim is that the average likelihood of a response under the current policy is enough to stand in for the unknown human reward. Concretely, MPPO sets $r_{\\mathrm{MPPO}}(x,y) = \\left(\\prod_{i=1}^{|y|} \\pi_\\theta(y_i \\mid x, y_{<i})\\right)^{1/|y|}$ and feeds this scalar into the pairwise comparison $\\sigma(r(x,y_w) - r(x,y_l))$ in place of DPO's policy-to-reference ratio. The authors argue this is more direct than DPO's implicit reward, needs no reference model or KL regularizer, and extends naturally to $N+1$ responses per prompt: the highest-scored response is the positive sample and all other $N$ responses are negative. Their experiments on UltraFeedback with Llama3-8B show the Pair-MNM objective, which compares the positive response against the merged sum of negative rewards, gives the best MT-Bench result (6.16) and a strong Arena-Hard win rate (21.6), leading them to call it state of the art.","pith_inferences":["The geometric-mean reward is only length-normalized in the exponent: longer and shorter responses can still have very different per-token probability scales, so MPPO may implicitly favor short, generic, or high-probability text; testing a length-matched or length-penalized variant would separate this bias from genuine preference learning.","If the reward proxy is only approximately monotone in human preference, then the Pair-MNM objective could be improved by score-weighted suppression of negatives or by a small held-out verification reward; the paper does not test these extensions.","The conclusion that only the single best response should be promoted suggests a preprocessing heuristic for sparse preference data: discard middle-scoring responses instead of forming random pairs among them, which could be validated by ablating on UltraFeedback."],"forward_implications":["Preference optimization can run without a reference model, cutting GPU memory and removing KL-regularization hyperparameters; MPPO's only tuned hyperparameter is the learning rate.","Multi-response preference data can be used in full: instead of discarding extra rejections, the best response is paired against all $N$ rejected responses in a single objective.","Merging rejections into one term (Pair-MNM) separates chosen and rejected likelihoods more sharply than summing independent log-sigmoid terms, which the authors connect to the best MT-Bench score.","Because the same geometric-mean reward can be inserted into pointwise, pairwise, or listwise losses, the paper's comparison offers guidance on which preference-information format is worth collecting and optimizing."],"supporting_citations":[{"why":"Introduces DPO and the reference-model reparameterization of the reward that MPPO replaces; this is the baseline objective in Eq. (3) and in the benchmark table.","marker":"Rafailov et al., 2023"},{"why":"Introduces SimPO, the reference-free length-normalized baseline that MPPO compares against and whose public Llama3-8B SFT weights initialize training.","marker":"Meng et al., 2024"},{"why":"Provides the UltraFeedback dataset, the sole preference-training source with four GPT-4-scored responses per prompt.","marker":"Cui et al., 2023"},{"why":"Provides the Llama3-8B model family used for the SFT and all preference-optimization runs.","marker":"Abhimanyu et al., 2024"},{"why":"Introduces ORPO, a reference-free baseline whose Arena-Hard result MPPO surpasses.","marker":"Hong et al., 2024"},{"why":"Introduces KTO, a baseline that handles unbalanced preference data and is compared on both benchmarks.","marker":"Ethayarajh et al., 2024"}],"fun_headline_variants":["MPPO drops reference model, beats DPO on MT-Bench","Average likelihood reward lifts LLMs without reference model","Pair-wise MPPO hits MT-Bench 6.16, no reference needed","Reference-free MPPO outranks DPO, ORPO, SimPO","MPPO: pairwise preference with arbitrary negatives"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on treating the geometric mean of the model's per-token probabilities as a faithful, monotone proxy for human preference, so increasing the chosen response's geometric mean while decreasing rejected ones will actually make outputs better.","fun_headline_variants_meta":{"raw":{"variants":["MPPO drops reference model, beats DPO on MT-Bench","Average likelihood reward lifts LLMs without reference model","Pair-wise MPPO hits MT-Bench 6.16, no reference needed","Reference-free MPPO outranks DPO, ORPO, SimPO","MPPO: pairwise preference with arbitrary negatives"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1612,"prompt_tokens":997,"completion_tokens":615,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":526}},"tokens_in":613,"tokens_out":615,"duration_ms":5973,"temperature":1.0,"reasoning_tokens":526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:16:09.270205+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the rank correlation between $r_{\\mathrm{MPPO}}(x,y) = \\left(\\prod_i \\pi_\\theta(y_i \\mid x, y_{<i})\\right)^{1/|y|}$ and held-out human or GPT-4 preference scores over a fixed pool of responses; if the correlation is not positive and monotone, or if a length-confounded training set makes the MPPO loss drop while win rates fall, the reward proxy has failed to track preference.","supporting_citations":[],"review_version":1}