{"id":"95a73bd8-d53a-46f7-ae6e-174a559081b5","arxiv_id":"2505.08507","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"InfoPO's mutual information derivation fails: its rejected-response regularizer is constant in expectation, leaving plain supervised fine-tuning on chosen responses.","lead":"InfoPO introduces a preference optimization loss inspired by mutual information estimation and reports gains over DPO and SimPO on reasoning benchmarks. The paper's own equations show the loss reduces in expectation to supervised fine-tuning on chosen responses, so the claimed mechanism and theory do not hold up.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Under the paper's stated sampling model, the rejected-response term in Eq. (13) has zero expected gradient, so InfoPO reduces to SFT on chosen responses; Theorem 4.1's proof uses the same term as if it contributed a reverse-KL gradient.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: under the paper's own sampling model, the rejected-response term in Eq. (13) has zero expected gradient, so InfoPO collapses to supervised fine-tuning on chosen responses. The paper's equations support this reading without requiring any external assumptions: Section 4.1 states y_l ~ pi_ref(y|x), and Appendix A repeats the same sampling for Eq. (23). For any normalized pi_theta, E_{pi_ref}[pi_theta/pi_ref] = 1, making the term constant in expectation and its gradient zero. This directly undercuts the claimed preference signal, the claimed conservative unlearning, and the proof of Theorem 4.1, since Eq. (23) is forward cross-entropy plus a constant rather than reverse KL. The DPO-as-InfoNCE connection in Section 4.1 is internally consistent and the empirical recipe is sufficiently clear to re-implement, but those points do not repair the central mechanism. The experiments also omit the SFT-on-chosen baseline that this reduction makes essential, and the limitations section does not flag the sampling-model issue. I see no way to rescue the main claim without changing the sampling model or the loss, so I agree with the reader's REJECT and recommend no change to the verdict.","tokens_in":15444,"tokens_out":5951,"duration_ms":59172,"concrete_test":"Derive the population gradient of Eq. (13) under the paper's stated sampling model: compute E_{y_l~pi_ref}[pi_theta(y_l|x)/pi_ref(y_l|x)] and its gradient. If this equals 1 and 0 respectively, re-run the Table 1 experiments with L = -log pi_theta(y_w|x) only, using identical hyperparameters; if InfoPO's results are statistically indistinguishable from this SFT-on-chosen baseline, the rejected term is inert in expectation and the central preference-learning claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing failure is in the derivation of Eq. (13) and the proof of Theorem 4.1. Section 4.1 and Appendix A both specify that the rejected response is drawn as y_l ~ pi_ref(y|x). For any normalized pi_theta, E_{y_l~pi_ref}[pi_theta(y_l|x)/pi_ref(y_l|x)] = 1, so the second term in L_InfoPO is constant in expectation. Consequently its gradient is zero in expectation: E[grad_theta (pi_theta(y_l)/pi_ref(y_l))] = grad_theta E[pi_theta(y_l)/pi_ref(y_l)] = grad_theta 1 = 0. The objective therefore has the same population gradient as supervised fine-tuning on the chosen responses, not a preference contrast and not the claimed 'conservative unlearning' of rejected responses. Any finite-sample effect is noise, not a designed preference signal. The same error appears in Eq. (23): E_{pi_chosen}[-log pi_theta] + E_{pi_ref}[pi_theta/pi_ref] = E_{pi_chosen}[-log pi_theta] + 1, which is forward cross-entropy plus a constant, not the reverse KL D_KL(pi_theta || pi_chosen). Thus Theorem 4.1 is unproven. Secondary issues (the beta parameter from Eq. (9) is dropped in Eq. (13), and the critic in Eq. (9) does not depend on c as written) reinforce that Eq. (13) is not the claimed NWJ-based preference objective. To make the preference signal real, the second term would need to be conditioned on y_l being rejected under a distribution where the ratio has nontrivial expectation, such as y_l sampled from pi_theta or with an explicit rejection model; no such sampling model is given.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InfoPO, a preference fine-tuning method for LLMs based on maximizing conditional mutual information I(Y;C|X) with the NWJ estimator, claiming to avoid the Bradley-Terry assumption. The method is derived as L_InfoPO = -log πθ(yw|x) + πθ(yl|x)/πref(yl|x), with a stated advantage that it prevents the chosen-response likelihood from decreasing. The authors reinterpret DPO as InfoNCE-based MI maximization, provide a gradient analysis to motivate conservative unlearning of rejected responses, prove a theorem that InfoPO minimizes reverse KL to the chosen-response distribution (Theorem 4.1), and report experiments on reasoning, chat, and summarization benchmarks where InfoPO outperforms DPO and SimPO.","tokens_in":15772,"tokens_out":5832,"duration_ms":51305,"significance":"If the central derivation and theorem were correct, the MI-maximization perspective on preference learning could be a useful conceptual contribution, and the empirical gains on reasoning benchmarks would be of practical interest. The paper also ships a relatively simple training objective and reports evaluations across multiple models and tasks. However, the significance is entirely conditional on the validity of the derivation of Eq. (13) and the proof of Theorem 4.1, which are the load-bearing components of the paper. As detailed in the major comments, those components fail under the paper's own stated sampling model, so the proposed method does not actually perform preference learning in expectation.","major_comments":[{"comment":"Under the sampling model stated in §4.1 ('we sample x~p(x), (yw,c)~π(y,c=1|x), and (yl,c)~πref(y|x)p(c|x)'), the second term of L_InfoPO has zero expected gradient. For any normalized πθ, E_{yl~πref}[πθ(yl|x)/πref(yl|x)] = 1, so the population objective is E[-log πθ(yw|x)] plus a constant. The InfoPO objective therefore reduces to supervised fine-tuning on the chosen responses, with no preference contrast and no 'conservative unlearning' of rejected responses. Any finite-sample effect of the second term is noise, not a designed preference signal. This invalidates the paper's central claim of a BT-free preference learning method.","section":"§4.1 and §4.3, Eq. (13)"},{"comment":"The derivation of Eq. (13) from the NWJ bound (Eq. (12)) is algebraically inconsistent unless β=1 and several constant terms are dropped. Substituting the critic f = β log(πθ/πref) from Eq. (9) into the NWJ objective yields -β log(πθ(yw|x)/πref(yw|x)) + (πθ(yl|x)/πref(yl|x))^β - 1, which contains β and the reference-policy term for the chosen response. Eq. (13) contains neither β nor log πref(yw). The paper never states that β is fixed to 1, and Appendix B.2 describes searching β in [0.5, 1.0, 2.0] for 'our InfoPO' although the loss in Eq. (13) has no β parameter. The claimed objective does not follow from the stated estimator.","section":"§4.3, Eqs. (9), (12), (13)"},{"comment":"The proof of Theorem 4.1 is invalid. Eq. (23) is E_{π_chosen}[-log πθ] + E_{π_ref}[πθ/π_ref]. The second term equals 1 identically, so the expression is forward cross-entropy from the chosen distribution to πθ plus a constant. The reverse KL divergence D_KL(πθ || π_chosen) is E_{πθ}[log(πθ/π_chosen)], which is a different functional; the 'change of variables' from Eq. (18) to Eq. (23) is not derived and is not valid. Consequently Theorem 4.1's claim that InfoPO minimizes reverse KL is unproven, and the mode-seeking interpretation lacks support.","section":"Appendix A, Eq. (23) and Theorem 4.1"}],"minor_comments":[{"comment":"The critic fϕ(x,c) is written as independent of c; clarify whether the preference label c has any role in the critic or whether the contrast is entirely carried by the choice of yw versus yl in the expectation.","section":"§4.1, Eq. (9)"},{"comment":"The constant '+1' in the NWJ estimator disappears without comment in Eq. (13). While constants do not affect the gradient, the paper should state that this is an intentional omission for clarity.","section":"§4.3, Eq. (12)"},{"comment":"The hyperparameter description 'For SimPO and our InfoPO, the β in SimPO was selected through a search within the range of [0.5, 1.0, 2.0]' is ambiguous. Since Eq. (13) contains no β or length-normalization parameter, it should be stated explicitly which hyperparameters were tuned for InfoPO and how the length normalization mentioned in the same paragraph enters the loss.","section":"Appendix B.2"},{"comment":"The quantity 'Margins' is not defined in the text or captions. Specify whether it is the difference in average log-likelihoods, the reward margin, or something else, and report the standard errors or number of runs, since the curves appear to be based on a single training run.","section":"Figures 1 and 2"},{"comment":"On Mistral-7B, CPO achieves a GSM8K score of 33.06, which is higher than InfoPO's 32.07; the text's claim that InfoPO consistently outperforms all baselines is therefore overstated. Please qualify the claim or perform significance testing across multiple seeds.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"This paper cannot be accepted in its current form because the central objective is, under the paper's own sampling model, equivalent to supervised fine-tuning on chosen responses. The NWJ derivation in §4.3 and the proof of Theorem 4.1 in Appendix A both contain load-bearing errors. The empirical results, even if reproducible, would then be explainable by SFT on the chosen subset rather than by the proposed MI-based preference mechanism. The issues are not local presentation problems; they concern the fundamental validity of the method. A revision would need to change the sampling model or the loss itself, along with the theoretical analysis, which goes beyond a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take on 2505.08507: the paper's central claim does not survive contact with its own equations. InfoPO's loss in Eq. (13) is derived from NWJ, but with the stated sampling model (yl ~ pi_ref), the rejected term is constant in expectation, so the objective is just SFT on chosen responses. Theorem 4.1's proof commits the same error: Eq. (23) is forward cross-entropy plus a constant, not reverse KL.\n\nWhat is actually new: the observation that DPO can be read as InfoNCE with a particular critic is a legitimate reframing, and the paper states it cleanly. The loss itself is simple enough to reimplement, and the benchmark numbers are at least internally consistent. If the theoretical problems were fixed, the empirical recipe could be worth a follow-up.\n\nWhere it falls apart: first, the NWJ derivation drops beta and the reference term; Eq. (13) only matches Eq. (12) at beta=1, and beta is never set to 1 in the experiments—Appendix B.2 tunes it in {0.5,1.0,2.0} for a loss that contains no beta. Second, under Section 4.1's sampling model, E_{yl~pi_ref}[pi_theta(yl)/pi_ref(yl)] = 1, so the rejected term has zero expected gradient. At initialization, its actual gradient increases rejected likelihood, which is the opposite of the claimed conservative unlearning. Third, the proof of Theorem 4.1 confuses forward and reverse KL; Eq. (23) is not D_KL(pi_theta || pi_chosen). Also, the critic in Eq. (9) does not actually depend on c, which makes the \"preference\" label in the InfoNCE bound more a formality than a modeling choice.\n\nThe experiments lack an SFT-on-chosen baseline, which is essential once the reduction to SFT is understood. The citation pattern is okay—DPO, SimPO, NCE, NWJ all cited—and the limitations section is honest, except it does not mention any of these gaps.\n\nWho it is for: anyone working on preference optimization who wants to see a concrete example of a mutual-information estimator argument gone wrong. It is not a usable method as written. I would send it to review only because the flaw is subtle enough to deserve a referee's time; after review, the theory needs to be redone or the paper reframed as an empirical loss with an SFT baseline. I would not cite it in its current form.\n\nBest.","headline":"InfoPO's rejected-response term is constant under the paper's own sampling model, so the loss reduces to SFT on chosen responses and the reverse-KL theorem is unproven.","tokens_in":16431,"tokens_out":4090,"would_cite":false,"duration_ms":39426,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InfoPO aligns LLMs by maximizing mutual information, not Bradley-Terry odds","keywords":["InfoPO","preference optimization","mutual information","NWJ estimator","Bradley-Terry model","LLM alignment","reasoning benchmarks","reverse KL divergence"],"falsifier":"Train InfoPO on a preference dataset and measure the empirical average of $\\pi_\\theta(y_l|x)/\\pi_{\\mathrm{ref}}(y_l|x)$ over the rejected responses at each checkpoint. If this average stays at $1$ and the gradient contribution of that term is numerically zero while the loss trajectory matches training on $-\\log \\pi_\\theta(y_w|x)$ alone, then the rejected-response mechanism is not doing the unlearning work the paper claims.","tokens_in":1702,"feed_emoji":"🧮","tokens_out":1903,"duration_ms":116751,"temperature":0.7,"pith_summary":"InfoPO is a new preference fine-tuning method for large language models that drops the Bradley-Terry assumption used by DPO and SimPO. Instead, it directly maximizes the conditional mutual information between response and preference label given the prompt, using the NWJ estimator. The resulting loss keeps the chosen response likely while applying a gentle penalty to rejected responses, so the model unlearns bad answers without driving down the probability of good ones. The paper argues this prevents the known failure mode where DPO-style training lowers chosen-response likelihood and hurts math and coding performance. It reports that InfoPO consistently outperforms DPO, SimPO, and related baselines on reasoning, instruction-following, chat, and summarization benchmarks.","feed_headline":"InfoPO swaps Bradley-Terry for mutual info, wins on math","feed_subtitle":"A mutual-information objective keeps chosen responses likely and lifts GSM8K and MATH.","key_machinery":"The load-bearing object is the NWJ estimator of conditional mutual information, $I(Y;C|X) \\ge \\mathbb{E}[f] - \\mathbb{E}[e^f] + 1$, used with the critic $f_\\phi(x,c) = \\beta \\log(\\pi_\\theta(y|x)/\\pi_{\\mathrm{ref}}(y|x))$. With that critic, InfoNCE becomes exactly the DPO loss, and NWJ becomes $L_{\\mathrm{InfoPO}} = -\\log \\pi_\\theta(y_w|x) + \\pi_\\theta(y_l|x)/\\pi_{\\mathrm{ref}}(y_l|x)$. This identity is what lets the paper move from a contrastive, margin-based objective to a direct keep-chosen, gently-demote-rejected objective. Theorem 4.1 then claims that minimizing this loss is equivalent to minimizing the reverse KL divergence $D_{\\mathrm{KL}}(\\pi_\\theta \\| \\pi_{\\mathrm{chosen}})$, the mode-seeking behavior the paper argues alignment should have. The machinery simultaneously explains DPO as a special case and supplies the replacement objective.","core_discovery":"The central claim is that direct preference optimization can be reframed as mutual-information maximization. DPO's loss is exactly an InfoNCE estimator of the conditional mutual information between response and preference label given the prompt when the critic is $\\beta \\log(\\pi_\\theta/\\pi_{\\mathrm{ref}})$. Swapping in the NWJ estimator gives the InfoPO objective $L_{\\mathrm{InfoPO}} = -\\log \\pi_\\theta(y_w|x) + \\pi_\\theta(y_l|x)/\\pi_{\\mathrm{ref}}(y_l|x)$. The first term keeps the chosen response probable; the second produces a smaller, more conservative gradient on rejected responses than DPO's reciprocal-probability gradient. The paper proves that minimizing this loss encourages mode-seeking behavior via reverse KL divergence between the model and the chosen-response distribution, and shows empirically that this preserves chosen likelihoods while improving reasoning, coding, and chat benchmarks.","pith_inferences":["A direct comparison the paper leaves implicit: InfoPO trained on preference pairs should be compared against plain supervised fine-tuning on the chosen responses alone; if the rejected-response term has zero expected gradient, the two should match.","The mutual-information framing suggests a family of alignment losses indexed by MI estimator; interpolating between InfoNCE and NWJ could trade bias for variance in a controlled way, though the paper does not explore this.","If reverse-KL mode-seeking is what the loss implements, InfoPO should reduce the entropy of generated responses relative to the reference model; measuring generation diversity across training would test this prediction."],"forward_implications":["If InfoPO is correct, preference alignment can be performed without assuming the Bradley-Terry model, so the chosen-response likelihood decline is addressed by construction rather than by an added regularization term.","The reverse-KL characterization implies InfoPO concentrates probability mass on a focused set of high-reward responses, which the paper argues is the right inductive bias for reasoning and task-specific generation.","Because the loss contains an explicit chosen-response log-likelihood term, InfoPO should require fewer manually tuned regularization hyperparameters than methods that add NLL penalties on top of a contrastive loss.","The reported gains on GSM8K, MATH, HumanEval, and MBPP suggest that preference fine-tuning with InfoPO preserves pretrained reasoning abilities while still improving instruction following and chat quality."],"supporting_citations":[{"why":"Defines the NWJ estimator, the mutual-information lower bound whose instantiation with the paper's critic yields the InfoPO loss.","marker":"Nguyen et al., 2010"},{"why":"Defines InfoNCE/contrastive predictive coding, which the paper shows reduces to the DPO objective under its critic.","marker":"Oord et al., 2018"},{"why":"Introduces DPO, the main baseline and the source of the closed-form optimal-policy identity used in the proof of Theorem 4.1.","marker":"Rafailov et al., 2024"},{"why":"Introduces SimPO, a primary comparison baseline, and supplies the experimental configuration and length-normalization approach used in the evaluation.","marker":"Meng et al., 2024"},{"why":"Supplies the bias-variance comparison of NWJ versus InfoNCE estimators that motivates replacing InfoNCE with NWJ.","marker":"Poole et al., 2019"},{"why":"Establishes the optimal-discriminator/log-density-ratio relation invoked in the theorem's proof.","marker":"Song and Ermon, 2020"},{"why":"Documents the chosen-likelihood decrease failure mode of DPO that InfoPO is designed to avoid.","marker":"Pal et al., 2024"}],"fun_headline_variants":["InfoPO reframes DPO as mutual info to fix overfitting","InfoPO: mutual info keeps chosen responses likely","InfoPO: mutual info boosts reasoning over Bradley-Terry","InfoPO: conservative mutual-info gradients improve LLMs"],"cache_read_input_tokens":18304,"weakest_assumption_plain":"The load-bearing premise is that the rejected-response penalty in the InfoPO loss actually pushes the model during training; if rejected responses are effectively sampled from the reference distribution, the average penalty is constant, its gradient is zero, and the method reduces to supervised fine-tuning on chosen responses.","fun_headline_variants_meta":{"raw":{"variants":["InfoPO reframes DPO as mutual info to fix overfitting","InfoPO: mutual info keeps chosen responses likely","InfoPO: mutual info boosts reasoning over Bradley-Terry","InfoPO: conservative mutual-info gradients improve LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3225,"prompt_tokens":856,"completion_tokens":2369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":2302}},"tokens_in":472,"tokens_out":2369,"duration_ms":20184,"temperature":1.0,"reasoning_tokens":2302,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:55:13.003675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train InfoPO on a preference dataset and measure the empirical average of $\\pi_\\theta(y_l|x)/\\pi_{\\mathrm{ref}}(y_l|x)$ over the rejected responses at each checkpoint. If this average stays at $1$ and the gradient contribution of that term is numerically zero while the loss trajectory matches training on $-\\log \\pi_\\theta(y_w|x)$ alone, then the rejected-response mechanism is not doing the unlearning work the paper claims.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the NWJ estimator, the mutual-information lower bound whose instantiation with the paper's critic yields the InfoPO loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the bias-variance comparison of NWJ versus InfoNCE estimators that motivates replacing InfoNCE with NWJ."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the optimal-discriminator/log-density-ratio relation invoked in the theorem's proof."}],"review_version":1}