{"id":"f11457ab-2062-426e-b907-f802c9e9d05d","arxiv_id":"2506.05260","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LeanPO improves Video-LLM alignment by using a reference-free average-likelihood reward, self-generated winning/losing pairs, and dynamic label smoothing, yielding gains on six video benchmarks.","lead":"LeanPO is a training method for video-language models that replaces DPO's ratio-based reward with the average log-likelihood of a response, paired with self-generated preference data and dynamic label smoothing. It reports consistent benchmark gains over two Video-LLM baselines, though the core loss closely resembles SimPO.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eqs. 2–3 define the reward with the frozen SFT model π_sft, not the trainable policy π_θ; as written the LeanPO loss is constant w.r.t. θ, so the central training objective is either a typo or unimplementable.","rationale":"I focused on the formal definition of the training reward because the entire paper's contribution—mitigating likelihood displacement by using average likelihood as reward—depends on the reward being a function of the policy being trained. The manuscript contains a direct contradiction between Sec. 1 ('policy model') and Eqs. 2/3 (π_sft). This is more load-bearing than the reader's conceptual concern about likelihood-quality alignment because it is a definite internal inconsistency: either the equations are wrong or the method cannot learn. The reader's concern is still valid and partially overlaps: even if the π_θ version is intended, Fig. 3 shows likelihood is not a general quality proxy, so the method depends on the GT-hint data pipeline. That dependency is acknowledged by the paper though. Since the likely resolution is a typo (the code presumably uses π_θ), and the data-pipeline contribution could still be valid, the correct verdict remains CONDITIONAL: require the authors to correct Eqs. 2/3, release the loss code, and clarify the relationship to SimPO. I therefore keep the reader's CONDITIONAL verdict (UNCHANGED) but flag a different, more concrete reason.","tokens_in":19410,"tokens_out":8221,"duration_ms":90807,"concrete_test":"Inspect the released code (github.com/Wang-Xiaodong1899/LeanPO): locate the loss computation and verify whether the log-likelihoods in Eqs. 2/3 are evaluated with the trainable policy network θ or a frozen SFT checkpoint. Concretely, run one training step with a randomly initialized policy and check the gradient norm of the LLM backbone; if it is exactly zero, the loss is not a function of θ and the reported training is impossible. Also diff the loss against SimPO to enumerate the exact algorithmic changes, e.g., whether the only difference is the dynamic label-flip term from Eq. 6.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that maximizing the margin between average log-likelihoods of winning and losing responses updates the policy. However, Eqs. 2 and 3 define r(V,q,y)=β/|y| Σ log π_sft(...), where π_sft is the frozen SFT model (used in Eq. 1 to generate y_init). Section 1 explicitly says the reward is 'the average likelihood of the response generated by the policy model,' and the training loss Eq. 7 must be a function of θ to produce gradients. But Eqs. 4–7 reuse r without redefining it; if r is the π_sft-based quantity, p(y_w≻y_l) and z_q are constants w.r.t. θ, making L_LEANPO non-differentiable and training impossible. If r is instead computed with π_θ, then Eqs. 2/3 are misstated and the method reduces to SimPO plus dynamic label smoothing—so the claimed novelty of the 'average-likelihood reward' is not actually novel. Either way, a central component of the submission is incorrectly specified. Fig. 3's observation that GT answers have the lowest likelihood reinforces that the reward is only meaningful relative to the constructed data distribution, not as a general quality proxy, but the immediate technical defect is the π_sft/π_θ inconsistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LeanPO, a reference-free preference optimization method for Video-LLMs. The key idea is to replace the DPO-style ratio reward with the average log-likelihood of a response under the policy model, pair it with a self-generated preference data pipeline that injects ground-truth hints into winning responses and uses video augmentation for losing responses, and add a dynamic label-smoothing rule based on the reward margin. The authors report consistent gains over two video-LLM baselines on multiple-choice and open-ended benchmarks, including improvements of 12.4% on Video-MME and 10.8% on NeXT-QA for LLaVA-NeXT-Video-7B, plus generalization experiments on Qwen-VL and image-based LLaVA models.","tokens_in":19697,"tokens_out":3920,"duration_ms":50479,"significance":"If the formulation is corrected and the empirical claims hold, LeanPO is a practical and inexpensive alignment recipe: it removes the reference model, avoids external reward models or GPT-based labeling, and shows gains across several backbones and benchmarks. The authors also release code and test on both video and image LLMs, which strengthens the paper's reproducibility and scope. The main caveats are that the central objective as written is non-differentiable, that the reward reformulation is essentially SimPO, and that several hyperparameters and tuning choices are underreported, so the specific novelty and the robustness of the reported margins need to be clarified.","major_comments":[{"comment":"The reward in Eqs. (2) and (3) is defined with the frozen SFT model π_sft, but the final objective (7) optimizes π_θ. Taken literally, r(V,q,y_w) and r(V,q,y_l) do not depend on θ, so p(y_w ≻ y_l) and the pseudo-label z_q are constants with respect to θ and L_LEANPO has zero gradient. The abstract and Section 1 explicitly state that the reward is the average likelihood under the policy model, so this is very likely a typo, but as written the central training objective is unimplementable. Please correct Eqs. (2) and (3) to use π_θ and verify that all subsequent equations are consistent.","section":"Sec. 3.2, Eqs. (2), (3), and (7)"},{"comment":"After replacing π_sft with π_θ in Eqs. (2)-(3), Eq. (4) is exactly SimPO's reference-free objective with a margin γ: -log σ(β/|y_w| log π_θ(y_w) - β/|y_l| log π_θ(y_l) - γ). The paper's claim that reformulating the reward as the average likelihood is a contribution is therefore not correct, since SimPO already uses this reward. The genuinely new components are the self-generated data pipeline and the dynamic label smoothing in Eqs. (5)-(6). The paper should acknowledge this overlap explicitly and position the contribution accordingly.","section":"Sec. 3.2, Eq. (4) and Related Work Sec. 2.2"},{"comment":"Fig. 3 shows that ground-truth answers, which the paper itself regards as having the highest trustworthiness, receive the lowest average-likelihood reward. This directly contradicts the premise that the average likelihood is a reliable proxy for response quality. The reward-trustworthiness correlation is therefore not a property of the reward itself but an artifact of the constructed data pipeline, where GT hints are injected into winning responses and augmentation is used for losing responses. The authors should either provide evidence that likelihood and trustworthiness are generally aligned on unmodified model outputs, or restrict the claim to the proposed data distribution and explain why the method would transfer to other preference-data sources.","section":"Sec. 3.1 and Fig. 3"},{"comment":"The reward margin γ and the label-flipping threshold d appear in Eqs. (4)-(5) but are never reported in the implementation details or the appendix. In addition, the smoothing factor α is selected in Tab. 7 by evaluating on Video-MME, which is the same benchmark used to report the final gains; this is test-set tuning and can inflate the results. All reported numbers appear to be single runs with no error bars or significance tests; given that some improvements are small (e.g., 0.8% overall for LLaVA-Video), the authors should provide multiple seeds or at least standard deviations, and should evaluate α on a held-out validation set rather than on Video-MME.","section":"Sec. 4.1, Tab. 7, and Fig. 5"}],"minor_comments":[{"comment":"Several table entries have missing spaces or fused numbers, e.g., '28.238.422.3' in Table 3 and '59.855.9' in Table 9; please reformat the tables for readability.","section":"Sec. 4.2, Tables 3 and 9"},{"comment":"The caption contains a typo: 'for on Video-MME' should be 'on Video-MME'. Also, the figure reports a single trajectory with one run per size, so it is unclear whether the non-monotonic medium-duration behavior is significant.","section":"Fig. 5 caption"},{"comment":"References [15] and [21] are the same DPO-shift paper, and references [16] and [20] are the same 'Preference fine-tuning of LLMs should leverage suboptimal, on-policy data' paper; duplicate citations should be removed.","section":"References"},{"comment":"The text refers to 'More details are shown in Appendix A', but Appendix A only describes the benchmarks and contains no implementation details, hyperparameter settings, or training configurations; please either add the missing details or fix the cross-reference.","section":"Sec. 4.1 and Appendix A"},{"comment":"The notation [V,q] is used inconsistently for conditioning in Eqs. (2)-(3) and for the concatenation of the video and question in Eq. (1); please define the notation once and use it uniformly.","section":"Sec. 2.2 and Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"The central equations are almost certainly a typo, but the paper as submitted contains a non-differentiable objective, so the technical correction is mandatory. Once corrected, the method is a SimPO variant with a clever data-generation scheme and dynamic labels; the empirical gains are interesting but need robustness checks, especially because α is tuned on the test benchmark and the reported margins are sometimes small. I would encourage the editor to request a revision that fixes the reward definition, adds the missing hyperparameters and error bars, and reframes the novelty claim honestly relative to SimPO."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Interesting read. The empirical story is stronger than the theory. The data pipeline — injecting GT answers as hints, generating the winning response via reflection, and using video augmentation for the losing response — is the real contribution. The ablations show it helps DPO and SimPO too, so that part deserves credit. The dynamic label smoothing is a minor add-on but harmless.\n\nThe problem is that the equations don't match the text. The reward in Eqs. 2–3 is defined with π_sft, the frozen SFT model, not the trainable π_θ. Under that definition, the Bradley-Terry objective and the final loss in Eq. 7 are constant with respect to θ; training is impossible. This isn't a nitpick — it's a load-bearing inconsistency. If they meant π_θ (which they clearly did, given the introduction), then the loss is exactly SimPO with a length-normalized reward and a margin, plus their label-smoothing trick. Either way, the paper as written cannot be used to reproduce training.\n\nThe reported gains are plausible but under-supported: single runs, no error bars, no significance tests, and the smoothing factor α is selected on Video-MME itself. Hyperparameters d and γ are never reported. There's also no contamination check between the Hound training data and the evaluation benchmarks, and the TPO citation [61] points to a different paper (Temporal Alignment Networks, not Temporal Preference Optimization). The Image-LLM table also has garbled column headers.\n\nIf you can look past the equation issue as a typo, the pipeline alone seems worth a closer look. But the paper needs a serious revision before the numbers can be trusted. I'd recommend sending it to peer review with a request for major changes, since the empirical claims are substantial and the data pipeline is genuinely useful. Right now it's not reproducible.\n\nFor your reading group, I'd wait for the revision.","headline":"Useful data pipeline and plausible gains, but the equations as written make the loss a constant; major revision needed.","tokens_in":20226,"tokens_out":3160,"would_cite":false,"duration_ms":32178,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LeanPO replaces DPO's ratio-based reward with the policy model's own average response likelihood, stopping the likelihood-displacement collapse in Video-LLM alignment and lifting LLaVA-NeXT-Video-7B from 38.8 to 43.6 on Video-MME.","keywords":["preference optimization","video large language models","likelihood displacement","direct preference optimization","self-generated preference data","dynamic label smoothing","video question answering","hallucination"],"falsifier":"Take a preference pair in which the human-preferred response has a lower average log-likelihood than the rejected one (Fig. 3 shows such inversions are routine for ground-truth answers), optimize the LeanPO loss on it, and record the chosen response's likelihood: if it decreases, the average-likelihood reward is misaligned with quality outside the pipeline's specially generated pairs, and the reported gains would depend on the GT-hint data construction rather than on the reward reformulation alone.","tokens_in":19208,"feed_emoji":"🎬","tokens_out":13209,"duration_ms":131415,"temperature":0.7,"pith_summary":"The paper sets out to show that the main obstacle to efficient preference alignment of Video-LLMs is likelihood displacement: during DPO training the probabilities of both the chosen and the rejected response fall, because the objective only rewards the gap between them. Its proposal, LeanPO, replaces DPO's ratio-of-likelihoods reward with the policy model's own length-averaged log-likelihood, making the winning response's reward stable instead of falling, and it pairs this reward with a data pipeline that keeps likelihood and trustworthiness aligned: the winning response is generated with the ground-truth answer as a hint and refined by self-reflection, while the losing response comes from a video-augmented pass. A dynamic label-smoothing rule down-weights the rare pairs where the losing response accidentally outscores the winner. If the claims hold, video preference alignment becomes cheaper and more stable, with no reference model and no external reward labels, and the paper reports LLaVA-NeXT-Video-7B rising from 38.8 to 43.6 on Video-MME and from 53.9 to 59.7 on NeXT-QA, plus consistent gains on LLaVA-Video, Qwen2-VL, and Qwen2.5-VL checkpoints.","feed_headline":"Video-LLM alignment fix lifts Video-MME by 12.4%","feed_subtitle":"Rewarding each answer by its own average likelihood stops both chosen and rejected responses from slumping in training.","key_machinery":"The load-bearing object is the reformulated implicit reward, the length-normalized average log-likelihood $r(V,q,y)=\\frac{\\beta}{|y|}\\sum_{i=1}^{|y|}\\log\\pi_\\theta(y_i\\mid [V,q], y_{<i})$, which replaces DPO's reference-ratio reward and makes the training objective a margin maximization over likelihoods. Around it sits the reward-trustworthiness-correlated self-generated preference data pipeline: for each video-text triplet $(V,q,a)$ the model is prompted with the ground-truth answer as a hint, produces an initial response, and then rewrites that response in a self-reflection pass to form the winning answer $y_w$, while the losing answer $y_l$ is generated from an augmented copy of the video; the design goal is that $y_w$ carries both higher likelihood and higher trustworthiness than $y_l$. The third component is dynamic label smoothing: a pseudo-label $z_q$ is set to 1 when the likelihood margin falls below a threshold $d$, indicating a possibly noisy pair, and the smoothed preference probability $\\tilde{p}(y_w\\succ y_l)=(1-z_q\\alpha)\\,p(y_w\\succ y_l)+z_q\\alpha\\,p(y_l\\succ y_w)$ replaces the raw Bradley-Terry term, with a margin term $\\gamma$ further separating the two responses in Eq. 4. These three pieces, reward form, pair construction, and dynamic labels, carry the argument, and each is ablated in Table 6.","core_discovery":"The paper's central claim, stated on its own terms, is that likelihood displacement is more severe for Video-LLMs than for LLMs: the redundant complexity of video inputs pushes both $\\log\\pi_\\theta(y_w\\mid x)$ and $\\log\\pi_\\theta(y_l\\mid x)$ down faster during DPO, and the root cause is the ratio form of the DPO reward, $\\log(\\pi_\\theta(y)/\\pi_{\\mathrm{sft}}(y))$, which becomes erratic when both likelihoods fall and the gap to the SFT model widens. LeanPO therefore redefines the implicit reward as the average likelihood of the response under the policy model itself, $r(V,q,y) = \\frac{\\beta}{|y|}\\sum_{i=1}^{|y|}\\log\\pi_\\theta(y_i\\mid [V,q], y_{<i})$, turning the training objective into a direct maximization of the likelihood margin between winning and losing responses. Because ground-truth answers, despite being most trustworthy, receive the lowest likelihood (Fig. 3), the paper argues the reward is reliable only for pairs in which likelihood and trustworthiness are correlated, and contributes a pipeline that manufactures such pairs. The final piece is a dynamic label-smoothing objective in which a pseudo-label $z_q$ flags pairs whose reward margin falls below a threshold $d$, and the smoothed preference probability $(1-z_q\\alpha)\\,p(y_w\\succ y_l) + z_q\\alpha\\,p(y_l\\succ y_w)$ replaces the raw Bradley-Terry term.","pith_inferences":["The paper leaves implicit that its reward is calibrated only inside its own pipeline: Fig. 3 shows ground-truth answers, despite maximal trustworthiness, receive the lowest likelihood, so running LeanPO unchanged on human- or GPT-labeled pairs would test whether the reward generalizes, and it would likely push chosen responses downhill whenever the chosen answer is a low-likelihood ground truth.","If the explanation that video redundancy amplifies displacement is right, the severity of likelihood displacement should scale with input complexity: more frames, longer context, or denser temporal events should produce larger likelihood drops, which the paper does not measure directly.","The dynamic label smoothing can be viewed as an online noise filter, which suggests a simpler alternative worth testing: discard pairs whose margin falls below the threshold $d$ instead of smoothing them, at lower implementational cost.","Because the winning response is always generated with the ground-truth answer as a hint, the pipeline inherits the quality of the source annotations; on subjectively judged or open-ended video questions, the reflection step could propagate annotation bias rather than remove it."],"forward_implications":["The reference-free reward removes the need to keep the SFT reference model during alignment, and the paper shows the winning response's reward curve stays stable or rises during training instead of falling.","Any existing video QA or video caption corpus can be converted into paired preference data by the pipeline, without human annotators, GPT judges, or a reward model; the paper uses 17k triplets from LLaVA-Hound and trains on a single pass.","The gains transfer across model families and capacities: LLaVA-NeXT-Video-7B, LLaVA-Video-7B, Qwen2-VL-7B, Qwen2.5-VL-7B, and the image model LLaVA-v1.5-7B all improve on the reported benchmarks.","The largest reported gains land on the temporal-reasoning benchmarks (Video-MME and NeXT-QA), the settings where the paper argues displacement and hallucination are most visible.","Scaling the preference dataset further improves results, with the medium-duration Video-MME slice benefiting most as data grows (Fig. 5)."],"supporting_citations":[{"why":"The DPO baseline whose likelihood-displacement failure motivates LeanPO; its ratio-form reward is the object being replaced.","marker":"[1]"},{"why":"Coined and analyzed likelihood displacement in LLMs, the phenomenon LeanPO extends to Video-LLMs and claims is amplified there.","marker":"[17]"},{"why":"Supplies the 17k video-text triplets used to generate preference data and the Hound win/lose pairs used as comparisons in Fig. 3 and the ablations.","marker":"[13]"},{"why":"The closest reference-free preference optimization baseline; LeanPO's reward reformulation is positioned against it.","marker":"[32]"},{"why":"The current video-DPO method (Temporal Preference Optimization) that LeanPO compares against and outperforms on video benchmarks.","marker":"[14]"},{"why":"The LLaVA-NeXT-Video-7B backbone on which the main results, likelihood curves, and ablations are run.","marker":"[6]"},{"why":"The second Video-LLM backbone used to demonstrate that gains transfer beyond a single model family.","marker":"[8]"},{"why":"The Video-MME benchmark that supplies the headline numbers and the short/medium/long-duration breakdowns.","marker":"[47]"}],"fun_headline_variants":["Video-LLM alignment fix lifts Video-MME by 12.4%","LeanPO boosts Video-LLMs by curbing DPO likelihood drops","Reference-free LeanPO aligns Video-LLMs with trusty preferences","Dynamic label smoothing and self-preference data lift Video-LLMs","Stop Video-LLM preference collapse with LeanPO's likelihood reward"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the premise that a response's average likelihood under the model is a dependable measure of its quality, but the paper's own Fig. 3 shows the most trustworthy answers (the ground truths) receive the lowest likelihood, so the reward only tracks trustworthiness for the specially engineered pairs the pipeline creates.","fun_headline_variants_meta":{"raw":{"variants":["Video-LLM alignment fix lifts Video-MME by 12.4%","LeanPO boosts Video-LLMs by curbing DPO likelihood drops","Reference-free LeanPO aligns Video-LLMs with trusty preferences","Dynamic label smoothing and self-preference data lift Video-LLMs","Stop Video-LLM preference collapse with LeanPO's likelihood reward"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1713,"prompt_tokens":1190,"completion_tokens":523,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":806,"completion_tokens_details":{"reasoning_tokens":428}},"tokens_in":806,"tokens_out":523,"duration_ms":6301,"temperature":1.0,"reasoning_tokens":428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:21:06.315824+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a preference pair in which the human-preferred response has a lower average log-likelihood than the rejected one (Fig. 3 shows such inversions are routine for ground-truth answers), optimize the LeanPO loss on it, and record the chosen response's likelihood: if it decreases, the average-likelihood reward is misaligned with quality outside the pipeline's specially generated pairs, and the reported gains would depend on the GT-hint data construction rather than on the reward reformulation alone.","supporting_citations":[{"cited_title":"Direct preference optimization: Your language model is secretly a reward model","cited_arxiv_id":null,"evidence_quote":"The DPO baseline whose likelihood-displacement failure motivates LeanPO; its ratio-form reward is the object being replaced."}],"review_version":1}