{"id":"c3c39e15-b2c1-4348-9c49-5fa525f3f0ee","arxiv_id":"2411.14797","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"nSFT matches multimodal RLHF performance by converting rejected responses into corrective SFT data, without pairwise preference optimization.","lead":"The authors introduce nSFT, a way to align vision-language models by turning the model's wrong answers into corrective training conversations and then fine-tuning with a plain supervised loss. They report that nSFT matches or outperforms multimodal RLHF methods such as DPO and PPO while requiring about half the GPU memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The theoretical bridge from DPO to nSFT is invalid: Eq. 20-23 miscompute the DPO gradient, and the nSFT loss contains no term evaluated on the rejected response, so the claim that nSFT captures RLHF's negative supervision is unproven.","rationale":"The reader's weakest assumption—that G(·) extracts the same negative supervision as DPO—is exactly the point on which the paper is least secure. The paper's own derivation is internally inconsistent: Eq. 20-23 omit the sigmoid derivative and produce a scaling factor that does not match the actual DPO gradient. A corrected derivation would still leave a gap, because nSFT trains on the LLM's reconstructed conversation rather than applying an explicit penalty to the rejected sequence. The empirical campaign is a real strength: controlled baselines across three data sources, multiple model sizes, and DPO/PPO variants, plus the VEC ablation showing the codebook matters. However, none of those experiments isolates whether the rejected response yr is necessary, so the paper's central explanatory claim is not yet demonstrated. This does not warrant rejection of the empirical result that nSFT performs comparably to DPO/PPO; it warrants conditional acceptance pending a corrected derivation, a yr-free control, and release of data and code. The reader's CONDITIONAL verdict therefore stands unchanged, and the reader correctly identified the same load-bearing concern.","tokens_in":22831,"tokens_out":13819,"duration_ms":136642,"concrete_test":"Run the Table 1 LLaVA-150k experiment with a control construction G'(yc, Q) that uses the same LLM prompt and codebook but omits the rejected response yr, generating the same number of corrective QA pairs from GT alone; keep all training settings identical. If nSFT-without-yr matches nSFT on the nine benchmarks, then yr is not the active ingredient and the negative-supervision mechanism is not supported. If performance drops, the rejected-response information is doing the work and the central claim survives this test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A.1 (Eq. 20-23) derives the DPO gradient as (1/p_dpo) times the difference of two SFT gradients. This is not the derivative of L_d = -log sigma(beta p_dpo). The correct chain rule gives dL_d/dtheta = -beta sigma(-beta p_dpo) dp_dpo/dtheta, a sigmoid coefficient, not 1/p_dpo. The claimed characterization of DPO as a linear combination of SFT gradients with a 1/p_dpo scaling is therefore not established. More substantively, the proposed nSFT loss L_nSFT = L_sft(yc) + L_sft(G(yr; yc, Q)) contains no loss term evaluated on the rejected response yr itself; it increases the likelihood of the LLM's reconstructed correction. DPO's defining behavior is to decrease the likelihood of the rejected response, weighted by beta sigma(-beta p_dpo). The paper never proves that training on G(yr) yields the same gradient direction on yr, and no ablation isolates the contribution of yr: Table 6 removes the vision error codebook and the chosen response, but never removes the rejected response from the construction. The empirical comparisons across data sources, model scales, and RLHF variants are extensive and useful, and the VEC ablation shows the codebook matters. However, without a demonstration that the rejected-response information is the active ingredient, the central causal claim that nSFT matches RLHF by excavating the same negative supervision is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the success of multimodal RLHF (DPO/PPO) in continual VLM alignment is largely due to the negative supervision provided by rejected responses. It proposes nSFT, which uses an LLM (GPT-4) guided by a vision error codebook to construct corrective conversations from the rejected responses, and trains the VLM with a standard SFT loss on the chosen caption plus the constructed conversation. The paper reports experiments across three alignment data sources, multiple base VLMs (LLaVA-1.5-7B/13B, LLaVA-NeXT-13B), and various RLHF baselines (GT-DPO, SeVa, SIMA, PPO, iterative DPO), finding that nSFT matches or exceeds the RLHF baselines while requiring only one model in training. The central claim is that nSFT 'strictly matches' multimodal preference alignment by excavating the same negative supervision.","tokens_in":23199,"tokens_out":8025,"duration_ms":70347,"significance":"If established, the claim that a simple continual SFT loss on LLM-reconstructed corrective conversations matches DPO/PPO would have substantial practical value, since nSFT avoids the memory overhead of maintaining reference, policy, and reward models. The empirical coverage is a strength: the paper evaluates on nine benchmarks, three data sources, two model scales, and four RLHF variants, and the VEC ablation indicates that the codebook contributes to the gains. However, the theoretical derivation that forms the paper's stated basis for the negative-supervision equivalence contains a calculus error, and the empirical design does not directly isolate the contribution of the rejected response, so the causal claim as stated is not yet supported.","major_comments":[{"comment":"The derivative of -log sigma(beta p_dpo) with respect to theta is computed incorrectly. Since d/dz[-log sigma(z)] = -sigma(-z), the prefactor in Eq. (20) is -beta sigma(-beta p_dpo), not -1/p_dpo. Consequently, Eq. (9) and Eq. (23) are wrong as written. A corrected derivation would still express the DPO gradient as a linear combination of the chosen and rejected SFT gradients, but with a sigmoid coefficient rather than 1/p_dpo scaling. This does not invalidate the overall idea, but the specific mathematical claim used to justify the equivalence is incorrect and must be fixed.","section":"Appendix A.1, Eqs. (20)-(23)"},{"comment":"The nSFT loss L_nSFT = L_sft(yc) + L_sft(G(yr; yc, Q)) is evaluated on the constructed correction G(yr), not on the rejected response yr itself. DPO's negative supervision is realized by decreasing the likelihood of yr. The manuscript does not provide a gradient-level argument showing that increasing the likelihood of G(yr) has an equivalent effect on the model's tendency to produce the original error, and Table 6 does not include an ablation that removes the rejected response from the construction; it ablates the vision error codebook and the chosen response. To support the central causal claim, the authors should add a 'w/o rejected response' condition or an explicit derivation linking the gradient of L_sft(G(yr)) on the error-related tokens to the DPO negative gradient.","section":"Section 3.3, Eq. (15); Table 6"},{"comment":"The conclusion that DPO is 'biased towards how to reject samples' is based on the ratio of partial derivatives with respect to the abstract variables t1 and t2. This ratio alone does not determine the relative contributions to the parameter gradient, because the Jacobians dt1/dtheta and dt2/dtheta are not equal. Please either provide a derivation in terms of the parameter gradient or explicitly defer to the argument in [7] rather than presenting Eqs. (11)-(13) as a self-contained proof.","section":"Section 3.2, Eqs. (11)-(13)"}],"minor_comments":[{"comment":"The phrase 'a huge discrepancy of mutlimodal RLHF literature' contains a typo: 'mutlimodal' should be 'multimodal'.","section":"Section 4.1"},{"comment":"The phrase 'with faster gradient updating rate of t1' appears to contradict the ratio in Eq. (13), which indicates |∂L/∂t2| > |∂L/∂t1| when t2 < t1; please correct the direction or rephrase.","section":"Section 3.2"},{"comment":"The calculation of the 'total' score is not defined; please specify whether it is the sum of per-column improvements over the baseline.","section":"Table 1 and Table 9"},{"comment":"The 15k nSFT data are merged with the LLaVA-665k SFT data during training, which differs from the continual-learning setup in Table 1; this should be explicitly acknowledged in the main text rather than only in the caption.","section":"Section 4.4, Table 2"},{"comment":"The number of constructed conversations per image (5) is a free parameter; no sensitivity analysis is provided for this choice.","section":"Appendix B.1"},{"comment":"The phrase 'fruitful of ablations' should be rephrased, e.g., 'fruitful ablations' or 'a wealth of ablations'.","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on reference [46] (SeVa) for baselines and data construction, and the first author of this manuscript is also a co-author of [46]. This is not inherently problematic, but the novelty of nSFT relative to SeVa should be stated more clearly. The core issues are the incorrect theoretical derivation in Appendix A.1 and the missing ablation that isolates the contribution of the rejected response; if these are addressed, the paper could be a valuable contribution to the field."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the empirical package is stronger than the theory. The paper claims continual SFT can match DPO/PPO for VLM alignment if you inject 'negative supervision' via LLM-constructed corrective conversations. The experiments are broad—three data sources, two model families, three DPO variants plus PPO and iterative DPO—and the recipe is simple, memory-efficient, and plausibly useful. If the numbers reproduce, this is a practical alternative to RLHF.\n\nWhat's new: the nSFT construction pipeline with the vision error codebook is genuinely new, and Table 6 shows the codebook matters. The cross-scale and cross-method comparisons are a solid empirical effort; the in-domain evaluation (Table 3) is a thoughtful addition. Self-citation to SeVa exists but is not load-bearing.\n\nThe soft spots are significant. Appendix A.1's derivation is wrong: d/dz[-log sigma(z)] is -sigma(-z), not -1/z, so Eq. (9) and Eqs. (20–23) are incorrect. A corrected derivation would still give a sigmoid-weighted combination of positive and negative SFT gradients, so the qualitative conclusion survives, but the paper should not present that chain as proof. More importantly, nSFT never trains on the rejected response yr itself—only on the LLM's reconstruction G(yr). DPO directly suppresses yr; the paper never shows that training on G(yr) produces the same gradient direction on yr, and no ablation isolates the contribution of yr. Table 6 removes the codebook and the chosen response, but not the rejected response. So the central causal claim—that nSFT matches RLHF by excavating the same negative supervision—is not actually demonstrated. The 'total' aggregate in Table 1 also sums improvements across benchmarks with very different scales (MME and SQA together), which is opaque; no code or data is released either.\n\nWho gets value: VLM alignment researchers, especially those who want a cheaper SFT-based alternative to DPO. The recipe is easy to try. But the paper needs a corrected derivation, an ablation that removes the rejected response, and clearer aggregation before I'd trust the headline. I'd still send it to review—the empirical scope justifies referee time—but with the expectation of major revision.","headline":"Broad empirical comparison, but the theory is wrong and the causal claim about negative supervision is not actually supported by an ablation.","tokens_in":23684,"tokens_out":3928,"would_cite":false,"duration_ms":35911,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that the success of multimodal RLHF comes mostly from negative supervision, and that a simple SFT loss on LLM-constructed corrective conversations matches DPO and PPO while using fewer models.","keywords":["negative supervised finetuning","multimodal RLHF","direct preference optimization","vision-language models","hallucination correction","preference alignment","LLM data construction"],"falsifier":"Take a held-out set of preference pairs; compute the gradient of the DPO loss with respect to the policy and the gradient of the nSFT loss restricted to the constructed negative term, and measure their cosine similarity across examples. If many examples show near-zero or negative similarity, the claimed equivalence fails; alternatively, replace the GPT-4 construction function with a random or non-error-aware captioner and check whether nSFT's gains over continual SFT disappear.","tokens_in":22646,"feed_emoji":"🖼️","tokens_out":6286,"duration_ms":55590,"temperature":0.7,"pith_summary":"The paper tries to establish that multimodal RLHF is not superior to continual SFT because of reinforcement learning machinery; the decisive ingredient is the negative supervision carried by rejected responses. It proposes nSFT, which runs the base vision-language model to produce responses, asks an LLM with a vision error codebook to identify the errors in those rejected responses, and constructs corrective question-answer conversations. The model is then trained with an ordinary SFT loss on the chosen captions plus the constructed corrections. The authors report that nSFT matches or beats DPO, PPO, and iterative DPO variants across three data sources, three model scales, and nine benchmarks, while requiring only one VLM in training instead of two or four.","feed_headline":"SFT with negative supervision matches multimodal RLHF","feed_subtitle":"Training on LLM-corrected rejected responses beats DPO and PPO on nine benchmarks with one model instead of two or four.","key_machinery":"The load-bearing object is the negative supervision disentangled from the DPO logit: the term $-L_{sft}(y_r)$ that appears when the DPO loss is rewritten as a difference of two next-token-prediction losses. Since that term is a sequence-level SFT loss on the rejected response, nSFT replaces it by $L_{sft}(G(y_r; y_c, Q))$, where $G(\\cdot)$ is an LLM prompted with the rejected response, the chosen response, and a vision error codebook $Q$ listing instance-level and image-level error types (object identity, attribute, action, location, relative position, background, events, reasoning). The construction function converts the pairwise preference signal into plain corrective conversation data, so the alignment stage can run with one model and a standard SFT loss.","core_discovery":"The central claim is that the key success factor of multimodal RLHF (e.g., DPO) mostly attributes to the negative supervision in the rejected responses. The paper derives that DPO's gradient is a linear combination of two SFT gradients, one for the chosen response and one for the rejected response, so continual SFT falls behind mainly because it lacks the rejected-response term. nSFT recovers that missing term by disentangling it from the pairwise DPO logit and expressing it as an SFT target: it uses a construction function $G(\\cdot)$ that identifies the wrong statements in a rejected response, guided by a vision error codebook and the chosen response, and writes them into a corrective conversation. The final loss is $L_{nSFT} = L_{sft}(y_c) + L_{sft}(G(y_r; y_c, Q))$, and the paper claims this strictly matches multimodal preference alignment methods (both DPO and PPO) under different datasets, base VLMs, and evaluation metrics.","pith_inferences":["If the equivalence holds, the practical distinction between SFT and RLHF becomes a difference in data selection rather than optimizer: any preference signal that can be written as corrective text is trainable with cross-entropy.","The construction step shifts cost from GPU memory to an LLM API; a natural extension is to distill the construction function into a smaller local model, or to generate corrective conversations on the fly during training.","A direct way to test the paper's mechanism is to measure the gradient cosine similarity between DPO's negative logit term and the nSFT negative term on the same preference pairs; the paper's formal derivation (Eq. 20-23) skips the sigmoid derivative, so that similarity is not yet established.","The same recipe may transfer to NLP alignment, where the error types are toxicity, style, or factual mistakes rather than visual hallucinations; the paper lists this as future work."],"forward_implications":["Preference alignment for VLMs can be done with an SFT loss once the rejected-response errors are turned into corrective text, so no reward model, reference model, or policy-reference pair is required.","Training memory drops from two models (DPO) or four (PPO) to one, and the measured training time is substantially lower.","nSFT improves over pure continual SFT on hallucination benchmarks, with the largest gains on POPE, CHAIR, and MMHal across OCRVQA, TextCaps, and LLaVA-150k data.","The approach transfers to stronger and larger VLMs (LLaVA-1.5-13B and LLaVA-NeXT-13B) and outperforms iterative DPO (CSR) and PPO on the tested benchmarks.","Adding a per-token KL constraint to nSFT further improves results, suggesting the method composes with RLHF-style regularization."],"supporting_citations":[{"why":"Supplies the theoretical result that for regular preference pairs $t_2/t_1 < 1$, so DPO's gradient is biased toward optimizing the rejected response; the paper leans on this to argue negative supervision is the key.","marker":"[7]"},{"why":"Defines the DPO loss and reward-model formulation that the paper rewrites as a difference of two SFT losses.","marker":"[23]"},{"why":"SeVa, a DPO-style multimodal alignment method that uses noised model outputs as rejected samples; the paper compares against it and reproduces it as a baseline.","marker":"[46]"},{"why":"SIMA, a self-improvement preference optimization method; used as a baseline and its official implementation is reproduced.","marker":"[31]"},{"why":"Ground-truth DPO style with self-response as negative; motivates the chosen/rejected setup analyzed in Section 3.","marker":"[22]"},{"why":"Factually augmented RLHF/PPO baseline that nSFT is compared against; also source of the MMHal evaluation metric.","marker":"[29]"},{"why":"CSR, an iterative DPO variant; provides the multi-iteration comparison that nSFT outperforms.","marker":"[42]"},{"why":"Token-level DPO; the per-token KL constraint used in the nSFT ablation is adopted from this line of work.","marker":"[38]"}],"fun_headline_variants":["Negative-supervised SFT matches RLHF with one model","nSFT turns rejected responses into SFT gains vs DPO/PPO","SFT on rejected logits matches multimodal RLHF","One-model SFT rivals RLHF via negative supervision","Rejected response training makes SFT match RLHF"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that GPT-4's error identification, guided by the vision error codebook, extracts the same corrective signal that DPO's rejected-response logit would provide, and the paper's derivation of this equivalence (Eq. 20-23) does not fully prove it because it drops the sigmoid derivative.","fun_headline_variants_meta":{"raw":{"variants":["Negative-supervised SFT matches RLHF with one model","nSFT turns rejected responses into SFT gains vs DPO/PPO","SFT on rejected logits matches multimodal RLHF","One-model SFT rivals RLHF via negative supervision","Rejected response training makes SFT match RLHF"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000843,"raw_usage":{"total_tokens":3672,"prompt_tokens":944,"completion_tokens":2728,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":2645}},"tokens_in":560,"tokens_out":2728,"duration_ms":19466,"temperature":1.0,"reasoning_tokens":2645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:52:06.175206+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of preference pairs; compute the gradient of the DPO loss with respect to the policy and the gradient of the nSFT loss restricted to the constructed negative term, and measure their cosine similarity across examples. If many examples show near-zero or negative similarity, the claimed equivalence fails; alternatively, replace the GPT-4 construction function with a random or non-error-aware captioner and check whether nSFT's gains over continual SFT disappear.","supporting_citations":[{"cited_title":"Direct preference optimization: Your language model is secretly a reward model","cited_arxiv_id":null,"evidence_quote":"Defines the DPO loss and reward-model formulation that the paper rewrites as a difference of two SFT losses."},{"cited_title":"Token-level direct prefer- ence optimization","cited_arxiv_id":null,"evidence_quote":"Token-level DPO; the per-token KL constraint used in the nSFT ablation is adopted from this line of work."}],"review_version":1}