{"id":"6382860e-ccef-464b-a882-78ef1e995f55","arxiv_id":"2505.22172","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RPO reverses the constraints a response fails to turn any sampled response into a perfect chosen example, improving DPO on multi-constraint instruction following by 2.5 to 4.6 points.","lead":"This paper introduces Reverse Preference Optimization (RPO), a training method that flips unsatisfied constraints in an instruction so that a model's own imperfect responses become perfect training examples. It reports gains over standard DPO on two complex instruction-following benchmarks, and a 70B model that beats GPT-4o on those benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The noise-free guarantee in Section 4.3 fails on a nontrivial fraction of constraints by the paper's own audit: only 85.2% of reversals are perfect, so the chosen response is not always perfect and strictly dominating.","rationale":"I read the paper as claiming two distinct things: a formal property (reversal makes the chosen response perfect and strictly dominating, thereby eliminating noise) and an empirical benefit (RPO improves over DPO on SysBench and Multi-IF and scales to 70B). The formal property is the more load-bearing because it motivates the entire construction and is stated as a guarantee, not as a heuristic. The reader's weakest assumption correctly identifies this: the reversal must be a faithful logical complement. My independent check of the appendices finds the paper's own audit in D.3 confirming that the guarantee is approximate, not exact: roughly 15% of reversals are not perfect, with 8.4% bad and 2.4% non-reversible. The empirical results may still be valid and the method may still work, but the central 'noise-free' advantage is not established by the submitted evidence. I therefore see no reason to move the reader's conditional verdict; the concern is real but warrants further experiments rather than rejection. The proposed concrete test directly measures the formal property at pair level and would settle whether the reversal-quality concern actually changes performance.","tokens_in":22317,"tokens_out":6981,"duration_ms":79256,"concrete_test":"Sample 500 constraints from the training data and, for each, draw 10-20 responses from Llama-3.1-8B-Instruct. Apply the paper's reversal prompt and the same fine-grained evaluator to obtain binary adherence for original and reversed constraints. Compute at the pair level: (i) fraction of chosen responses that are perfect on the reversed instruction; (ii) fraction of pairs in which the chosen strictly dominates the rejected one. Compare these pair-level rates with the 85.2% constraint-level 'perfect reversal' audit; if they fall below 90%, the strict-dominance guarantee fails on more than a negligible fraction of training data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing property is the claim in Section 4.3 that after reversing the constraints a response fails, the chosen response is perfect and strictly dominates the rejected one, 'thus eliminating noise' (Equation 4). This requires two things for every constraint used in training: (1) the adherence labels in S_y are correct, and (2) the reversed constraint is the exact logical complement of the original over the whole response space. The paper's own audit (Table 7, Appendix D.3) is direct evidence that condition (2) fails: only 85.2% of reversals are perfect, 4.0% are slight-error, 8.4% bad, and 2.4% non-reversible. In a bad case from Table 10, 'Replies must end with an exclamation mark' is reversed to 'Replies must end with a period,' whose complement should be 'must not end with an exclamation mark'; a response ending in a question mark violates both, so the chosen response is not perfect and does not dominate. If such pairs are kept in training, Equation 4 is optimized on pairs with the same noise RPO is designed to remove. The paper's Limitations section concedes bad cases 'lead to error propagation.' Without a filtered-reversal ablation, the empirical gains cannot be attributed to the noise-free property; they might come from larger margins or data augmentation. This does not refute the empirical results, but it removes the theoretical centerpiece.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Reverse Preference Optimization (RPO), a modification of DPO for multi-constraint instruction following. For each response that fails some constraints, RPO rewrites the instruction by reversing the failed constraints so that the response becomes a \"perfect\" chosen response, while another response that differs in at least one constraint adherence serves as the rejected response. The objective in Eq. (4) adds an adaptive margin proportional to the number of constraint differences. Training data are generated through a role-driven self-play pipeline that yields a 30K system-prompt corpus (SysBank). On SysBench and Multi-IF, RPO improves over DPO across Llama-3.1 and Qwen-2.5 at the 8B/7B and 70B/72B scales, and the 70B RPO model surpasses GPT-4o on several metrics. The paper also reports sample-efficiency, data-volume, and general-ability analyses.","tokens_in":22595,"tokens_out":6210,"duration_ms":60895,"significance":"The central idea—turning a failing response into a perfect one by reversing the constraints it violates—is elegant and practically attractive, since it avoids expensive sampling of perfect responses. If validated, RPO would be a simple, scalable drop-in modification of DPO for multi-constraint alignment. The paper is commendably transparent: it includes an honest Limitations section, a detailed error analysis in Appendix D.3, sample-efficiency comparisons, and general-ability checks. The SysBank resource and the self-play construction recipe are potentially useful to the community. However, the paper's headline theoretical claim is not fully supported by its own audit, and the empirical evidence lacks uncertainty quantification; these gaps need to be addressed.","major_comments":[{"comment":"The \"noise-free\" property is load-bearing. Section 4.3 states that after reversal the chosen response \"dominates the rejected ones and performs no worse than the rejected response in any aspect, thus eliminating noise,\" and Eq. (4) is derived under this assumption. Appendix D.3 (Table 7) reports that only 85.2% of reversals are perfect, with 4.0% slight-error, 8.4% bad, and 2.4% non-reversible. The bad case in Table 10 (\"Replies must end with an exclamation mark\" reversed to \"Replies must end with a period\") is a direct counterexample: a response ending in a question mark violates both the original and the reversed constraint, so the \"chosen\" response is not perfect and does not dominate. The Limitations section concedes that bad cases \"lead to error propagation.\" Since roughly one in ten training pairs does not satisfy the premise of Eq. (4), the paper needs either to filter such reversals and re-run the experiments, or to quantify how often bad reversals occur in the actual training set and show that the results are insensitive to them. Without this, the empirical gains cannot be attributed to the noise-free mechanism rather than to the enlarged margin or data augmentation.","section":"§4.3, Eq. (4), Appendix D.3, Limitations"},{"comment":"The main results are all single runs without seeds, variance, or significance testing. The headline improvements over DPO are 2.5 points on Multi-IF average and 2.5-6.6 points on SysBench components for Llama-3.1-8B; for Multi-IF Step 1 the Llama-8B RPO improvement over DPO is about 2.9 points, which is small relative to run-to-run variability in fine-tuning. The paper should report means and standard deviations over at least three seeds, or otherwise justify that the differences are outside noise. This is needed to support the wording \"significant improvements\" in the abstract and Section 6.5.","section":"§6.5, Table 1"},{"comment":"The margin hyperparameter γ is tuned directly on the evaluation benchmarks. Table 6 sweeps γ on SysBench and Multi-IF, and the main results in Table 1 use γ=0.05, the value selected on those same benchmarks. Since the test sets are used for model selection, the reported improvements may be optimistic. The paper should either fix γ before seeing the benchmarks, tune on a held-out validation subset, or report the full sensitivity and show that the conclusions are robust across the γ range.","section":"§6.1, §6.6, Table 6"},{"comment":"Table 4 reports that the Reverse method achieves a 1.00 Perfect rate, but this is true by construction only if every reversal is a perfect logical complement. Appendix D.3 shows this is not the case, so the 1.00 figure overstates the sample-efficiency advantage. The Perfect column should be recomputed using the reversal-quality audit, or the table should be restricted to the 85.2% perfect reversals.","section":"§6.6.3, Table 4"}],"minor_comments":[{"comment":"The sentence \"Across the three metrics of SysBench, RPO outperforms the DPO baseline by 2.9, 2.6, and 2.1 points, respectively\" appears in the paragraph about Multi-IF and is inconsistent with the preceding sentence; clarify whether these numbers refer to Multi-IF steps or to SysBench.","section":"§6.5"},{"comment":"The text says \"As shown in Table 1, across different training data volumes\" but the relevant results are in Table 3; the cross-reference should be corrected.","section":"§6.6.2"},{"comment":"There is a missing space in \"γ, βare hyper-parameters\"; also, g should be defined more precisely before first use, although the surrounding paragraph does explain it.","section":"Eq. (4)"},{"comment":"Typo: \"enrichh\" should be \"enrich\".","section":"§5.1"},{"comment":"The audit is based on 500 constraints; report the number of annotators and inter-annotator agreement, and clarify whether the 85.2% figure was computed at the constraint level or at the preference-pair level.","section":"Appendix D.3"},{"comment":"The figure caption contains raw JSON-like text that is not readable; consider moving this to an appendix or formatting it as a proper example.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal, and the core idea has merit. The main risk is the mismatch between the theoretical noise-free claim and the paper's own reversal-quality audit. I recommend major revision rather than rejection because the empirical method may still be sound once the reversal-quality issue is handled with a filtered ablation and uncertainty quantification. The benchmark-tuning concern is also serious; the editor may wish to ask the authors for code and check whether the γ sweep was performed before or after the main experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read. The core idea is simple and genuinely new: instead of sampling or filtering for perfect chosen responses, RPO reverses the constraints a response failed, turning that response into a perfect one for the reversed instruction. That turns weak samples into clean preference pairs cheaply, and the adaptive margin is a sensible addition. The experiments are broad—two benchmarks, three model families/sizes, ablations on data size and margin, plus a general-ability check—and the reported gains over DPO are plausible and internally consistent. The SysBank corpus, 30K system prompts from real GPTs, is also a useful resource.\n\nThe soft spots are real but not disqualifying. The load-bearing claim in Section 4.3 is that the chosen response is perfect and strictly dominates the rejected one, \"thus eliminating noise.\" The paper's own Table 7 undercuts that: only 85.2% of reversals are perfect, 8.4% are bad, and 2.4% are non-reversible. In the bad case from Table 10, the reversed constraint is not the logical complement, so the chosen response can violate both the original and the reversed constraint. The Limitations section already concedes that bad cases \"lead to error propagation,\" so this is not a hidden flaw, but it does mean the theoretical centerpiece is weaker than advertised. A filtered-reversal ablation—train on only perfect reversals versus all reversals—would be the obvious way to separate the benefit of the reversal trick from the benefit of data augmentation or larger margins. Table 1 also reports single runs with no variance or seed averaging, and gamma is tuned on the same benchmarks via Table 6, which nudges the reported numbers upward. No code or data was released at submission, so the exact magnitudes are hard to verify from the paper alone.\n\nNone of this refutes the empirical direction. RPO is a practical method with clear motivation, and the improvements over DPO are consistent across model sizes and datasets. The stress-test note holds up: the noise-free guarantee fails on a nontrivial fraction of pairs, so that specific claim should be softened. But the method itself deserves serious referee time, and the flaws are addressable with an ablation, error bars, and released artifacts.\n\nRecommendation: send it to peer review. Ask for code/data, seed-averaged results, and the filtered-reversal ablation.","headline":"RPO is a genuinely clever data-construction trick with real gains, but the paper oversells the noise-free guarantee that its own audit undercuts.","tokens_in":23171,"tokens_out":1368,"would_cite":true,"duration_ms":16367,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By reversing the constraints a response fails, RPO builds preference pairs whose chosen response is perfect and strictly dominates the rejected one, yielding multi-constraint instruction-following gains over DPO and, at 70B scale, results…","keywords":["instruction following","preference optimization","DPO","constraint reversal","multi-turn dialogue","system prompts","multi-constraint alignment","synthetic data"],"falsifier":"Take the constraint 'at least three keywords must appear' and its LLM-reversed 'at most three keywords must appear'; a response with exactly three keywords satisfies both, disproving perfect complementarity for that case. A systematic version is to sample reversed constraint pairs from the paper's training data and check whether any response can satisfy both the original and the reversed constraint; if the overlap rate is substantially above zero, RPO's central 'chosen is perfect and rejected is strictly worse' property fails on those pairs.","tokens_in":22104,"feed_emoji":"🔄","tokens_out":5284,"duration_ms":56025,"temperature":0.7,"pith_summary":"The paper argues that noisy preference pairs are the main obstacle to aligning language models with complex, multi-constraint instructions, and that this noise can be removed without expensive sampling of perfect responses. Its method, Reverse Preference Optimization (RPO), takes any response, finds the constraints it violates, and rewrites those constraints into their opposites, so the same response becomes a flawless answer to the new instruction. Such reversed pairs have a chosen response that is perfect and strictly dominates the rejected one, which the paper says eliminates misleading optimization signals. On SysBench and Multi-IF, RPO beats the DPO baseline by about 4.6 and 2.5 average points on Llama-3.1 8B, and a 70B RPO model surpasses GPT-4o. A reader should care because the method makes preference-data collection cheaper and more reliable for multi-turn instruction following.","feed_headline":"Reversed constraints turn failed responses into perfect training examples","feed_subtitle":"RPO beats DPO on complex instruction following; a 70B model surpasses GPT-4o.","key_machinery":"The central object is the reversed instruction $x_{S_i}$, built by negating the set of constraints that a response $y_i$ fails. The key identity is that if $S_{y_i} \\neq S_{y_j}$, then for $x_{S_i}$, $y_i$ satisfies every constraint and strictly dominates $y_j$. The training loss is the DPO objective with an adaptive margin $\\gamma_g$ that depends on the number $g$ of constraint-level differences between the two responses, sharpening the preference signal. Reversal is performed by an LLM prompt that asks for the exact logical complement, covering the entire opposite domain rather than a partial negation. The machinery also includes a fine-grained LLM evaluator that labels each constraint True or False and a role-driven self-play data generator that produces the SysBank corpus of 30K system prompts.","core_discovery":"The central claim is that when two responses differ on which constraints they satisfy, the original instruction makes at least one of them imperfect, so any preference pair built from it carries noise: the chosen response may violate constraints that the rejected response satisfies, teaching the model to ignore those constraints. RPO reverses every constraint the would-be chosen response fails, constructing a new instruction for which that response is perfect and, by construction, no worse than the other response on any constraint. The paper asserts that this eliminates noise and enlarges the chosen-rejected gap, making the DPO-style gradient direction clearer and more robust to evaluation errors. Experiments on two multi-turn benchmarks support the claim, including scaling to 70B parameters, where the RPO model surpasses GPT-4o on most metrics.","pith_inferences":["The paper leaves implicit that the 8.4% bad reversals and 2.4% non-reversible constraints found in its own audit directly weaken the claimed noise-free property; adding an automatic overlap check, which rejects reversed constraints that are also satisfied by the original response, is a testable extension that should further close that gap.","The failure cases in Appendix D.3, such as reversing 'at least three keywords' into 'at most three keywords', suggest that numeric and threshold constraints need rule-based complement handling before LLM reversal, since a response with exactly three keywords satisfies both the original and the reversed constraint.","Because RPO makes any sampled response perfect by construction, it could combine naturally with online DPO or iterative self-refinement loops where a weak model produces poor responses; this may reduce the need for strong teacher models in preference-data collection.","The SysBank corpus of 30K system prompts drawn from real GPTs, independent of the RPO training objective, is a reusable resource for future work on system-prompt following and multi-turn agent alignment."],"forward_implications":["Preference pairs can be built from any two responses that differ on at least one constraint, so RPO removes the need to sample a perfect response; the paper reports that reversal yields perfect pairs 100% of the time versus 59% for direct sampling.","Because the chosen response is perfect and dominates the rejected one, the training signal no longer teaches the model to ignore constraints that the chosen response happens to violate.","Larger chosen-rejected gaps improve performance: models trained on preference pairs with a gap of at least three constraints outperform those trained on pairs with a gap of one on both SysBench and Multi-IF.","RPO scales across model sizes (7B to 72B) and model families (Llama-3.1 and Qwen-2.5), with the 70B RPO model surpassing GPT-4o on most SysBench and Multi-IF metrics.","General capabilities on AlignBench, GSM8K, and HumanEval do not degrade under RPO training, and at 70B scale RPO can even improve some of these benchmarks relative to the base Instruct model."],"supporting_citations":[{"why":"Supplies the DPO objective that RPO modifies by reversing constraints and adding an adaptive margin.","marker":"Rafailov et al., 2024"},{"why":"Supplies the SysBench multi-turn system-prompt benchmark used for the in-domain evaluation.","marker":"Qin et al., 2024a"},{"why":"Supplies the Multi-IF multi-turn, multilingual instruction-following benchmark used for out-of-domain evaluation.","marker":"He et al., 2024b"},{"why":"Supplies the IFEval evaluation methodology and the loose/strict accuracy metrics reused for Multi-IF.","marker":"Zhou et al., 2023"},{"why":"Supplies the KTO baseline and its binary preference construction that RPO is compared against.","marker":"Ethayarajh et al., 2024"},{"why":"Supplies the persona data and character-driven generation approach that seeds the SysBank user and system profiles.","marker":"Ge et al., 2024"},{"why":"Supplies the Llama-Factory training framework used to run the SFT, DPO, KTO, and RPO experiments.","marker":"Zheng et al., 2024"}],"fun_headline_variants":["RPO flips constraints to perfect chosen responses and boost IF","Reverse constraints to fix preference noise in instruction following","Making chosen responses perfect by reversing instruction constraints","Cleaner preference pairs via constraint reversal for LLMs","RPO: Turn any response into a perfect example by reversing instructions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that reversing a constraint produces its exact logical complement: every response that violated the original must satisfy the reversed one, and the reversed one must cover the entire opposite domain with no overlap. The paper's own manual audit finds this holds perfectly for only 85.2% of reversals, so roughly one in ten preference pairs does not have a perfect, strictly dominating chosen response.","fun_headline_variants_meta":{"raw":{"variants":["RPO flips constraints to perfect chosen responses and boost IF","Reverse constraints to fix preference noise in instruction following","Making chosen responses perfect by reversing instruction constraints","Cleaner preference pairs via constraint reversal for LLMs","RPO: Turn any response into a perfect example by reversing instructions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1361,"prompt_tokens":899,"completion_tokens":462,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":384}},"tokens_in":515,"tokens_out":462,"duration_ms":5549,"temperature":1.0,"reasoning_tokens":384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:13:21.261262+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the constraint 'at least three keywords must appear' and its LLM-reversed 'at most three keywords must appear'; a response with exactly three keywords satisfies both, disproving perfect complementarity for that case. A systematic version is to sample reversed constraint pairs from the paper's training data and check whether any response can satisfy both the original and the reversed constraint; if the overlap rate is substantially above zero, RPO's central 'chosen is perfect and rejected is strictly worse' property fails on those pairs.","supporting_citations":[],"review_version":1}