{"id":"0e2fdec3-4202-4806-8081-12f688c2bf25","arxiv_id":"2504.14177","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DAR aligns LLMs through advantage-weighted supervised fine-tuning on online AI scalar rewards with dual KL regularization, and reports win-rate gains over online RLHF and online preference methods.","lead":"This paper introduces DAR, an alignment method that improves a language model by re-weighting its own sampled responses with scores from an AI judge, then running plain supervised fine-tuning. It reports that scalar AI scores agree with human preference more often than pairwise AI choices do, and that DAR beats online RLHF and online preference baselines in AI-judged win-rate tests.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Harmlessness reward direction is internally inconsistent: Table 8 defines higher score as more harmful, while Algorithm 1 maximizes the score, so DAR may optimize harmfulness on its third headline task.","rationale":"The reader's conditional verdict already identifies the reward-sign risk, and I agree that it is the load-bearing concern because it is the one issue that, if real, reverses a headline result rather than merely weakening it. The derivation (Theorem 4.1, Eq. 7) is mathematically sound under its stated assumptions, and the TL;DR and Helpfulness comparisons are noisy but suggestive. However, the Harmlessness setup couples a training prompt that grades harmfulness (10 = harmful) with an objective that maximizes the label and an evaluation prompt that selects the most harmful response. No code or data are released, so the only way to know whether an inversion exists is to inspect training logs or rerun the experiment. I therefore keep the reader's CONDITIONAL verdict: the proposed check should settle whether this is a documentation error or a substantive failure of the Harmlessness result.","tokens_in":26440,"tokens_out":8555,"duration_ms":81608,"concrete_test":"Re-run Algorithm 1 on the Harmlessness setup with the Table 8 prompt (Qwen2-72B-Instruct labels, Qwen2-7B policy, same hyperparameters, 1k test subset) and record the mean extracted harmfulness score of the policy's samples at each online iteration. If the mean score increases over training, DAR is maximizing harmfulness and the Harmlessness claim is invalid as written. If it decreases, the code must contain an undocumented sign inversion (e.g., reward = 10 - SCORE); that transformation should be added to the paper and verified on the saved checkpoint by checking that DAR responses are judged less harmful than reference responses under a correctly worded preference prompt. This single run adjudicates the contradiction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DAR's headline results include an 85.84% Harmlessness win rate (Table 2). The harmlessness reward prompt (Table 8) instructs the annotator to return a SCORE from 1 to 10 rating harmfulness, with 1 = completely harmless and 10 = extremely harmful. Algorithm 1 computes A(x,y) = r(x,y) - (1/K) * sum_i r(x,y_i), normalizes it, and upweights responses by exp(A_norm/(alpha+beta)) in Eq. 7. No sign inversion or '10 - SCORE' transformation is described in Section 4, Algorithm 1, or Appendix D. Under the printed definitions, DAR maximizes harmfulness. The evaluation prompt for this task (Table 11) also asks GPT-4-Turbo to choose the most harmful response, so the reported 'reference win rate' would count DAR responses as winning when they are more harmful. This is not merely an external-validity concern about AI reward bias; the sign of the training signal is opposite to the stated alignment goal. Either the implementation applies an undocumented negation, or the Harmlessness result is the opposite of the claimed alignment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Direct Advantage Regression (DAR), an online alignment algorithm for LLMs that uses scalar AI rewards to reweight a supervised fine-tuning objective. The update rule in Eq. (7) is derived from a dual-KL-constrained advantage maximization objective, with a closed-form optimal policy obtained via Gibbs' inequality, and is implemented as weighted SFT with Monte-Carlo baseline estimation, advantage normalization, and weight clipping. Experiments compare AI reward labels against AI preference labels for agreement with human preferences, and report reference win rates on TL;DR, Helpfulness, and Harmlessness, plus MT-Bench scores after fine-tuning on HelpSteer2 with a pretrained reward model. The paper claims that DAR outperforms online DPO/IPO/SLiC with AI preference and online PPO/RLOO with AI reward while avoiding reward over-optimization.","tokens_in":26504,"tokens_out":5472,"duration_ms":51919,"significance":"If the empirical claims hold, DAR is a simple, RL-free alternative to PPO/RLOO with a clean theoretical derivation: the proof of Theorem 4.1 in Appendix B is correct, and Eq. (7) genuinely follows from the stated objective rather than being fitted post hoc. The head-to-head comparison of AI reward versus AI preference as supervision signals (Table 1) is a useful contribution. However, the reported Harmlessness result appears to rely on an inverted reward direction, the main win-rate comparison conflates algorithm and supervision type, and the PPO baseline is admitted to be suboptimal. These issues place a substantial burden on the experimental claims, although the central algorithm idea remains defensible for the other two tasks.","major_comments":[{"comment":"The harmlessness reward prompt in Table 8 instructs the annotator to return a SCORE from 1 to 10 where 10 means 'extremely harmful,' while Algorithm 1 and Eq. (7) maximize the advantage computed directly from that score with no sign inversion. The evaluation prompt in Table 11 also instructs GPT-4-Turbo to select the 'most harmful' response. Under these printed definitions, the reported 85.84% Harmlessness win rate in Table 2 measures harmfulness, not harmlessness, and the training signal would actively maximize harmful behavior, contradicting the stated alignment goal. The high human-AI agreement for Harmlessness in Table 1 is likewise inconsistent with this orientation, since the human-preferred response in the Harmlessness dataset is the less harmful one. Please state explicitly whether a sign inversion (such as using 10 - SCORE or negating the advantage) was applied in the implementation; if not, the Harmlessness experiments need to be redone with the correct reward direction.","section":"Algorithm 1, Table 8, Table 11, Section 6.2"},{"comment":"The main win-rate comparison between DAR with AI reward and DPO/IPO/SLiC with AI preference changes both the algorithm and the supervision type simultaneously, so the conclusion that 'AI reward is a better form of AI supervision' is not actually supported by the win-rate results. Table 1 does isolate the supervision signal through human-AI agreement, but the headline performance comparison in Table 2 cannot separate algorithmic advantage from supervision advantage. Please add an ablation that keeps either the algorithm or the supervision signal fixed, or rephrase the claims to describe the joint combination rather than attributing the gain to AI reward alone.","section":"Section 6.2, Table 2"},{"comment":"The PPO baseline is described as 'suboptimal' by the authors and is not run for the HelpSteer2 experiment in Table 3. The claim that DAR outperforms online RLHF therefore rests on a PPO implementation that the authors themselves judge to be inadequate, including the absence of a separate value model. Please either tune PPO more carefully (for example, with a separate value model and moving-average reference) or restrict the claim to comparisons against RLOO and SFT+Best-of-N.","section":"Appendix D.3.2, Table 2, Table 3"}],"minor_comments":[{"comment":"There is a typo in 'proper on-policy learnring approach'; it should be 'learning'.","section":"Section 1"},{"comment":"'This approach further simplies the computational complexity' should be 'simplifies'.","section":"Section 4.3"},{"comment":"The claim that the positional-bias difference is 'statistically significant' is not accompanied by a test statistic or p-value; please provide the test used.","section":"Appendix E.2"},{"comment":"Human-AI agreement numbers are reported without confidence intervals or significance tests; given the small subset size (1k) and the relatively small gaps between AI reward and AI preference, some uncertainty quantification would strengthen the head-to-head claim.","section":"Table 1"},{"comment":"The evaluation uses a 1,000-sample random subset for win rates, but no standard errors adjusted for clustering by prompt are reported; the provided standard deviations over seeds are helpful but do not capture within-prompt correlation.","section":"Appendix D.4"}],"recommendation":"major_revision","confidential_remarks":"Major Comment 1 is the most serious issue: if the implementation did not apply a sign inversion, the entire Harmlessness section, including Table 1's harmlessness columns and Table 2's 85.84% claim, is invalid for the stated alignment goal. I would ask the authors to provide the exact reward extraction code or training logs demonstrating the orientation of the harmlessness reward. If they cannot, the paper should be rejected or the harmlessness experiments removed. The other two major comments are addressable with additional experiments or relaxed claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here is my read of arXiv:2504.14177 (DAR). The core idea is sound and potentially useful: DAR is advantage-weighted regression plus a reference-KL term, solved in closed form, and the paper includes a genuinely useful systematic comparison of scalar AI reward versus pairwise AI preference across nine annotator models and three datasets. That comparison is the real contribution — the finding that reward labels achieve higher human-AI agreement than preference labels is citable and will influence how RLAIF pipelines are built. The math is clean: Theorem 4.1 is the standard closed-form KL-constrained policy solution, and the derivation in Appendix B follows by Gibbs' inequality. The ablations on alpha/beta ratio, weight clip, and MC sampling size are thoughtful.\n\nBut there is a serious sign problem in the harmlessness task, and I do not think it is a minor external-validity concern. The reward prompt in Table 8 instructs the annotator to give a SCORE from 1 to 10 where 1 is completely harmless and 10 is extremely harmful. Algorithm 1 computes advantage as reward minus baseline and upweights responses by exp(A/(alpha+beta)). No sign inversion or 10-minus-score transformation is described anywhere. The evaluation prompt (Table 11) also asks the judge to choose the most harmful response. So the reported 85.84% harmlessness win rate appears to mean DAR trained to become more harmful and won by being more harmful. Either the implementation negated the score outside the printed algorithm, in which case the paper must say so, or the headline harmlessness result is the opposite of what is claimed. This needs to be checked before anything else.\n\nOther soft spots are more ordinary. The PPO baseline is confessed to be suboptimal (Appendix D.3.2), so \"beats online RLHF\" is a weak claim; RLOO and SFT+Best-of-N are the meaningful comparisons, and DAR's margin over SFT+Best-of-N is negligible on TL;DR and modest on Harmlessness. Hyperparameters are tuned on the same 1k test sets used to report final numbers. Human-AI agreement in Table 1 has no significance tests. No code or data are released. And Section 6.3 says 8.572 while Table 3 reports 8.526 — a minor inconsistency.\n\nIf the harmlessness sign is corrected, the remaining paper is a solid empirical study of a simple, RL-free alignment method with a clean derivation. The paper deserves a serious refereeing process, but the referee needs to push on the sign issue and the evaluation split. My recommendation: send to peer review with major revision, requiring code/data release, human evaluation, a fixed PPO baseline or a more careful claim, and a corrected harmlessness reward direction.","headline":"Useful reward-vs-preference comparison and a clean derivation, but the harmlessness result appears to train toward harmfulness because the reward prompt is never inverted.","tokens_in":27229,"tokens_out":2902,"would_cite":true,"duration_ms":23750,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Direct Advantage Regression replaces online RLHF with one weighted supervised learning objective.","keywords":["direct advantage regression","online AI feedback","AI reward","weighted supervised fine-tuning","dual KL regularization","direct alignment from preferences","RLHF","LLM alignment"],"falsifier":"Take a DAR-tuned model and the same 1,000-prompt test subsets, but have human raters judge the model's outputs against the reference instead of GPT-4-Turbo; a large drop from the reported 98.27/92.67/85.84 percent win rates would show that the AI-reward supervision, not human preference, drove the gains.","tokens_in":26039,"feed_emoji":"🤖","tokens_out":10250,"duration_ms":78793,"temperature":0.7,"pith_summary":"Direct Advantage Regression (DAR) is an alignment algorithm that replaces the two-stage RLHF pipeline—reward collection followed by policy optimization with a policy-gradient method—with a single weighted supervised fine-tuning step. Each on-policy response is assigned a weight equal to the product of an advantage-based reward term and a policy-ratio discount that jointly enforces two KL constraints, one against a fixed reference policy and one against the current sampling policy. The paper's central empirical claim is that this online, RL-free update outperforms both online direct-preference methods (DPO, IPO, SLiC with AI preference) and online RL methods (PPO, RLOO with AI reward), reaching 98.27 percent reference win rate on TL;DR, 92.67 percent on Helpfulness, and 85.84 percent on Harmlessness as judged by GPT-4-Turbo, while keeping response lengths bounded. A supporting claim is that scalar AI reward is higher-fidelity supervision than binary AI preference, showing consistently higher human-AI agreement across several annotator models. If correct, fine-grained AI supervision can replace both human labels and RL machinery in online alignment.","feed_headline":"Weighted regression loss beats online RL and DPO for LLM alignment","feed_subtitle":"Uses scalar AI reward with dual KL control, hitting 98.27 percent on TL;DR without any RL machinery.","key_machinery":"The engine is the DAR regression weight in Equation (7): each on-policy response is up-weighted by the advantage exponential $\\exp(A/(\\alpha+\\beta))$ and discounted by the ratio $(\\pi_{\\mathrm{ref}}(y|x)/\\pi_t(y|x))^{\\alpha/(\\alpha+\\beta)}$, and the product becomes the weight for ordinary next-token maximum-likelihood training on $\\log\\pi_\\theta(y|x)$. The two KL coefficients $\\alpha$ and $\\beta$ play distinct roles: their sum $\\alpha+\\beta$ scales the temperature of the advantage weight, while the ratio $\\alpha/(\\alpha+\\beta)$ sets how strongly the update stays near the fixed reference policy rather than the moving sampling policy. This construction converts the dual-constrained reward objective into a supervised loss, avoiding value networks and policy-gradient estimators.","core_discovery":"The central claim is that LLM alignment can be solved as iterative weighted supervised regression on replies sampled from the current policy, with weights encoding both the advantage of a reply and a dual KL regularization. For the objective $\\max_\\pi \\mathbb{E}_{x,y\\sim\\pi}[A(x,y)] - \\alpha D_{\\mathrm{KL}}(\\pi\\|\\pi_{\\mathrm{ref}}) - \\beta D_{\\mathrm{KL}}(\\pi\\|\\pi_t)$, the paper proves (Theorem 4.1) that the optimum is $\\pi^*(y|x) \\propto \\pi_{\\mathrm{ref}}(y|x)^{\\alpha/(\\alpha+\\beta)} \\pi_t(y|x)^{\\beta/(\\alpha+\\beta)} \\exp(A(x,y)/(\\alpha+\\beta))$, and projecting onto a parametric policy by KL minimization yields the DAR update $\\pi_{t+1} = \\arg\\max_\\theta \\mathbb{E}_{(x,y)\\sim D_{\\pi_t}}[(\\pi_{\\mathrm{ref}}(y|x)/\\pi_t(y|x))^{\\alpha/(\\alpha+\\beta)} \\exp(A(x,y)/(\\alpha+\\beta)) \\log\\pi_\\theta(y|x)]$. With Monte-Carlo advantage estimation, advantage normalization, and weight clipping, the paper reports that this update attains the highest reference win rate among all compared methods on TL;DR, Helpfulness, and Harmlessness, and avoids the length inflation seen in the SFT and online-preference baselines.","pith_inferences":["Because DAR's weights are built from the scalar $r(x,y)$ directly, the same objective could be applied to any continuously scored feedback—graded rubrics, safety scores, or human Likert ratings—not just LLM-generated rewards; testing this would show whether the dual-KL weighting, rather than the AI annotator, is what drives DAR's gains.","The paper's Table 1 agreement figures (60–80 percent) imply that DAR's measured win rates may overstate progress toward human preference; a natural next experiment is the same three test sets judged by human raters rather than GPT-4-Turbo.","The dual-KL trick could be retrofitted to offline preference algorithms: applying the $\\pi_{\\mathrm{ref}}/\\pi_t$ ratio weight inside an offline DPO or IPO loss would give such methods a monotone-improvement guarantee analogous to DAR's.","The paper's bias analysis for pairwise preference (second-position bias) suggests a similar audit is needed for the single-response reward prompts; checking whether reward scores drift with response length or formatting would expose whether DAR amplifies annotator artifacts."],"forward_implications":["AI reward is a more informative supervision signal than AI preference: across Qwen2, Llama-3, Mistral, Gemma-2 and GPT-4 annotators, reward-based labels agree with human judgments more often than preference-based labels on all three datasets.","An RL-free learner can match or beat online RLHF: DAR's weighted SFT reaches higher GPT-4-Turbo reference win rates than both online PPO and RLOO while using the same scalar reward.","Dual KL regularization checks reward over-optimization: DAR keeps response lengths near reference levels, unlike SFT+best-of-n on TL;DR and online DAP methods on Helpfulness.","The same update works with a human-trained reward model: on the HelpSteer2 setting DAR obtains the best MT-Bench score among the trained methods compared.","Methods learning from AI reward, DAR included, need 3-5 times fewer online annotations than preference-based online DAP methods to reach their plateau.","If the dual-KL weighting is what stabilizes DAR, the same trick can be grafted onto offline preference losses such as DPO or IPO, giving them a similar monotone-improvement guarantee without changing their data requirements."],"supporting_citations":[{"why":"Supplies the Advantage-Weighted Regression (AWR) framework that DAR extends with a reference-policy KL term.","marker":"Peng et al., 2019"},{"why":"Establishes the direct-RLAIF setting of learning from LLM-generated reward labels, which DAR adopts online.","marker":"Lee et al., 2024"},{"why":"Defines OAIF, the online AI preference baseline that DAR is compared against.","marker":"Guo et al., 2024"},{"why":"Provides DPO, one of the online direct-preference baselines, and the broader direct alignment formulation.","marker":"Rafailov et al., 2024"},{"why":"Provides RLOO, an online RLHF baseline using Monte-Carlo advantage estimation without a value model.","marker":"Ahmadian et al., 2024"},{"why":"Provides PPO, the online RLHF baseline, and motivates the current-policy KL trust region used in DAR.","marker":"Schulman et al., 2017b"},{"why":"Supplies the TL;DR summarization dataset and frames reward over-optimization as a key failure mode.","marker":"Stiennon et al., 2022"},{"why":"Provides the Helpfulness and Harmlessness datasets used for the alignment and evaluation experiments.","marker":"Bai et al., 2022a"},{"why":"Supplies HelpSteer2, the dataset and pre-trained reward model used in the reward-model alignment setting.","marker":"Wang et al., 2024b"},{"why":"Provides the MT-Bench benchmark used to evaluate the reward-model fine-tuning setting.","marker":"Zheng et al., 2023"}],"fun_headline_variants":["DAR: AI reward regression beats RLHF and DPO","Weighted regression with AI reward tops online RLHF and DPO","RL-free: Advantage-weighted AI reward improves alignment","Scalar AI reward outperforms preference in LLM alignment","DAR: Simpler, RL-free, and better than online RLHF"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the scalar score an LLM annotator gives a single answer is a faithful, bias-free proxy for human preference; if that score is skewed by length, style, position, or safety wording, DAR will simply amplify the skew.","fun_headline_variants_meta":{"raw":{"variants":["DAR: AI reward regression beats RLHF and DPO","Weighted regression with AI reward tops online RLHF and DPO","RL-free: Advantage-weighted AI reward improves alignment","Scalar AI reward outperforms preference in LLM alignment","DAR: Simpler, RL-free, and better than online RLHF"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2477,"prompt_tokens":989,"completion_tokens":1488,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":1404}},"tokens_in":605,"tokens_out":1488,"duration_ms":12366,"temperature":1.0,"reasoning_tokens":1404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:55:41.580270+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a DAR-tuned model and the same 1,000-prompt test subsets, but have human raters judge the model's outputs against the reference instead of GPT-4-Turbo; a large drop from the reported 98.27/92.67/85.84 percent win rates would show that the AI-reward supervision, not human preference, drove the gains.","supporting_citations":[{"cited_title":"Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024","cited_arxiv_id":null,"evidence_quote":"Provides RLOO, an online RLHF baseline using Monte-Carlo advantage estimation without a value model."}],"review_version":1}