{"id":"b7b29186-826a-4338-b1a6-d7b4b6ec6ead","arxiv_id":"2507.20181","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SGPO uses one shared model to refine its own responses and then optimize with DPO on those self-generated preference pairs, outperforming DPO and SPIN on AlpacaEval 2.0 and Arena-Hard without external preference labels.","lead":"This paper introduces SGPO, a training method where a single LLM both improves its own responses and then learns from those improvements, removing the need for human-written preference data. If it holds up, it offers a cheaper path to aligning instruction-following models with human intent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 'significantly outperforms DPO' rests on a DPO baseline trained with β=0.01 versus SGPO's β=0.5; this 50x mismatch in the DPO loss scale is not controlled, so the reported gains may be an artifact of an under-tuned baseline.","rationale":"The reader's CONDITIONAL verdict already identifies the need for released code, repeated runs, and independent evaluation, and I agree with that overall assessment. However, the most load-bearing technical flaw is not the improver-transfer assumption, which has some supporting evidence in Figure 8 and Table 6, but the uncontrolled 50x difference in the DPO β parameter between the baseline and the proposed method. Since the DPO loss scales linearly with β, the baseline and SGPO are trained under very different optimization regimes, making the headline 'significantly outperforms DPO' unreliable as stated. This is a concrete, fixable issue: a proper β sweep for the baseline would settle whether SGPO's advantage survives a fair comparison. The paper otherwise presents a coherent method, useful ablations, and clear writing, so the appropriate verdict remains CONDITIONAL rather than ACCEPT or REJECT.","tokens_in":18205,"tokens_out":14552,"duration_ms":175395,"concrete_test":"Retrain the DPO baseline with β=0.5 (matching SGPO) under the same learning rate 5e-7 and one epoch, and also run a small β sweep {0.01, 0.1, 0.5} for DPO; report AlpacaEval 2.0 LC/WR and Arena-Hard WR for each configuration. If the tuned DPO reaches or exceeds SGPO's scores on either benchmark, the headline comparative claim is invalidated. Additionally, rerun SGPO with β=0.01 to separate the effect of β from the method itself.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2.1 sets the DPO baseline to β=0.01, while Section 4.2.2 sets SGPO's preference-optimization step to β=0.5, with the same learning rate (5e-7) and one epoch. Because the DPO loss scales multiplicatively with β, the baseline and the proposed method are not trained under comparable optimization pressure; a 50x smaller β can produce substantially weaker alignment, and indeed the DPO baseline reaches only 9.2 LC for Qwen2.5-7B while SGPO reaches 25.2 LC. No hyperparameter sweep, multiple seeds, or error bars are reported for any method. The central claim in Section 1 that SGPO 'significantly outperforms baseline preference optimization methods including DPO' is therefore not yet supported: the comparison could be dominated by the β mismatch rather than by the proposed self-improver. This concern is distinct from the reader's transfer concern, which is at least partially addressed by Figure 8 (the trained improver is evaluated on π_θ* responses, showing roughly 70% win rate) and by Table 6 (one additional Step 2), although repeated-iteration transfer is not directly measured.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SGPO, an alignment framework in which a single language model acts as both the policy and a response improver. In Step 1, the initial policy model generates responses, and GPT-4 Turbo is prompted to produce improved responses under explicit constraints: the improved response should stay close to the policy response in edit distance and should not exceed the SFT reference in quality. The resulting targets, filtered by perplexity-based IQR, are used to train the unified improver-policy model. In Step 2, the trained model generates both a policy response and an improved response, which are used as rejected and chosen samples, respectively, in a DPO-style objective. The method is evaluated on AlpacaEval 2.0 and Arena-Hard with Qwen2.5-Base (7B), Llama3-Base (8B), and Qwen2-Base (1.5B), and it is compared against DPO, SPIN, and two SGPO variants. The central claim is that SGPO significantly outperforms DPO and prior self-improving methods without human preference annotations.","tokens_in":18397,"tokens_out":7030,"duration_ms":80480,"significance":"If the results are upheld after a controlled comparison, SGPO is a practically interesting contribution: it removes the need for human preference pairs, uses a single shared model for both generation and refinement, and provides a concrete recipe for constructing gradually improving targets. The manuscript has notable strengths: the ablation structure isolating the improver-policy integration and the improved-response supervision (Tables 4 and 5) is well designed and internally consistent; Figure 8 provides a useful transfer diagnostic showing that the trained improver refines current-policy responses better than the initial-policy improver; and Table 6 demonstrates one additional self-boosting step without retraining the improver. The main weakness is that the headline comparison against DPO is built on a hyperparameter mismatch and on single-run results with no variance estimates, so the significance claim is currently not supported. The method also relies on GPT-4 Turbo as both teacher and judge, which should be discussed as a potential benchmark-confounding factor.","major_comments":[{"comment":"The DPO baseline is trained with beta=0.01 (§4.2.1), while SGPO's preference-optimization step uses beta=0.5 (§4.2.2), with both using one epoch and a learning rate of 5e-7. Because the DPO objective in Eq. (6) scales multiplicatively with beta, the baseline is trained under 50x weaker preference-update pressure, so the large LC/WR gaps in Table 3 may reflect an under-tuned baseline rather than a property of SGPO. The Section 1 claim that SGPO 'significantly outperforms baseline preference optimization methods including DPO' is therefore not established by the reported comparison. Please add a beta sweep for DPO (including beta=0.5, with the same learning rate and epoch count) or otherwise demonstrate that the reported superiority is not dominated by the beta mismatch.","section":"§4.2.1, §4.2.2, Table 3"},{"comment":"All reported numbers appear to come from a single run, with no seeds, standard deviations, or statistical tests. The word 'significantly' in the abstract and Section 1 is therefore not statistically grounded. Please report results over multiple seeds (at least three) with means and standard deviations for the main comparisons and key ablations, or provide a paired significance test over the 805/500 benchmark prompts. This is necessary to support the consistency claims made throughout Section 4.4.","section":"Tables 3, 4, 6; §4.4.1"},{"comment":"GPT-4 Turbo is both the teacher that produces the improver training targets in Eq. (2) and the judge on AlpacaEval 2.0 (as GPT-4-Preview-1106, it is also the judge on Arena-Hard, per Table 2). This train-judge overlap means the reported improvements may partly reflect the model imitating GPT-4 Turbo's stylistic response preferences rather than aligning better with human preferences. Since no human evaluation or independent judge is provided, the practical claim that SGPO improves model quality is at risk of being confounded by judge-model alignment. Please add a discussion of this limitation and, if feasible, an evaluation with an independent LLM judge or a small human study.","section":"§3.1.1, §3.1.1.4, Table 2"},{"comment":"The claim in Section 5 that 'the self-improver does not suffer from degraded refinement performance during the policy updates' is supported by only one additional Step 2 and by Figure 8, which evaluates the improver on responses from pi_theta* rather than on responses from later policies such as pi_theta**. The discussion of 'fully iterative learning' in Section 5 is therefore extrapolated beyond the tested regime. Please either narrow the claim to the number of iterations actually measured or add a diagnostic that evaluates improver win rate on responses from the later policy in SGPO + Step 2.","section":"§4.4.3, Table 6, §5"}],"minor_comments":[{"comment":"The edit-distance constraint is attributed to reference [31], but that reference (TLCR) is about token-level continuous rewards, not edit distance; the citation appears mismatched and should be corrected or replaced.","section":"§3.1.1, reference [31]"},{"comment":"Reference [25] is labeled as the Qwen2 technical report but the arXiv ID 2412.15115 corresponds to the Qwen2.5 technical report; please correct the bibliographic data to avoid confusion with reference [30].","section":"References [25] and [30]"},{"comment":"Several SGPO responses in Tables 10-12 contain stray '2019s' artifacts (e.g., 'Here's 2019s a suggested agenda', 'the day's 2019s priorities', 'Yesterday's 2019s Achievements'); these appear to be formatting or encoding errors that should be cleaned before publication.","section":"Tables 10-12"},{"comment":"The phrase 'approximately 80% higher quality' is imprecise given the reported win rates of 87.00/86.50 for improved-versus-initial and 50.50/45.00 for improved-versus-SFT; stating the percentage-point differences directly would be clearer.","section":"§3.1.1, Table 1"},{"comment":"No code or data release is mentioned; including a reproducibility statement with training/inference code or model checkpoints would be valuable given the many nonstandard choices (prompt templates, IQR filtering, edit-distance constraints).","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope and the core idea is attractive, but the current experimental protocol does not support the headline claim. The beta mismatch between DPO and SGPO is the clearest fixable blocker, followed by the absence of any variance information. I would not require a full human study, but an independent judge would materially strengthen the evaluation. The transfer question raised by the reader is partly addressed by Figure 8 and Table 6, so I would treat it as a scope limitation rather than a fatal flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read SGPO. The genuinely new thing is that a single model switches between policy and improver via prompt control, trained on GPT-4 Turbo refinements of its own initial-policy responses under an edit-distance constraint, with the SFT reference excluded from the training input. That design is coherent, and the internal ablations (Tables 4, 5, 7) support the contribution of the unified model and the incremental-improvement signal. The paper is also honest in the Discussion about needing an external LLM to regenerate improver training data for iterative learning, which tempers the 'self-generated' branding.\n\nThe soft spots are real. The strongest one is the baseline comparison. DPO is trained with beta=0.01 while SGPO uses beta=0.5 (Section 4.2). That 50x mismatch in the DPO loss scale alone can explain the reported gap; there is no sweep over beta, no seeds, no error bars, and no code release. So the Section 1 claim that SGPO 'significantly outperforms' DPO is not yet supported. This is fixable: tune DPO's beta, report a couple of seeds, and the comparison becomes informative.\n\nThe second soft spot is the GPT-4 Turbo loop. It generates the improver's training targets and also serves as the AlpacaEval judge, and the improver's own quality is measured with the same annotator. That overlap makes the magnitude of the gains suspect until a judge-independent evaluation is added.\n\nThe transfer concern the reader raised is weaker. Figure 8 evaluates the trained improver on the updated policy's responses and gets roughly 70% win rate, and Table 6 shows one additional Step 2 works without retraining the improver. Repeated-iteration transfer is still unmeasured, but the immediate transfer worry is partially answered.\n\nI don't think the central mechanism is circular or fitted. The method is a plausible alignment recipe that removes human preference labels. But the headline numbers are not trustworthy as reported.\n\nThis is worth a serious referee. Send it to review with a request to fix the baseline tuning and add seeds and judge-independence. It needs heavy revision, not a desk reject.","headline":"Clever self-improvement loop with a serious baseline-tuning flaw: the DPO comparison uses a 50x different beta, so the headline 'significantly outperforms' claim is not yet supported.","tokens_in":19001,"tokens_out":2603,"would_cite":true,"duration_ms":29737,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single LLM refines its own answers into preference data and beats DPO alignment.","keywords":["self-improving alignment","direct preference optimization","on-policy preference data","synthetic preference data","self-improver","LLM alignment","response refinement","SFT reference"],"falsifier":"Concrete check: take the post-Step-2 policy $\\pi_{\\theta^{**}}$ and have the original improver $\\pi_{\\theta^*}$ refine 500 of its responses; if GPT-4 Turbo judges the refinements as better than the originals at a rate near 50%, or if the chosen responses in $\\mathcal{D}_P$ are not rated above the rejected ones, the central transfer claim collapses. A cleaner test: retrain the improver on refinements of the updated policy's responses and compare SGPO's benchmark scores, since unchanged scores would indicate that the one-time improver training is not the load-bearing component.","tokens_in":17921,"feed_emoji":"🔄","tokens_out":4325,"duration_ms":45730,"temperature":0.7,"pith_summary":"The paper tries to show that preference alignment can be driven entirely by a model's own improved responses, removing the need for human-written preference pairs. It proposes SGPO, in which one model first answers an instruction and then refines that answer into a clearly better version by referencing an SFT response, and the pair is used as on-policy preference data for DPO. The improver is trained once, on incremental refinements of the initial model's outputs generated by a stronger external LLM, constrained to stay close to the initial distribution. If the claim holds, alignment becomes cheaper and more online: no external preference annotation, no separate reward model, and preference data that always match the current policy's output distribution. The reported results on AlpacaEval 2.0 and Arena-Hard support this claim across three base model sizes.","feed_headline":"One model rewrites its own answers to beat DPO alignment","feed_subtitle":"SGPO turns a model's own refinements into preference data, lifting AlpacaEval 2.0 and Arena-Hard win rates without human labels.","key_machinery":"The central mechanism is the integrated improver-policy model: one shared model, invoked with different prompts, first produces an answer and then a revised version of that same answer. The revision is trained with a supervised objective on targets from an external LLM that satisfy two constraints—an edit-distance cap keeping the revision close to the original response, and a quality ceiling at the SFT reference level—so the improvement signal is incremental rather than aggressive. A perplexity-based interquartile-range filter removes targets that lie far outside the initial policy's output distribution, and DPO then treats the revision as the chosen response and the original as the rejected response, making every preference pair on-policy.","core_discovery":"SGPO claims that a single language model can act as both generator and improver: it samples a response to an instruction, then refines that same response toward an SFT reference, and the refined-versus-original pair serves as an on-policy preference pair for direct preference optimization. Trained on target refinements produced by GPT-4 Turbo that are edit-distance-constrained to stay close to the initial policy's outputs and are judged to be no better than the SFT reference, the improver learns gradual, attainable improvements rather than large leaps. The paper reports that this design significantly outperforms DPO and prior self-improving approaches on AlpacaEval 2.0 and Arena-Hard across Qwen2.5-Base (7B), Llama3-Base (8B), and Qwen2-Base (1.5B), with gains up to 16.18 points in length-controlled win rate and 17.3 points in Arena-Hard win rate, all without human preference labels.","pith_inferences":["Editorial inference: the one-time improver training is the method's likely ceiling; a direct diagnostic comparing refinement win rates on initial versus updated policy responses would reveal how quickly the improver's advantage decays as the policy shifts.","Editorial inference: SGPO still depends on a stronger external LLM and SFT references for its training targets, so 'without external preference data' should not be read as 'without external supervision'; a fully self-contained upgrade would need the model to propose its own refinement targets.","Editorial inference: the edit-distance and 'no better than SFT' constraints are verified only on sampled win rates and histograms; an ablation that removes the edit-distance cap while holding everything else fixed could isolate how much of the gain comes from incremental targets versus from on-policy pairing."],"forward_implications":["SGPO outperforms DPO on AlpacaEval 2.0 by up to 16.18 points in length-controlled win rate and on Arena-Hard by up to 17.3 points in win rate, without using any human preference annotations.","Both chosen and rejected responses come from the current policy, and replacing either side with SFT or initial-policy responses degrades performance, showing that on-policy pairing is the main driver of the gains.","The two design choices—unifying improver and policy into one model, and training on improved responses rather than only external-LLM responses—each contribute; the full SGPO beats both ablations on every benchmark.","SGPO can be iterated: applying a second Step-2 preference optimization without retraining the improver further improves most reported numbers, suggesting the loop is self-sustaining.","The method needs only a small SFT dataset for improver training and the SFT split of UltraFeedback for policy updates, avoiding preference labels and separate reward models."],"supporting_citations":[{"why":"Supplies the DPO objective in Eq. (6) that SGPO applies to its self-generated preference pairs.","marker":"[23]"},{"why":"SPIN is the main self-play baseline that SGPO compares against and claims to improve by using on-policy improved responses.","marker":"[5]"},{"why":"SynPO is the prior separate-improver framework whose divergence from the policy SGPO is designed to overcome.","marker":"[10]"},{"why":"Provides the rationale that overly large preference gaps hinder effective DPO training, motivating the incremental refinement targets.","marker":"[28]"},{"why":"GPT-4 Turbo is the external LLM that generates the target improved responses used to train the self-improver.","marker":"[21]"},{"why":"The DPO-Mix-7K SFT dataset supplies the reference responses used for building improver training targets.","marker":"[3]"},{"why":"The UltraFeedback SFT split supplies prompts and chosen responses for policy finetuning and refinement, without using its preference labels.","marker":"[8]"},{"why":"AlpacaEval 2.0 is the primary evaluation benchmark reporting length-controlled and raw win rates.","marker":"[17]"},{"why":"Arena-Hard is the second evaluation benchmark used for win-rate comparison.","marker":"[16]"},{"why":"The IQR method is the statistical filter used to remove outlier responses from the improver training data.","marker":"[27]"}],"fun_headline_variants":["One model refines its own responses to beat DPO","Self-improver trains DPO without human labels","Rewriting own answers: SGPO outperforms DPO","Self-improver: better alignment without human labels","LLM self-refinement beats DPO on AlpacaEval and Arena-Hard"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The improver, trained once on refinements of the initial policy's responses, remains effective when asked to refine the responses of the updated policy during preference optimization.","fun_headline_variants_meta":{"raw":{"variants":["One model refines its own responses to beat DPO","Self-improver trains DPO without human labels","Rewriting own answers: SGPO outperforms DPO","Self-improver: better alignment without human labels","LLM self-refinement beats DPO on AlpacaEval and Arena-Hard"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001279,"raw_usage":{"total_tokens":5228,"prompt_tokens":945,"completion_tokens":4283,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":4199}},"tokens_in":561,"tokens_out":4283,"duration_ms":33318,"temperature":1.0,"reasoning_tokens":4199,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:41:19.663619+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Concrete check: take the post-Step-2 policy $\\pi_{\\theta^{**}}$ and have the original improver $\\pi_{\\theta^*}$ refine 500 of its responses; if GPT-4 Turbo judges the refinements as better than the originals at a rate near 50%, or if the chosen responses in $\\mathcal{D}_P$ are not rated above the rejected ones, the central transfer claim collapses. A cleaner test: retrain the improver on refinements of the updated policy's responses and compare SGPO's benchmark scores, since unchanged scores would indicate that the one-time improver training is not the load-bearing component.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the rationale that overly large preference gaps hinder effective DPO training, motivating the incremental refinement targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The DPO-Mix-7K SFT dataset supplies the reference responses used for building improver training targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The IQR method is the statistical filter used to remove outlier responses from the improver training data."}],"review_version":1}