{"id":"a60b20c8-2d01-423e-8794-7db808084d4b","arxiv_id":"2506.21495","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Semi-online DPO, which syncs the generation model every few update steps, performs nearly as well as fully online DPO and GRPO, while strongly beating offline DPO.","lead":"This paper compares offline, semi-online, and fully online reinforcement learning for finetuning Llama-3.1-8B with DPO and GRPO on math and instruction following. It finds that semi-online DPO nearly matches fully online training, and that mixing verifiable and non-verifiable rewards improves both tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Checkpoint selection by length-normalized Athene-RM-8B scores may inflate the online-vs-offline gap, but the gap is corroborated by verifiable math results.","rationale":"The reader's weakest assumption is precisely the checkpoint-selection concern I identify: use of length-normalized Athene-RM-8B rewards for selection introduces a channel through which reward-model bias could inflate the apparent advantage of online and semi-online methods on non-verifiable tasks. I agree this is the most load-bearing soft spot. However, I do not believe it warrants changing the conditional-accept verdict. The verifiable-task results (Table 1) use a clean accuracy-based selection rule and show the same qualitative pattern: online and semi-online DPO outperform offline DPO by roughly 4-5 points on Math500 and 3 points on NuminaMath, with semi-online matching online. GRPO also matches online DPO on verifiable tasks. The non-verifiable results are further corroborated by two independent benchmark judges (GPT-4-1106 and GPT-4o) with length-controlled AlpacaEval, reducing the risk that the gap is purely length-driven. The abstract's wording about GRPO on AlpacaEval is somewhat strong given Table 2 (GRPO is 5-8 points below online DPO on AlpacaEval), but the conclusion as stated ('fully online DPO and GRPO perform comparably') is approximately true overall and exactly true on verifiable tasks. The requested additional evidence — reporting results under a neutral selection rule or releasing checkpoints — is a reasonable condition for full acceptance, but the core claim is already supported enough for conditional acceptance. Thus I recommend UNCHANGED relative to the reader's CONDITIONAL verdict, while affirming the reader's concern as the right one to test.","tokens_in":17181,"tokens_out":1966,"duration_ms":17700,"concrete_test":"Re-run the WildChat non-verifiable experiments and evaluate, for each training method and at multiple fixed step counts (e.g., the same 10 checkpoints per method), the AlpacaEval LC winrate and ArenaHard score under the paper's evaluation protocol. If the mean or best-over-checkpoints performance of online/semi-online DPO remains above offline DPO by a margin comparable to Table 2 when checkpoints are selected by fixed step or by an independent heldout judge, the selection-bias concern is resolved. Alternatively, report checkpoints in Table 2 without any reward-model-based selection (e.g., final checkpoints). The minimum requirement to settle the concern: recompute Table 2 with final-checkpoint evaluation (no reward-model selection) and compare.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that semi-online and online DPO substantially outperform offline DPO rests heavily on Table 2 for non-verifiable tasks. The paper's checkpoint selection rule (Section 3.1) picks the checkpoint with the highest length-normalized Athene-RM-8B reward on a heldout set, and the same Athene-RM-8B reward model provides the training rewards. The paper itself acknowledges (Section 4.2) that Athene-RM-8B has a length bias and that response lengths generally increase over training in non-verifiable settings. Thus selection may favor checkpoints that exploit the reward model's length bias. The reported evaluation metrics (AlpacaEval LC and ArenaHard) are length-controlled and relatively robust, which mitigates this concern, but the comparison is not fully protected: a checkpoint selected for high reward-model scores could also have higher quality on benchmark judges, or could be specifically overfit to Athene-RM-8B's notion of quality rather than to human preference. For verifiable math (Table 1), checkpoint selection is based on heldout math accuracy with no length normalization, and the same offline-vs-online gap appears; that task provides independent support. A key missing check is whether the ranking of methods under a neutral selection rule (e.g., fixed training budget, or selection by an independent judge like GPT-4 on a heldout set) preserves the conclusion that semi-online DPO matches online DPO and both beat offline DPO. Without that, the non-verifiable headline could be an artifact of reward-model exploitation during selection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically compares offline, semi-online, and fully online post-training algorithms (DPO and GRPO) for LLMs on verifiable math and non-verifiable instruction-following tasks. Using Llama-3.1-8B-Instruct as the seed model, it reports that semi-online DPO with periodic generator synchronization (s=10 or 100 for math, s=5 or 10 for instruction following) performs comparably to fully online DPO (s=1) and GRPO, and that all of these substantially outperform offline DPO. It also studies reference-model synchronization, entropy collapse, and joint training with verifiable and non-verifiable rewards, reporting that multi-task training yields gains on non-verifiable benchmarks while roughly preserving verifiable performance.","tokens_in":17612,"tokens_out":6742,"duration_ms":71214,"significance":"If the central result holds, the paper provides a practically valuable finding: much of the benefit of fully online RL for LLM alignment can be captured by semi-online DPO at lower inference cost. The verifiable-task results are independently corroborated by checkpoint selection based on heldout math accuracy rather than reward-model scores, and the use of external benchmarks (Math500, NuminaMath, AMC23, AlpacaEval, ArenaHard) with reported standard errors is a strength. The paper also contains useful analysis of length collapse, reference-model sync, and entropy degeneration. However, the non-verifiable headline rests on a checkpoint-selection rule that shares its reward model with the training signal, and the verifiable comparison confounds online rollouts with reference-model synchronization, so the central claim is not yet fully established. The absence of code or data release and the use of a single base model further limit immediate reproducibility.","major_comments":[{"comment":"Checkpoint selection for non-verifiable tasks uses the highest length-normalized Athene-RM-8B reward on a heldout set, and the same Athene-RM-8B model supplies the training rewards. Section 4.2 acknowledges that Athene-RM-8B has a length bias and that response lengths increase in online/semi-online training. Although AlpacaEval LC and ArenaHard control for length, they do not control for overfitting to Athene-RM-8B's specific notion of quality. The claim that online and semi-online DPO substantially outperform offline DPO on non-verifiable tasks is load-bearing for the paper, so the selection rule could be inflating the gap. Please report results under a neutral selection rule (e.g., fixed training budget, or checkpoint selection by an independent judge such as GPT-4 on a heldout set), or at minimum report the best checkpoint per external benchmark. Without such a check, the non-verifiable headline remains conditional on the reward model used for both training and selection.","section":"Section 3.1, Section 4.2, Table 2"},{"comment":"In the verifiable-task experiments, all online and semi-online DPO runs use reference-model synchronization, while offline DPO does not; Table 4 shows this difference, and Figure 2 shows that online DPO without reference sync suffers length collapse and lower validation reward. This confounds the comparison: the large verifiable-task advantage of online/semi-online DPO over offline DPO could be driven by reference-model sync rather than by fresh on-policy rollouts. The manuscript does not report benchmark scores for online DPO without reference sync on Math500/NuminaMath/AMC23. Please add those benchmark numbers, and ideally for semi-online DPO without sync as well, so the independent contributions of on-policy data and reference-model stabilization can be separated.","section":"Section 4.2, Table 4, Figure 2"},{"comment":"The abstract states that joint training on verifiable and non-verifiable rewards yields 'improved performance across both task types,' but Table 3 shows that the NM+WC model has Math500 accuracy 57.3 versus 58.7 for NM-only training and AMC23 accuracy 31.7 versus 32.9 for NM-only training, i.e., numerically lower verifiable performance. The body text in Section 4.1 more carefully says the result is 'comparable' to individual-task training with 'slight improvements in the non-verifiable evaluations.' Please align the abstract's claim with the data, or clarify that the improvement is on non-verifiable benchmarks only.","section":"Abstract, Section 4.1, Table 3"}],"minor_comments":[{"comment":"No code, training data, or model checkpoints are released; for an empirical methods paper with comparisons of this sort, a public repository would substantially improve reproducibility and should be provided.","section":"Throughout"},{"comment":"The reported standard errors and confidence intervals appear to reflect evaluation-sample noise (e.g., N=50 generated solutions per problem) rather than variance across independent training runs; please state this explicitly so readers do not interpret the intervals as training-seed variance.","section":"Tables 1-2"},{"comment":"The statement that 'we find s = 100 to be too unstable with our non-verifiable hyperparameters' is not accompanied by any plot, table, or numerical evidence; please provide the supporting results or label this as an informal observation.","section":"Section 4.1"},{"comment":"There is a typo in the second paragraph: 'it it was only relatively recently' should read 'it was only relatively recently.'","section":"Introduction"},{"comment":"The text says 'GPRO' where 'GRPO' is intended; please correct this typo in the sentence 'in both DPO and GPRO experiments' and in the Figure 3 caption.","section":"Section 4.2 and Figure 3"},{"comment":"The final paragraph says 'transitioning from offline to offline learning'; this should read 'from offline to online learning.'","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is plausible and the math task provides an independent anchor, but the non-verifiable checkpoint-selection issue and the reference-sync confound are both directly relevant to the headline result. I would ask the authors to run the neutral-selection robustness check and the no-reference-sync benchmark before acceptance, rather than treating this as a purely editorial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical study. It systematically varies the sync interval s for DPO on verifiable math and non-verifiable instruction following, adds GRPO baselines, and finds that semi-online DPO (s=10 or 100) lands close to fully online DPO, with both clearly beating offline DPO. The math results are the strongest evidence: there, checkpoint selection is by heldout accuracy, not by the reward model, and the gap appears consistently across Math500, NuminaMath, and AMC23. The non-verifiable results are more vulnerable. Checkpoints are chosen by length-normalized Athene-RM-8B reward on a heldout set, and the same reward model supplies the training rewards; the paper itself notes this reward model has a length bias. So part of the Table 2 gap could come from selection favoring checkpoints that exploit the reward model. The evaluation judges are length-controlled, which mitigates the worry but does not remove it. I would like to see a fixed training budget or selection by an independent judge before fully trusting the non-verifiable headline. The joint verifiable+non-verifiable training results are interesting but only one recipe is tested, so I read those as preliminary.\n\nWhat is actually new: the sync-interval sweep on verifiable math, the GRPO comparison, and the mixed-reward training. The paper builds honestly on Xu et al. 2024b and related work, and the negative results on NLL terms, GroupDPO, and entropy regularization are reported without overclaiming. Standard errors are given and the compute is substantial. The practical observations about reference model sync and Adam epsilon will be useful to people running these methods.\n\nSoft spots, in proportion: one base model only; no code or data released; and the abstract slightly overstates the GRPO/DPO similarity on AlpacaEval, where online DPO leads by several points under both judges. The multi-task claim is modest in effect size.\n\nOverall, the central offline-vs-online result is real and practically important. This paper deserves serious refereeing. I would recommend conditional accept: require a neutral checkpoint-selection check (or a clear statement of this limitation), and ideally code or data release. A practitioner deciding how much online compute to budget will get real value from this work.","headline":"Useful, well-run empirical paper: the offline-vs-online gap is real, and the semi-online finding is worth taking seriously, but the non-verifiable checkpoint selection deserves a robustness check before the headline is fully trusted.","tokens_in":18057,"tokens_out":1733,"would_cite":true,"duration_ms":20031,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the gap between offline and online reinforcement learning for LLMs is mostly set by how often the response generator is synchronized with the trainer, and that semi-online DPO—syncing every 5 to 100 steps—recovers…","keywords":["offline-to-online reinforcement learning","direct preference optimization","group relative policy optimization","semi-online training","LLM post-training","verifiable rewards","reward models","instruction following"],"falsifier":"Re-run the offline, semi-online, and online DPO recipes but select checkpoints by held-out benchmark scores (math accuracy and instruction-following winrates) instead of the length-normalized reward score; if offline DPO then matches or beats the online variants, the reported gap is a checkpoint-selection artifact rather than a training-regime effect.","tokens_in":16979,"feed_emoji":"🤖","tokens_out":10467,"duration_ms":94187,"temperature":0.7,"pith_summary":"The paper sets out to determine how much of the benefit of online reinforcement learning for large language models survives when training is moved offline or semi-online, and whether the choice of objective (DPO versus GRPO) matters. Its answer, across both verifiable math and non-verifiable instruction following, is that the online/offline distinction matters much more than the objective: fully online DPO and GRPO perform comparably and both strongly beat offline DPO, while semi-online DPO—in which the generator model is synchronized with the trainer only every 5 to 100 steps—nearly closes the gap. A practitioner who wants most of the online gain without paying for synchronous rollouts at every step can therefore use semi-online DPO. The paper also claims that training on verifiable and non-verifiable rewards jointly improves both task types compared with training on either alone.","feed_headline":"Skip per-step syncs and keep most of online RL's gain","feed_subtitle":"Syncing the response generator every 5–100 steps captures most of the gains of online DPO and GRPO on math and instruction following.","key_machinery":"The paper's central object is the synchronization step $s$: the number of parameter updates the trainer performs before copying its weights to the generator that produces rollouts. $s=\\infty$ is offline DPO (pre-generated responses), $s=1$ is fully online training (generator always current), and intermediate values are semi-online. Varying $s$ isolates how much of the gain of online RL comes from training on the current policy's own responses; the paper's result is that periodic synchronization, not per-step synchronization, carries almost all of the benefit. A second mechanism is the reference model in DPO: syncing the reference model along with the generator prevents the response-length collapse seen in verifiable-task runs, and the paper treats this as a stability requirement rather than a performance booster.","core_discovery":"On the paper's own terms, the central discovery is that the synchronization interval between the model generating responses and the model being trained—not the choice between DPO and GRPO—is the main driver of post-training quality. With an 8-billion-parameter instruct-tuned seed, offline DPO improves over the seed but is left far behind by online DPO, online GRPO, and semi-online DPO; semi-online DPO with sync intervals of 10 or 100 steps on math and 5 or 10 steps on instruction following lands close to fully online DPO. The paper reports similar convergence and performance for the online and semi-online variants, finds no benefit from larger GRPO groups beyond eight responses, and shows that combining verifiable rule-based rewards with reward-model scores in a single online DPO run improves non-verifiable benchmarks relative to training on non-verifiable rewards alone.","pith_inferences":["If the reward score used for checkpoint selection is biased toward length, the absolute sizes of the reported gaps between online/semi-online and offline DPO may shrink under a length-agnostic judge; the qualitative ranking could survive, but the semi-online bridge might be narrower than it appears.","A natural extension is to treat $s$ as a tunable compute-quality knob: for a fixed inference budget, the optimal synchronization interval may depend on rollout cost and dataset epoch size, and the paper's setup makes that measurement straightforward.","The entropy-collapse observation suggests that stability, not objective choice, will be the binding constraint as online methods scale; testing adaptive synchronization or entropy regularization as a function of rollout entropy is a direct follow-up.","The multi-task result hints that mixing a verifiable signal into general instruction-following data may substitute for part of the reward model's role; a testable consequence is that the same benefit should appear with other verifiers such as code execution."],"forward_implications":["Semi-online DPO with $s=10$ or $s=100$ on math and $s=5$ or $s=10$ on instruction following can substitute for fully online DPO, retaining most of the benchmark gain while allowing rollouts to be generated in parallel between syncs.","Online DPO is a practical alternative to GRPO for verifiable math: it needs only one preference pair per prompt per step, matches GRPO's accuracy, and avoids the need for a group of responses.","Offline DPO should not be treated as equivalent to online preference optimization for post-training LLMs; the gap on math and instruction-following benchmarks is large and consistent.","A single model can be trained on verifiable and non-verifiable rewards together, and the combination improves non-verifiable evaluations over non-verifiable-only training, even though one-task-only checkpoints show little cross-task transfer."],"supporting_citations":[{"why":"Defines the DPO loss that all DPO variants in the paper optimize.","marker":"Rafailov et al., 2024"},{"why":"Defines GRPO, the group-relative policy-gradient objective used as the online comparison.","marker":"Shao et al., 2024"},{"why":"Supplies Athene-RM-8B, the reward model used for non-verifiable rewards and checkpoint selection.","marker":"Frick et al., 2024a"},{"why":"Provides WildChat-1M, the source of the non-verifiable instruction-following prompts.","marker":"Zhao et al., 2024"},{"why":"Provides NuminaMath, the source of verifiable math training problems and test data.","marker":"Li et al., 2024a"},{"why":"Shows that GRPO length normalization can bias optimization; the paper omits it for that reason.","marker":"Liu et al., 2025"},{"why":"Supplies prior evidence that iterative response regeneration improves over offline DPO.","marker":"Yuan et al., 2024"}],"fun_headline_variants":["Sync interval beats algorithm choice in LLM RL","Semi-online RL matches online for LLM fine-tuning","Offline lags: step-sync delivers most online gains","Multi-task rewards boost both math and instruction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated reward score used to choose checkpoints faithfully measures response quality; the paper itself notes that this reward model has a length bias, so if the score rewards length rather than quality, the reported gaps between training regimes could be an artifact of checkpoint selection.","fun_headline_variants_meta":{"raw":{"variants":["Sync interval beats algorithm choice in LLM RL","Semi-online RL matches online for LLM fine-tuning","Offline lags: step-sync delivers most online gains","Multi-task rewards boost both math and instruction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000779,"raw_usage":{"total_tokens":3398,"prompt_tokens":853,"completion_tokens":2545,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":2481}},"tokens_in":469,"tokens_out":2545,"duration_ms":19293,"temperature":1.0,"reasoning_tokens":2481,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:23:35.598981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the offline, semi-online, and online DPO recipes but select checkpoints by held-out benchmark scores (math accuracy and instruction-following winrates) instead of the length-normalized reward score; if offline DPO then matches or beats the online variants, the reported gap is a checkpoint-selection artifact rather than a training-regime effect.","supporting_citations":[],"review_version":1}