{"id":"8ccacdb1-1fa4-45c0-aff6-4b252ea98cff","arxiv_id":"2505.02142","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Applying LD-DPO to the DeepDistill-32B model improves average benchmark scores by 3.3 points, but the gains are uneven, the tables are inconsistent, and the evidence is preliminary.","lead":"This paper tests whether offline preference optimization (DPO and LD-DPO) can improve reasoning in a 32B language model without the cost of online reinforcement learning. It reports a 3.3 point average gain across five benchmarks, with the largest jump (10.1 points) on Arena-Hard.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 3.3% average and 10.1% Arena-Hard gains rest on a single run with alpha=0.3 chosen in preliminary experiments, with no seeds, confidence intervals, or Arena-Hard judge details; the headline numbers may reflect selection or evaluation noise rather than a stable method effect.","rationale":"The paper's contribution is an empirical claim about a training method, so the load-bearing requirement is that the reported benchmark deltas are stable estimates. The weakest spot is not the LD-DPO math (the loss in Eq. 3 follows DPO with a reparameterized likelihood) but the evidence connecting that loss to the headline numbers. A single run with a hyperparameter chosen in preliminary experiments, no error bars, and no judge details for Arena-Hard is insufficient to distinguish a 3.3% average gain from variance or selection. This is not an accusation of anything improper; the paper simply does not report the information that would let a reader assess stability. The reader's conditional verdict captures exactly this. I therefore do not change the verdict: CONDITIONAL remains appropriate, with release of seeds, code, and judge details as the condition. I also note the internal inconsistency that the abstract credits 'these simpler Offline RL methods' while Table 3 shows vanilla DPO underperforms the baseline on average; the supported claim is specifically about LD-DPO. This strengthens the need for the stability check rather than changing the verdict category.","tokens_in":14671,"tokens_out":4414,"duration_ms":51623,"concrete_test":"Reproduce Section 3.2 training with at least three random seeds for alpha=0.3 and for alpha in {0.0, 0.1, 0.5, 1.0}, using identical data and hyperparameters. Evaluate every run on all five benchmarks, reporting the Arena-Hard judge model and settings and giving mean±std. The concern lands if: (a) the mean Arena-Hard gain over the baseline is below about 5 points or fails a 95% significance test; (b) Arena-Hard varies by more than about 3 points across seeds at fixed alpha; or (c) the result at alpha=0.3 is not stable across neighboring alpha values. As a complementary check, repeat the alpha selection using only AIME2024, GPQA-Diamond, LiveCodeBench, and IFEval, then assess Arena-Hard on the held-out runs; if the Arena-Hard gain disappears, the headline 10.1% is a selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline quantitative claim—average +3.3%, Arena-Hard +10.1%—is the DeepDistill-32B-lddpo row of Table 2. For that claim to be true, these numbers must estimate a stable method effect rather than a selected or noisy outcome. The paper gives three reasons to doubt this. First, Section 3.2 says alpha=0.3 was 'empirically set ... based on preliminary experiments'; if those experiments used the same evaluation benchmarks, the reported gains are a selected maximum, not an unbiased estimate. Second, every benchmark score comes from a single training run, with no seeds, confidence intervals, or significance tests, so training/evaluation variance cannot be separated from the reported effect. Third, Arena-Hard is an LLM-judged benchmark; the paper reports no judge model, prompt, temperature, or number of judge samples, and the 10.1-point jump is the largest and most judge-sensitive number in the paper. Additionally, Table 3 shows vanilla DPO's average (70.9) is below the baseline (71.6), so the abstract's 'these ... methods' is not supported for DPO itself; the claim effectively rests on LD-DPO alone. The central claim is therefore conditional on evaluation reliability that the paper does not yet document.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether simple offline RL methods, specifically Direct Preference Optimization (DPO) and its length-desensitized variant LD-DPO, can improve reasoning in large language models. Using DeepDistill-32B as the base model, the authors construct a preference dataset from DeepSeek-R1 distilled outputs and train with DPO and LD-DPO, evaluating on AIME2024, GPQA-Diamond, LiveCodeBench, IFEval, and Arena-Hard. The central claim is an average improvement of 3.3 percentage points, with a 10.1-point gain on Arena-Hard, attributed primarily to LD-DPO. The paper also analyzes generation lengths and argues that increasing response length is only beneficial when accompanied by semantic richness. The analysis is framed as a preliminary study of offline RL for reasoning.","tokens_in":14987,"tokens_out":3574,"duration_ms":41710,"significance":"If the empirical claims are reliable, the paper would provide useful evidence that inexpensive offline preference optimization, particularly LD-DPO, can yield substantial reasoning gains over a strong base model, and it offers a practical datapoint on the length-bias problem in DPO. The paper's strengths include a detailed data-processing description, the use of external public benchmarks, and comparisons against several published models. However, the quantitative headline rests on a single training run per configuration, an ad hoc choice of the LD-DPO hyperparameter alpha, and an uncharacterized LLM-judged benchmark (Arena-Hard). These issues make the current evidence insufficient to support the strength of the claims. The paper is a reasonable pilot study, but the central quantitative conclusions need substantially stronger evaluation methodology before they can be accepted.","major_comments":[{"comment":"The hyperparameter alpha in LD-DPO is reported as 'empirically set to 0.3 based on preliminary experiments.' If those preliminary experiments used the same five evaluation benchmarks, then the reported gains (especially the 10.1-point Arena-Hard improvement) are a selected maximum, not an unbiased estimate of the method's effect. Please state explicitly whether alpha was chosen on a held-out validation split, report results across a range of alpha values (e.g., 0.0, 0.1, 0.3, 0.5, 1.0), and run multiple training seeds to demonstrate that the chosen value is not an artifact of selection.","section":"Section 3.2"},{"comment":"Every benchmark score in the paper comes from a single training run and a single evaluation pass, with no standard errors, confidence intervals, or significance tests. This is a load-bearing issue for the central 3.3% average and 10.1% Arena-Hard claims, because training and evaluation variance cannot be separated from the reported effect. In particular, Arena-Hard is an LLM-judged benchmark, yet the paper does not report the judge model, prompt, temperature, number of judge samples, or any measure of judge variability. Please provide these details and, ideally, multiple evaluation runs or bootstrap confidence intervals.","section":"Tables 2, 3, and 8"},{"comment":"There is an internal inconsistency in model labeling: Table 8 labels the LD-DPO results as 'Ours-32B-DPO,' while Table 3 labels the same configuration as 'DeepDistill-32B-lddpo' and uses 'DeepDistill-32B-dpo' for vanilla DPO. This makes it impossible for the reader to track which model produced the 74.9 average. Moreover, Table 3 shows that vanilla DPO's average (70.9) is below the baseline (71.6), so the abstract's claim that 'these simpler Offline RL methods substantially improve model performance' is not supported for DPO itself; the claim effectively rests on LD-DPO alone. Please correct the labeling and qualify the claim accordingly.","section":"Table 8 vs. Tables 2 and 3"},{"comment":"The paper repeatedly states that the method yields 'consistent enhancement' or 'consistent improvements' across benchmarks, but Table 2 shows GPQA-Diamond is unchanged (66.3 to 66.3), and Table 3 shows vanilla DPO's IFEval score drops from 72.8 to 59.7. The 'consistent improvement' claim is therefore not accurate for all benchmarks or for both methods. Please temper the wording to reflect the actual pattern, e.g., 'improvements on four of five benchmarks for LD-DPO, with no change on GPQA-Diamond.'","section":"Figure 1 caption and Section 5"}],"minor_comments":[{"comment":"There are typos in the subsection headings: 'acquring quries' should be 'Acquiring Queries'.","section":"Section 2.1"},{"comment":"The column header 'A VG' should be 'AVG' (average). This appears in Tables 2, 3, and 4.","section":"Tables 2, 3, 4"},{"comment":"The sentence listing benchmarks has missing punctuation: 'GPQA-Diamond[41], covering challenging problems in biology, physics, and chemistry LiveCodeBench[12] (2024-08–2025-01) for code generation tasks' needs a period or semicolon after 'chemistry'.","section":"Section 3.1"},{"comment":"Cross-references are inconsistent: 'Table 8 illustrates the performance differences between DPO and LD-DPO' should refer to Table 3, and 'Further analysis of response generation lengths in Table 8' should refer to Table 4. Please correct these references.","section":"Section 3.3.2"},{"comment":"The base model is called 'DeepDistill-32B' in Tables 2 and 3, but 'DS-Distill-32B' and 'DS-Distill-32BB' appear in Tables 2 and 8. Please unify the naming to avoid confusion.","section":"Tables 2 and 8"},{"comment":"The evaluation protocol is underspecified: please report the number of test samples per benchmark, decoding parameters (temperature, top-p, max tokens), the exact pass@1 implementation, and the version of Arena-Hard used. Without these, the reported scores are not reproducible.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is openly positioned as a preliminary study, which lowers the bar on scope but not on the reliability of the headline numbers. The current central claim (average +3.3%, Arena-Hard +10.1%) would be acceptable only with multiple seeds, confidence intervals, a clear hyperparameter-selection procedure, and full Arena-Hard judge details. The inconsistent labeling in Table 8 is easily fixable but should be caught before any acceptance. I also note that the paper's 'consistent improvement' language overstates what the data show even within the authors' own tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a preliminary empirical study of applying DPO and LD-DPO to the authors' own DeepDistill-32B, and it reports average +3.3% and +10.1% on Arena-Hard. The direction is sensible—offline RL for reasoning post-training is underexplored and worth testing—but the headline numbers are not supported by the evidence as reported. The stress-test concern lands: Section 3.2 says alpha=0.3 was 'empirically set based on preliminary experiments,' and every benchmark comes from a single training run, with no seeds, confidence intervals, or Arena-Hard judge details. So the central gains could easily be selection or evaluation noise rather than a stable method effect. What is genuinely useful: the data pipeline is described in detail (multi-domain queries, verify-score filtering, chosen/rejected construction with o4-mini-high verification), and the length-versus-quality analysis is a good observation—Table 4 shows DPO increases average generation length by ~25% while hurting IFEval, while LD-DPO improves performance with more moderate growth. That is a clear, practical insight. The soft spots beyond evaluation rigor are real but fixable. Table 8 labels the same numbers as 'Ours-32B-DPO' that Table 2 calls 'DeepDistill-32B-lddpo'; this needs correction. The abstract says 'these simpler Offline RL methods' improve performance, but Table 3 shows plain DPO's average (70.9) is below the baseline (71.6), so the claim effectively rests on LD-DPO alone. The Figure 1 caption claims 'consistent enhancement across all evaluated metrics' while GPQA-Diamond is unchanged (66.3 to 66.3). These are overstatements, not fatal flaws, but they need to be reworded. Is this paper worth a serious referee? Yes. The question—can offline RL give some of the benefit of online RL for reasoning at lower cost—is important, and this is a concrete datapoint on a 32B model. But I would send it back for major revision: add multiple seeds or at least confidence intervals, document Arena-Hard judge details, release code/data, fix the table labels, and qualify the claims to distinguish DPO from LD-DPO. If the numbers hold up after that, it becomes a useful reference for the community. If the authors cannot provide that evidence, the paper should not be accepted as is. I'd bring it to the reading group as an example of what counts as preliminary evidence in RL post-training, and I would cite it only if the reproducibility issues get resolved.","headline":"Plausible direction, shaky evidence: the single-run results with alpha tuned in preliminary experiments do not support the headline 3.3%/10.1% claims yet.","tokens_in":15513,"tokens_out":3215,"would_cite":false,"duration_ms":38621,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Offline preference training alone lifts LLM reasoning by 3.3 points on average.","keywords":["offline reinforcement learning","direct preference optimization","LD-DPO","length bias","LLM reasoning","preference optimization","Arena-Hard","instruction following"],"falsifier":"Rerun the exact one-epoch LD-DPO recipe from the same base model with several random seeds and neighboring $\\alpha$ values, and evaluate Arena-Hard with an independent judge. If the 89.0% result does not consistently clear the 78.9% baseline, or if a different judge shrinks the gap, the paper's headline gain is not reproducible.","tokens_in":14519,"feed_emoji":"🧠","tokens_out":7935,"duration_ms":88012,"temperature":0.7,"pith_summary":"The paper claims that cheap offline preference optimization—Direct Preference Optimization and its length-desensitized variant LD-DPO—can improve long-context reasoning in a strong 32-billion-parameter language model without the costly rollout loops of online reinforcement learning. On five reasoning benchmarks, a single epoch of LD-DPO training on verified preference pairs raised the model's average score from 71.6% to 74.9%, a gain of 3.3 percentage points, with the largest jump on Arena-Hard (78.9% to 89.0%, +10.1 points). The authors attribute the gains to LD-DPO's handling of output length: plain DPO biases models toward longer, often redundant answers and can hurt precision-oriented tasks, while the length-desensitized variant keeps outputs concise and meaningful. The paper also supplies a detailed recipe for building the preference data, including a verification step that rejects failed responses, and argues this recipe makes offline RL a practical alternative to online RL for reasoning.","feed_headline":"Cheap preference training lifts LLM reasoning 3.3 points","feed_subtitle":"One epoch of length-aware preference optimization adds 10.1 points on Arena-Hard without an online rollout loop.","key_machinery":"The load-bearing object is the length-desensitized likelihood of LD-DPO, defined as $\\hat{\\pi}_\\theta(y|x) = \\prod_{i=1}^{l_p} p(y_i|x,y_{<i}) \\prod_{i=l_p+1}^{l} p^{\\alpha}(y_i|x,y_{<i})$, where $l_p$ is the common length of the shorter response in a preference pair and $\\alpha \\in [0,1]$ controls how much the model cares about tokens beyond it. Plugging this likelihood into the DPO implicit-reward loss removes the incentive to pad responses, which is the paper's central explanation for stable gains. The second piece of machinery is the data pipeline: queries sampled from math, code, science, instruction-following, and general reasoning categories, with chosen and rejected answers drawn from a stronger teacher's distilled outputs, rejected answers required to fail verification, and a further cleaning step that corrects ground truth when the teacher's mode disagrees with it.","core_discovery":"The core claim is that a one-epoch, offline preference-optimization pass can materially improve an already-strong reasoning model across math, code, instruction following, and general reasoning, averaging +3.3 percentage points and reaching +10.1 points on Arena-Hard. The mechanism the authors identify is length desensitization: DPO's implicit reward favors longer outputs, and the extra tokens are often redundant, which explains why their plain-DPO run lost 13.1 points on IFEval (72.8 to 59.7) even as output length rose by about 25%. LD-DPO reparameterizes the policy likelihood $\\hat{\\pi}_\\theta(y|x)$ so that tokens beyond the shorter response's length are discounted by an exponent $\\alpha \\in [0,1]$; with $\\alpha = 0.3$, gains appear on four of five benchmarks while generations stay shorter than plain DPO's. The paper's thesis is that offline RL can improve long-context reasoning when response-length growth is aligned with semantic richness rather than pursued blindly.","pith_inferences":["Our inference: if the Arena-Hard gain is stable, the same offline recipe could be applied iteratively, using the improved model's own verified outputs to build the next round of preference pairs, potentially narrowing the gap to online RL without rollout infrastructure.","Our inference: the paper's 'length should track semantic richness' principle yields a testable prediction—under a fixed token budget, LD-DPO should outperform plain DPO by an even wider margin, because constrained decoding removes verbosity as a confound.","Our inference: since $\\alpha=0.3$ was chosen once from preliminary experiments, per-domain tuning of $\\alpha$ is the natural next experiment; the IFEval failure of plain DPO suggests instruction-following may need a different $\\alpha$ than code or math."],"forward_implications":["A one-epoch offline LD-DPO run on verified preference pairs can lift a strong base model above its SFT baseline on math, code, instruction following, and general reasoning.","Plain DPO's length bias can make instruction-following performance drop sharply (IFEval falls from 72.8 to 59.7 in this paper) even while generation length grows, so controlling length is a practical necessity.","The hyperparameter $\\alpha$ is a direct knob practitioners can use to balance conciseness and reasoning depth, with 0.3 documented as a working default.","Because the recipe needs no reward model and no online interaction, it can be reproduced with modest compute, which is the paper's stated advantage for open-source and academic settings."],"supporting_citations":[{"why":"Defines the DPO objective the paper trains with; every experiment depends on this implicit-reward loss.","marker":"[40]"},{"why":"Supplies LD-DPO, the length-desensitized likelihood and loss the paper uses for all main results.","marker":"[26]"},{"why":"Provides the base model and data-processing recipe, including the verify_score filtering, from which the paper's model and dataset are derived.","marker":"[48]"},{"why":"Supplies the teacher distillation outputs from which chosen and rejected answers are selected.","marker":"[3]"},{"why":"Defines the Arena-Hard benchmark where the paper's largest gain (+10.1 points) is measured.","marker":"[22]"},{"why":"Documents the Arena-Hard pipeline and live-data benchmark construction used for evaluation.","marker":"[23]"},{"why":"Defines LiveCodeBench, the code-generation benchmark used in the evaluation.","marker":"[12]"},{"why":"Defines IFEval and the prompt-strict score used for instruction-following evaluation.","marker":"[62]"},{"why":"Defines GPQA-Diamond, the science benchmark used in the evaluation.","marker":"[41]"},{"why":"Supplies the AIME problem set used for mathematical reasoning evaluation.","marker":"[4]"}],"fun_headline_variants":["Offline RL adds 3.3 points to LLM reasoning","DPO offline: +10.1 on Arena-Hard, cheaply","Length-aware DPO boosts reasoning without RL loops","Offline preference optimization lifts reasoning 3.3%","LD-DPO: cheap reasoning gains via length control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire reported advantage rests on a single training run with $\\alpha=0.3$ picked in preliminary experiments; if that run's 10.1-point Arena-Hard jump is evaluation noise rather than a stable method effect, the central claim softens substantially.","fun_headline_variants_meta":{"raw":{"variants":["Offline RL adds 3.3 points to LLM reasoning","DPO offline: +10.1 on Arena-Hard, cheaply","Length-aware DPO boosts reasoning without RL loops","Offline preference optimization lifts reasoning 3.3%","LD-DPO: cheap reasoning gains via length control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000743,"raw_usage":{"total_tokens":3315,"prompt_tokens":943,"completion_tokens":2372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2288}},"tokens_in":559,"tokens_out":2372,"duration_ms":18246,"temperature":1.0,"reasoning_tokens":2288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:59:44.106915+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the exact one-epoch LD-DPO recipe from the same base model with several random seeds and neighboring $\\alpha$ values, and evaluate Arena-Hard with an independent judge. If the 89.0% result does not consistently clear the 78.9% baseline, or if a different judge shrinks the gap, the paper's headline gain is not reproducible.","supporting_citations":[{"cited_title":"Gonzalez, and Ion Stoica","cited_arxiv_id":null,"evidence_quote":"Documents the Arena-Hard pipeline and live-data benchmark construction used for evaluation."},{"cited_title":"Aime_1983_2024 (revision 6283828), 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the AIME problem set used for mathematical reasoning evaluation."}],"review_version":1}