{"id":"22f23c58-d55c-454a-be8e-44c396ad61a2","arxiv_id":"2502.02095","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LongDPO uses MCTS-collected, critique-refined stepwise preference pairs with step-level DPO to improve long-form generation quality and length adherence.","lead":"This paper presents LongDPO, a training method that uses tree search and AI-generated critiques to create step-by-step preference data so language models can write longer, higher-quality text. The method improves long-form writing benchmarks over standard DPO while keeping general task performance stable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The full preference pipeline (node rewards, consistency checks, and critiques) is produced by one unvalidated LLM judge; if that judge's preferences diverge from human/GPT-4o judgment, LongDPO's reported gains may be overfitting to the judge rather than a general long-form ability.","rationale":"The reader's weakest assumption matches the load-bearing risk I see: the entire data-construction chain is mediated by one LLM judge, and that judge's reliability is never established. I do not see an internal inconsistency that would justify rejecting the paper. The main results are plausible: gains appear on two backbones, on two long-form benchmarks, with ablations that isolate the memory pool and the stepwise objective. The missing training-seed variance, the eta threshold chosen after looking at evaluation results, and the unreleased code are real but secondary; they support the conditional verdict without changing it. My recommended adjustment is therefore UNCHANGED: the paper should be conditionally accepted pending judge validation and released code.","tokens_in":23792,"tokens_out":9077,"duration_ms":91916,"concrete_test":"Sample 200 node-level preference pairs produced in Section 3.1 and have the three human annotators (using the Section A.4 guidelines) independently choose the better step for each pair; measure agreement with Theta's reward ordering via Cohen's kappa. If kappa is below about 0.6, the stepwise preference signal is too noisy to support the claim that process supervision is reliably better than outcome supervision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that step-level DPO with critique-augmented MCTS data outperforms outcome-level DPO for long-form generation. In Section 3.1 to Section 3.2, every piece of supervision is created by a single model Theta = Llama-3.1-70B-Instruct: node rewards (Eq. 3), fact extraction and consistency labels (Section 3.1.2), and critique suggestions (Section 3.2). If Theta is biased, for example favoring fluent, internally consistent text over actual instruction fulfillment, then the chosen steps, memory-pool updates, and refined candidates all carry the same bias, and the stepwise DPO objective (Eq. 9) amplifies it. The paper provides no agreement statistics between Theta and human judges, no correlation between Theta rewards and the GPT-4o quality scores used in Table 1, and no held-out reward-model validation. The human evaluation (Table 6) compares only final responses of LongWriter versus LongDPO; it does not verify that Theta's stepwise preferences are correct. Moreover, the outcome-DPO baseline also selects its chosen response with Theta, so the head-to-head comparison partly measures how much more effectively LongDPO optimizes Theta's idiosyncratic scoring function. The empirical pattern is consistent and the ablations are helpful, but without judge validation the magnitude of the improvement cannot be separated from reward overfitting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LongDPO proposes a process-supervision variant of DPO for long-form generation. The method first constructs stepwise preference pairs via Monte Carlo Tree Search (MCTS), using a global memory pool to enforce factual consistency across steps and an external LLM (Llama-3.1-70B-Instruct) to generate critiques that refine low-reward chosen candidates. It then trains with a step-level DPO objective (Eq. 9) rather than outcome-level DPO. The method is evaluated on LongBench-Write, LongGenBench, and general benchmarks (TruthfulQA, MMLU, GSM8K) on LongWriter-Llama-3.1-8B and LongWriter-Qwen2.5-7B, with an additional larger-model experiment in Table 13. The paper reports consistent improvements over the vanilla DPO baselines, and ablations attribute the gains to the critique component, the memory pool, and the stepwise objective.","tokens_in":24084,"tokens_out":6864,"duration_ms":57503,"significance":"If the reported effects are reliable, LongDPO is a useful recipe for step-level preference learning in open-ended long-form generation, where outcome-level DPO has previously shown limited or unstable gains. The pipeline is concrete and the ablations isolate the main components: external critiques improve over self-critique or no critique (Table 3), the memory pool helps on a fact-checking transfer test (Table 4), and the stepwise objective helps over vanilla DPO on LongGenBench (Table 5). The paper ships a reproducible protocol with fixed seeds, official evaluation scripts, and promises to release code and models. However, the central claim is weakened by the lack of validation of the single LLM judge that produces every supervision signal, the absence of error bars or significance tests, and the apparent selection of the important hyperparameter η on the evaluation benchmark.","major_comments":[{"comment":"The entire preference-construction pipeline — node rewards (Eq. 3), factual-content extraction, consistency judgments, and critique generation — is produced by a single model Θ = Llama-3.1-70B-Instruct. The paper provides no agreement statistics between Θ and human judges, no correlation between Θ rewards and the GPT-4o quality scores used in Table 1, and no held-out reward-model validation. The human evaluation in Table 6 compares only final outputs of LongWriter versus LongDPO and therefore does not verify that the stepwise preferences or critiques are correct. Because the outcome-level DPO baseline also selects its chosen response with Θ, the head-to-head comparison in Tables 1 and 2 partly measures how effectively LongDPO optimizes Θ's idiosyncratic scoring function rather than a general long-form quality. This is load-bearing for the central claim: without judge validation, the magnitude of the reported gains cannot be separated from reward overfitting.","section":"§3.1–§3.2, Eq. (3), Eq. (6), Table 6"},{"comment":"The paper reports a single run per condition with fixed seeds and describes results as 'significantly outperforms' without reporting confidence intervals, multiple seeds, or hypothesis tests. Several differences are small in absolute terms (e.g., LongGenBench STC1 for Llama: 27.59 vs 25.99; STC2: 18.45 vs 16.29; CR for Qwen at 16k: 98.51 vs 95.95), so it is unclear whether these are statistically reliable. The authors should report at least three training seeds with means and standard deviations, or a paired test across evaluation examples, before claiming statistical superiority.","section":"Tables 1, 2, 13; §4.2"},{"comment":"The threshold η = 2.5 used in the main experiments is not accompanied by a statement that it was chosen on a held-out validation split. The ablation in Section 4.4 evaluates η ∈ {2.0, 2.5, 3.0} on LongBench-Write, the same benchmark used for the headline results in Table 1, and the 'LongDPO' row in Table 3 is the average over these three values rather than a single configuration. If η was selected on this benchmark, the main comparison is partially tuned on the test set, which inflates the apparent improvement. The paper should specify the validation procedure for hyperparameters or report results for all η values transparently in the main table.","section":"§3.2, Eq. (6); §4.4; Table 3"},{"comment":"The comparison between 'w/o Stepwise' and 'w/ Stepwise' is meant to isolate the stepwise DPO objective, but the paper does not specify how the non-stepwise baseline consumes the same preference data. If 'w/o Stepwise' concatenates the step-level chosen and rejected segments into full responses and applies Eq. (8), the data distribution differs from the standard outcome-DPO baseline in Section 4.1; if it instead applies a different loss to the step-level pairs, the comparison is not a clean test of stepwise learning. The paper should specify the exact training objective and data representation for 'w/o Stepwise' to make this ablation interpretable.","section":"§4.6, Table 5"},{"comment":"The reward-evaluation template explicitly instructs Θ not to consider whether the response meets the user's length requirements ('You do not need to consider whether the response meets the user's length requirements in your evaluation'). Yet Table 1 reports consistent improvements in the length score Sl. The paper does not explain the mechanism by which stepwise preferences that ignore length improve length adherence. This is not a fatal inconsistency, but it should be discussed: either length gains arise indirectly from other criteria (e.g., completeness or formatting) or from the training distribution, and the claim that process supervision directly addresses length deviations (Section 1) needs qualification.","section":"Appendix A.1; §1 and Table 1"}],"minor_comments":[{"comment":"There is a typo in the affiliation: 'Pattern Recongnition Center' should be 'Pattern Recognition Center.'","section":"Author affiliations"},{"comment":"The reference model π_ref is not defined; the paper should state that π_ref is the base LongWriter model (or the SFT model) used for DPO.","section":"§3.3, Eq. (8)"},{"comment":"The caption says 'we set different values of η and report the average result,' which means the 'LongDPO' row is not the same configuration as in Table 1; this should be stated explicitly to avoid confusing the ablation with the main method.","section":"Table 3 caption"},{"comment":"Inter-annotator agreement is reported as pairwise percentages without a chance-corrected statistic such as Cohen's κ; adding κ would strengthen the claim of high agreement.","section":"Table 7"},{"comment":"The data-collection seed is described only as 'a fixed random seed'; the actual seed value should be reported for reproducibility.","section":"§4.1"},{"comment":"The Limitations section only mentions the 32K evaluation ceiling and does not acknowledge the single-judge dependence or the lack of significance testing; these are relevant limitations that readers should be told about.","section":"Limitations"},{"comment":"The notation s_{1∼i} is used before it is defined in the sentence following the equation; define it at first occurrence.","section":"§3.3, Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a plausible and technically detailed approach with internally consistent ablations, but the evidenciary basis for the headline claim is currently weaker than the text suggests. The single-judge dependency is the most serious issue: every step-level preference, memory update, and critique is produced by Llama-3.1-70B-Instruct with no validation against human judgments or an independent reward model, and the DPO baseline uses the same judge for its outcome-level choices. The lack of multi-seed statistics and the apparent test-set tuning of η further undermine the confidence that the reported gains reflect a general improvement rather than reward overfitting. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I would also ask the authors to clarify the 'w/o Stepwise' training setup in Table 5, as the current description leaves the comparison ambiguous."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nLongDPO is a solid engineering contribution: it applies stepwise preference learning to long-form generation via MCTS-collected pairs, a memory pool for factual consistency, and external critiques to refine low-reward steps. The pieces are known individually, but the combination for long-form text is new, and the ablations confirm each component earns its place. The main results are consistent and credible: on both Llama- and Qwen-based LongWriter backbones, LongDPO beats outcome-level DPO on LongBench-Write and LongGenBench, with roughly lossless general performance.\n\nThe paper is honestly written. The ablations are informative, the human evaluation supports the main finding (though it is small), and the reliability analysis of the GPT-4o quality score is a useful addition.\n\nThe real soft spot is the single-judge pipeline. Every piece of supervision — node rewards, fact extraction, consistency checks, critiques — comes from Llama-3.1-70B-Instruct. The paper provides no agreement statistics between this judge and humans, no correlation with GPT-4o scores, and no held-out reward-model validation. Since the outcome-DPO baseline also uses the same judge to pick chosen/rejected responses, part of the head-to-head gain may be better optimization of that judge's idiosyncratic scoring function rather than a general long-form ability. The human evaluation mitigates this only partially, because it compares final outputs, not stepwise preferences, and the sample appears small.\n\nMinor concerns: no error bars or training-seed variance on the main tables; the critique threshold η is tuned on the training data; the code is promised but not public. These are not disqualifying.\n\nWho should read this: practitioners working on long-form generation or stepwise preference optimization. It is a concrete recipe with useful ablations, not a conceptual breakthrough. The paper deserves serious peer review. A good referee should push for judge validation and seed variance, but should not reject on the current evidence.\n\nRecommendation: engage. Send to peer review.","headline":"A practical stepwise-preference recipe for long-form generation with consistent gains over DPO, but the single-judge pipeline and missing variance leave the exact improvement magnitude unresolved.","tokens_in":24658,"tokens_out":4046,"would_cite":true,"duration_ms":33710,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stepwise preference learning improves long-form generation length and quality while keeping general-task performance nearly lossless.","keywords":["long-form generation","process supervision","Monte Carlo Tree Search","direct preference optimization","stepwise preference learning","LLM critiques","memory pool","factual consistency"],"falsifier":"Run the same MCTS pipeline but score the step pairs with human annotators and train on those human-preference pairs: if the model-judge scores disagree with humans on enough pairs, the reported gains should shrink or vanish. A simpler check is to replace the 70B judge with a different judge and see if the quality advantage over outcome-level DPO persists.","tokens_in":23533,"feed_emoji":"✍️","tokens_out":7753,"duration_ms":64466,"temperature":0.7,"pith_summary":"Long-form generation—essays, reports, repo-level code—remains weak even in large models because feedback is usually given once, on the final text. LongDPO claims that supervising each intermediate step works better. The paper collects stepwise preference pairs with Monte Carlo Tree Search, filters candidates through a global memory pool to preserve factual consistency, refines low-reward chosen steps with external critiques, and then trains with step-level DPO. On LongBench-Write and LongGenBench, both Llama- and Qwen-based models improve in length and quality while keeping general benchmarks near lossless. If the claim holds, it offers a practical recipe for turning any long-context model into a better long-form writer.","feed_headline":"Stepwise DPO beats whole-answer DPO for long-form writing","feed_subtitle":"Llama and Qwen writers match length targets better and hold general-task accuracy, with gains largest past 4,000 words.","key_machinery":"The central mechanism is the stepwise preference pair built from a Monte Carlo Tree Search. The paper splits generation into steps, expands each node into child candidates, and scores each node with a reward model using seven quality principles. A global memory pool stores extracted factual statements from earlier steps and rejects candidate steps that contradict them, preserving consistency. For chosen steps with low rewards, external critiques generated by a stronger model supply writing suggestions that are prepended to the context when regenerating the step. Training then uses a step-level DPO objective that maximizes the log-likelihood of the chosen step and minimizes the rejected step conditioned on the query plus all preceding steps.","core_discovery":"LongDPO's central discovery is that outcome-level DPO gives weak and sometimes harmful supervision for long-form generation, whereas stepwise supervision with the same model family yields consistent gains. Concretely, the paper decomposes a response into steps, uses MCTS to generate candidate steps, keeps only factually consistent ones via a memory pool, refines low-reward chosen steps by injecting external critiques, and optimizes a stepwise DPO objective. Across LongWriter-Llama-8B and LongWriter-Qwen2.5-7B, this improves length matching and quality on LongBench-Write and completion and instruction adherence on LongGenBench, with almost no drop on TruthfulQA, MMLU, and GSM8K. The authors state this as evidence that fine-grained, step-level learning is better suited to long-form generation than sample-level preference learning.","pith_inferences":["A testable corollary of the paper's argument is that the gains are driven by step-level separability, not by the MCTS tree itself; any procedure producing locally ranked step pairs should reproduce part of the effect.","Because the entire pipeline depends on one judge's seven-principle scores, a low-cost empirical check would be to re-run the pipeline with a second judge on a subset of instructions and measure how much the chosen/rejected pairs change and how much downstream DPO performance shifts.","The memory pool regularizes factuality; one natural extension is applying the same consistency check to long-form summarization, where grounding documents are available, to see whether the factuality gains transfer.","The stepwise DPO objective could in principle be applied to other long-output domains such as repository-level code generation, though the paper only evaluates text writing."],"forward_implications":["On LongBench-Write, both Llama- and Qwen-based LongWriter models trained with LongDPO improve length score and quality score over the same models trained with vanilla DPO, with the largest length gains in the 4k–20k word range.","On LongGenBench, the method raises completion rate and instruction-following metrics at both 16k and 32k lengths, indicating that stepwise supervision helps models stick to long-range instructions.","Training with the global memory pool improves fact-checking performance on LLM-AggreFact compared to training without it, connecting the preference data collection step to reduced hallucination.","External critiques give more stable gains than self-critiques when refining low-reward chosen candidates, as shown by varying the refinement threshold η.","General benchmarks (TruthfulQA, MMLU, GSM8K) remain nearly lossless after LongDPO, whereas plain DPO in some cases degrades them (e.g., MMLU for the Qwen backbone)."],"supporting_citations":[{"why":"Supplies the LongWriter base models and the LongBench-Write benchmark used for long-form evaluation.","marker":"Bai et al., 2024b"},{"why":"Suri, the outcome-level preference optimization approach that LongDPO compares against and improves on.","marker":"Pham et al., 2024"},{"why":"The Monte Carlo Tree Search survey that motivates the tree-search procedure for collecting stepwise candidates.","marker":"Browne et al., 2012"},{"why":"Provides the process-supervision premise that supervising intermediate steps is more effective than supervising final outcomes.","marker":"Lightman et al., 2024"},{"why":"Step-DPO, the stepwise preference optimization objective that LongDPO adapts for generation steps.","marker":"Lai et al., 2024"},{"why":"LongGenBench, the benchmark measuring completion rate and instruction adherence at 16k and 32k lengths.","marker":"Wu et al., 2024c"},{"why":"WildChat, source of the 2.5K instructions from which stepwise preference pairs are collected.","marker":"Zhao et al."},{"why":"UltraFeedback, the additional preference dataset combined with the collected stepwise pairs for DPO training.","marker":"Cui et al., 2024"}],"fun_headline_variants":["Stepwise DPO with critiques boosts long-form writing quality","Critique-augmented stepwise DPO improves length and quality","LongDPO: step-level preferences beat outcome-level for long-form","MCTS and critiques make stepwise DPO excel for long outputs","Stepwise preference learning fixes long-form generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline trusts one LLM judge (Llama-3.1-70B-Instruct) to score every step on seven principles, extract factual statements, check consistency, and generate critiques; the paper does not validate these scores against human judgments, so if that judge is biased or noisy the preference pairs and refinements are corrupted before training.","fun_headline_variants_meta":{"raw":{"variants":["Stepwise DPO with critiques boosts long-form writing quality","Critique-augmented stepwise DPO improves length and quality","LongDPO: step-level preferences beat outcome-level for long-form","MCTS and critiques make stepwise DPO excel for long outputs","Stepwise preference learning fixes long-form generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000591,"raw_usage":{"total_tokens":2743,"prompt_tokens":888,"completion_tokens":1855,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":1771}},"tokens_in":504,"tokens_out":1855,"duration_ms":12282,"temperature":1.0,"reasoning_tokens":1771,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:20:35.232371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same MCTS pipeline but score the step pairs with human annotators and train on those human-preference pairs: if the model-judge scores disagree with humans on enough pairs, the reported gains should shrink or vanish. A simpler check is to replace the 70B judge with a different judge and see if the quality advantage over outcome-level DPO persists.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Suri, the outcome-level preference optimization approach that LongDPO compares against and improves on."}],"review_version":1}