{"id":"2d182ff7-7522-423b-a6ce-a5df67fe1ef3","arxiv_id":"2506.04180","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A 7B writing model trained on plan-write-refine thinking data with multi-stage preference optimization matches or beats several larger models on long-form generation benchmarks.","lead":"This paper trains a 7B language model to write long texts by planning, drafting, and revising in stages, then aligning it with preference feedback. It reports competitive scores against much larger models, but its headline state-of-the-art claim is not supported by its own benchmark table.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The hierarchical DPO signal rests on an unvalidated QwQ-32B judge; if that judge's scores do not track human quality, the preference pairs in §3.2 are misaligned and the +0.04 gain in Fig. 6 is not evidence of improved writing.","rationale":"The reader's weakest_assumption identifies the correct target: the paper's novel component is hierarchical DPO, and its training signal is entirely derived from QwQ-32B's scores. Without validation, the preference pairs are only guaranteed to reflect QwQ-32B's preferences. The final evaluations use different LLM judges (WritingBench critic and GPT-4.1), so correlated biases could persist even if the DPO training 'works'; the only human evaluation is against a same-size model and does not report agreement statistics. The proposed calibration study directly tests whether the reward tracks human quality. If it passes, the DPO contribution is credible. If it fails, the +0.04 gain in Figure 6 is not evidence of better writing, and the paper should be revised to de-emphasize the DPO result or retrain with a validated reward. This is consistent with the current CONDITIONAL verdict and does not require changing it.","tokens_in":25202,"tokens_out":11034,"duration_ms":102972,"concrete_test":"Sample 100 final outputs from the MCTS leaves (or from the 200 win-rate prompts), have three fresh annotators score them on the same six-dimension rubric used by Write-judge, and compute Spearman correlation between the average human score and QwQ-32B's score; additionally ask annotators to express preferences on a sample of the constructed DPO pairs (e.g., 50 pairs from P1/P2/P3). If the Spearman rho is below about 0.6 or human agreement with the DPO-pair directions is below about 65%, the reward model is not a valid proxy for human writing quality, and the hierarchical DPO result in Figure 6 cannot be interpreted as a genuine quality improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is the validity of the Write-judge reward used to build every hierarchical DPO preference pair. In Section 3.2, QwQ-32B scores each final leaf three times on six LLM-selected dimensions; the averaged score is converted to an ordinal reward via rank-percentile discretization (Step 1), propagated upward by averaging over children (Step 2), and used to sample preference pairs P1/P2/P3 (Step 3). If QwQ-32B is biased toward length, formatting, or stylistic fluency rather than coherence and planning quality, all three preference sets encode that bias, and the +0.04 gain attributed to hierarchical DPO in Figure 6 is alignment to the judge rather than evidence of better writing. The paper reports no human agreement for this reward, no inter-annotator reliability, and no check that child-averaging tracks human judgments of plans and drafts. Because the final evaluation also uses LLM judges (WritingBench critic, GPT-4.1), correlated biases would not be exposed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SuperWriter-Agent, a three-stage (Plan-Write-Refine) agent pipeline that produces thought-annotated long-form writing data, and SuperWriter-LM, a 7B model trained on this data via stage-wise SFT plus a hierarchical DPO procedure in which MCTS propagates final-output quality scores to intermediate plan, draft, and refine decisions. On WritingBench, SuperWriter-LM scores 8.51 on average, making it the best same-size model and second overall to DeepSeek-R1; the paper also reports pairwise win-rate evaluations against several baselines and a human evaluation against Writing-Model-Qwen-7B. The authors argue that these results demonstrate the value of explicit thinking steps and hierarchical preference learning for long-form generation.","tokens_in":25487,"tokens_out":4269,"duration_ms":41207,"significance":"If the central claims were fully supported, the paper would be a useful demonstration that a 7B model can match or approach much larger systems on long-form writing by distilling an agentic planning-writing-refining process, and that hierarchical DPO over tree-structured rollouts can improve multi-stage generation. Strengths include the explicit three-stage data format, the MCTS-based credit-assignment idea, the inclusion of a human evaluation, and the release of code and models, which are reproducibility pluses. However, the headline 'state-of-the-art' claim is not supported by the paper's own Table 1, and the validity of the training-time reward and the statistical significance of the key ablation are not yet established, so the strength of the claims currently exceeds the evidence.","major_comments":[{"comment":"The abstract claims that SuperWriter-LM 'achieves state-of-the-art performance, surpassing even larger-scale baseline models in both automatic evaluation and human evaluation,' but Table 1 shows SuperWriter-LM is second to DeepSeek-R1 on WritingBench (8.51 vs. 8.55), and the human evaluation in §4.3 compares only against Writing-Model-Qwen-7B, a same-size 7B model. The abstract's claim about surpassing larger-scale models in human evaluation is therefore not supported by any reported experiment. Please revise the claims to match the evidence, e.g., 'best among same-size models and competitive with much larger models,' and clearly state that the human evaluation was against a 7B baseline.","section":"Abstract and §4.2, Table 1"},{"comment":"The entire hierarchical DPO signal is built on the Write-judge reward: QwQ-32B scores each leaf three times on six LLM-selected dimensions, the scores are averaged and discretized into ordinal rewards, then propagated upward by averaging over children to construct preference pairs P1, P2, and P3. The paper reports no validation of this reward against human judgments, no inter-annotator or judge agreement statistics, and no check that child-averaging correctly ranks intermediate plans and drafts. If the judge is biased toward length, formatting, or stylistic fluency rather than coherence and planning quality, all three preference sets encode that bias, and the +0.04 gain attributed to hierarchical DPO in Figure 6 could reflect alignment to the judge rather than improved writing. Please add a human-agreement study for the Write-judge scores, report agreement/reliability metrics, analyze correlations with length and surface features, and validate the parent-node averaging assumption on human judgments of plans and drafts.","section":"§3.2, Steps 1–3 (Write-judge reward)"},{"comment":"The ablation improvement from hierarchical DPO is reported as a single point estimate: 8.47 with three-stage SFT versus 8.51 with hierarchical DPO, a difference of 0.04 on the WritingBench scale. No confidence intervals, repeated evaluation runs, bootstrapped standard errors, or significance tests are provided, and the evaluation itself uses an LLM critic. Given that the entire claimed benefit of the paper's central training contribution rests on this small increment, the current presentation does not establish that the improvement is real rather than noise. Please report variance or significance estimates, or temper the claim accordingly.","section":"§4.4, Figure 6"},{"comment":"Table 1 is captioned 'evaluated with our critic model,' which is ambiguous: it could mean the official WritingBench critic (a Qwen2.5-7B model described in the benchmark) or a custom critic built for this paper. If a custom critic was used, the scores are not directly comparable to published WritingBench numbers, and the comparison to baselines may not be meaningful unless the custom critic is validated against the official benchmark. Please clarify whether the official WritingBench evaluation harness was used, and if not, report the agreement between the custom critic and the official benchmark's critic, ideally reporting both sets of scores.","section":"§4.2, Table 1 caption and §4.1 Benchmark Setup"}],"minor_comments":[{"comment":"The text says 'As shown in Table 6' but the cited result is Figure 6; please correct the cross-reference.","section":"§4.4"},{"comment":"The phrase 'Lack of online reinforcement learning. Lack of online reinforcement learning.' is duplicated; please remove the repetition.","section":"§7, Limitation (3)"},{"comment":"The sentence 'we conduct an human supplementary assessment' contains a grammatical error ('an human' should be 'a human'), and the description of annotators as 'with undergraduate degrees.' should be made into a complete clause with details about their language proficiency and evaluation instructions.","section":"§4.3, Human Evaluation"},{"comment":"The phrase 'it better accommodate to real-world user workflows' should be 'it better accommodates real-world user workflows' or 'it better accommodates the workflows of real users.'","section":"§3.1, SFT Training"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a plausible and potentially useful method, but the abstract overstates the results relative to Table 1 and the human evaluation, and the unvalidated Write-judge reward is a core technical risk that should be addressed with direct evidence before publication. I would ask the authors to tone down the SOTA claim, validate the reward model against human judgments, and provide uncertainty estimates for the key ablation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the take. The genuinely new piece is the hierarchical DPO over an MCTS tree of plan→draft→refine paths, with leaf quality scores back-propagated to create preference pairs at every stage. That is a sensible extension of TPO/CHIP-style process preferences to long-form writing, and the authors cite those precedents. The SFT data pipeline (stage-segmented plan/write/refine with thinker-writer steps) is also useful, and they are honest that the agent pipeline is expensive, which motivates distilling it into a 7B model.\n\nThe empirical story is mostly solid. Table 1 shows 8.51 Avg, second only to DeepSeek-R1, best among 7B-class models. The ablations are monotone: 7.43 → 8.21 → 8.47 → 8.51. Credit where due: the three-stage SFT contributes most of the gain, and hierarchical DPO adds a small but consistent +0.04. That is a modest claim, and it holds up assuming the judge is measuring something real.\n\nSoft spots, in proportion. First, the abstract and conclusion overstate: 'state-of-the-art, surpassing even larger-scale' is not what Table 1 shows, since DeepSeek-R1 is ahead on Avg, and the 'wins over 98%' line comes from 200 manually curated queries with ties counted as half-wins. That should be fixed. Second, the reward model for hierarchical DPO is QwQ-32B with a six-dimension rubric, and there is no human-agreement or inter-annotator reliability check. The stress-test concern is real: if QwQ-32B rewards length or formatting rather than coherence, all three preference sets encode that bias. But the concern is partial—the SFT result is independent of that judge, and the DPO gain is small, so a biased judge undermines the DPO claim more than the overall recipe. Third, WritingBench's own critic is a Qwen2.5-7B trained on 50K human labels with 83% agreement, so the evaluation is not circular, but there are no error bars or significance tests anywhere.\n\nWho is this for? People building long-form generation pipelines. It is a methods paper with a reproducible-looking recipe, but no verified artifacts are on the page, just a GitHub URL. I would send it to peer review; with a validated judge, fixed overclaims, and released code and models, it could become a useful reference. The citation pattern is fine; the prior work is appropriately cited.","headline":"A useful 7B long-form writing recipe with a clean ablation story, but the headline claim overreaches and the hierarchical DPO reward is not validated against humans.","tokens_in":26002,"tokens_out":2379,"would_cite":true,"duration_ms":22822,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7B model trained to plan, draft, and revise approaches the quality of far larger writing models.","keywords":["long-form text generation","hierarchical direct preference optimization","Monte Carlo Tree Search","plan-write-refine agent framework","thinking data","LLM-as-a-judge","WritingBench","7B language model"],"falsifier":"Show a set of the sixty-leaf MCTS trees to human raters and measure agreement between their rankings and the QwQ-32B leaf scores; if agreement is near chance or driven by output length or formatting rather than coherence, the preference pairs built from those scores are misaligned. A cleaner test is to retrain the hierarchical DPO with a judge score that has been validated against human preferences and see whether the WritingBench and win-rate gains persist.","tokens_in":25011,"feed_emoji":"✍️","tokens_out":8797,"duration_ms":81770,"temperature":0.7,"pith_summary":"SuperWriter tackles the problem that language models generate long texts in one pass and their coherence decays as output length grows. The paper's proposal is to teach a small model the deliberate process a human writer uses—first plan an outline, then think before writing each paragraph, then check and revise—rather than to rely on scale alone. To do this the authors generate a plan–write–refine dataset with an agent, train a 7B model on it, and add a hierarchical DPO stage that converts final-output quality scores into preferences at every step using Monte Carlo Tree Search. On WritingBench the resulting SuperWriter-LM scores 8.51/10, second only to DeepSeek-R1 and ahead of all same-size models, and it wins or ties most pairwise comparisons against several larger open models on roughly 200 real user queries. If the result holds, a 7B model can reach near-frontier long-form writing quality by learning how to structure its generation, with consequences for efficiency and deployment.","feed_headline":"Plan first, write second: a 7B model rivals 100x bigger writers","feed_subtitle":"Training on plan-write-refine thinking plus hierarchical preference optimization lifts a small model to near-frontier writing scores.","key_machinery":"The load-bearing mechanism is the three-stage SuperWriter pipeline plus a hierarchical DPO procedure over an MCTS tree. For each writing prompt the agent produces five plans, four drafts per plan, and three refinements per draft, giving sixty leaf outputs; a six-dimension rubric scored by QwQ-32B (average of three runs) rates each leaf. Leaf scores are converted into ordinal rewards by percentile, then averaged upward first across refinements and then across drafts, so every stage gets a preference pair that reflects final quality; the policy is then optimized with the standard DPO loss on preferences harvested at the plan, draft, and refine levels. The design assumption is that good plans cause good drafts and good refinements cause good finals, so averaging leaf rewards down the tree is a valid credit-assignment signal.","core_discovery":"On the paper's own terms, the central discovery is that a 7B model can internalize an agent-style writing process and thereby match or beat much larger models at long-form generation. SuperWriter-LM, trained on the three-stage SuperWriter-Agent data and then aligned with hierarchical DPO, achieves an average WritingBench score of 8.51, second among all evaluated models behind DeepSeek-R1's 8.55 and first among models of its own size; it also matches DeepSeek-R1 on English and outperforms it in four of the six benchmark domains. In pairwise win-rate evaluations on roughly 200 real user queries, SuperWriter-LM beats long-form specialist LongWriter-8B and writing-tuned 7B models, and remains competitive against much larger systems including DeepSeek-R1. The ablation path—7.43 base, 8.21 with agent final outputs, 8.47 after stage-wise SFT, 8.51 after hierarchical DPO—is the paper's evidence that the structured thinking data, not scale, drives the gain.","pith_inferences":["If the QwQ-32B judge is partly rewarding length, formatting, or stylistic fluency rather than coherence, the DPO pairs inherit that bias; a direct test is to compare judge scores against human rankings on the same sixty-leaf trees.","The paper's comparison bundles plan, write, and refine into one three-stage package, so it does not isolate whether the planning stage alone or the refinement stage alone drives the gain; an ablation that trains on only two of the three stages would answer that.","The same recipe—agent-generated thinking data plus hierarchical DPO—should transfer to other base models and languages; testing it on an 8B or 14B backbone would show whether the gain is specific to Qwen2.5-7B or a general property of the method.","Three sequential forward passes cost latency, so the practical question the paper leaves implicit is whether users prefer the quality gain over a single-pass model when both are given the same wall-clock budget."],"forward_implications":["A single 7B model can replace a 30-to-40-call agent pipeline, so the quality gains of multi-stage reasoning can be distilled into one efficient inference-time model.","Because the SFT data is segmented into plan, write, and refine stages, users could be given intermediate outlines to edit before the model writes, aligning generation with human review workflows.","The ablation numbers imply that structured thinking data accounts for most of the gain (7.43 to 8.47 on WritingBench) and hierarchical DPO adds a further, smaller improvement (to 8.51).","Same-size and writing-specialist baselines are overtaken, suggesting that explicit planning plus step-level preference learning is a more effective use of a fixed model budget than scaling parameters."],"supporting_citations":[{"why":"Supplies the benchmark and critic model used for all main scores and the Writing-Model-Qwen-7B baseline.","marker":"[60]"},{"why":"Defines the long-form generation baseline and the output-length gap the paper addresses.","marker":"[6]"},{"why":"Serves as the strongest baseline, the filter for training instructions, and the reasoning-style paradigm.","marker":"[12]"},{"why":"Supplies the DPO loss that the hierarchical training extends to multiple generation stages.","marker":"[40]"},{"why":"Scores every leaf output on six rubric dimensions, generating the reward signal propagated through the tree.","marker":"[53]"},{"why":"Powers the SuperWriter-Agent that produces the plan–write–refine SFT data.","marker":"[31]"},{"why":"Provides the real-world instructions used to build the training set and the evaluation queries.","marker":"[68]"},{"why":"Provides the real-world instructions used to build the training set and the evaluation queries.","marker":"[69]"},{"why":"Provides the 7B backbone with a 128K context window on which SuperWriter-LM is trained.","marker":"[63]"}],"fun_headline_variants":["Small 7B model outwrites much larger LLMs in long-form tests","Plan-write-refine: 7B model beats giants in long-form generation","Rival to giant LLMs: 7B SuperWriter scores 8.51 on WritingBench","Writing mastery at 7B: hierarchical DPO pushes score to 8.51","Reflection-driven 7B writer beats larger models on long-form quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the judge's averaged six-dimension score on a final output is a true measure of writing quality and that averaging it down through refinements and drafts correctly identifies which earlier plan or draft decisions were good.","fun_headline_variants_meta":{"raw":{"variants":["Small 7B model outwrites much larger LLMs in long-form tests","Plan-write-refine: 7B model beats giants in long-form generation","Rival to giant LLMs: 7B SuperWriter scores 8.51 on WritingBench","Writing mastery at 7B: hierarchical DPO pushes score to 8.51","Reflection-driven 7B writer beats larger models on long-form quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000733,"raw_usage":{"total_tokens":3288,"prompt_tokens":964,"completion_tokens":2324,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2215}},"tokens_in":580,"tokens_out":2324,"duration_ms":15026,"temperature":1.0,"reasoning_tokens":2215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:45:33.809013+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Show a set of the sixty-leaf MCTS trees to human raters and measure agreement between their rankings and the QwQ-32B leaf scores; if agreement is near chance or driven by output length or formatting rather than coherence, the preference pairs built from those scores are misaligned. A cleaner test is to retrain the hierarchical DPO with a judge score that has been validated against human preferences and see whether the WritingBench and win-rate gains persist.","supporting_citations":[],"review_version":1}