{"id":"0ee2d976-36b4-40ef-9e87-adf1ee0d0b77","arxiv_id":"2606.03021","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"HDPO distills 'propose-select-think' reasoning into LLMs and uses GRPO-style RL with scheduled diversity and entropy-based reliability rewards, improving average benchmark scores by about 7 points over GRPO on 4B–8B models.","lead":"Scientists train language models to sketch several solution approaches to a problem, pick one, then reason in detail, and reward them for making the sketches diverse and selecting a confident one. On math and science tests it beats standard reinforcement-learning training on 4B–8B models, but the runs lack error bars and code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reliability reward may pay for confident-but-wrong selections: Eq. 11 applies r_rel even when r_acc=0, so entropy-based ranking can be reinforced without correctness; the paper's correlations are only against a proxy, not ground truth.","rationale":"I read the paper as making a two-part claim: HDPO improves final benchmark accuracy, and it improves the diversity and reliability of proposed solutions. The first part is empirically plausible but would be stronger with error bars and code. The second part is where the argument is least secure. The reader's weakest assumption correctly identifies the entropy-as-reliability proxy. I agree with that diagnosis and sharpen it by pointing to Eq. 11, where the reliability reward is paid even on incorrect trajectories whenever the group average accuracy is positive. This is not a mere philosophical worry: it is a structural property of the reward function that can reward confident-but-wrong selection. The paper's Table 3 correlation is against another model's sampled accuracy, not ground-truth correctness, and Table 4 measures alignment with that same proxy. The proposed concrete test—checking whether entropy rankings predict actual final-answer correctness on held-out problems—would directly settle whether the reliability reward is doing what the central claim requires. The paper does have independent supporting evidence: a consistent ablation pattern, three backbones, and a self-evolution experiment. These make the method worth conditional acceptance rather than rejection. No ad hominem is intended; the concern is about the reward definition and the proxy validation, not the authors' integrity.","tokens_in":23147,"tokens_out":7719,"duration_ms":86307,"concrete_test":"Take the final Qwen2.5-Math-7B HDPO checkpoint and sample 1,000 held-out problems. For each problem, have the model generate up to M=5 candidate outlines, rank them by token entropy per Eq. 9–10, then independently complete reasoning from each outline and score the final answers against ground truth. Compute the Spearman/Kendall correlation between entropy rank and actual answer correctness, and compare the accuracy of the model's <selected> choice against a random-choice baseline. If the selected-choice accuracy is not significantly above random, or the rank correlation is not clearly positive, the entropy-based reliability reward is not grounded in correctness and the central claim about reliable selection is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that HDPO enhances the model's ability to identify reliable solutions rests on the entropy-based reliability reward. Two related weaknesses make this the most load-bearing concern.\n\nFirst, Eq. 9–10 define reliability as low token entropy of a candidate outline. Table 3 reports Spearman correlations of 0.57–0.67 between this entropy ranking and Qwen3-1.7B-sampled accuracy on 10,000 training rollouts. That is a moderate correlation, and the criterion is itself a proxy, not ground-truth correctness. Thus the reward signal may rank confident-but-wrong outlines above correct but less fluent ones.\n\nSecond, Eq. 11 combines rewards as r_i = r_fmt + r_acc + 1[r_acc>0]·r_div + \\bar{r_acc}·r_rel. The reliability term is not gated by answer correctness: when r_acc=0, the third term vanishes but the fourth term remains positive whenever group-level average accuracy \\bar{r_acc} > 0. So an incorrect trajectory that selects a low-entropy candidate still receives positive reliability reward. Under group-relative normalization this can be overcome by correct trajectories, but it creates a direct incentive to favor fluently generated, confident-looking outlines even when they lead to wrong answers. The paper's Appendix J argues that RLVR eventually corrects this, but that argument is informal: the reward function itself does not enforce it, and no evidence is provided that entropy rankings correlate with final answer correctness after training.\n\nTable 4, which is offered as evidence that the reliability reward improves selection, measures agreement with the same Qwen3-1.7B sampled-accuracy proxy used to define reliability, not agreement with actual answer correctness on held-out problems. The Limitations section also acknowledges that confidence-based reliability estimation is left to future work. Therefore the component most responsible for the 'reliable selection' claim is not yet grounded in verifiable correctness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HDPO, a two-stage RLVR pipeline for LLM reasoning. In the first stage, the model is SFT on teacher-distilled ``propose-select-think'' trajectories: it generates multiple candidate solution outlines, selects one as a hint, and then produces a full solution. The distilled data is filtered by final-answer correctness and by a lightweight-model-based reliability check of the selected candidate. In the second stage, the model is fine-tuned with GRPO using a composite reward that adds a scheduled diversity reward (based on embedding similarity of candidate outlines) and a reliability reward (based on inverse entropy rank of the selected outline) to the usual format and accuracy rewards. Experiments on three backbones (Qwen3-4B, Qwen2.5-Math-7B, DeepSeek-R1-Distill-Llama-8B) and nine benchmarks report consistent improvements over GRPO and several prior RLVR baselines. Ablations cover the cold start, the RL phase, the diversity and reliability rewards, the scheduling, and several design variants.","tokens_in":23472,"tokens_out":9432,"duration_ms":102612,"significance":"If the results are confirmed, the paper makes a useful empirical contribution: it shows that a reasoning policy can internalize a explore-then-select cycle at zero inference-time overhead, and that a diversity incentive plus an entropy-based reliability incentive can improve both final-answer accuracy and the quality of the selected candidate. The method is described in enough detail to reimplement, and the ablation coverage is broad (cold start, RL, diversity, scheduling, reliability, candidate count, embedding model, and alternative policy-optimization algorithms). A self-evolution variant that removes the external teacher is also explored. However, the strength of the central claim is currently limited by the absence of statistical uncertainty estimates, by the lack of a clear validation protocol for hyperparameters, and by the reliance on a moderate, proxy-based correlation to justify the reliability reward. The paper is a solid candidate in an active line of work, but these load-bearing issues need to be addressed before the reported gains can be taken at face value.","major_comments":[{"comment":"The reliability reward is not gated by correctness. In Eq. (11), the term \\bar{r_acc}·r_rel is nonzero for every trajectory in a group whenever at least one answer in the group is correct, regardless of that trajectory's own r_acc. Thus an incorrect trajectory that selects a low-entropy candidate still receives positive reliability reward. The paper's defense in Appendix J is informal; it argues that RLVR eventually corrects this, but no experiment demonstrates that the entropy ranking of the selected candidate correlates with the final answer's correctness after training. Table 3 reports only moderate Spearman correlations (0.57–0.67) between the entropy ranking and the Qwen3-1.7B-sampled accuracy, and that criterion is itself a proxy, not ground truth. Table 4's selection-accuracy metric uses the same lightweight sampler as the cold-start filter, so the observed improvement may reflect","section":"§3.4, Eqs. (9)–(11); Table 4"},{"comment":"No error bars, confidence intervals, or multiple seeds are reported for any result. For example, on AIME 25 the Qwen3-4B HDPO score is 27.92 against GRPO's 25.78; with 30 problems and a pass@3 estimate from 16 samples, this 2.14-point gap is very plausibly within sampling noise. Several other per-benchmark gaps are similarly small. The abstract and conclusion claim that HDPO \"greatly boosts\" reasoning ability, but single-run results cannot support that strength of claim. Please run at least three independent training runs per configuration and report means and standard deviations (or bootstrap confidence intervals), including for the baselines; state the number of seeds used throughout.","section":"§4.2, Table 1"},{"comment":"The two-stage design is not isolated from the additional SFT data. HDPO adds 83,279 teacher-distilled trajectories during cold start, while the GRPO baseline receives only RL. The ablation \"w/o Cold Start\" shows that the RL rewards alone already outperform GRPO (e.g., AIME 25: 28.63 vs. 24.55), and \"w/o RL\" shows that cold-start SFT alone performs worse than GRPO. This leaves open the possibility that the gains of HDPO over GRPO come mainly from the reward design and not from the specific \"propose-select-think\" cold start. To attribute the gain to the structured trajectory, include a control that performs SFT on the same 83k trajectories in a standard chain-of-thought format (or with randomly selected candidates) before the same RL stage. Additionally, ablate the reliability filter in the cold-start data construction (e.g., keep all correct trajectories regardless of whether the selected","section":"§3.3 and §4.3"},{"comment":"Hyperparameters appear to be selected directly on the evaluation benchmarks. Table 9 chooses the maximum number of candidates M=5 using AIME 25, Math-500, Olympiad-Bench, and GPQA; Figure 5 selects the diversity strength µ=0.5 using Olympiad-Bench; Table 6 selects the embedding model using the same benchmarks. If these test sets were used for model selection, the reported numbers are optimistically biased and the ablations in Table 2 are not independent. Please specify a held-out validation set (e.g., a split of the training data or a separate set such as MATH or Minerva) that was used for all hyperparameter and design choices, and report final test-benchmark results only after those choices were fixed. If a validation split was already used, state it explicitly in Section 4.1 or Appendix B.","section":"§4.2, Appendix D/G; Table 9; Figure 5"}],"minor_comments":[{"comment":"The metric is called \"selection accuracy\" but is not defined in the text. Please define it precisely, e.g., the fraction of problems for which the model's selected candidate has the highest Qwen3-1.7B-sampled accuracy among the generated candidates, and note that this is a proxy-based metric.","section":"Table 4"},{"comment":"The diversity schedule uses t, twp, and tmax but the text does not explicitly state that t is the RL training step counting from 0. Please add a sentence clarifying the step indexing and that the sine factor reaches its maximum at t=tmax.","section":"Eq. (8)"},{"comment":"The evaluation says \"unbiased pass@3\" with 16 samples; please state the estimator formula or cite Chen et al. (2021) precisely, and specify how the 16 samples are used to compute pass@3.","section":"§4.1"},{"comment":"The Hit@N curves are not described in the caption: how many problems, what sampling temperature, and what is the policy used for each curve? Add these details to make the figure interpretable.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a potentially interesting empirical contribution, and the authors should be encouraged to revise rather than discouraged. The most serious issue is the lack of any variance information and the likely use of test benchmarks for model selection; both undermine the strength of the central claim. The reliability-reward proxy is a conceptual weakness, but it is partially mitigated by the breadth of ablations; the requested direct validation (entropy rank vs. final-answer correctness) would make the mechanism credible. I also recommend asking the authors to release code and data-processing scripts, as the method's value depends on the reproducibility of the cold-start filtering and reward computation. The scope is appropriate for the journal as an empirical RLVR paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core recipe is real: a cold-start SFT stage that teaches a policy to output multiple candidate solution outlines, select one, then think, followed by GRPO-style RL with a scheduled diversity reward and an entropy-rank reliability reward. The ablations are mostly consistent—removing cold start, RL, diversity, scheduling, or reliability each hurts—and the gains hold across three backbones and several GRPO-family optimizers. That is a solid empirical package worth a serious look. Second, the reliability-reward story is weaker than the abstract suggests, and the stress-test note is on target.\n\nThe most load-bearing soft spot is the entropy proxy. Equation 11 applies the reliability reward even when the current sample's answer is wrong, as long as the group average accuracy is positive. That directly rewards selecting low-entropy but possibly incorrect outlines. The paper's Table 3 shows only moderate correlations (Spearman 0.57–0.67) between entropy ranking and Qwen3-1.7B sampled accuracy, and Table 4 validates the model's selection against that same proxy, not against ground-truth correctness. So the evidence shows the model learns to match the proxy, not that it learns to pick correct reasoning paths. The authors' Appendix J reply is informal; it might be true that outcome rewards eventually correct this, but the paper provides no data on selection accuracy against true answers after training. Their own Limitations section concedes the confidence-proxy issue. This is a real gap, but not a fatal one—the overall gains do not depend on the reliability reward being perfectly calibrated, and the ablation shows it contributes.\n\nOther issues are more standard: no code or artifacts, no seeds or error bars, and hyperparameters like μ=0.5 and M=5 were tuned on the same evaluation benchmarks appearing in Table 1. AIME-level deltas of a few points can easily be noise. The paper is still worth citing for the diversity-scheduling and propose-select-think internalization idea, but I would not cite it for the reliability claim.\n\nWho is this for? Anyone working on RLVR, reasoning diversity, or structured reasoning trajectories. It deserves a serious referee—conditional accept with requests for code, multiple seeds, and a reliability metric evaluated against true answer correctness on held-out problems.","headline":"A genuinely useful RLVR training recipe with consistent ablations, but the entropy-based reliability reward is only weakly validated and the headline 'reliable selection' claim overstates what the evidence shows.","tokens_in":24089,"tokens_out":1620,"would_cite":true,"duration_ms":21152,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HDPO claims that LLM reasoning improves when a model is explicitly trained to propose multiple candidate outlines, select the most reliable one, and then reason, with diversity and confidence rewards, so exploration happens inside a single","keywords":["LLM reasoning","reinforcement learning with verifiable rewards","propose-select-think","solution diversity","reliability reward","cold-start supervised fine-tuning","policy optimization","mathematical reasoning"],"falsifier":"On a fixed benchmark set, take every candidate outline the trained model generates, compute its token entropy and (by sampling many continuations) the probability that it reaches the ground-truth answer, then rank the candidates both ways. If the rank correlation is near zero, or if low-entropy outlines repeatedly lead to wrong answers on problems where other outlines succeed, the reliability reward is rewarding confidence rather than correctness. The paper reports Spearman correlations of 0.57–0.67, so the decisive check is whether those correlations hold on held-out problems and in the tails","tokens_in":22990,"feed_emoji":"🧠","tokens_out":6852,"duration_ms":73072,"temperature":0.7,"pith_summary":"HDPO sets out to fix a blind spot in reinforcement learning with verifiable rewards: checking only final answers gives no signal for exploring different strategies or choosing a reliable one. The paper's claim is that training an LLM to first sketch several candidate solution outlines, pick the most promising one, and only then produce the detailed reasoning—with rewards for accuracy, diversity among candidates, and selecting low-entropy, high-confidence outlines—makes reasoning both more accurate and more diverse. If true, this turns an inference-time search strategy into a single-pass behavior learned during training, and it explains why 4B–8B models trained this way outperform stronger baselines on math and general reasoning benchmarks. The authors also argue that the propose-select-think pattern is what lets the model recover when its first idea is wrong.","feed_headline":"Propose-first training lifts small LLM reasoning by 8 points","feed_subtitle":"Teaching models to sketch candidate strategies, pick the most promising one, then reason step-by-step boosts accuracy.","key_machinery":"The load-bearing object is the propose-select-think trajectory, which splits reasoning into three identifiable parts: a candidate-solutions block listing 1–5 high-level outlines, a selected tag naming one outline, and a thinking block with the final derivation. This structure lets RL assign separate rewards to distinct behaviors. The diversity reward measures pairwise cosine similarity between embedded candidate outlines and is applied through a sine schedule that rises from zero after a warm-up; the reliability reward ranks outlines by mean token entropy and pays 1/rank for selecting the lowest-entropy outline, scaled by group-level accuracy. Together these convert a costly inference-time s","core_discovery":"The paper argues that accuracy-only RLVR fails to incentivize the explore-then-commit behavior that human problem solving uses, and that this behavior can be taught directly into policy weights. Its central proposal is HDPO, a two-stage training scheme: first, cold-start supervised fine-tuning makes the model emit a set of candidate solution outlines, select one outline, and only then produce full reasoning; second, reinforcement learning shapes this behavior with format, accuracy, diversity, and reliability rewards. The paper reports that 4B–8B models trained this way outperform standard outcome-reward RL baselines on nine benchmarks—for one 7B backbone, average accuracy rises from 53.71 to","pith_inferences":["Extension: If the entropy proxy is the true bottleneck, blending it with a cheap external verifier or a learned reliability classifier should push selection accuracy beyond the numbers reported in the paper; a test like that would separate the reliability-reward design from the propose-select-think structure itself.","Extension: The paper evaluates on fixed math and general-knowledge benchmarks; applying HDPO to code generation or multi-step planning, where verifiable rewards exist but solution spaces are larger and answers are not a single number, would test whether the diversity-and-selection mechanism transfers beyond closed-form math.","Extension: The diversity reward is gated by final-answer correctness, so the model is only rewarded for diversity when it already gets the answer right; an un-gated variant would reveal whether diversity is learned as a reasoning strategy or merely as a side effect of fitting the training distribution."],"forward_implications":["If HDPO is right, RLVR does not have to be limited to outcome-level correctness; process-level rewards for exploration and selection can be folded into policy weights without extra inference-time sampling.","The gains appear to transfer across model families and sizes (4B, 7B, 8B) and across math and general reasoning benchmarks, suggesting the mechanism is not tied to one dataset or architecture.","The self-evolution result implies that dependence on a large teacher can be reduced over iterations, making the overall pipeline cheaper to scale.","Because the method wraps a group-relative policy optimizer, its reward design could be added to existing RLVR pipelines rather than requiring new optimization machinery."],"fun_headline_variants":["Propose-select-think training lifts small LLM reasoning by 8 points","Teaching LLMs to sketch, select, then reason yields 8-point boost","Diverse solution proposal improves LLM reasoning by 8 points","Propose-first RL lifts small LLM reasoning by 8 points"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"That token-level confidence (low entropy) of a candidate outline is a faithful proxy for the probability that the outline leads to the correct answer, and that the lightweight model used in cold-start filtering measures that probability accurately; if low-entropy outlines are merely confident-but-wrong, the reliability reward teaches the model to commit to plausible errors.","fun_headline_variants_meta":{"raw":{"variants":["Propose-select-think training lifts small LLM reasoning by 8 points","Teaching LLMs to sketch, select, then reason yields 8-point boost","Diverse solution proposal improves LLM reasoning by 8 points","Propose-first RL lifts small LLM reasoning by 8 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000825,"raw_usage":{"total_tokens":3423,"prompt_tokens":704,"completion_tokens":2719,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":2641}},"tokens_in":448,"tokens_out":2719,"duration_ms":19812,"temperature":1.0,"reasoning_tokens":2641,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:29:08.004751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed benchmark set, take every candidate outline the trained model generates, compute its token entropy and (by sampling many continuations) the probability that it reaches the ground-truth answer, then rank the candidates both ways. If the rank correlation is near zero, or if low-entropy outlines repeatedly lead to wrong answers on problems where other outlines succeed, the reliability reward is rewarding confidence rather than correctness. The paper reports Spearman correlations of 0.57–0.67, so the decisive check is whether those correlations hold on held-out problems and in the tails","supporting_citations":[],"review_version":2}