{"id":"4fecfdcd-5a3f-441d-91de-ad8c4c974241","arxiv_id":"2607.16206","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"PPO-HSC adds a novelty reward based on distance to a library of correct trajectories to PPO for LLM fine-tuning, reporting marginal Pass@5 gains on GSM8K and SVAMP.","lead":"PPO-HSC adds a novelty bonus to PPO-based RL fine-tuning of LLMs, rewarding correct solutions that are far from previously verified ones. The paper reports small gains in coverage on math and code tasks, but the supporting evidence is thin and one core reward term is mis-specified.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reader's KL-sign objection is a red herring; the load-bearing flaw is the contradiction between §3.2's curated library admission and Algorithm 1's unconditional add, which blocks a well-defined novelty signal.","rationale":"I read the paper in good faith. The reader's verdict is REJECT, and I agree that the paper should not be accepted as a research claim in its current form, but for a different primary reason. The reader's weakest_assumption focuses on the sign of P_KL, but that is a misreading. The expression -Σ log(πθ/πref) is the standard per-token KL penalty: when used as an additive reward (β·P_KL), maximizing it is equivalent to minimizing Σ log(πθ/πref), whose expectation is the forward KL divergence. Thus the 'plausibility constraint' is correctly specified as a penalty, not a reward of low-confidence tokens.\n\nHowever, the proposed mechanism has a serious internal inconsistency between the prose specification (§3.2) and the algorithm (Algorithm 1) regarding library maintenance. The HSC reward's entire purpose is to encourage novelty relative to a curated set of diverse, verified solutions. If the library is updated unconditionally, the novelty signal is ill-defined and likely to vanish as duplicates accumulate. This is a technical, load-bearing flaw that affects the method's validity, not just the strength of the evidence. In addition, the experimental evaluation is too weak to support the strong claims in the abstract: no error bars, best-checkpoint selection, only one baseline, and one metric showing a decrease in Pass@1 on GSM8K. These issues compound the central concern.\n\nI recommend keeping the reader's REJECT verdict, hence UNCHANGED. The concrete test—inspecting the released code and measuring d_min over training—would settle whether the mechanism is actually implemented as claimed, and if so, whether it provides any meaningful novelty signal. The paper does deserve credit for releasing code, which makes this test feasible.","tokens_in":7378,"tokens_out":7654,"duration_ms":86717,"concrete_test":"Inspect the released code at https://github.com/JJayshum/PPO-HSC in the training loop. Determine whether a correct trajectory τ is added to the library (a) unconditionally, (b) only after a deduplication check per §3.2, or (c) with Nmax-based eviction. Also instrument the code to log the average d_min(τ,D) over correct trajectories during GSM8K training. If the code follows Algorithm 1's unconditional update and d_min collapses to <0.01 within a few hundred steps, the novelty reward is effectively dead and the central claim fails. If the code instead implements deduplication, then Algorithm 1 is inaccurate, and the paper's reproducibility is compromised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest assumption is not the sign of P_KL. P_KL = -Σ log(πθ/πref), so the HSC reward includes +β·P_KL = -βΣ log(πθ/πref), which is exactly the standard KL penalty in RLHF (maximizing -Σ log ratio is equivalent to minimizing the forward KL). Thus the reader's claim that this 'rewards low-confidence choices' is a misunderstanding: the expected value of Σ log(πθ/πref) under πθ is KL(πθ||πref) ≥ 0, so the term is a genuine penalty on average.\n\nThe real load-bearing concern is internal inconsistency in the core mechanism. Section 3.2 states that a trajectory is admitted to the dynamic library D only if R_obj=1 *and* it is not already contained at the semantic/symbolic level, and that when |D| ≥ Nmax a diversity-greedy eviction is used. Algorithm 1 line 11, however, unconditionally executes D_k ← D_{k-1} ∪ {τ} for every correct trajectory, with no deduplication check, no cap, and no eviction. If the implementation follows Algorithm 1, the library becomes an ever-growing bag of all correct samples; then d_min(τ,D) tends to collapse as duplicates accumulate, so the HSC novelty reward no longer provides a meaningful signal for discovering 'low-similarity yet high-validity' paths. If the implementation follows Section 3.2, the published pseudocode is not the actual algorithm and the method is not reproducible. Either way, the central claim is not underpinned by a well-specified mechanism.\n\nThe empirical evidence is also insufficient: single seed, best-checkpoint reporting, no error bars, and only one baseline. GSM8K Pass@1 actually drops from 0.300 (Base) to 0.280 (PPO-HSC), and the uniqueness ratio is comparable to the base model, so 'significantly enhances diversity and state-space coverage' is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PPO-HSC, a modification of PPO-based RLVR in which a gated reward term is added to verified-correct trajectories. The extra reward combines a novelty bonus based on the minimum distance between the current trajectory and a dynamic library of previously verified correct trajectories, together with a token-level KL term against a frozen reference model, intended as a plausibility constraint. Experiments on GSM8K and SVAMP, including code-generation variants, compare PPO-HSC with a base model and a standard PPO-only RLVR baseline, reporting Pass@5, SC@5, and Syntax@5 metrics. The central claim is that PPO-HSC significantly improves solution diversity and state-space coverage while maintaining or improving accuracy and syntax integrity.","tokens_in":7861,"tokens_out":6150,"duration_ms":61953,"significance":"If properly validated, the idea would be a useful contribution: it directly targets mode collapse in RLVR, a recognized problem, and it connects quality-diversity / novelty-search ideas to gradient-based LLM fine-tuning. The gating of the exploration bonus behind the verification reward is a clean design choice, and the paper is readable enough that the intended mechanism is largely understandable. The manuscript also ships a pseudocode description and a code URL, which is helpful for reproducibility. However, in its current form the central claim is not supported by the evidence: the experiments are too thin, the evaluation protocol is biased or at least unclear, and the core library mechanism is specified inconsistently between text and pseudocode. The approach may be salvageable, but substantial additional work is needed.","major_comments":[{"comment":"The library update rule is internally inconsistent. Section 3.2 states that a trajectory is admitted only if R_obj=1 and it is not already contained at the semantic/symbolic level, and that when |D_t| >= Nmax a diversity-greedy eviction is used. Algorithm 1 (lines 7–15), however, unconditionally executes D_k ← D_{k-1} ∪ {τ} for every correct trajectory, with no deduplication, no capacity bound, and no eviction. If the implementation follows Algorithm 1, the library becomes an unbounded bag of all correct samples, duplicates accumulate, d_min(τ,D) collapses, and the novelty signal degenerates. If the implementation follows §3.2, the published pseudocode does not describe the actual algorithm. Either way, the central mechanism is not well specified. Please reconcile the description and pseudocode, and state which update rule was used in the experiments.","section":"§3.2 vs. Algorithm 1"},{"comment":"The empirical evidence does not support the claim that PPO-HSC 'significantly enhances' diversity or coverage. The table reports a single number per metric, with no error bars, no multiple seeds, and no statistical tests. The footnote states that PPO-HSC is reported at the best-performing checkpoint, but it is unclear whether the Base and PPO-only baselines are also selected at their best checkpoints. If not, the comparison is biased. The raw deltas are small (e.g., GSM8K Pass@5 0.405 vs. 0.395; SVAMP Pass@5 0.685 vs. 0.665), and on GSM8K Pass@1 PPO-HSC is lower than the Base model (0.280 vs. 0.300). Moreover, the abstract claims superiority over 'state-of-the-art RL baselines,' but the only RL baseline is a standard PPO-only variant. Please provide means and standard deviations over multiple seeds, a fixed checkpoint-selection rule, and additional strong baselines, or temper the claims","section":"§4.2, Table 1"},{"comment":"The syntax-stability claim is not demonstrated. Table 2 shows Syntax@5 = 1.000 for all models, including the Base and PPO-only models, so there is no observed degradation for PPO-HSC to 'preserve.' The statement that 'our Plausibility Constraint effectively filters out low-quality trajectories' is therefore unsupported without an ablation that removes the KL term. In addition, the 'Unique Code Ratio' is mentioned as approximately 0.333 but is never defined or tabulated, and the code-generation experiment is only reported for SVAMP even though the setup says the same datasets are processed through a code-execution verifier. Please define the metrics, report the full results, and include an ablation of α, β, and the KL term.","section":"§4.3, Table 2"},{"comment":"The KL term is described as a 'constraint,' but it is a soft reward term, and its expected effect should be clarified. Maximizing +β·P_KL with P_KL = -Σ log(πθ/πref) is equivalent to maximizing -βΣ log(πθ/πref), whose expectation under πθ is -β·KL(πθ||πref); in expectation this is a standard KL penalty, so the sign is not inherently wrong. However, P_KL is written with πθ, which changes inside the PPO update loop. A reward should be a scalar fixed at sampling time, so it should be computed with the behavior policy π_old (or the reference should state that the reward is re-evaluated after each gradient step, which would be unusual). Please specify exactly which policy distribution is used in the reward computation.","section":"§3.3, Eq. (4) and Algorithm 1 line 9"}],"minor_comments":[{"comment":"Key metrics are used without definitions: Pass@5, SC@5, Syntax@1, Syntax@5, and Unique Code Ratio. Please define them precisely, including whether they are computed by sampling k times per prompt at evaluation or by a different procedure.","section":"Throughout"},{"comment":"The hyperparameter list includes k (samples per prompt) and Nmax, but k is never used in the pseudocode and Nmax is only mentioned in §3.2, not in the algorithm. Either use them in the pseudocode or remove them from the requirements.","section":"Algorithm 1"},{"comment":"The section heading says 'High-k Sampling Coverage Reward,' while the paper elsewhere uses 'High-order Sampling Coverage.' The terminology should be consistent.","section":"§3.3"},{"comment":"Implementation details are sparse: values for α, β, σ, LoRA rank, learning rate, batch size, and number of PPO update steps are not given. Without these, the experiments are not reproducible. Please provide a full hyperparameter table in an appendix.","section":"§4"},{"comment":"Figure 3 is not referenced in the text, and its caption ('tianzige-style visualization') is not informative for a scientific figure. Either reference and explain it, or remove it.","section":"Figures"},{"comment":"Reference [6] is an unusual citation for entropy regularization in LLM RL; consider citing a more standard recent treatment. Also, there are several typographical spacing issues throughout the text (e.g., 'T able').","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core idea is relevant and potentially interesting, but the paper in its current form is not ready for publication. The internal inconsistency between §3.2 and Algorithm 1 goes to the heart of the proposed mechanism, and the empirical evaluation is far too thin to support the abstract's claims of significant improvement. I recommend major revision rather than outright rejection because these problems are addressable: the authors can align the algorithm description, run a multi-seed evaluation with proper error bars and checkpoint selection, add baselines and ablations, and clarify the KL term. If the authors cannot provide such evidence after revision, rejection would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable novelty-search-flavored extension of PPO for LLM reasoning. Gated novelty reward with embedding distance to an archive of verified solutions is a natural combination of QD/novelty search and RLVR. The authors cite the right literature. The 'plausibility constraint' is actually a standard KL penalty: maximizing +β·P_KL with P_KL = -Σ log(πθ/πref) is equivalent to minimizing the forward KL on average, so the reader's sign objection doesn't land. Good.\n\nBut there are two real problems. First, the library update is not well-specified. §3.2 gives a strict admission rule (only unique, semantically novel trajectories) and a diversity-greedy eviction when size exceeds Nmax. Algorithm 1, line 11, unconditionally adds every correct trajectory to D_k, with no dedup, no cap, no eviction. If the code follows Algorithm 1, the library becomes an ever-growing bag of duplicates and d_min collapses, so the novelty signal dies. If the code follows §3.2, then the pseudocode is not the algorithm, and the method as written is not reproducible. Either way the central mechanism isn't pinned down. That's load-bearing.\n\nSecond, the empirical claim of 'significant' improvement is not supported. Single seed, no error bars, no significance test, best-checkpoint reporting, and one baseline. The numbers are modest: GSM8K Pass@5 0.405 vs 0.395; Pass@1 drops from 0.300 to 0.280; SC@5 0.320 vs 0.315; unique code ratio comparable to base. Syntax@5 is 1.000 for all models, so the plausibility constraint isn't shown to be doing anything. The paper needs more runs, more baselines (e.g., a simpler entropy bonus or a novelty baseline without the library), and a proper evaluation protocol.\n\nWhat's genuinely useful: the gating (novelty reward only on correct trajectories) is a sensible way to adapt QD ideas to RL fine-tuning, and the embedding-distance archive is straightforward. The direction is worth exploring. But this draft doesn't establish that PPO-HSC works as claimed.\n\nRecommendation: I'd send it to review — the core idea is plausible and the literature grounding is honest — but I'd expect the reviewers to demand a corrected algorithm description, the actual implementation details, and a properly powered comparison. If you're looking for a quick desk reject, this isn't one; if you're looking for a solid paper, it's not there yet.","headline":"The KL-sign objection is a red herring — the term is a standard KL penalty — but the paper has a real reproducibility problem: Algorithm 1 contradicts the library-update rules in §3.2, and the experimental support is thin.","tokens_in":8333,"tokens_out":4088,"would_cite":false,"duration_ms":37363,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PPO-HSC rewards novel but correct solutions, broadening LLM reasoning without sacrificing accuracy or syntax.","keywords":["reinforcement learning","PPO","exploration","mode collapse","solution diversity","LLM fine-tuning","verifiable rewards","quality-diversity"],"falsifier":"Train PPO-HSC with the KL term set to zero (or with the sign flipped) on the SVAMP code task and measure Syntax@5 and Pass@5. If syntax stays at 1.000 and Pass@5 still improves, the plausibility constraint is not what preserves syntax; if gibberish appears, the sign issue is confirmed.","tokens_in":7317,"feed_emoji":"🎲","tokens_out":4119,"duration_ms":40879,"temperature":0.7,"pith_summary":"Standard reward-based fine-tuning of large language models tends to collapse onto a few high-reward solution paths. This paper proposes PPO-HSC, which adds a High-order Sampling Coverage (HSC) reward to PPO so that, once a trajectory is verified correct, the model earns extra reward for being unlike previously seen correct solutions. The effect is to shift optimization from exploiting a single path to covering a wider solution manifold. On math word problems and code generation, the authors report higher pass@5 and self-consistency than PPO-only RLVR, with syntax rates unchanged.","feed_headline":"Novel-correct reward lifts LLM pass@5 and keeps syntax at 1.000","feed_subtitle":"PPO-HSC adds a gated coverage bonus to PPO, expanding the reasoning manifold without sacrificing verified accuracy.","key_machinery":"The HSC reward is the engine: a dynamic trajectory library with greedy diversity eviction supplies the reference set; min-neighbor distance (cosine or Levenshtein) measures novelty; tanh bounds the novelty bonus; a token-level KL divergence relative to the base/reference model acts as a plausibility guard; and an indicator on R_obj=1 gates the whole bonus behind correctness. Together these terms turn 'be different' into 'be different and correct'.","core_discovery":"The paper's central claim is that mode collapse in reinforcement learning from verifiable rewards can be mitigated by gating exploration behind correctness. A dynamic trajectory library D stores verified unique solutions; a correct trajectory receives an intrinsic bonus tanh(d_min/sigma) based on its minimum distance to the library, plus a KL-based plausibility term relative to a frozen reference model. The total reward is R_obj + alpha * 1[R_obj=1] * (tanh(d_min/sigma) + beta * P_KL(τ)), so only correct solutions receive diversity pressure. The authors report that this beats PPO-only baselines on pass@5 and self-consistency while preserving syntax.","pith_inferences":["The plausibility term as written, P_KL(τ) = -Σ_t log(π_θ(a_t|s_t)/π_ref(a_t|s_t)), is positive when the current policy assigns lower probability than the reference, so it rewards low-confidence token choices; if this sign is not a typo, the claimed syntax protection is not obviously doing what the text says.","A clean ablation that sets beta=0 (or flips the sign of P_KL) on the SVAMP code task would reveal whether syntax integrity comes from the KL constraint or from the fact that only verified correct trajectories are rewarded.","The same novelty-minus-similarity scheme could be carried to open-ended generation where correctness is learned rather than binary, provided a reliable verifier exists.","The diversity gains are reported mainly through pass@5; counting exact duplicate generations would sharpen what the semantic distance metric actually buys."],"forward_implications":["If correct, RLVR fine-tuning can be steered toward multiple valid reasoning chains rather than a single collapsed mode.","The gating mechanism provides a reusable template for adding novelty bonuses to other sparse-reward LLM tasks.","The diversity-greedy library offers a bounded-memory way to track solution coverage during training.","The reported results suggest that solution diversity and syntax integrity need not be in strong tension."],"fun_headline_variants":["Correct-only novelty reward lifts LLM pass@5","Gated diversity bonus expands LLM reasoning manifold","PPO-HSC: novelty reward for correct trajectories boosts exploration","RL fine-tuning with novelty gating preserves syntax, adds diversity","Wide-area policy coverage via novelty bonus in PPO"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The framework assumes P_KL(τ) = -Σ_t log(π_θ(a_t|s_t)/π_ref(a_t|s_t)) acts as a plausibility constraint that keeps outputs syntactically valid; with this sign, it instead rewards tokens the current policy finds less likely than the reference, so the syntax claim rests on this equation behaving as described.","fun_headline_variants_meta":{"raw":{"variants":["Correct-only novelty reward lifts LLM pass@5","Gated diversity bonus expands LLM reasoning manifold","PPO-HSC: novelty reward for correct trajectories boosts exploration","RL fine-tuning with novelty gating preserves syntax, adds diversity","Wide-area policy coverage via novelty bonus in PPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1134,"prompt_tokens":722,"completion_tokens":412,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":466,"tokens_out":412,"duration_ms":4697,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T14:34:32.318729+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train PPO-HSC with the KL term set to zero (or with the sign flipped) on the SVAMP code task and measure Syntax@5 and Pass@5. If syntax stays at 1.000 and Pass@5 still improves, the plausibility constraint is not what preserves syntax; if gibberish appears, the sign issue is confirmed.","supporting_citations":[],"review_version":1}